You are on page 1of 21

Comprehensive and biaised comparison of OpenBSD and FreeBSD

Antoine Jacoutot Baptiste Daroussin
ajacoutot@ OpenBSD. org bapt@ FreeBSD. org
The OpenBSD project The FreeBSD project

Abstract AJACOUTOT: Right, so while discussing the re-
productive appendix of these horses to know
This paper will look at some of the differences between which came first...
the FreeBSD1 and OpenBSD2 operating systems. It is OK who am I kidding here. It was something
not intended to be solely technical but will also show like midnight and...
the different ”visions” and design decisions that rule
BAPT: Actually I think it was more like 2am.
the way things are implemented. It is expected to be a
AJACOUTOT: Anyway, we were having our 10th
subjective view from two BSD developers and does not
pint of Heineken in a Pub somewhere...
pretend to represent these projects in any way.
BAPT: Duvel, it was Duvel. Be careful what you
say here, its a beer land!
We don’t want it to be a troll talk but rather a casual
AJACOUTOT: Right right. So I started complaining
and friendly exchange while nicely making fun of
that the Duvel didnt taste anything, of course it
each other like we would do over a drink. Of course,
was our 10th pint so...
we shall try and hit where it hurts when that makes
BAPT: No no no, I told you cant say things like this.
sense. Obviously, we both have our personal subjective
Or OK, maybe the last one was Heineken after
preferences and we will explain why. Showing some of
the weaknesses may encourage people to contribute in
AJACOUTOT: See! So, as I was saying, the beer did
some areas.
not have any flavor anymore. And thats when
it hit me... Flavor... wait a minute... OMG this
Most of the topics discussed here could warrant their
is exactly like FreeBSD. You guys dont have
own paper and talk and as such some may not get the
FLAVORed packages!
deep analysis they deserve.
[ Trolls must now leave the audience]
This is a totally biased talk from two different perspec-
tives. ACT I
2 Ports & packages
1 The History behind the story
AJACOUTOT: FLAVOR3 is an awesome concept. I
AJACOUTOT: Before we begin, let us tell you a lit- am not aware of any other package manager in
tle story about how we ended up here today... UNIX land that implements the same function-
Once upon a time in the countryside on a sunny ality.
afternoon, in the mid-summer time, Bapt and I Basically, it allows providing packages com-
were having some herbal tea with... piled with a different set of options. Thats very
What was it already, sponge cake? convenient from a dependency point of view
BAPT: Absolutely, we were talking about the inter- since according to the options you want, the fi-
action between ponies and unicorns. Which nal binary will change as it might end up link-
one appeared first etc... ing to different libraries.
Just another one of our regular meetings argu- So you want to be able to control that but pro-
ing on important philosophical concepts. viding packages compiled with a good set of
defaults can be next to impossible sometimes in mind that we do not release a package set,
(its hard compromise) and thats where FLA- but the ports tree is a rolling release not tight
VORs kick in. For example you can install to a FreeBSD release. Which reminds me, you
Sendmail with or without support for SASL guys do not support upgrading packages on a
and / or LDAP: there are different (conflict- given release!
ing) packages for it (e.g. ”sendmail” versus AJACOUTOT: Actually we do support upgrading
”sendmail–ldap-sasl”). packages on a given release.
Its different from subpackages which are just What we do not provide are precompiled
split packages. These are typically used when binary packages. So its true that you need
you have a software that comes with a huge some kind of a build box where you can
amount of data (like audio and graphics for distribute packages updates from.
games for example) which rarely change while But theres work in progress in this area and
the main binary does each update. I am confident that within a year we will be
Instead of having to update the complete set, able to provide binary upgrades for packages
you just update one small subpackage. It can on supported releases.
also be used for things that ship modules or So keeping your packages up-to-date will just
add on components that have a whole lot of be a matter of running ”pkg add -u”.
different dependencies. PHP is a good exam-
ple. On OpenBSD we compile it with pretty Talking about pkg add(1)4 , its important
much all options, so the build requirements are to note that a lot of operations are done
big but the port only needs to be built once and as different unprivileged users (fetching,
since the package is split into multiple ones, extracting etc), we dont really want to go on
you only get to install what you need (e.g. php- the Internet as root while you guys still do
imap, php-mysql...). not drop privileges when using libfetch(3)5 in
BAPT: You really took the wrong example here. pkg(8)6 . Care to comment about that?
For php we have fine grained split the pack- BAPT: That is not true actually pkg(8) uses cap-
ages so you want php with imap? Just pkg in- sicum to sandbox everywhere it is possible.
stall php5-imap and done. And pkg(8) also uses unprivileged user to fetch
AJACOUTOT: But my point is actually that sub- packages for examples and everywhere it is
packages allow you to not have to deal with possible.
fine grained packages. If I understand cor- While sandboxing is supported for a long time,
rectly how your ports tree works you still need unprivileged users mechanism were added re-
to build each PHP module separately one by cently before OpenBSD did it ... by a few days
one, while we do it all at once. :)
BAPT: Correct, these are not really subpackages AJACOUTOT: Maybe you and Marc (Espie; the
and would be nicer with them. For FLAVORs author of OpenBSD pkg add(1)) are sharing
a good example could be the OpenLDAP client brains!
compiled with or without SASL :)
AJACOUTOT: Right. So if I need SASL support in Back to our topic, we also try very hard
OpenLDAP on FreeBSD, I assume I need to to provide proper upgrade paths for our users,
build it myself right? Thats fine I guess, but but the difference is that we dont have to deal
my fear is what will happen when I run ”pkg with legacy / unmaintained wrappers against
update && pkg upgrade”? the pkg tools (portmaster7 , portupgrade8 ,
BAPT: Bad things... ˆ
portinstall, portguruwhatever...) . Our pkg
BAPT: So yes it is true that we do not yet have FLA- tools are designed to provide everything one
VORs nor subpackages but there is some on- needs for package management without the
going work to provides those features in the need for an external helper. Dont forget we
ports tree. That work is pretty hard as it breaks have a long tradition of providing proper bi-
the design of some tools that are heavily used nary packages over pushing people to compile
by users like portmaster and portupgrade and their own.
which are barely maintained; meaning we need BAPT: This is also true for FreeBSD now, most of
to find an upgrade path which will not break the tools you are speaking about are mostly
those. We try to always have an upgrade path to use the ports tree directly as a live system
for users and try to avoid hard breakage. Keep rather than using binary packages (which is a

usage not supported by OpenBSD right?). This is all very confusing, no support can be
AJACOUTOT: Its perfectly supported by OpenBSD. good if you dont understand how it works.
We just dont encourage it it because for BAPT: Common that is easy:
99.99% of the cases, it is not needed. Normal Releases which are published from
BAPT: But one thing about the FreeBSD ports tree a -STABLE branch were supported by the
is that it works on all supported versions of Security Officer for a minimum of 12 months
FreeBSD. Ah well... true you only support one after the release, and for sufficient additional
version of OpenBSD and only for 6 month. time (if needed) to ensure that there is a newer
release for at least 3 months before the older
3 Release model & engineering Normal release expires.
Extended Selected releases (normally every
AJACOUTOT: You are correct, for packages we second release plus the last release from each
only support the current release which at this -STABLE branch) were supported by the
time is is 6.0; the operating system itself is sup- Security Officer for a minimum of 24 months
ported for two releases. after the release, and for sufficient additional
And aside from current/snapshots, our ports time (if needed) to ensure that there is a newer
tree is indeed tagged to a specific release. Extended release for at least 3 months before
Thats a design decision. We do not want our the older Extended release expires.
users to be forced to upgrade to a new major Nah... I agree this is a mess, further more it
version of a software on a given release and was causing lots of issue when maintaining
we do not want to maintain 4 different versions the ports, as we had to wait for at the very least
of the same software to prevent that. Which 2 years before being able to use modern tools
is why our ports tree does not follow a rolling available in newer base. Also issuing a new
release model (except for -current obviously). release for important features that did not fit
Considering the fact that we have regular re- en errata was having a huge impact: extending
leases every 6 months, our packages dont have the support for the branch for at least 2 years.
time to get too old anyway.
That said, theres ongoing discussion about Actually I should have used a past tense,
supporting the previous release as well (as we because this has changed recently with
do for the base system). FreeBSD 11.0 the model has changed:
Our support policy is pretty easy: * Base: n each major version stable branch is explicitly
and n -1 * Packages: n By support I mean that supported for 5 years, while each individual
we will fix and commit code to the correspond- point release is only supported for three
ing tag but we will not provide updated binary months after the next point release. Given we
sets nor packages. That sucks I agree, but if ensure binary compatibility on a given major
all goes well, things are going to change pretty branch, this allows users to benefit stability
soon with syspatch(8), a base system update for 5 years, and at the same time benefit new
tool. Coupled with binary packages updates, features.
maintaining OpenBSD will become even eas- AJACOUTOT: It looks like FreeBSD support model
ier than it is now. keeps newer features out of the hands of users
Its true that this means you must upgrade at because it can be years before they actually see
least once a year if you want to run a sup- them in a release. It reduces real-world testing
ported release. I dont think thats a bad thing: and benefits no one. In my opinion, OpenBSD
Id rather upgrade often in small steps than once model is easy to understand and keeps the code
every 5 years having to handle multiple inva- stable but still pretty fresh. Whatever is in cur-
sive changes at once. rent at a certain period in time will end up in
I do understand the need for LTS releases, I the next release (which would take at the very
just dont like it when I am the one responsible worst, 6 months).
for upgrading them... BAPT: You seems to have misunderstood the new
That said while it can be good to have a some release model, new feature not breaking binary
long term support WTF is your policy? compatibility are merged into the stable branch
Extended support versus normal support? pretty quickly and we will be able to issue new
How come FreeBSD 10.1 was still supported release on those branch whenever we want to
while 10.2 wasnt? have releases supporting those new features.

AJACOUTOT: Well you did say that you ”had to mentioned syspatch(8)9 which is a small util-
wait for at the very least 2 years before be- ity that will fetch, verify and install binary up-
ing able to use modern tools available in newer dates (tarballs containing the updated binaries
base for ports”. We do things differently, we / libraries...). We intend to release it as a tech-
are not afraid of breaking binary compatibility nology preview for the 6.1 release. It wont
in -current which will end up being the next be as clever nor technologically interesting as
release (no merge / backports necessary nor freebsd-update(8)10 but thats how we like our
wanted). But as you put it, your process is a tools: simple, stupid and safe (in no particular
mess so yes, I may have misunderstood. order); we do not want to fight with the oper-
ating system but get on to our task at hand.
For packages, we just need to agree on the
4 Binary upgrades
workflow, the team responsible for them and
BAPT: Given we do support binary upgrades it easy get the required infrastructure. Once we have
to upgrade from a minor to another minor (by that, we will start providing updated packages
the way binary upgrades works also nicely for all of our supported releases which should
across major releases) but hey, here I must be happen for 6.1 (6.2 at the worst).
losing you as you do not support binary up- BAPT: On the FreeBSD side we do support in
grades neither for security fixes, neither for up- place binary upgrade via freebsd-update. The
grade across releases, seriously how tedious it basic design behind freebsd-update is to create
should be to have OpenBSD boxes in produc- binary diffs and binary patch your system.
tions? It is very convenient to receive security fixes.
AJACOUTOT: While I see where you’re going, the freebsd-update also allows to upgrade
way you put it is not entirely true. Binary FreeBSD across releases ”in place” the proce-
upgrades between releases are perfectly sup- dure is then the following:
ported. What is not supported are in-place up-
grades of the base system from one release to • First install the new kernel
another. But is that really an issue? Supporting
• Reboot on it
in-place major upgrades would not prevent us
from having to reboot anyway. • Install the userland
The upgrade process of OpenBSD is one of the • Merge configuration files
easiest and fastest Ive ever encounter. Just re-
boot on the new release bsd.rd (ramdisk ker- • Remove the files not in the release any-
nel) and it will do the rest for you in like 5 more
minutes. It works very nicely but fetching all the binary
Also, thanks to our auto-install / auto-upgrade patches when upgrading can be very very long.
functionality, you can give it the path to an ”an- Another issue is that producing the patches is
swer file” (via dhcp) and you wont even need really not trivial. That is why we are looking
manual interaction. forward to be able to provide a new mechanism
So you can rely wipe and reinstall or up- based on pkg(8).
grade an unlimited number of machines in a To be honest there is also a drawback about
snap. And it can also upgrade your packages in place upgrades we need to maintain strong
to match the new release! Add this to the fact backward compatibility on some kernel inter-
that sysmerge(8) (or mergemaster equivalent) faces vs the related userland tools.
is automatically run early at boot time after an In particular ZFS11 is an interesting place.
upgrade, you are really left with almost noth- Illumos12 consider the version in kernel goes
ing to do... with a given version of the userland tools. So if
BAPT: Maybe but that only works from release to they modify a kernel interface they modify the
release (or -current), not when you want to userland related tools to use that new interface
keep your currently running release up-to-date and this is done. On FreeBSD we have to add
with security patches. a compatibility shim in the kernel because oth-
AJACOUTOT: Indeed that is true. I said earlier, up- erwise rebooting on the new kernel one would
dates within a particular release are only pro- end up with userland tools that does not work
vided in the form of CVS patches. Yes that with the kernel until the userland itself as also
part sucks but we are working on it. I already been upgraded.

AJACOUTOT: In-place upgrades can be interesting a multi-node setup for building packages. It
when the upgrade process itself takes a while. allows us to provide -current binary packages
In theory you would prevent long down times. daily (in reality we only ship new packages ev-
But in reality, production should be resilient ery 3 days unless theres a good reason).
and services should be redundant so its gen- As most OpenBSD daemons, it comes with
erally not such of a problem. Also the upgrade privilege separation: a different unprivileged
process on OpenBSD is literally the fastest one user is used for fetching, building and doing
Ive ever used. specific builder tasks. While pkg add(1) still
If in-place upgrades meant and updating and runs as root, dpb(1) allows building each pack-
reloading the entire userland and kernel with- age in its own chroot(2)15 using the proot(1)16
out rebooting, now that would have my vote! tool; the end goal being to actually build each
BAPT: An interesting policy we have on FreeBSD package under its own sandbox on the fly. Also
is POLA13 (Principle Of Least Astonishment) its worth noting that on our official build setup,
this is a strong policy on FreeBSD saying we the build user is prevented from reaching the
should try very very hard to make upgrades as network; the only process that can go online
seamless as possible and avoid breaking end is the one that fetches the distfiles for building
user boxes as much as possible. from ports (and that process runs as a different
AJACOUTOT: In general we are not afraid of break- user).
ing backward compatibility when we think it
BAPT: Historically, we were using a tool which a
makes sense and help push ideas forward. That
bit like dpb was distributing the builds across
said I can understand the need for a POLA but
multiple machines. This has been changed
not when it goes against basic security. For
years ago when we switched to poudriere(8)
example, just to satisfy third party clients you
which is designed to work on a single box dis-
guys kept DSA encryption, AES-CBC cipher
tributing the builds across multiple CPU cores.
and SSH1 support in your OpenSSH a long
Doing that we have benefit faster builds and
time after upstream (aka OpenBSD) dropped
way simpler code. The packages are signed on
it. That is the kind of compromise we are not
a box without internet access, the main differ-
willing to make.
ence is probably that we do not signed every
BAPT: That is not entirely true. POLA does not
packages but the repository metadata which
prevent from making such modifications, but
contains hash of all the packages.
they should be driven as in the users should be
warned long enough before. This is in partic- AJACOUTOT: Same here, all of our packages are
ular true on a given branch, big changes usu- cryptographically signed using our signify(1)
ally happen in new major branches. On newer tool on a dedicated machine that has no ac-
branch they were just removed. cess to the internet. The difference is indeed
AJACOUTOT: Well, according to FreeBSD SVN that we do not want to rely on metadata so the
revision 303716 (Wed Aug 3 16:08:21 2016 packages themselves are signed; it also makes
UTC): it easier for sharing packages.
”Remove DSA from default cipher list and
BAPT: Since we switched to poudriere we are able
disable SSH1.
to provide updates of binary packages almost
everyday for the stable and head branches of
Upstream did this a long time ago, but
the ports tree.
you kept DSA and SSH1 in FreeBSD for
While working on that we have leverage all
reasons which boil down to POLA.
FreeBSD features:
Now is a good time to catch up.” • Jails: all build are done inside a jail with-
out network access
5 Package building & delivery
• ZFS: to fast populate the jails and ensure
AJACOUTOT: Speaking of upgrades delivery, it we are always building in a clean room
would be interesting to know how you guys • tmpfs(5)17 : for fast I/O
build and ship packages. On OpenBSD we use
a tool called dpb(1)14 , the Distributed Package • smp(4)18 : yes we can use more than 24
Builder. Its a perl daemon that orchestrates cores :)

6 SMP & scheduling AJACOUTOT: Well our entire SCSI stack and ker-
nel profiling are already fully SMP for ex-
BAPT: Speaking of using multiple cores, you are ample and recently theres been some huge
still using the Giant lock almost everywhere? progress on making our network stack more
That should be a big problem for you to scale SMP friendly. The goal of the NET LOCK()
isnt it? being able to use multiple threads to forward
AJACOUTOT: For a regular ”Desktop” usage, our traffic.
”Big lock” SMP implementation model (KER- I dont think you will argue to the fact that mod-
NEL LOCK()) is good enough. The reason is ern scheduling is hard.
that most of the time on a workstation you only Most operating systems had to do it then re-do
have a handful amount of cores (between 2 and it multiple times. We are learning from others
8) and only one socket. Our actual scheduler experience and as usual with OpenBSD we are
isn’t bad, it’s just that it is a bit old and was trying to implement something simple that will
written for real SMP machines. Meaning it match the project goals. As you said, it wasnt
does not consider the cache distance between an easy road for you.
cores and that’s the main reason why machines That explains why sometimes we may seem
with several sockets often have lower perfor- behind on certain aspects, we take our time to
mance than the ones with only one socket (less do it well according to our standards, or we
ping-pong involved). don’t do it.
So we can also use 24 cores, but in the con-
text of bulk builds, we have much better per- 7 The base system, part I
formance concatenating the work of 6*4 cores
machines. This will change someday, for bet- BAPT: While you are here, you can maybe explain
ter or worth its just not been a priority. something I do not understand: you seems
BAPT: How such a change can be done in the to be keeping writing your own tools when
OpenBSD model? I mean on FreeBSD it took there are already BSD licensed counterparts:
a lot of time to get most of the architectural httpd(8)20 , smptd(8)21 , vmm22 , etc. Arent you
work landing. it took us several years in- suffering from the NIH23 syndrome?
AJACOUTOT: Thats actually very good question. I
cluding releases that got very bad reputation:
FreeBSD 5. Im not saying the road we took is am glad you asked because there are IMHO
the one true way, but I cant see how this can be objective reasons to do that. The first one
achieved in an incremental way. would be trust: we have a coding style, prac-
tice and process that make us confident in what
AJACOUTOT: A few things can be incremental be- we develop. Another reason is also control: we
cause they are not directly related to the sched- know someone will not decide to change the
uler itself. software license one day to the next or start
Several scheduling issues on OpenBSD come adding knobs for each and every crazy cor-
in fact from spin locking in librthread (our ner case that is out there that only two people
POSIX 1003.1c thread implementation) and would use. Looking at how many CVEs im-
are actively being worked on. And contrary to pacted ntpd or OpenSSL in the last couple of
the myth, OpenBSD scales very nicely on big years should be a good hint as to why Open-
userland workloads. Now, is raw performance NTPD and was created and LibreSSL forked
on par with FreeBSD? No, but our priorities from OpenSSL for example. If I am not mis-
are different and we are OK with loosing a bit taken, a lot of your security advisories have
in this area if it gains us more security. Actu- been related to the NTP daemon included in
ally Id go further and even say that wed rather your base system (known as the ”reference im-
crash / panic if we detect an unsafe behavior. plementation”); is it really having NIH syn-
BAPT: To be honest SMP is something that is al- drome wanting to write a secure implementa-
ways evolving, right now our next challenges tion?
are better NUMA(4)19 support, and improved BAPT: Right it makes some kind of sense for the
the overall locking systems using lock less one you talked about, as there were no ”BSD
mechanism where possible via for example quality” alternatives, but what about bhyve? It
Concurrency Kit. What is the current status on is working it can ”easily” fit your security re-
OpenBSD? quirements about security and sandboxing.

So why yet another implementation? AJACOUTOT: Most daemons run with privilege
AJACOUTOT: Theres been an initial porting trial a separation: that is most of the code is run ch-
few years ago but compiling a single file ended rooted and as a non-privileged user. OpenSSH
up being a huge amount of work (let alone led the way on this topic. They also use priv-
work); when youre confronted to such code, ilege revocation (or dropping) to drop privi-
you need to ask yourself whether its not worth leged as soon as possible. A good and sim-
just creating something from scratch instead of ple example to look at is ntpd(8): it is a
porting. The situation may have changed since standard OpenBSD daemon so it is written
(see xhyve: the lightweight virtualization so- with principle of least privilege in mind and
lution for OS X) but that was another reason so not only does”privsep”and ”privdrop” but
vmm(4) was created. I would add that it also more than this: it has a completely privilege
helps us providing a base system that in our separated tls speaker for the ”constraint” fea-
definition ”Just Works”. If you ever feel the ture with no memory sharing nor fd passing
need to tune or harden something, then you and with a limited amount of allowed system
are clearly on the wrong operating system and calls thanks to pledge(2). The constraint fea-
OpenBSD is not for you... ture makes ntpd(8) query the time from trusted
web servers over HTTPS reducing the risk of
BAPT: Right but another example would be cap-
a man-in-the-middle attack by comparing that
sicum, you decided to bake your own sandbox-
time with the one that the remote NTP server
ing mechanism instead of porting it.
gave us.
AJACOUTOT: Theres actually a very good reason BAPT: For ntp you are right but at the price of hav-
for that but lets talk about pledge versus cap- ing a half baked implementation, one that can-
sicum later since you brought the subject of not authenticate the peer it is receiving the time
security, lets talk about it. from (using the standardized mechanism), and
miss half of the features ntp is supposed to pro-
vide (yes I agree most people wont care). That
8 Security & sandboxing said I would love that we get rid of ntpd.
AJACOUTOT: You just described exactly OpenNT-
BAPT: Lets start with our malloc24 implementation, PDs strength; it actually does not need to au-
while it is not a direct security related tool it thenticate thanks to ”constraints” and it does
helps a lot driving your development in a safe not need to implement the kitchen sink because
way. On developer can fully tune it to find no one cares and if one does, then it can install
out memory leaks via prof leak (very similar the reference implementation.
to what you can get with gperftool, heap profil- On a related topic, OpenBSD is well known for
ing, and way more things, you can activate all it numerous exploit mitigation techniques. It’s
this via malloc.conf and/or MALLOC CONF important to note that all of these have been
environment variable. enabled by default for years and are very hard
AJACOUTOT: Our memory allocator has a lot of or even impossible to disable. In no particular
countermeasures as well. order, we have (non-exhaustive list):
Some are not enabled by default because they
• Address space randomization (ASLR26 ):
are very costly in terms of performance but de-
randomly arranges the address space to
velopers often run with them enabled because
prevent buffer overflow attacks
it allows catching several issues beforehand.
The ”S” option in malloc.conf(5)25 is perfect • WX ˆ (memory page can be either writable
for this: ”Enable all options suitable for secu- XOR executable)
rity auditing.” ”Guard pages” can be enabled, • in the kernel since years
these provide overruns detection by creating
• now in userland (”wxallowed” mount op-
an unreadable, unwritable page ”fence” after
tion on /usr/local for now)
malloc(3)ed chunks (accessing that region will
trigger a segmentation fault). • ProPolice (Stack-smashing protection
BAPT: So it sounds like both malloc are quite in the using bounds checking and canary)
same shape regarding debugging and memory • Position Independent Executables (PIE)
related potential security issues. But in a dif- (also for static binaries): the executable
ferent way. is loaded at a random address instead of

a fixed one address which stops return-to- possible to audit. Its not a FreeBSD specific
libc type attacks against functions in the problem by the way, Linux has similar facili-
main program (another space randomiza- ties and most people disable them...
tion technique) What about LibreSSL? Are you guys planning
on integrating it? I know its a long subject but I
• Strong random generator
am just asking because just from a regular user
The way we approach security is that we al- point of view it has been a great success:
ways assume to be running in a hostile envi-
• usually only a handful of the CVEs im-
ronment. Thats also what started our constant
pacting OpenSSL impacts us
source audit (which started in 1995, and still
ongoing) which allowed us to detect unsafe • provides new crypto algorithms like
patterns and / or bugs and fix them everywhere CHACHA20-POLY1305 and X25519
in our tree. It looks like security mitigations We are seeing more and more people and
in FreeBSD are next to nonexistent, even your companies adopting it in their projects and
(non-committed) ASLR is intentionally weak agreeing to work with the ”So and so from
(using non-aggressive settings). Is that true? OpenBSD”.
BAPT: We also have strong random generator BAPT: LibreSSL is pretty well integrated into the
and Propolice. While we do not have ASLR ports tree even if not the default. For the base
yet, I can’t argue about the HardennedBSD system, the problem is LibreSSL development
implementation but I would be surprised if model does not fit with our ABI compatibil-
it is true given they implemented the PaX27 ity rules. The approach in the future would
recommendation as far as I know while PaX be probably to drop OpenSSL in favour of ei-
seems to consider the OpenBSD implementa- ther LibreSSL or any more lightweight alter-
tion as an ”incomplete” implementation which natives like mbedtls into a ”privatelib” aka not
should be named ASR apparently. By the exposed to the ports/packages so ports / pack-
way we have an implementation of ”ASR” ages can have the flexibility to use whatever tls
currently in review we should hit the tree: stack we need to have at the time which yes can be libressl.
BAPT: Now on the sandbox part, Capsicum(4)28 is
On our side we do have the MAC framework a very nice sandboxing mechanism we have in
which is a very interesting: Mandatory Ac- FreeBSD leveraging the concept of capability.
cess Control which allows you to restrict many What is really nice about it is once you are in-
things. With MAC we can: * Restrict entirely side a sandbox, there is no way to exit, even
what a user can see in the system to noth- child process inherits the capabilities.
ing but its own resources (including not see- It is really design for developers to secure their
ing sockets open by other users for example) own applications by strictly restricting the ca-
* Have a firewall-like policy on file systems pability of the application to only what it needs
* Restrict network interfaces access that plays to be able to do. We have started converting
nicely with IDS * Limit the scope of the pro- most of the base system to use it. But we are
cess one can see compartmentalizing them into not entirely there yet.
partitions * Restrict the information flow in a Because the capsicum design allow no com-
running system (very handing for example to promise, it is not always easy to convert to ex-
ensure things like nagios can only access what isting software to capsicum.
an admin planned to make it acces and nothing AJACOUTOT: Thats where I wanted to go actually:
more. it seems that capsicum is too complicated and
Another very interesting security feature we thats why it doesn’t get much use by default.
do have OpenBSM: a full feature audit system The pledge(2)29 syscall is actually a very good
which can monitor everything that has been summary of what the OpenBSD project is
done on a system. It can also distribute the au- about: develop affordable security. By ”af-
dit log To a central server. fordable” I mean simple and easy to imple-
AJACOUTOT: All these technologies you mention ment and hence enabled by default almost ev-
are interesting in theory, I agree. But in real erywhere in the system (30% of the base sys-
life, I almost never see them in action because tem was pledged after only two month; to-
they are too complex to use and next to im- day 85 to 90% of base and even some ports

like chromium...). It was designed to be easy proposals or the projects bylaws.
for the programmer to add support for it in The FreeBSD foundation is a 501, US based,
his software, otherwise no one would use it. non-profit organization dedicated to support-
pledge(2) itself is designed to serve two goals: ing and promoting the FreeBSD Project and
encourage refactoring towards safer models community worldwide.
and reduce code exposure to the kernel: it en- Funding comes from individual and corporate
courages the developer to audit and understand donations and is used to fund and manage
the software initialisation and mainloop to add projects, fund conferences and developer sum-
pledge(2) calls where needed and move code mits, and provide travel grants to FreeBSD de-
around if needed. One can assume that any- velopers.
thing making its way into the OpenBSD base The Foundation purchases hardware to im-
system will end up being pledged. People of- prove and maintain FreeBSD infrastructure
ten compares it to seccomp-bpf on Linux al- and publishes FreeBSD white papers and mar-
though pledge(2) is not a system calls firewall keting material to promote, educate, and advo-
but rather a facility to explicitly allow a group cate for the FreeBSD Project. The Foundation
of system calls in the form of promises (sort also represents the FreeBSD Project in execut-
of predeclared profiles, e.g. inet, chown, dns, ing contracts, license agreements, and other le-
tty). It can also inspect arguments. A pledged gal arrangements that require a recognized le-
process is forced into a restricted-service oper- gal entity
ating mode where abilities can be reduced but AJACOUTOT: To quote it website: ”The OpenBSD
never regained and if a restricted operation is Foundation is a Canadian not-for-profit cor-
attempted, then the process is killed with an poration which exists to support OpenBSD
uncatchable SIGABRT. In theory capabilities- and related projects such as OpenSSH, Open-
based security may seem more advanced, and BGPD, OpenNTPD, OpenSMTPD, LibreSSL,
it probably is. But what good is a feature if it and mandoc”.
is too complex to be widely implemented? It is a fundraising entity only: it does not own
BAPT: Another security feature we do have for
any copyright over the code even for spon-
a while are jails, it allows to create a sored people. Its responsible for funding the
prison/container, we have it in FreeBSD for day to day needs of the project, hackathons,
very very long, and it very simple and easy to etc and works solely through donations (from
use. Almost everything can be disabled (and is users and companies). It has no influence over
by default) or restricted inside a jail: network, the project itself.
file system, CPU, memory, routing tables, etc. Speaking about the country of origin, while it
Nowadays jails can even be nested, have they may not seem like an important detail at first
own zfs datasets, have virtual network. but it actually is when it concerns the operat-
AJACOUTOT: Well in that regard I agree that we
ing system itself. OpenBSD being based in
suck. OpenBSD has no container-like tech- Canada means that we are not subject to the US
nology. There has been an initial effort years crypto export regulations and all. As a conse-
ago called sysjail but it was abandoned because quence, it was illegal to re-export OpenBSD
of an inherent design issue in systrace(1) on from the USA early on and no US citizens
which it was based that would allow evading a were allowed to work on crypto in the project.
jail by exploiting a race condition. If you wonder why we ended up with our
Anyway I think its time to move on to some- own ACPI implementation instead of Intels,
thing else. that one of the explanation as well. At the
time it was unclear whether their implemen-
9 Project governance tation contained embargoed code or not be-
cause they triple-licensed their code and the In-
BAPT: Good idea, lets talk about project gover- tel license explicitly stated that ”Licensee shall
nance and funding for example. not export, either directly or indirectly, any of
The FreeBSD project is a community driven this software ¡...¿ to any country for which
project, the active developers elect every 2 the United States government or any agency
years a new core team of 9 people responsi- thereof requires an export license”. While that
ble for deciding the project’s overall goals and chunk was obviously not part of the two other
direction as well as validating new committers possible licenses (BSD and GPLv2), export

regulation compliance is license agnostic, so 10 Project organisation
in doubt...
AJACOUTOT: OpenBSD has only one true commit
BAPT: Interesting. Regarding the governance bit. Youre a developer (with commit access)
strictly speaking, The OpenBSD project seems or youre not, period. Thats good because it
to suck: kind of dictatorship? One guy decides encourages people to contribute in all parts
for every volunteers? And what would happen of the tree, not just their area of expertise.
if he resigns? The project is comprised of 4 repositories: src
AJACOUTOT: Well, thats a bit of an exaggeration (base system), www (websites hosted by the
dont you think? Obviously, one guy does project), xenocara (Xorg with a BSD build sys-
not decide for everything, most of the deci- tem) and ports (ports tree).
sions are taken collegially. That said one guy, Since every commit requires at least one ex-
namely Theo, does have a veto over the en- plicit OK from a fellow developer, were pretty
tire project. From my point of view, this is safe with regards to people committing in area
good, it speeds up the decision process, pre- they usually dont. The policy is a bit more re-
vents stalling and make sure we have a safe- laxed in the ports tree where we only require an
guard. We really cant afford losing ourselves explicit OK when importing new ports, making
in over-administrative processes which might invasive changes or similar. Of course since
lead to no decision. It is true that we like hack- you own what you commit, if you made a mis-
ing more than getting ourselves caught into take and did not have an OK from anyone, be
committee-induced compromises. prepared to suffer the consequences...
Regarding the future of the project, the only BAPT: We have split the repository into 3 parts:
thing I have to say about it is that we are ports, base and docs when doing that we have
fully open source so I wouldnt worry about it. split the commit bits as well. But note that the
Maybe something is planned, maybe not... difference is only administrative because tech-
Look at the longest living Linux distribution nically as soon as one has an access he can
as of today: Slackware; its been run since the commit to all the repositories. This separa-
very beginning by a benevolent dictator and is tion is more to enforce a policy, if one has been
still alive! I think its safer for anyone, be it awarded a commit for docs it is because of the
a regular user or a company to use OpenBSD good work he has done on this side, the com-
than any company-baked project; look at what mitter. He is not supposed to commit in other
just happened to Oracle Solaris... So regarding area without a format approval from a commit-
the future of the operating system, Id worry ter that has a bit in that area. And usually after
much more about using products from such a bunch of good patches he is granted easily
companies than OpenBSD. with a commit bit in that area. The barrier is
very soft.
BAPT: Well in the end it all depends on devel-
BAPT: While we are on administrative, I always
opers for us, so not quite different than you,
wondered what is the process of becoming an
core@ is elected by them every 2 years so core
OpenBSD committer?
have confidence from the developers to drive
AJACOUTOT: Theres no real defined process.
the project, take final decision if there are no
technical agreement among developers on a OpenBSD is a meritocracy so if you keep send-
given subject, the team also have to deal with ing good patches to the mailing lists for a
disputes among developers and/or community while, theres a good chance one developer will
and for the project policies. All core decisions contact you and ask you whether youd like to
are taken over votes inside the team. join the gang. At that point, it usually means
Last thing core is responsible for voting on de- that youll get invited to ICB (our internal chat)
velopers. On this in particular, core focuses on where you will get introduced to other peo-
source committers and delegate ports commit ple. The way it works then is that the devel-
bits to the portmgr team and docs commit bit oper who invited you becomes your de-facto
to the doceng team. mentor and is responsible to teach you the few
things about our development workflow and
AJACOUTOT: Wait... What? You have different commit process etc. At that point, you get your
types of commit bits? You guys really love OpenBSD account pretty fast.
making things more complex than needed. Thats pretty much it, there is not ”mentor-

ing period”, your mentor is only here to get BAPT: All the FreeBSD development happens in
you started and take responsibility if you break the head branch, on the subversion repository.
something just a couple of days after getting For big features people often create dedicated
your commit bit... svn branches. Lots of people are also us-
BAPT: On FreeBSD when a contributor has sent a ing the git forks of the tree, works on their
set of good patches, usually a committer gets branches there and pushed back the changes
pissed of with committing those patches he into FreeBSD via small readable chunks via git
generally proposes the contributor to become a svn.
committer. If the contributor is interested, the We also have a perforce repository but almost
committer sends a formal request to the core noone uses it anymore and was mainly used
team for base, portmgr for ports and doceng before the svn/git time due to CVS nightmare.
for docs. The members of the team will anal- The only case where one can see very big
yse its contributions, take a vote, and if granted commits instead of smaller changes are branch
will ensure the now new committer gets men- merges, merge from p4 (which hasnt happened
tor(s) to drive him into the project: policies, for a while) on importing new version of exter-
knowing what resources he can access to, who nally maintained softwares.
is responsible for what, ensure that he probes Once the code has hit the head branch if
the proper developers (the experts of the areas) needed or willed than it is merged into the sta-
for reviews before committing. When the new ble branches.
committer is confident enough he is released Before each major BSD conferences, the
from mentorship. project organize dev summits usually 2 days.
Speaking of commiters and contributors, can It is a unique occasions for developers to meet
you tell us a word about leading edge develop- and to be able to discuss in live issues they
ment in OpenBSD? Do you use branches or ex- faced, discussing how to make a new feature
ternal repositories which are then merged into happening or project directions. Aside from
the main tree or? those devsummits, there are some other dev-
AJACOUTOT: As a general rule, all OpenBSD summits organised around the world sometime
driven software projects development is done very small sometime large. Those devsummits
directly in CVS (-current). Like OpenSSH, often turns into some kind of hackathon, some
OpenNTPD... time are only in the hackathon form.
We do not use branches because, well, CVS. AJACOUTOT: It is my personal opinion that the
Fun anecdote: I think we are one of the oldest reason OpenBSD is successful despites its
and biggest running project still using CVS. Im few amount of developers is because of
not saying I am proud of it, I just wanted to hackathons. Considering our distributed
mention it :-) Anyway, when invasive changes development model, it is a requirement.
need to be done we usually make a patch avail- The project is sort of a pioneer in that regard,
able to other developers for review and the in- we had regular hackathons before they became
tegration is done incrementally (by splitting a ”thing”. You can see it as a gathering of
the patch into smaller chunks). Of course people coming from all over the world, with
sometimes it may end up being hard or next their own different background but with one
to impossible to do that in which case we often common goal: improve OpenBSD in all area.
distribute snapshots including the uncommit- In reality I see it as a bunch of friends coming
ted bits for wider testing until we are confident together to have fun doing what they like to
we can integrate that one big ”patch” into the do. Thats a very important aspect I think and
tree. is what makes our developer community so
There’s been a few exceptions in the past vibrant and passionate. Theres always some
where a particular piece of OpenBSD software yelling, shouting, laughing, crying, despair,
was developed outside of our tree. hope and desilution... its software after all.
OpenSMTPD is such an example. But af- But as we said in the past: ”its not torture if
ter a couple of years struggling with having you do it to yourself”.
to manually merge changes one by one ”the Anyway, its extremely intense and motivating:
OpenBSD way”, development went back into its just amazing how many things are achieved
the tree which greatly helped general integra- during these events.

ACT II • GNU roff while mandoc is now used to
render manpages, some manpages still
11 ”BSDification” requires a full roff toolchain, we are go-
ing to replace it with heirloom doctools
BAPT: You guys claim a lot about license purity, but
you suck on that area, FreeBSD is very close • GNU grep (BSD grep is still too buggy
to provide a base system with only permissive but already imported)
licenses (mostly BSD). The only few compo- Which is why we have hope that FreeBSD 12
nents left (for tier 1), progress can be followed will be the first release without any GNU com-
here: ponents which is good users as well as they
AJACOUTOT: Before we go further, could you ex- will benefit being able to simply use the mod-
plain to me what ”tier 1” mean exactly? ern version of those tools instead of the old
BAPT: Tier 1 are the main supported architecture GPLv2 version.
for now i386 and amd64, various arm are close If we speak about tiers 2:
to pass in the Tier 1, It means we do ensure
binary packages, binary security fixes for those • gcc/libstdc++ is still required
arches. In term of support from source there is • dtc for embedded, beside the fact we have
not much difference. a BSD licensed dtc which is not yet 100%
AJACOUTOT: On OpenBSD, if a platform is listed fully compliant but mostly usable.
as supported, it’s at the same level as amd64 /
AJACOUTOT: Besides a few exceptions, most of of
i386 (for the base system at least; ports is usu-
userland is BSD. We still have (some fork of)
ally harder even if we try very hard to fix and
GNU CVS, although our RCS has been rewrit-
support non mainstream architectures). There
ten a few years ago and is under a BSD license.
is no concept of tier platforms for us, we sup-
Mandoc takes care of all our man needs, I am
port an architecture or we dont.
not aware of anything in base that would re-
BAPT: The reason I talked about tier 1 is because
quire a full roff(7) toolchain. I think we can
most of the tier 2 platform are not buildable
thank our documentation Gods for that: Jason
with LLVM/Clang meaning they have the old
McIntyre and Ingo Schwarze.
gcc/libstdc++ which means they have more
Regarding the toolchain you are clearly ahead
GPL softwares.
in that regard but we finally did jump on the
Basically we are very pushing on having a full
bandwagon thanks to the arm64 architecture.
BSD toolchain. Since FreeBSD 10.0 we are
I will not go too much into details because
using LLVM/clang as a main compiler, libc++
that effort is somewhat recent but what I can
as C++ stack, libcxxrt for the C++ runtime.
say is that we are going pretty much the same
We also develop and maintain the BSD license
direction as you are (lld, libc++ etc). Of
elftoolchain which replaces most of the tools
course, we still have to maintain somewhat
provided by GNU binutils, and is built around
”exotic” architectures that are not supported by
our BSD licensed libelf. With the main excep-
LLVM/Clang and as such our old (¡GPLv3)
tion of the linked, which will be most certainly
GNU toolchain is not going to go away any-
replaced by LLD in FreeBSD 12.
time soon.
Other left non BSD components where GNU
In the subject of how each of our base system
texinfo which we removed in FreeBSD 11.0
is evolving, whats the deal with that libxo(3)
replacing the useful info pages with proper
manpages (the only ones were the binutils
We removed GNU CVS in FreeBSD 10 and 12 Over engineering
GNU rcs will be removed from FreeBSD 12.0.
GDB is slowly being phased out replaced by BAPT: Well libxo is actually very useful. Have
LLDB. you ever tried to parse the netstat output from
So for now really the left components are: scripts? Having a programmatically parseable
• GNU diff (not that sdiff has been re- output is very handy to write software on top
placed by a mix of the OpenBSD and of FreeBSD quickly.
NetBSD one + patches for GNU diff AJACOUTOT: That sounds like a vendor require-
compatibility) ment... But anyway, even if Id agree with you,

does ls(1) really need to have a libxo(3) out- Remember it is all about flexibility, and yes
put? That looks like over engineering at its flexibility have a price, it can confuse users.
peak. All scripting languages provides API to AJACOUTOT: In OpenBSD we like simplicity. I
be able to deal with filesystems so I dont see agree that we may miss a few feature here and
the relevance here. there sometimes but we are ready to accept if it
BAPT: I agree that this one is arguable but it is actu- keeps us clean, simple and self-contained. It is
ally somehow useful to deal with BSD exten- simply unsustainable to keep accommodating
sions like chflags, where most of the scripting ”choice”
languages are not aware of. and in general, complexity is the worst enemy
AJACOUTOT: I am kind of torn about this. Its not of security which is one of our primary goals.
because one can implement something that one BAPT: Flexibility does not necessary mean com-
should. Oh and about about this one: plexity :) we also like simplicity.

ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [-D format] [file ...]
13 Storage & file systems
BAPT: Well. For this one I have no arguments :) BAPT: FreeBSD supports multiple filesystems. The
The fun thing is I discovered the options when native one being the original UFS but one that
looking at a slide in last year EuroBSDCon and have evolved to fits modern time: supporting
we all thought it was a typo :) journaling, soft updates, trim etc.
AJACOUTOT: See from my perspective, this all We also of course do support very modern
looks very over engineered. filesystem as ZFS as a first class citizen.
I mean even just for the jail administration, Other filesystems we do support are msdos,
there are probably 10+ third party tools to man- ext2,3,4 (without journal), nandfs.
age them. Why? Why can’t you include one AJACOUTOT: On OpenBSD we are stuck with the
good one, instead of ten written by random traditional BSD file system: UFS/FFS. That
people and of different quality? There’s sim- said, it has been extremely stable for us. Af-
ilarly a huge list of tools for controlling dif- ter 15+ years of OpenBSD in production in all
ferent aspects of ZFS, some in base and some kind of different setups, I have never lost one
not. Oh and sorry for mentioning this again file.
but: three firewalls... It can be very confusing We obviously also support NFS but only ver-
to know which tool to use, and whether you sion 2 and 3 since version 4 is a complete
can mix them or not especially when none of rewrite requiring pieces we really do not want
them are enabled by default. From an outsider, to support in base (like Kerberos). Our auto-
IPFW should be the native firewall (it was writ- mounter is still amd(8) from 4.4BSD! Its not
ten by FreeBSD for FreeBSD) so why keep IP- very efficient but it gets the job done. I for one
Filter and PF (which as far as I know has not would welcome a more modern implementa-
been synced with the upstream code in years). tion... Regarding Internet SCSI, we have a na-
BAPT: For jail the default provided tools base are tive iSCSI initiator implementation: iscsid(8).
good enough to manage them, they are flexible I would not argue that file system is not really
and nice enough. All external tools are mostly where we shine, we have not journaling, we do
providing extra features like provisioning etc have sot updates but we still have some bugs
which I dont think belong to base, because first in that code path... Ive experienced a few very
they can evolve faster if developped available hard to diagnose and reproduce (hence hard to
from ports (remember a branch will live for fix) panics so I usually do not activate them on
around 5 years). Concerning ZFS, the only one my servers. Note that a standard OpenBSD in-
I can have in mind is beadm, this one will prob- stallation does not enable softdep on any file
ably end up in base after a huge rewrite. system.
Concerning the firewalls, true ipfw has been BAPT: Wow you still have bugs with softdep???
developed by and for freebsd, but others are AJACOUTOT: Yeah but thats not fair, you guys have
willing to maintain ipfilter and they do. Con- Kirk (McKusick)! Its hard to compete! ;-)
cerning pf while it hasnt been synced that is BAPT: Under the hood we have the GEOM frame-
true, it is still actively maintained and have work which is very powerful giving us support
many features not available on OpenBSD: for multipath (gmultipath, encryption (geli,
SMP, VNET for example. gbde), mirroring (gmirror), network transport

(ggate) or even faking the hardware with gnop. AJACOUTOT: Our swap is encrypted by default
We have a very good support for iSCSI targets since many years without the need to do any-
(via ctld including HA) and initiator (iscsid) thing. It’s my opinion that swap should al-
On network file system we have very good sup- ways be encrypted. Your SSH or PGP private
port for NFS (all versions) server and client. key could actually end up there at some point...
We provide native tools to deal with storage There is a sysctl(8) to disable swap encryption
devices: SES, SAS cards etc. if really needed. I am mentioning it because
As a client of storage we also have the tra- that is an exception to the general OpenBSD
ditional amd(8) and since recently we have a rule where security enhancements are enabled
shiny new autofs support (automountd). by default and cannot be disabled.
And that is only speaking of user facing fea-
tures and utility.
AJACOUTOT: But ZFS is not just a filesystem, it’s
14 The base system, part II
much more. Is it a good thing? Is it bad
AJACOUTOT: Speaking of ”default” settings, that
thing? I honestly dont have a strong opinion
is really something I love about this operat-
on the matter so I dont know. That said, it
ing system: they make sense and are very well
is a very nice piece of technology and comes
thought out. We regularly have lengthy argu-
with some very nice features but there are a
ments about what feature should be available
few things that worry me about it: the kitchen
by default and in which in way it should work
sink approach and the fact that it strongly
out of the box. Thats how we came up with so
recommends using ECC memory. Software
many simple daemons.
RAID support is generally in a good shape on
The network related ones are a very good ex-
OpenBSD except for the RAID5 discipline on
ample because they solve hard problems in
which we cannot boot from... I would agree
simple ways: * vxlan, relayd, bgpd, rtables,
that it’s a bit embarrassing.
ospfd, carp, ipsec, spamd, httpd, dvmrpd, iked,
BAPT: About encryption we have multiple choices
around the GEOM layer: Ive seen people understand how networking
• geli works by just reading the configuration and
man page of OpenBGPD and similar.
• gbde
BAPT: While it is true that OpenBSD is full fea-
Because they are on the geom layer, then we tured in the network side FreeBSD is also very
can put any filesystem on top of it. very full featured: vxlan, carp, FIB, vimage,
For geli since recently (FreeBSD 11.0) we can netmap, ipsec, netgraph
support full disk encryption decrypted by the BAPT: Back to the base system,
boot loader using a passphrase. that makes the base system huge maybe some
We also support (not in base by developed for of those tools could be just installed via ports
FreeBSD) filesystem level encryption via pefs why having them in base?
which allows to encrypt only some directories
AJACOUTOT: We consider OpenBSD a general
whatever the filesystem is under the hood.
purpose OS that provides a useful number of
AJACOUTOT: The way we do disk encryption in services out-of-the-box. Its an important de-
OpenBSD is similar to how we manage RAID sign decision because it means that all these
devices (software or hardware) by using the tools are developed together. A change in
bioctl(8) management interface utility over a the kernel or a library will immediately trig-
softraid(4) encrypting discipline virtual de- ger some modification to these daemons when
vice. So it is already familiar to people and need be. Also, anything that is part of base is
the process is very simple (one command re- audited, pledged and follows our standards; the
ally). We have support for full disk encryption, situation is totally different in ports.
decrypted by the boot loader either by using a Also it encourages code synchronisation and
passphrase (interactive) or a ”keydisk” (ideal sharing amongst the base system. That would
for non-interactive use, using a USB dongle or be a totally different thing is these were part
similar). of the ports tree: runtime breakage may be-
BAPT: Btw swap encryption on FreeBSD is also come unnoticed for some time... If you look at
very simple just use geli or gbde. Linux, its the exact opposite: kernel and each

and every userland tools are developed sepa- that make FreeBSD more a toolkit to build an
rately. operating system than a proper one?
I for one am very happy to have so many fea- BAPT: No FreeBSD by default is a full featured OS
tures in the base system; just look at the ACME and on the binary form we do only support
client for example: not only it is fantastic to the plain full FreeBSD as served by the ISOs.
have it working out of the box, but also it im- but keep in mind that FreeBSD is also widely
proved so much since it got integrated (as op- used in embedded environments or inside ap-
posed to being a regular external package). pliances, both needs lots of flexibility (either
Oh and the funny thing is that OpenBSD base stripping down the size for embedded, or be-
system is still smaller than of FreeBSD. ing able to replace one feature we do provide
So it is quite surprising that you speak about by their own implementations). Also we do
a huge base... On one end you strip down encourage a lot vendors to work closely with
your base system (removing texinfo, perl, etc) their upstream meaning us.
but on the other end you have three firewalls, AJACOUTOT: Yes FreeBSD encourages vendors to
three! contribute and commit: it wants to be a ”uni-
We on the other end like to concentrate on hav- versal” operating system like Linux and satisfy
ing one way to do things. the maximum amount of people. If you look at
BAPT: The base system for us is a coherent out of it from a different perspective, you could also
box general purpose system. For us a full fea- say that FreeBSD does what your employer
ture version of given server can be installed wants while OpenBSD does what its develop-
from packages and there is no reason why we ers want.
should bloat the base system with those tools. Anyway, I like the fact that I can have ”func-
Having a full featured MTA in base is almost tional problem solving box” in no time with-
impossible: how one would add the feature out the need to install any third-party packages
required via tons of external libraries which while staying very small in size. For instance
are not available in base (ldap connection, an- I dont need to install proper shell which I can
tivirus scan, etc). Why would I bother having work with... and I am not even talking about a
an http server installed on my storage server? super featured shell ala zsh, but ”/bin/csh” as
Of course if a FreeBSD developers maintains the default root shell you guys... really?
that code, then there is no need to remove it, BAPT: If someone is willing to push a new feature
that is how we ended up with 3 firewalls. With and/or a new KNOB to improve flexibility as
exception of course of servers closely tied with long as it does not break FreeBSD directions
the kernel like iSCSI target (ctld), nfsd, etc. and usability we will accept it. That gives us
But in that case we do provide KNOBS so one lot of flexibility in the base system and is very
can build a stripped down version of FreeBSD nice to make freebsd usable for very niche us-
if they need it. age. For example making a very thin storage
AJACOUTOT: I think that is a major difference be- only server on ramdisks (like we do in Gandi)
tween our operating systems: we do not want Which makes me think I see that OpenBSD
to provide knobs for everything, we want to does not support kernel modules, I find those
provide what ”we” consider sane defaults and very handy for flexibility as I can load or un-
best practices to enforce ”our” way of seeing load features on demand if I need it without
IT. Is it the job of the operating system to pro- having to rebuild the kernel we also have sysctl
vide embeded and/or stripped ”editions”? and tunables to be able to interact with the ker-
An OpenBSD base installation is a basically nel configuration. Isnt it painful to tune your
complete fork kit in itself, i.e. kernel?
you have everything you need to continue de- AJACOUTOT: While its true that we removed sup-
veloping the operating system (utilities, com- port for loadable kernel modules, we still have
pilers, distribution tools, CVS, source code, dynamicity thanks to our config(8) utility. Its
etc). Similarly to FreeBSD, the non-BSD/ISC used to modify a kernel without recompilation
licensed parts are isolated in the source tree or rebooting. Device parameters that are usu-
so you can easily and safely make commer- ally hard-coded in the kernel can be changed,
cial products and appliances from OpenBSD added, removed... It does not allow you to in-
without putting the responsibility of providing ject new code into the kernel but it gives you
build knobs on us. Talking about knobs, doesnt the choice to enable or disable existing code.

To give you a ”Joe User” example, its often But back on virtualization what is the current
used to disable ulpt(4) (USB printer) support situation on OpenBSD?
to make ones printer seen as a regular ugen(4) AJACOUTOT: Lets start with vmm(4) since its
usable by libusb. been mentioned a couple of times already.
Its somewhat currently in its infancy: it can
only reliably run OpenBSD guests (and with
15 Virtualisation & running alien software
some hacks, NetBSD). That said, it works
BAPT: On FreeBSD we do support running linux really nicely and have a very friendly user
binaries natively via the linuxulator or linux interface with vmctl(8). Since it is chrooted
emulation. Right now we do support both and pledged, breaking out of the VM monitor
linux i386 and amd64, there are patches to sup- means ending up in a chroot with a very
port linux on arm. limited list of allowed syscalls... not much one
Also we do support running old FreeBSD bi- can do.
naries on newer FreeBSD meaning one can run Theres also some ongoing work to be able
FreeBSD 4 userland and binaries on FreeBSD to have qemu(1) working on top of vmm(4)
11, inside a jail if they need to for whatever mostly to take advantage of existing VM
reason. management tools; but theres nothing concrete
On the virtualisation front we are very well yet.
featured, we do support native virtualization The other virtualisation technology that
via bhyve It has many nice features like netmap OpenBSD supports is sun4v logical domains
iface support, vnc server and can run almost (LDOM). Its hardware virtualisation (i.e.
any OS as guest as long as it supports virtio. the resources are partitioned directly at the
But we also support external virtualization hardware level), is supported on SPARC V9
mechanism: xen dom0 and domU (natively), processors and is very secure by design since
virtualbox (via ports). theres no software ”hypervisor” involved:
AJACOUTOT: Well, having Citrix people working
the processor itself runs in hyper-privileged
as developers does help having Xen Dom0 execution mode. It can natively run any
support. I think it boils down to the fact that sparc64 operating systems (Linux included).
you guys made the decision a long time ago
to not fuck with the hardware as much as pos- 16 Out of the BSDs
sible but let hardware manufacturers do it and
instead concentrate on something else. BAPT: Right ok, so we speak about our project in-
On OpenBSD, we prefer to try and convince ternals, but what about the relationship with
the vendors to open their specifications to let external projects.
us write our own drivers. Your approach is In general for us most upstream are happy to
probably more ”pragmatic” but as you prob- receive FreeBSD related patches and integrate
ably know by now, we like to be in control of them even big ones.
our operating system. Not everything is black For example dealing and upstreaming patches
and white of course so this is to be taken with a with LibreOffice is very easy they are very
grain of salt since there is cross-pollination be- welcomed and even tried to setup a CI based
tween our projects and we do use some parts on it.
of your vendor-written drivers (ix(4) for exam- On the GNOME/GLib front while I dont deal
ple). with them much we have very good relation-
BAPT: Being open to vendors is in my opinion a ship with the GLib maintainer who came to us
pragmatic point of view and does not make us to ensure GLib is BSD friendly he even has set
lose control of our sources actually we have up the GNOME CI on FreeBSD to ensure ev-
really good relationship with plenty of them, erything builds as fine as possible.
their committers running through our usual AJACOUTOT: We share a same experience I guess.
process to recruit new developers (mentorship While some upstreams are more opened than
etc). We review the code they commit, and others to non Linux contributions, in general
even extend it sometime. It is not unusual to they are happy to integrate our patches (and
see vendors committers continuing to work on actually sometimes learn a thing or two about
FreeBSD after they stopped working for the portability).
said vendor. Its true that a few identified people within the

FOSS community takes some pleasure in bash- and functions used by debugging tools.
ing anything that is not GNU or Linux but they We also have a ctfdump(1) and ctfconvert(1)
are far from being the norm, fortunately for the implementation and we can dynamically acti-
ecosystem! vate kernel profiling using DTrace-like probes.
Sometimes, upstream can be very close... To So we have the bedrock for DTrace on
give you a few examples, OpenBSD devel- OpenBSD. Fun fact is that OpenBSD is also
oper Jonathan Matthew is also the developer of its own upstream for several software like
GNOMEs Rhythmbox; Jasper and myself have OpenSSH, PF, OpenSMTPD, LibreSSL, man-
commit access to GNOME, Landry Breuil is a doc... which are of course used by other oper-
Mozilla and XFCE developer and Robert Nagy ating systems.
has commit access to LibreOffice. BAPT: It looks like you guys are getting there
That said, some applications can be very com- which is great. So both projects offer nice de-
plex to port so its important to be able to in- bug capabilities, now what we both need are
teract nicely with upstream. And when things more contributors! The tooling is here!
start crashing, well, thats when our debugging Anyway, since we mentioned GNOME and
tools come into action! GLib, lets talk about how we shine (or not) on
I will not go into the details of such tools which the Desktop and multimedia side.
are probably similar to yours, Ill just mention
that we are building the entire base system with
debug symbols (”-g”). We are thinking about 17 BSD as a Desktop
doing the same for ports and provide different
sets of packages (regular and debug ones). AJACOUTOT: Sure, Ill start with audio if you dont
BAPT: FreeBSD is pretty well stuffed for debug- mind... On OpenBSD, system audio and MIDI
ging. By default in base we have ktrace and support is handled by the sndio(7) program-
truss to be able to track what an application ming interface.
is doing. In particular when debugging a cap- Just like FreeBSD OSS, it supports device
sicumized application ktrace is able to tell the sharing, conversions, resampling, per-program
developer what capability his application is volume knob and most features that modern
missing if any. systems have. However, while OSS is a large
More importantly we now also have dtrace in pure kernel implementation, sndio(7) is built
base, which is a very powerful tool to be able around a small user-space daemon (of the size
to debug. When building the FreeBSD kernel of getty(8)): no code does signal processing
for example all debug flags are converted to the with kernel privileges but instead it is done us-
CTF format, so are dtraceable. ing the unprivileged ” sndio” user. It is com-
To help debugging we also build by default the prised of 17 functions (sio xxx) that do one
base system with debug flags (-g) but because thing and while being relatively small, it gives
we do care about embedded systems we extract you network transparency, per application au-
into a separate file. Lldb and gdb are config- dio mixing and virtualization (audio devices
ured to know where to look for them. and MIDI ports are available on the network
Concerning the ports tree, now lots of appli- transparently which allows virtual machines to
cation are dtrace aware, but the ports tree it- use the host soundcard for example), record-
self is not built with debug flags yet, but there what-you-hear device, MIDI ports virtualiza-
are work in progress to be able to extract them tion... In a word, it tried very hard to stay
from the binaries just like in base and provide within the Unix KISS philosophy.
-debug packages. If I am to be honest, there’s one thing that is
AJACOUTOT: That is yet another area that has seen still missing in sndio(7), mixer support: we
a lot of development lately. Our ddb(4) (the still need to use the old SunOS API for that.
kernel debugger) just got basic support for Also when it was first implemented, it did re-
CTF. No that has nothing to do with Capture quire some porting effort in ports compared to
The Flag... OSS (pre-2002 default Linux API that a lot of
Once being included into the binaries, CTF (or programs already supported) but one says: ”no
Compact/Compressed Type Format) will pro- pain, no gain” and as far as I know, OSS still
vide a subset of the information from DWARF has no way to obtain a device block size not to
debugging sections like definitions of data type recovers from underrun and overruns.

BAPT: FreeBSD has a full-featured sound subsys- actively working on the kernel part. Mean-
tem, giving low-latency in-kernel mixing al- ing in FreeBSD vanilla kernel we can only
lowing multiple applications to play sound support up do haswell GPU which is... We
(with independent volume controls enabled by have an external repository where some people
default) with no configuration. are actively working on bringing newer drm
It has an OSS compatible API from userland drivers to FreeBSD with that project one can
perspective. Which makes (made) porting run FreeBSD on nearly all intel chips but this
code to it very easy. is far from being mergeable.
It has multi channel audio support enabled by So I would say that for this we both us sucks.
default (It can go up to 18 channels but is cur- AJACOUTOT: Yes indeed, graphics support can also
rently set to 8 - which correspond to 7.1 sur- be problematic for us; its again mostly due to
round sound), software equalizer. only having a handful of people working on
About the device block size, there are mech- it. Im actually impressed by the work theyve
anism in OSS for that it was even in been doing. But while we make a good job on
the old linux implementation, via SND- having and the userland stack up-to-date
CTL DSP BLKSIZE (which we do support on (version 1.18.4 at this date), we cannot always
freebsd) along with our old AIOGSIZE. In track the most recent changes or features re-
case an application does not take that into ac- quiring kernel assistance. I think on that topic
count (by getting first the block size) a user we are pretty much in the same boat.
can workaround the issue by using a sysctl: BAPT: On the userland part we are not that bad
hw.snd.latency. now. We are on xorg 1.18.4 as well aka al-
The only thing we lack here is network most the latest, we have finally made our duty
transparency where because sndio works on and provided a patch to the xorg configura-
FreeBSD as well, we can use your stuff to have tion mechanism so it can directly detect inputs
this feature. So we can have all our native fea- drivers and hotplugging via devd instead of re-
tures and also benefit all your features for the lying on HAL (something you did long ago).
price of simply installing the sndio package. We also now have evdev support in kernel (will
AJACOUTOT: We do have some limitations when it be in FreeBSD 11.1) which makes it even eas-
comes to the Desktop and as far as I know, you ier to deal with input drivers.
guys have similar ones. The first one would be Recently we have gain wayland support, it has
wireless support. While there are lots of differ- been partially committed it, it is usable as a
ences in supported features and hardware (dif- library and via the scfb driver on EFI with
ferent 11n rate scaling implementation...), Id vanilla kernel. For fully accelerated one, it re-
agree that FreeBSD is now a bit ahead of us in quires features only available for now on the
that regard; mostly due to the fact that you have external project I talked to you about. What
more people working on it. A few years ago, is your situation regarding wayland? And btw
when Damien Bergamini was frantically work- can you explain me how you deal with xorg?
ing on WiFi, it was a different story and I be- I heard of xenocara, is it a fork of Xorg? Is it
lieve you guys did base some of your WLAN part of the base system?
drivers on ours. Anyway we are slowly closing AJACOUTOT: Xenocara is not a fork of Xorg no.
the gap though but we are still missing support In essence its a BSD build infrastructure used
for bwn(4) and mwl(4) as well as Tx aggre- to natively compile and provide Xorg (and a
gation and 40 MHz channels. But even with couple of other graphical projects like fvwm
or without it, I think both projects do lag a bit or cwm) on OpenBSD. It was created by
behind in general when it comes to supporting OpenBSD developer Matthieu Herrb who is
recent WiFi hardware. also an Xorg developer and member of the
BAPT: Well the wireless situation on FreeBSD Foundation. Xenocara was born at the
is not really better on FreeBSD than on time Xorg was split into multiple different
OpenBSD Im not even sure we have more peo- modules and I think it was the easiest way
ple on it than you do :). Often our drivers to maintain it on OpenBSD while keeping
are first imported from OpenBSD and then upstream hierarchy and hence ease ”up and
adapted and extended. down” merging of code. Remember that our
Another issue is graphic drivers, we are al- X(7) server has been running with privilege
ways lagging behind with very few people revocation and separation (using a dedicated

x11 user) for ages. (LC ADDRESS, LC MEASUREMENT...).
Anyway, Xenocara is not part of the base sys- Speaking of which, I am sure you have a nice
tem per se but is part of the default OpenBSD story about implementing LC COLLATE in
installation. Its available in the same way as FreeBSD libc...
our base system sets, that is there are no X BAPT: Well on FreeBSD most LC * were mostly
packages. You can see it as in-between ports abandoned for a while, I recently decided to
and base. update those and because it is a nightmare to
The important thing is that it is assumed to deal with all the individual encodings I decided
be found on a standard OpenBSD installation to revive a very old project we have which
which is an important fact for packages(7) for basically means take from CLDR all the re-
example. sources we will never be able to adapt ourself
BAPT: I see. By the way, since we are talking about and generates those files from there. Along
the Desktop and by extension, the user experi- with adding unicode collation support to our
ence, how is your localisation support? libc which is a nice story of collaboration be-
AJACOUTOT: Regarding character encoding sup- tween FreeBSD, Dragonfly and Illumos but
port, the OpenBSD base system deliberately that is another story.
supports US-ASCII and UTF-8 only. It allows It is really great to have unicode collation in
for much better error handling and much more particular for database indexes and more.
stable operation than a system supporting A fun side effect is that apparently no one ex-
arbitrary character encodings. where a single pect what unicode collation means and that
damaged byte in a text file often results in ASCII is not the only characters available. So
all the rest of the file to be unreadable. It is sorting in a locale aware unicode world results
required that the system is UTF-8 only, or in a behaviour which can be different from
you don’t get the benefit. I believe that on what people may expect. In particular for basic
FreeBSD, if there is a single invalid byte, unix tools like tr and friends, where [a-z] does
cut(1) loses the complete rest of the input file not mean the same as in ASCII.
even when you use LC CTYPE=en US.UTF- All our base tools are locale aware but given
8. Because ASCII is compatible with UTF-8, we did not have unicode collation then no
we can enable UTF-8 by default in most one noticed that, since we have unicode col-
programs while making the terminal safer lation people tends to get surprised. On the
and more resilient against control character GNU world they ”fixed” that making those
attacks. A visual example of the differences tools not locale aware. I prefer to make peo-
is pod2man(1), which is UTF-8 enabled by ple learn about classes which were made for
default on OpenBSD, which FreeBSD cannot that: [:lower:] for example.
do because they support arbitrary locales. AJACOUTOT: Lets go back to our current topic: the
Consequently, non-ASCII characters work Desktop. Regarding the GNOME Desktop en-
in Perl manuals on OpenBSD, but not in vironment, a huge amount of work has been
FreeBSD; compare ”man perlunicook” on done to port it by Jasper and myself because
both systems. we had a need to deploy and maintain a few
Of course, it’s OpenBSD so there’s also the thousands OpenBSD-based desktops. I think
question of security. If terminals are allowed weve been ahead in that regard for a few years
to use arbitrary encodings, no text in any until you guys catched up thanks to Koop Mast
encoding can be safely displayed, because and the FreeBSD GNOME team. We actually
depending on the encoding, anything might be collaborated quite a lot and I am still wander-
a control character. With UTF-8 only, ASCII ing on the freebsd-gnome IRC channel! I re-
is always safe for display. ally appreciate when we are able to work to-
To summarize, FreeBSD attempts gen- gether for the greater good of the BSDs in gen-
eral locale(1) LC * support. OpenBSD eral.
deliberately does not, completely ignor- Besides that its surprising but OpenBSD actu-
ing LC COLLATE, LC MESSAGES, ally supports most ”fashion desktop comput-
LC MONETARY, LC TIME, NLSPATH, ing” softwares like DBus, ConsoleKit, PolKit,
and in particular LC NUMERIC. I hope chromium, Firefox, LibreOffice, KDE, XFCE,
that you guys are not going to implement CUPS...
the insane non-standard GNU extensions BAPT: About desktops we are pretty much up to

date on GNOME (mainly maintained by Koop OpenSSH security advisories, most of them
Mast whom collaborate with you for upstream- are related to PAM.
ing), KDE, xfce, mate are available in their BAPT: In the case of OpenSSH security advisories,
latest version as far as I know. Firefox is I think it is also probably related to the fact that
well maintained (probably more contributors upstream is paying less attention (for the obvi-
would help) the nice thing about Firefox is ous reasons you stated above) but that is true
the collaboration between the FreeBSD and the that pam API is not simple to use and so eas-
OpenBSD folks. LibreOffice is also in pretty ily error prone. In the OpenPAM implemen-
good shape very few patches necessary. The tation, while it is compatible with the official
main issue I can see in ports now is proba- API there is also plenty of helpers to simplify
bly Google chrome, while it works and I have adding support for pam in applications so that
to express a very big Kudos to the maintain- it is less error prone.
ers, upstream is really unfriendly is part of the For name services we do use NSS which pro-
very few upstreams that actually completely vides us lots of flexibility through its modular
ignore the BSD related patches most of the nature. Note that our NSS api are not 100%
time. To have it in ports we do have to main compatible with the GNU libc one.
484 patches.... Large part of them are very easy We also provide a nscd daemon which can
to upstream... cache name service response (per user), but is
not limited to that, It can also performs the re-
18 Authentication, authorisation quests which means the modules are no longer
loaded into the libc but through that dedicated
BAPT: For the authentication we do use PAM, ac- daemon.
tually the OpenPAM implementation (devel- AJACOUTOT: Regarding authorisation and virtual
oped and maintained by a FreeBSD developer) system users, we only have support for tradi-
which is compatible with the PAM specifica- tional YP/NIS. OpenBSD does not use NSS
tion. It gives us access to lots of various ex- (nsswitch.conf(5)) for basically the same rea-
ternal modules even if for most of them we do sons it does not use PAM, we do not want dy-
prefer relying on our own implementation. The namically loaded modules play games in our
nice thing is it makes it simple to port common libc and resolved. We do have support for
softwares as they all support pam. getting users from an LDAP server thanks to
AJACOUTOT: BSD Auth originally came from ypldap(8). Its a daemon providing YP maps
BSD/OS (BSDi) and was later adopted by using LDAP as a backend. It is a replace-
OpenBSD. One of the differences between ment for the traditional ypserv(8) daemon and
BSD Auth and PAM is that PAM ”modules” is compatible with any ypbind(8) implementa-
are libraries which must be loaded into the tion . Actually I see that you guys imported it
application. BSD Auth ”modules” are effec- some time ago and I am interested why, was
tively separate applications or scripts located nss-ldap not enough or is it again for satisfy
under /usr/libexec/auth/ that are run as a sepa- ”flexibility”?
rate process from the one authenticating allow- BAPT: I still wonder myself. It does not hurt but...
ing them to communicate over a simple IPC hey. I know at least one university that was
interface. That means we never expose the happy to see it so they could have simpler steps
credentials store to possibly buggy software. to migrate to full LDAP according to one of its
It matches the traditional privilege separation administrator.
model we are used to on OpenBSD while still
being able to provide different way to authenti-
cate (LDAP, Kerberos, YubiKey, RADIUS...). 19 conclusion
While PAM may be a bit more flexible and way
more commonly found (so there are a lot of AJACOUTOT: So, all things considered, I think its
different authentication modules readily avail- pretty obvious that my BSD sucks less than
able), it usually requires elevated privileges to yours!
authenticate. BAPT: I agree to disagree on that one, its clear my
Applications using BSD Auth only need to BSD sucks less than yours!
be in the ” auth” group to be able to run the That said, I think there are areas where we suck
/usr/libexec/auth/ helpers. Just look at your equally, like wireless or display drivers...

AJACOUTOT: That is very true, we do suck on these I suppose there is some truth in both stereo-
aspects! types.
BAPT: But as much as we like to make fun of each I see OpenBSD as some kind of incubator and
other, we are not only sharing bad things. bedrock for new technologies that is not afraid
AJACOUTOT: Indeed, I think cross-pollination be- to break things; sort of a destroy to build ap-
tween our two operating systems works quite proach. I love its simplicity and the fact that I
well. The ix(4) driver comes to mind, or can get on to my task at hand instead of having
mandoc(1)... we actually do exchange lots of to fight the operating system.
things and it would not make sense to list ev- I would encourage people to it try out as a
erything here. power user or developer. Not just install it but
BAPT: OpenBSD has imho an important role, in really try using it. In my experience, besides
the Open Source land and more, You guys are the obvious benefits OpenBSD is known for
tackling very important project which would (proactive security and all), it has been one of
probably never have happened otherwise. The the easiest and best documented operating sys-
most famous one that comes to my mind is tems I have ever used, especially considering
OpenSSH, I really like how open you are the amount of features it offers you on a de-
to portability for those softwares given the fault installation.
amount of extra work it requires to do that: BAPT: To conclude, we just wanted to mention that
OpenSMTPD, tmux, mandoc, sndio are very most topics we covered here could have had
good example of that. Often teaching up- their own paper.
streams about good (and secure) coding prac- AJACOUTOT: Indeed and thats the reason some
tices. were not covered very deeply or at all.
AJACOUTOT: I think FreeBSD is important in the
global ecosystem. Its a real ”enterprise” oper- Notes
ating system and I think it is slowly filling the
spot left by Solaris. It bundles some amazing
pieces of technology and in some area is still 3
on the edge of innovation. Some very large en- PortsFlavors
tities use it (WhatsApp, Netflix...) and thanks 4
to FreeBSD, a lot of people have been made 6
aware of the BSD community in general. 7
For me it is a good weapon to make people 8

aware that ”fringe” operating systems are cer- 9
tainly not lagging behind Linux. 11
BAPT: For me in the FreeBSD is a wonderful Op- 12
erating Systems very flexible making attractive 13

for almost all use cases. The project is very freebsd-glossary.html#pola-glossary
open, and everyone from vendors to individual 15
have their place, while there are lots of vendors 16
that contributes to FreeBSD, the project re- 17
mains completely community driven and indi- 19
vidual can easily find their path in the project. 20
As an example in less than 2 years I have been 21
able to bring very important modifications in 22
the project and drive lots of the directions the 24
project has taken. 25
AJACOUTOT: Were a small project but I am proud 26

when I see that in some areas twenty or so layout_randomization
hackers can compete with a huge project like 28
FreeBSD and sometimes deliver things ahead. 29
We do serious things without taking ourselves
too seriously.
You guys say our performance sucks, we say
your security sucks.