Sharing desktops one window at a time

Thomas Thurman
Collabora Ltd
thomas.thurman@collabora.co.uk

Compsoc — Durham University — 26 October 2010

Abstract
VNC, a technology to interact with arbitrary windows on a remote computer, is widely deployed.
However, it is capable only of sharing the entire desktop at once, which is not always necessary or
desirable. In addition, because it commonly uses TCP as a transport, it requires knowledge of the
network location of the remote computer, and it has difficulty in operating in environments using
network address translation. In this paper I present xzibit, a system which addresses both these
concerns.

Introduction
Virtual Network Computing (VNC) [Richardson, 2009] is a system developed by Olivetti in the
1990s. It sends an image of the entire desktop on one computer over the network to another
computer, and keeps this image continuously updated by sending a delta from time to time. This
image may either fill the display of the remote computer or appear in a window. The receiving user
(henceforth the remote user) may interact with the image of the desktop, and these remote mouse
actions and keypresses are sent back over the network to the originating user (henceforth the host
user).

This system is useful and widely deployed. However, it suffers from two major problems. Firstly,
it is not possible to send windows one at a time. The entire desktop must be sent at once. Secondly,
the choice of TCP as a transport means that the network location of the remote user must be known
to the originating user. It also implies that the remote user's computer must have an address which
is routable from the host user's computer On modern networks this is often not the case. For
example, the remote user may be behind a firewall which uses network address translation (NAT) to
share a few publicly-routable IP addresses amongst many private addresses. These private
addresses are not publicly routable, and therefore VNC to them becomes a difficult proposition.

In this paper I present xzibit [Thurman, 2010], a program which deals with both these issues. Its
primary use is to send images of windows, rather than of desktops, across the network. It uses a
novel protocol for this purpose which frames the VNC protocol, RFB. In addition, it uses
Collabora's system of Telepathy Tubes. This allows the protocol to be transported over instant
messenger connections, thus allowing connectivity to anyone on the host user's roster of contacts,
whether behind a firewall or on a public IP address.

Method
xzibit is implemented as a plugin for Mutter, a compositing window manager. In order to
composite the visible windows, Mutter keeps an image of each one in main memory. It is intended
that xzibit will be capable of reading these images in order to produce VNC streams, but at present
this is not possible from a Mutter plugin. Therefore,, at present xzibit reads images directly from
the X server, with a considerable performance penalty. This will be addressed in a later release.

Sharing desktops one window at a time 1 T. Thurman

Both the back and the front end are implemented with standard libraries. The front end is supplied
by gtk-vnc, as used in GNOME's vinagre package. This work is delegated for reasons of
encapsulation to a separate process, xzibit-rfb-client.

The back end is supplied by libvncserver, as used in GNOME's vino package. However, standard
libvncserver is unable to communicate other than over TCP sockets; we use a patched version
which may also communicate over Unix domain sockets, thus allowing us to multiplex the supplied
VNC data with VNC data from other windows, and with the control channel. This patch is supplied
in xzibit's source control, and is under consideration upstream.

The question may be raised as to how xzibit differs from ordinary X11 forwarding. In ordinary X11
forwarding, an application running on one host may display windows on a display attached to
another host. There remains only one window. Under xzibit, there are always two windows on two
displays, either of which may be interacted with by the remote or host users.

User interface
The original user interface involved a patch to Mutter's theming code to draw a coloured envelope
on the window's title bar. A black envelope meant the window was unshared; a red envelope meant
it was currently being shared; a cyan envelope meant the window was being received; no envelope
meant it was not a shareable window. (For example, xzibit's own dialogue boxes are marked in this
way.)

However, a patch to the window manager itself proved unwieldy, and this system was abandoned.
In its place we have an application which presents a window containing a single button. Clicking
this button causes a pointer grab, which allows the host user to select a window to share. If the
chosen window is already shared, it becomes unshared; otherwise, the host user is presented with a
further dialogue containing a list of contacts who have the capability to receive the window.

Protocol
Serious consideration was given to using the extension system built into RFB, the existing protocol
for VNC, but it proved difficult to represent multiple windows over such a connection. One
possibility would have been to composite all the window images on an imaginary desktop, and to
send extension messages to cut the appropriate rectangles out from the whole. This was abandoned
as being unwieldy when windows were resized.
Instead, a novel protocol was developed which multiplexes one or more streams. The primary
stream is the control channel, which is always open. Messages may be sent representing events, or
requesting the creation of new channels. A second type of channel is the RFB channel; which
contains VNC data representing a single window. One of these channels exists in the main
connection for every window shared. Other types of channel, such as audio channels representing
the sounds produced by the clients owning each window, may be created from time to time.
The multiplexed stream is then forwarded over Telepathy Tubes [McQueen, 2008]; this involves the
whole multiplexing stream being base-64 encoded and transmitted over an instant messaging
system such as XMPP.

Cursors
Mouse clicks and keypresses need only be transmitted from the remote user to the host user.
However, mouse motion notifications need to be transmitted in both directions. This is because

Sharing desktops one window at a time 2 T. Thurman

xzibit displays two mouse pointers for each window: one is the usual X pointer, and the other is a
doppelganger pointer, which represents the position of the ordinary X pointer on the other screen.
When that pointer is not over a shared window, the doppelganger is invisible.
VNC already sends mouse motion notifications from the remote user to the host user. Thus we
must send motion notifications from the host user to the remote user in a different way, outside the
VNC protocol. In addition, a notification that the remote user's pointer has left the shared window
cannot be expressed by means of the standard VNC notifications. Both these problems are solved
by use of a specialised message on the control channel.
To facilitate the building of doppelgangers, each xzibit instance will attempt to locate an icon
representing its user, and will send it over the control channel on connection. This image is known
as a avatar. A useful extension would be to plot text spelling the user's login name if no other
avatar could be created.

Challenges
In this section we discuss some of the reasons this problem is difficult.

Menus
Under X11, menus are traditionally implemented using override-redirect windows. This means that
the window manager does not pass certain events on to the window manager, which makes such
windows difficult to reproduce from within the window manager. Experiments with the creation of
actual override-redirect windows were fruitless. A further plan is to paint ersatz menus onto the
rectangular surface of their parent windows, and to use the shaped window extension [Packard,
1989] if the menu extends beyond the edges of that rectangle.

Resizing windows
Each VNC connection multiplexed in an xzibit stream believes the window it represents to be an
entire desktop. Since VNC has provision for resizing desktops¸ there is no great difficulty in theory
in resizing windows. However, this has not yet been tested: all xzibit windows remain of fixed
dimensions.

Faking events
When xzibit receives notification of a keypress or mouse click by the remote user, it must fake an
appropriate keypress or click event on the real window, found on the host user's desktop, so that the
controlling X client believes the transmitted event happened locally.
The standard X11 extension for faking events in this way is xtest [Drake, 1992]. Although xtest is
widely available, it is not ideal for our purposes: it allows keyboard input only to the window with
the focus. A much more useful, but much less widespread and much less stable solution is found in
the new xinput2 extension [Hutterer, 2008]. xzibit attempts to use xinput2 if available, though it
can be configured to use xtest if not.

Testing
The body of unit tests which were originally developed for xzibit presupposed the use of TCP as a
transport. They generally remain to be ported to the newer architecture based on Telepathy Tubes.

Sharing desktops one window at a time 3 T. Thurman

A unit test requires the ability to send and receive from the same host; therefore a pair of accounts
must be found, both of which are logged in from the current machine, and both of which have the
other on their contacts roster. Such a pair is known as a loopback. There is existing code to find
these pairs.

Permissions
It is unhelpful for windows simply to appear without warning on the remote user's desktop. It is far
preferable if the remote user is given a dialogue box asking permission to share windows. To this
end, the protocol has provision to refuse service to the host user until the remote user's end sends a
permission frame. This has not yet been implemented, however.
The common implementation will ask once at the first occurrence of sharing, and implicitly give
permission for all later windows. Some users have requested to be asked at the sharing of every
new window. This is problematic because it would require all transients— that is, all dialogues, and
even all menus— to request permission before appearing, unless some way could be found to give
implicit permission to all transients and to no intransients, and to force windows to keep the
transiency they originally claimed. However, the protocol is capable of supporting such an
arrangement.

Conclusion
It is demonstrably possible to share some subset of a desktop's windows, rather than sharing the
entire desktop, over the network with minimal lag. The system is already useful, but much more
work remains to be done before xzibit is ready to share with the general user.

References
• Drake, 1992. Xtest extension library.
http://www.x.org/releases/current/doc/Xext/xtestlib.pdf
• Hutterer, 2008. Redefining input in X.
http://linux.conf.au/conf/2008/programme/thursday.html
• McQueen, 2008. Breaking the silence: making applications talk with Telepathy.
http://people.collabora.co.uk/~robot101/telepathy-lca-2008.pdf
• Packard, 1989. X Nonrectangular Window Shape Extension Library.
http://www.x.org/docs/Xext/shapelib.pdf
• Richardson, 2009 The RFB protocol. http://www.realvnc.com/docs/rfbproto.pdf
• Thurman, 2010. Xzibit source control. http://git.collabora.co.uk/user/tthurman/xzibit/.git
• Thurman, 2010. Xzibit. http://telepathy.freedesktop.org/wiki/Xzibit.

Sharing desktops one window at a time 4 T. Thurman