Xen vchan port

Given support from the hypervisor, virtual machines (VM) running on the same virtual machine monitor can communicate through shared memory, in addition through network. Xen provides the grant table mechanism, a per-VM (domain) handle table that Xen actively maintains. Each entry points to a machine page that can be shared between domains. In addition, Xen event channels–an asynchronous notification mechanism–can be associated with the shared memory region. A VM can choose either polling or being notified on shared memory communication.

This shared memory (Xen grant table) plus asychronous notification (Xen event channel) setup has been seen quite commonly in Xen paravirtualization interface. For instance, it is used Xen console (DomU - Dom0), Xen Store (Domu - Dom0), and networking (front end - back end). One could setup shared-memory based communication between two random (unpriviledged) domains use the exact same mechanism. That is, rely on some directory service (Xen Store) for peer discovery and set up the data structure (e.g. a ring buffer) for communication. Sending and receiving is done by copying data to and from the shared memory. This is what Xen’s libvchan does.

libvchan origiated from Qubes project, mainly used for inter-domain communication. It later got adopted by Xen and has been substrantially rewritten since. libvchan out-performs the original version in Qubes, and is the default after Qubes 3.0.

2 Xen libvchan interface

Shared memory communication on libvchan involves two parties–a server and a client. Server offers the memory used for communication, and advertises its credentials (grant ref(s) and event channel) in the directory service Xen Store provides. Assuming the client knows the server’s domain id and service path, client picks that information from the Xen Store and maps the server’s offering into its own virtual address space. After performing the rendezvous, they can start to talk to each other.

Communication happens on separate rings. Server and client each has their own dedicated ring buffer for writes so they won’t race against concurrent writes. Server reads from client’s ring buffer and vise versa. The credentials for each ring is maintained in a shared control data structure.

Sharing is based on page granuality. libvchan allows user to specify the size of the ring, and it is bounded by the grant table size limit for each domain. Server and client can choose their behavior to be blocking or non-blocking. Figure 2 shows an overview of communication setup using libvchan.

libvchan provides two sets of interfaces for streaming-based communication as well as packaet-based one. We only concern about packet-based interface in this document.

2.1 Setup and teardown

1/**
2 * Set up a vchan, including granting pages
3 * @param logger Logger for libxc errors
4 * @param domain The peer domain that will be connecting
5 * @param xs_path Base xenstore path for storing ring/event data
6 * @param send_min The minimum size (in bytes) of the send ring (left)
7 * @param recv_min The minimum size (in bytes) of the receive ring (right)
8 * @return The structure, or NULL in case of an error
9 */
10struct libxenvchan *libxenvchan_server_init(struct xentoollog_logger *logger,
11                                            int domain, const char* xs_path,
12                                            size_t read_min, size_t write_min);
13/**
14 * Connect to an existing vchan. Note: you can reconnect to an existing vchan
15 * safely, however no locking is performed, so you must prevent multiple clients
16 * from connecting to a single server.
17 *
18 * @param logger Logger for libxc errors
19 * @param domain The peer domain to connect to
20 * @param xs_path Base xenstore path for storing ring/event data
21 * @return The structure, or NULL in case of an error
22 */
23struct libxenvchan *libxenvchan_client_init(struct xentoollog_logger *logger,
24                                            int domain, const char* xs_path);
25/**
26 * Close a vchan. This deallocates the vchan and attempts to free its
27 * resources. The other side is notified of the close, but can still read any
28 * data pending prior to the close.
29 */
30void libxenvchan_close(struct libxenvchan *ctrl);

Listing 1: xen/tools/libvchan/libxenvchan.h

2.2 Send and receive

1/**
2 * Packet-based send: send entire buffer if possible
3 * @param ctrl The vchan control structure
4 * @param data Buffer for data to send
5 * @param size Size of the buffer and amount of data to send
6 * @return -1 on error, 0 if nonblocking and insufficient space is available, or $size
7 */
8int libxenvchan_send(struct libxenvchan *ctrl, const void *data, size_t size);

10/**
11 * Packet-based receive: always reads exactly $size bytes.
12 * @param ctrl The vchan control structure
13 * @param data Buffer for data that was read
14 * @param size Size of the buffer and amount of data to read
15 * @return -1 on error, 0 if nonblocking and insufficient data is available, or $size
16 */
17int libxenvchan_recv(struct libxenvchan *ctrl, void *data, size_t size);

19/**
20 * Waits for reads or writes to unblock, or for a close
21 */
22int libxenvchan_wait(struct libxenvchan *ctrl);

Listing 2: xen/tools/libvchan/libxenvchan.h

2.3 Query

1/**
2 * Query the state of the vchan shared page:
3 *  return 0 when one side has called libxenvchan_close() or crashed
4 *  return 1 when both sides are open
5 *  return 2 [server only] when no client has yet connected
6 */
7int libxenvchan_is_open(struct libxenvchan* ctrl);
8/** Amount of data ready to read, in bytes */
9int libxenvchan_data_ready(struct libxenvchan *ctrl);
10/** Amount of data it is possible to send without blocking */
11int libxenvchan_buffer_space(struct libxenvchan *ctrl);

Listing 3: xen/tools/libvchan/libxenvchan.h

3 libvchan operations

4 Ethos Port

In order to support libvchan on Ethos, we need to provide operating system dependant hooks for Xen-PV operations below:

Normally this is done by providing OS hooks in individual libraries under xen’s xc interface, i.e. libxenevtchn, libxengnttab, libxenstore. By doing so, libvchan can be used in user space. We don’t need to support this semantics in Ethos, so we can short-circuit by directly plug-in Ethos kernel functions to implement desired operations.

Xen libraries wrap around a layer that operates on “handles”, an indirection of the above-mentioned PV operations. The hanlde is essentially a file descriptor that indexes to the file descriptor table. In order to port libvchan, we need to support this abstraction by operating on the integral handles.

4.1 Xen Event Related

4.2 Xen Grant Table Related

Similar to Xen event channel operations, grant table library operates on grant table “handles”–one layer above the raw grant table interface. The operations on Xen grant table hanldes are:

The rest of grant table operations can be divided into two sets: the server side which offers the physical backing memory for sharing and advertises the shared memory, and the client side which takes the advertised credentials and maps the shared memory.

4.3 Xen Store

Xen Store is the directory service that is used for server advertising and client rendezvousing. Server writes the credentials associated with a vchan to Xen Store, and client reads them off.

Xen Store operations also happen on a handle. The handle related operations are:

4.4 The Rest

Also, low-level operations to modify fields atomically in shared data structure:

1 Overview