Integrated Bubblewrap Support in LiteSpeed Web Servers

The LiteSpeed family of high performance web servers offers a number of ways to isolate requests, thereby providing protection against attack. In upcoming releases of LiteSpeed Enterprise (6.0) and OpenLiteSpeed (1.6.15), we have added integrated support for bubblewrap by Flatpak, providing even greater isolation and protection against attack.

What’s so great about bubblewrap?

Bubblewrap is a lightweight sandbox application written by Flatpak, and is described in their wiki here. What it does is implement Linux namespaces, which basically gives the application (LiteSpeed in this case) a full sandbox, which includes operating-system-supported isolated mounts, user/group IDs, interprocess communications, users, cgroups, host names and more.

For example, with isolated mount namespaces: each request can only see the file system designated for that request and can even be denied write access to any files within any directories you specify.

With isolated users and groups, the user will only be able to see their own user ID, group ID and capabilities. The default even creates independent /etc/passwd and /etc/group files with the contents you specify.

With isolated interprocess communications each process can only see the named pipes, or Unix Domain Sockets you specify, which limits the applications that a process can contact. It’s impossible to communicate with applications without specified file system access.

Similar to, but perhaps less comprehensive than, CloudLinux CageFS, a bubblewrap’d process is utterly independent. Thus significantly reducing the opportunities for an attack.

Why integrate it?

Use of bubblewrap may be configured into previous versions of LiteSpeed (or other web servers), however, it requires complex reconfiguration of your server in order to call the bwrap program directly, and call your program from it. By integrating it into LiteSpeed, bubblewrap works for a large number of application types and requires a very simple configuration change.

Integration allows it to operate on CGI applications as well as FCGI applications, which would not be possible without integration.

How to bubblewrap in LiteSpeed Web Server

In most cases you can enable bubblewrap in your existing LiteSpeed configuration by setting Bubblewrap Container to On in your security configuration. Details for OpenLiteSpeed are here.

If you do not customize a Bubblewrap Command (and most users will not) it will use the default of:

/bin/bwrap --ro-bind /usr /usr --ro-bind /lib /lib --ro-bind-try /lib64 /lib64 --ro-bind /bin /bin --ro-bind /sbin /sbin --dir /var --dir /tmp --proc /proc --symlink../tmp var/tmp --dev /dev --ro-bind-try /etc/localtime /etc/localtime --ro-bind-try /etc/ld.so.cache /etc/ld.so.cache --ro-bind-try /etc/resolv.conf /etc/resolv.conf --ro-bind-try /etc/ssl /etc/ssl --ro-bind-try /etc/pki /etc/pki --ro-bind-try /etc/man_db.conf /etc/man_db.conf --ro-bind-try /home/$USER /home/$USER --bind-try /var/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock --bind-try /home/mysql/mysql.sock /home/mysql/mysql.sock --bind-try /tmp/mysql.sock /tmp/mysql.sock  --unshare-all --share-net --die-with-parent --dir /run/user/$UID '$PASSWD 65534' '$GROUP 65534'

The key to the isolation is the bubblewrap parameters --unshare-all and --share-net.

• --unshare-all specifies all of the isolation mentioned above.
• --share-net is required to allow your application access to networking.

Let’s see the isolation this gives you. The first parameter is simply the location of the bubblewrap program. The rest of the parameters will:

• Mount the following directories (and their subdirectories) as read-only:
  • /usr
  • /lib
  • /lib64 (if it exists)
  • /bin
  • /sbin
  • /etc/ssl (if it exists)
  • /etc/pki (if it exists)
  • /home/(your user’s directory) (if it exists)
• Mount the following files as read-only:
  • /etc/localtime (if it exists)
  • /etc/ld.so.cache (if it exists)
  • /etc/resolv.conf (if it exists)
  • /etc/man_db.conf (if it exists)
• Mount the following files as read-write:
  • /var/lib/mysql/mysql.sock (if it exists)
  • /home/mysql/mysql.sock (if it exists)
  • /tmp/mysql.sock (if it exists).
• Create the following empty directories:
  • /var
  • /tmp
  • /run/user/(your user’s ID)
• Create symbolic links:
  • from ../tmp to var/tmp
• Provide minimal access to:
  • /proc
  • /dev
• Terminate when LiteSpeed terminates
• Create custom:
  • /etc/passwd file with only your user’s entry and 65534 (nobody)
  • /etc/group file with only your group’s entry and 65534 (nogroup)

Most programs will be able to run with these defaults and the result is great protection with minimum configuration.

If your program is not in one of the mounted directories, uses a file not in the list of files or directories, or needs write access to a file not in the writable list, your program won’t be able to access what it needs and will fail. In that case you will need to create a customized command line.

How to customize it

The customization is all in the Bubblewrap Command command line. Basically it uses the bubblewrap parameters from your version of bubblewrap with some additional tokens which LiteSpeed adds to provide better customization:

• $USER is replaced with the actual user name of your user.
• $UID is replaced with the user ID of your user.
• $GID is replaced with the group ID of your group.
• $PASSWD is replaced with the creation of a custom /etc/passwd file with a single user line in it, for the user you’re running as. If you enclose this option in single quotes you can add space-separated additional users. The default is ’$PASSWD 65535’ which creates a 2 line /etc/hosts file, with your user and the nobody user.
• $GROUP is just like $PASSWD but creates a custom /etc/group.

Note that most users will be able to use the default Bubblewrap Command by leaving it blank.

A simple example

The simplest example is the worst because it gives your application no additional protections. Set Bubblewrap Container to On and Bubblewrap Command to:

/bin/bwrap --dev-bind / /

A better example

This is a much better example of a Bubblewrap Command as it mounts even fewer files and directories than the default. It includes no /var, no /etc/man_db.conf, no database sockets, and only your user in /etc/passwd and your group in /etc/group:

bin/bwrap --ro-bind /usr /usr --ro-bind /lib /lib --ro-bind-try /lib64 /lib64 --ro-bind /bin /bin --ro-bind /sbin /sbin --dir /tmp --proc /proc --dev /dev --ro-bind-try /etc/localtime /etc/localtime --ro-bind-try /etc/resolv.conf /etc/resolv.conf --ro-bind-try /etc/ssl /etc/ssl --ro-bind-try /etc/pki /etc/pki --ro-bind-try /home/$USER /home/$USER --unshare-all --share-net --die-with-parent --dir /run/user/$UID $PASSWD $GROUP

Give it a try!

To try it, upgrade to a version of LiteSpeed Enterprise or OpenLiteSpeed that supports bubblewrap, and turn it on for an application. This will show you the power of an integrated sandbox like bubblewrap inside of a powerful web server like LiteSpeed.