Tutorial - The Definitive Guide to Chrooting
The Definitive Guide To Chrooting
INTRODUCTION
What
This document describes a general process for chrooting services running on a Linux based operating system. It is assumed that you are already familiar with basic UNIX systems administration, although some explanations are given where I fell they are warranted. If you are not already familiar with this, I would not recommend undertaking this process at this time, as the errors which will inevitably occur will be a nightmare to troubleshoot without this knowledge. You do not need to be a C guru, or for that matter a networking guru. But a basic famliarity with the processes described in here is definitely needed.
It was written because, although I was able to find documents and HOWTOs explaining how to chroot individual services, I could find no general framework document explaining the process behind it. Several services will be used as examples here, as they will apply in different situations. Keep in mind though, that this process must be interpreted and adapted for each service you will be chrooting using this method. No one service can be chrooted in the same way as another.
It is not a guide to securing your services. While some suggestions will be given for the sake of clarity, I cannot possibly cover the security of each and every application you may wish to chroot with this process. Chrooting in and of itself is another layer of security, but it is not the last. There are plenty of good quality guides and HOWTOs available on how to secure most services available for your system. I suggest reading the documentation on the developer website, or looking at The Linux Documentation Project, at www.tldp.org
Why
I will not explain the benefits and risks of chrooting your services. I assume that if you are reading this, you already know what these are. However, I would like to comment on one distinct advantage of chrooting all your services, as opposed to chrooting only a few to limit exposure to risk. Once all your services have been chrooted, it becomes a simple matter to control inter process communication via a firewall over the loopback interface. This provides for a much finer grain of control over your process communications and data flow than do normal POSIX permissions.
Where
There is no official home for this document as of yet. Hopefully somebody will offer me one.
How
I wrote this document based on my experiences after chrooting all my public services on my home server. I was rather frustrated with lack of documentation on the process, and with the assumption that what applies to one version of a service will apply to the next. I found that these documents went out of date rather quickly. After chrooting most of my services on several server build, I learned a general process for doing so by trial and error. This document is the end result of all those errors. It is the order that comes from my chaos.
While this method works for me, on my systems, it may not work for you. Every attempt has been made to make this document as general as possible, but a process such as this can never apply to everything. I welcome any and all comments and suggestions for improvements. Even insults are welcome if they make me laugh.
CREATING A TEMPLATE
The first and most helpful step in this process is to create a template to use as a reference when building the chroot jail. To do this, compile the application as normal, but install it to an independent directory, such as /usr/local/named, in the case of bind. The idea is to have available to you a sample of what the jail will look like once it has been built. Do not configure the application to install to the root of your filesystem, such as with a prefix of / or /usr. This will intersperse the installed files with the rest of the system and make them nearly impossible to find. If they are sitting on their own in a seperate, independent directory, you will easily be able to see which files you need to install to the chroot jail.
Once the tamplate has been made, do a make distclean in the root of the source tree of the application you are working on. This will delete any files created by the configuration and compilation process, and any saved configuration and cache files as well. In other words, you will be starting from scratch. If distclean is not a valid make target, a make clean will usually work. Otherwise, you can always just delete the source tree and re-extract it. Once a template has been made, you are ready to begin the process of building the application to function within the chroot jail.
CONFIGURING THE APPLICATION
The first rule, and most important rule, of chrooting any service, is to build that service from source code. Most applications come with easily configurable and compilable source code packages ready for download. Have a look at the README and INSTALL files for the steps necessary to compile each service you will chroot with this method. It would be quite difficult, if not impossible, to chroot most services using precompiled packages, as the directory search paths are quite often hardcoded into the binaries and not user configurable. The second rule of chrooting a service is:
DO NOT CONFIGURE SERVICES TO INSTALL TO YOUR CHROOT DIRECTORY!
That is, suppose you are installing your name server to /chroot/named. Do not configure it with a configure option such as
This will create a whole host of references within the application itself to /chroot/named, but while chrooted, /chroot/named is actually the root of named's filesystem. While this can be solved, if necessary, by creating a symlink within the chroot directory, such as /chroot/named => /, this will force every last library lookup and function call to resolve that reference, using unnecessary CPU time. It also creates configuration nightmares. This is a very common mistake. In general, configure everything to the root directory, or /usr, if that is what you prefer. Make sure also that system configuration directories are explicitly specified on the command line, such as /etc, or /var/mysql in the case of mysql. A simple configuration line for a chrooted mysql server would look like this, to avoid unnecessary /chroot/mysql lookups and storing databases in /chroot/mysql/var/mysql:Code:./configure –prefix=/chroot/named
Other configuration options can be specified as needed, usually without problems. The goal is to eliminate the need for a /chroot/[app] => / symlink within the jail, creating an environment which will be indistinguishable to the service from your system. Once configured, compile the application as you normally would.Code:./configure –prefix=/usr –sysconfdir=/etc –localstatedir=/var/mysql
INSTALLING THE APPLICATION
Some packages have an option available which allows you to install the compiled code to a directory other than the one you specified on the configure line, usually with something like:
This is the easiest method. Unfortunately, not all packages come with this option. You could edit the Makefiles manually, however, this often proves to be rather complicated. Failing the ability to install to a directory you did not configure the package for, you must install the needed files manually by copying them into the chroot jail. This is easier if you first configure and build the application to install to an independent directory, such as /usr/local/mysql. You can then check that directory for any files which are installed normally. Then, after configuring & compiling to the root, you have a template of what the chroot jail should look like. Be aware, however, that some configuration scripts ignore the /usr/local/mysql prefix and install configuration files to /etc anyway. Check certain likely directories for the presence of these files, such as /etc /var, and others which may be subject to this behaviour.Code:INSTALL_DIR=/chroot/mysql make install
Setting up the jail environment
You should have some idea of what the jail will eventually look like, having first set up a template installation. The first step in creating the jail is to create the necessary directories within it. A basic directory structure within the jail will include directories for configuration files, logfiles, binary files, etc.... In the case of Apache httpd, the following is a list of the basics:
Check the template directory you created for a complete list of the directory structure you will need; be sure to also set the permissions and ownerships on these directories appropriately. /tmp and /var/logs, for example will need to be writeable by the user that Apache is running as. Apache may also need read permissions for the /home directory, if you are serving user pages as well, and will surely need read access to the document root (usually /var/www). The next step is to copy any of the needed binaries that were compiled earlier into the chroot jail. For MySQL, this would include the mysql client and the mysqld daemon. Apache httpd is more complicated. It requires the lt-httpd (or httpd) server binary, the apachectl control script, php (usually), if compiled with php support, apxs, and others. Each application will require a different set of binaries to be installed to the jail in order to function. This is the purpose behind first installing the application to an independent directory, such as /usr/local/httpd (in the case of Apache httpd). Some binaries may be located in /bin, and others in /usr/bin. Others still will be in /usr/libexec. They may also be installed to /usr/bin and /usr/sbin if appropriate configuration options were given to the configuration script. But most importantly, keep the binaries in the same relative location in the chroot jail that they have been configured to, as many paths, for other binaries, scripts, and library files, will be hardcoded into the binaries. Once the binaries have been installed, we are ready to create a basic chroot jail environment.Code:/chroot/httpd/lib /chroot/httpd/bin /chroot/httpd/tmp /chroot/httpd/var /chroot/httpd/var/log /chroot/httpd/var/logs /chroot/httpd/etc /chroot/httpd/dev /chroot/httpd/home
Before the application can run, certain necessary files, beyond the application binaries, must be in place. Let us take the example of chrooting mysql. In most cases, mysql is run as the mysql user. In order for the mysql daemon to run as the mysql user, it must have available in the /etc directory a copy of the passwd, shadow, and group files containing the necessary entries that make the mysql user valid on your system. Copy the lines from those files which are relevant to the application you are chrooting. Next, there are a few other files in /etc you should copy to the chroot jail. These include, but are not limited, to the following:
This will give the chrooted application a working environment. One trick I use is to create a generic /chroot/etc directory, and then hard link these files from there. That way, I only need to change any one of the files in /chroot/etc, and the change is instantly reflected across all the chrooted services.Code:HOSTNAME nsswitch.conf localtime hosts.allow hosts.deny hosts.equiv magic ld.so.conf services pam.d/* (if you use PAM for authentication)
Now, once the files and directories are in place, we still do not have an appropriate user environment. We must still set the necessary permissions on many of these objects. /tmp, for example, must be writeable by whatever applciation will be using the jail. /etc/shadow must not be world readable. In the case of mysql, the /var/mysql directory must be writeable by the mysql user. In the case of apache httpd, the /var/www directory must be readable by whatever user apache runs as. Make sure the necessary permissions are set to ensure a sane operating environment.
Installing the Libraries
We now have a basic environment set up and ready for an application to run in. Next comes the most mundane, and to the beginner, most tricky part of configuring a chroot jail. The libraries. Unless you have compiled you application statically, which I do not recommend as it makes updating services files a nightmare (trust me...), your applications will be using shared libraries; they are likely configured to read them from /lib, or /usr/lib... most applications will try several places before failing. Generally if a library is in any standard location, your application will find it.
Before we begin installing the libraries needed by the application itself, we must install the NSS libraries. If you are running a service that does not perform any user, name, or DNS lookups, you will not need these libraries. They are, however, quite secure and pose very little risk, especially when the passwd and shadow files are properly secured, with only the bare necessary accounts in them. Now, for any application that will require access to passwd, shadow, and group files, we will install these libraries. These are the libraries that allow Linux to look up user information, and hostname information. (When you see a “user@hostname” prompt, Linux will first match your user id to a user name by consulting the user database. This is done with the function calls located in the NSS libraries.) In short, these libraries will be used, in most cases, whether you know it or not, as they are dynamically linked at run time. Even statically compiled applications will require these libraries. While the exact technical details are not important, what is important is that you copy these libraries to the chroot jail. These libraries include, but are not limited to:
In short, all of your nss libraries. These can be copied to the chroot jail with the commandCode:libnss_compat.so libnss_files.so libnss_dns.so libnss_hesiod.so libnss_nis.so libnss_nisplus.so
In the case of mysql. The -d switch causes the cp command to not resolve any symbolic links, but copy the links as they are. Now check the symbolic links you just copied to make sure they are valid. They may still reference the old location of the libraries before you copied them, or they may be a relative reference which is no longer valid. If so, delete them and replace them with valid links. More than likely, they are already valid.Code:cp -d /usr/lib/libnss* /chroot/mysql/lib
After installing the NSS libraries, we must determine which libraries the application itself calls. This can be done with the ldd command. Assume we have copied the mysqld command out of the mysql source tree. To determine which shared libraries this application requires, we run the command “ldd mysqld”. This will produce output similar to what is below:
This is a listing of every shared library (aside from the NSS libraries) which is referenced by the mysqld command, and where they currently are on the system. These libraries must be copied from the location specified to /chroot/[app]/lib. In most cases, applications will search /usr/lib and /lib, among other directories, for shared libraries before giving up. Again, if they are in nearly any standard location such as these, most applications will find them. If not, you can either create the directory where the application looks for these libraries and place them there, or put symbolic links in their place.Code:root@elizabeth:/usr/src/mysql-5.0.15/sql# ldd mysqld linux-gate.so.1 => (0xffffe000) librt.so.1 => /lib/tls/librt.so.1 (0xb7f4d000) libz.so.1 => /usr/lib/libz.so.1 (0xb7f3a000) libdl.so.2 => /lib/tls/libdl.so.2 (0xb7f36000) libssl.so.0 => /usr/lib/libssl.so.0 (0xb7f05000) libcrypto.so.0 => /usr/lib/libcrypto.so.0 (0xb7e04000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7df2000) libcrypt.so.1 => /lib/tls/libcrypt.so.1 (0xb7dc4000) libnsl.so.1 => /lib/tls/libnsl.so.1 (0xb7dad000) libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0xb7cf5000) libm.so.6 => /lib/tls/libm.so.6 (0xb7cd2000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xb7cc9000) libc.so.6 => /lib/tls/libc.so.6 (0xb7bad000) /lib/ld-linux.so.2 (0xb7f5d000)
Here again, I prefer to create a single /chroot/lib directory for all of the shared libraries used by my chrooted services, and hard link each individual service's libraries from there. When on library is updated, they all are that way. For example, assume you have apache-httpd with mod-php installed and configured with mysql support. Now you wish to upgrade mysql. Since apache will be using libmysqlclient.so.* from /chroot/httpd/lib, which is hard linked from /chroot/lib, when you upgrade mysql, it will upgrade that library. There is no need to also upgrade the library in /chroot/httpd/lib, since it is hard linked to /chroot/lib. This solves a lot of configuration problems which occur when you forget to roll the upgrade out to all the chrooted services.
Installing The Device Files
Finally, we must install the necessary device files for the application. Some applications require specific device files, such as ssh and emacs requiring console access with /dev/console or something similar, but in most cases, zero, null, random, andurandom will suffice, and should be installed as a bare minimum anyway. The device files can be created with the following commands:
Should your application require any additional device files, look them up in your system's /dev directory and use similar commands to create them. I get the following listing when looking up the above devices:Code:mknod c 1 3 null mknod c 1 5 zero mknod c 1 8 random mknod c 1 9 urandom
Where 1 is the major device number and 3,5,8, and 9 are the minor device numbers. If we needed a console device, we would look it up in /dev with ls -l /dev/console, getting the following output:Code:crw-rw-rw- 1 root root 1, 3 2005-11-08 02:31 null crw-r--r-- 1 root root 1, 8 2005-11-08 02:31 random crw-r--r-- 1 root root 1, 9 2005-11-08 02:31 urandom crw-rw-rw- 1 root root 1, 5 2005-11-08 02:31 zero
We would then create the device in /chroot/[app]/dev with the following command:Code:crw------- 1 root tty 5, 1 2005-11-08 02:32 console
Make sure the device permissions for your newly created devices match those found in the /dev directory. In the case of /dev/console, I would set the permissions withCode:mknod c 5 1 console
This will create a basic environment in which the application is capable of running. Most applications will also require several configuration files to be placed in various directories as well. Bind, for example, requires the zone files to be placed in /etc, as well as the named.conf and rndc.conf files. Apache httpd requires httpd.conf, and mysql requires my.cnf. Each application will have its own unique configuration files which will need to be placed in the jail to run properly. We must determine what these files are and place them there. The method for each applciation will be different.Code:chmod 600 /chroot/mysql/dev/console
LOGGING
Once a basic user environment is installed and the applciation can run, the next item on the agenda is to ensure that the applciation is capable of logging to your usual logging facility, or we may not be able to properly diagnose errors which will inevitably occur as we try to chroot each application. I will assume that you are using the sylogd logging dameon for your system logging, and that, at this point, syslog itself has not been chrooted (in which case we would employ normal syslog logging over the network). Applications will look to the /dev/log device (actually a socket), and send logs there for eventual logging to /var/log/syslog (or another logfile). When we start the syslogd daemon, we can specify additional locations for this socket anywhere we like, which will all cause logs from the relevant application to be sent to /var/log/syslog (again, assuming you have not chrooted syslogd). This can be specified with the -a [socket] switch. For example,
Will not only create the usual /dev/log socket, but also an additional socket in /chroot/mysql/dev/log, which MySQL will see as /dev/log while running chrooted to /chroot/mysql. This will allow MySQL to send logs to the /var/log/syslog file much like every other application using the syslogd dameon on your system. Look through your system initialization scripts for the line that starts syslogd and add the -a argument for any services you are chrooting. Placing this command in individual control scripts will not work, as each script would restart syslog, which would delete any sockets that are already present. If we had /chroot/httpd/dev/log, starting syslog with only the argument to put a logging socket in /chroot/mysql/dev/log would delete the socket already in the httpd jail. Any application which uses the syslogd logging daemon will require these additional sockets if you wish to see the logs it generates, which are invaluable in diagnosing problems that will inevitable occur as you get the service chrooted. Without these additional sockets, the logs will disappear into computational oblivion.Code:/usr/sbin/syslogd -a /chroot/mysql/dev/log
Unfortunately, some applications do not use the system logging interface by default. The squid proxy server is one such example, storing logs instead in /var/logs as flat text files. Apache httpd does this as well, again, storing logs as flat text files. There are several options available to us in this case. I will use the apache http server as an example here
1. Do nothing. Look up the logs wherever the application stores them; in this case, /chroot/httpd/var/logs
2. Have a separate partition for /var/log, and mount this inside each jail you wish to log to this directory. However, this rather defeats the point of having a chroot jail in the first place – your system logs are now a part of those jails.
3. Some applications may have an option to use syslog logging instead of the default. Compile the application with this option, and specify an additional /dev/log device for these applications.
4. For the über leet, modify the source code to support syslog logging
We now have a basic application environment installed, and can diagnose most errors the application will provide for us. We can begin the process of starting the chrooted service and, more or less, stepping through the errors one by one and getting it running.
INSTALLING A WORKING ENVIRONMENT IN THE JAIL
It may help at this point to install a working environment inside the chroot jail, complete with a shell and basic commands such as ls and rm. I recommend you obtain the coreutils package and compile it statically. (So it does not depend on shared system libraries which may not be present in the jail). You will also need a statically compiled shell, such as bash. Place these utilities in the chroot jail, but make sure they are in a seperate directory, such as /chroot/[app]/static-bin, so they may be easily removed later. You will need at least a symlink from /bin/bash to /static-bin/bash if you are to be able to chroot into the jail environment. Once these utilities are installed to /static-bin (inside the jail), and bash is symlinked from /bin (again, inside the jail), you should be able to chroot to the new environment with the following command:
The root directory will now be /chroot/mysql. You are seeing exactly what the MySQL server will see when it runs chrooted. MAKE SURE TO REMOVE THESE TOOLS FROM THE JAIL WHEN YOU NO LONGER NEED THEM. Their presence could be exploited remotely if the service is compromised. Hopefully, we will not need this temporary environment, but it will not hurt to have it, just in case. I have required it to diagnose errors on more than one occasion.Code:chroot /chroot/mysql
STARTING THE SERVICE FOR THE FIRST TIME
Here is where our paths will diverge. As I mentioned earlier, the process at this point consists of stepping through and correcting any error messages we get one by one. Everybody's environment is different, and so everybody will have different error messages. What follows is a general guide on how to diagnose errors which may crop up as you are starting services chrooted for the first time.
Correct Any Errors Appearing on the Console
First, just try running the service. You may get lucky and have it work as is. Generally, if a daemon mode is available, choose to run it in console mode instead. It will provide some kind of message, in most cases, to tell you the service has started. If not, it may write some error messages to the screen. First correct any error messages which may appear on the console if the application does not start successfully.
Look Through the Log Files for Errors
If no error messages are given on the console, or if you have corrected them all, next check /var/log/syslog, /var/log/messages, and /var/log/secure outside of the jail (which I hope is working, since you have already set up the syslogd daemon to work with this application) for any error messages which may have been sent to the logging dameon. Quite often the service can be started using these error messages for troubleshooting alone.
When All Else Fails
This is where it gets tricky. Assuming you can start the service when not chrooted, but still can't start it in the chroot jail, there is almost certainly a permissions problem or a missing file somewhere, but you can't figure out where yet. You need to truss the program to find any error messages returned by function calls which are not being logged or written to the console. To do this, you must copy the strace program from wherever it is located on your system to the static binaries directory in the chroot jail. Place it in the static binaries directory so it may be removed later easily. I suggest copying the less and grep commands as well. You should already have the necessary libraries in the jail to run these utilities, as they were likely copied during the library installation process for the application itself. If not, follow the library installation procedure above for these tools as well.
To truss the application for any errors, run the following command, assuming the main application binary is located in /chroot/[app]/bin. (remember that the second argument to the chroot command is given relative to the chroot root, not the system root):
And you now have a whole bunch of gobbledygook to sort through for error messages. This is where less and grep come in handy. You can just pipe the whole damn thing though less, and examine it one screen at a time, but here are some strings to grep for which may speed up your search:Code:strace chroot /chroot/[app] /bin/[app]
I'm sure there are other good strings to search for, but I can't think of any at the moment. Let me use MySQL as an example of how to use strace to diagnose a startup error. After installing a basic environment as described above, I attempt to start the mysql daemon with the following command:Code:open read connect ENOENT ENOACCESS
This is both good and bad. Bad, of course, because it's not working. But this also means that all the necessary libraries are in place, else we would see an error message complaining about missing libraries. My first thought was that mysql.host must be a missing file somewhere, so I checked the whole system for that file. Of course, it wasn't there. mysql.host refers to the host table within the mysql database. Next, I checked the /var/log/syslog, /var/log/messages, and /var/log/secure log files for any useful information. No information about the error was recorded in my logfiles, probably because the developers felt that the console error message was sufficient. Unfortunately, I had no idea at this point what was actually causing the error. So my next step was to truss the mysqld daemon as it started up and check if there were any files it could not find. I did so with the following command:Code:root@elizabeth:/chroot/mysql/var/mysql/mysql# chroot /chroot/mysql /bin/mysqld -u root 051111 23:41:02 InnoDB: Started; log sequence number 0 43655 051111 23:41:02 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
This starts mysqld from within the chroot jail with the argument “-u mysql”, which tells it to start as the mysql user. “2&>1” tells bash to redirect the 2nd output stream to the 1st output stream (in other words, redirect standard error to standard output), so I can filter it with grep, which normally operates on standard input. Grep then filters the output, displaying only those lines with the string “open” in them. I got the following output:Code:strace chroot /chroot/mysql /bin/mysqld -u mysql 2>&1 | grep open
AHA! I can see the error message that was written to the console... but it was not generated from a function call return error. This is merely how MySQL interprets the error. More descriptive error messages would have been nice. We can now see that in fact, the error is being caused because MySQL expects to find a file called ./mysql/host.frm, which means it is looking for it in a subdirectory of whatever directory it is working in. I guessed, since this looks like a database file, that it was in /var/mysql (within the jail), so it was looking at /var/mysql/mysql, which is the mysql system database, located in the database directory. MySQL stores a fair amount of information in its own, self-titled database. I then searched my system for the host.frm file. It was in /usr/local/mysql/var/mysql/mysql. /usr/local/mysql, which is where I installed MySQL originally, so I would have an idea of what the jail was supposed to look like for MySQL to run in it. I then copied the entire MySQL database from /usr/local/mysql/var/mysql/mysql to /chroot/mysql/var/mysql/mysql, and after doing so, the mysqld daemon started without error.Code:root@elizabeth# strace chroot /chroot/mysql /bin/mysqld -u root 2>&1 | grep open open("/etc/ld.so.cache", O_RDONLY) = 3 open("/lib/tls/libc.so.6", O_RDONLY) = 3 <snip> open("./ib_logfile1", O_RDWR|O_LARGEFILE) = 7 open("/tmp/iboSN3lL", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 8 open("./mysql/host.frm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("./mysql/host.frm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) write(2, "051111 23:46:43 [ERROR] Fatal er"..., 108051111 23:46:43 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
Of course, there were other steps involved in setting up the MySQL server, such as installing the database (which I apparently did in the wrong place), but these are specific to MySQL. This document is intended as a general guide to chrooting, which is why those steps were not included.
This tutorial is licensed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs 2.5 license
Regards
Reply With Quote
Social Networking Bookmarks