In operating systems, the process is the abstraction that separates the kernel from applications and applications from each other. The process is the scope of resource management and identity; for example, most operating systems impose per-process memory limits and perform per-process access checks. Traditionally, process separation--the principle of preventing malicious or incorrect code from accessing protected data--has been enforced by hardware protection mechanisms. The memory mapping hardware in these systems can check individual read or write accesses to a page of memory. By manipulating the memory mapping permissions, a kernel can give different access privileges to different processes. Such low-level enforcement successfully isolates both malicious and faulty processes. Fine-grained sharing, however, is awkward, expensive, or impossible because data sharing between processes can only be protected on coarse, page-based boundaries. This property of hardware-supported operating systems, coupled with the cost of switching permission tables, encourages an application structure that minimizes the number of process "boundary crossings" and reduces interprocess sharing to coarse page-based methods [49].
As an alternative to hardware-enforced separation, software-based separation methods perform memory access permission checks in software. Cedar [44] from Xerox PARC was the first to rely on software-based protection; Cedar allowed users to change or add system features via "extensions." Extensions in Cedar and other systems are externally defined modules that extend or modify the behavior or available functionality of a system. Other extensible systems include SPIN [9], the HotJava WWW browser [42], Oberon [52] and even the Emacs editor [41]. While web browsers and text editors are not operating systems in a traditional sense, both of these systems may need to manage and control their extensions in the same manner as an operating system.
Software-based protection mechanisms are also useful in systems that have no memory protection hardware. For example, the Apple Newton's NewtonOS [50] uses software-based protection mechanisms to separate the "kernel" from applications because the Newton does not include hardware for enforcing memory protections.
Software-based protection mechanisms can take several forms, including type-safe languages, annotated code systems, and checkable (or provable) code accompanied by a proof. All of these systems use compile-time, load-time or run-time checks to prevent a program from making illegal memory accesses. Annotated code systems, such as SFI [2], and provable code systems, such as PCC [34], have the advantage that they support legacy programming languages as the annotations (or proof) can be generated after the code is compiled. Currently, safe languages are more mature than either annotated code or provable code systems.
Although most software-based operating systems are defined in terms of "extensions" and operating system models are defined in terms of "processes," the only distinction seems to be a subjective measure of the size or independence of the module. A running application is usually considered a process while a modification to a larger system is usually considered an extension, but the distinction is arbitrary and the terms can be interchanged.
Existing software-based operating systems tend to be designed for single-user environments. Cedar, for example, was designed to make the operating system of the Xerox Alto computer extensible by the sole user of the system, thus avoiding many trust and safety issues. Oberon [53] makes a critical design assumption that restricts it to a single-user environment: tasks are not preemptible. The protections provided by languages for these inherently single-user environments can effectively shield the system from the majority of erroneous applications, but provide opportunities for malicious programs to bring a system down. As operating systems employing software-based protection mechanisms become used in more widely networked and multiuser environments, protecting a system from merely incorrect applications is insufficient. Protection needs to be extended to contain malicious and destructive applications, especially in the area of resource management.
Protection from and separation of mutually untrusting applications is the same problem that traditional, hardware-based multiuser operating systems face, and thus solutions from that domain should be applicable to language-based operating systems. To demonstrate that language-based extensible systems can benefit from the models developed for traditional operating systems, this thesis incorporates the process model and API from the Fluke operating system [20, 22] into Alta, a new virtual machine based on the Java virtual machine (JVM) [27]. The result is a JVM that supports nested processes and direct, hierarchical resource controls--the same set of services and protections that the Fluke model provides in a hardware-supported microkernel.
I chose to base Alta on the Fluke process model, or nested process model, for several reasons. First, the model incorporates per-process resource controls. Second, the nested process model infrastructure enables system services to be provided outside of the kernel, a feature appropriate for mobile-code platforms [45]. Third, the model is a complete process model: it specifies interprocess communication and synchronization primitives in addition to the basic process abstraction. Finally, I have experience with Fluke [21, 22, 46, 47].
Java was chosen as the "host" language for Alta because it provides the type-safety, language-level access control, and run-time environment necessary for an extensible system [32]. Java is also popular, and is used in a number of widespread systems, such as web servers and web browsers, that can benefit from more comprehensive support for extensions and resource controls.
The term "virtual machine" has different meanings in the context of Fluke, Java, and Alta, all of which are fundamentally similar. In Java the phrase "virtual machine" usually describes the Java bytecode interpreter/compiler and run-time support because these components act as a machine simulator. In Fluke, which is modeled on recursive virtual machines [26], the phrase "virtual machine" is defined as the environment that encapsulates a process because the process runs in the context of a virtualized machine. Both of these definitions are, in essence, about the presentation of a machine-like, virtual interface for an application. In this thesis, the phrase "virtual machine" describes a generic Java virtual machine.
The remainder of this chapter describes the Alta sandbox, an extended example of how an application might take advantage of Alta's process model. A sandbox is a controlled environment for untrusted code to execute in. The code gets to play the in safe and simple sandbox but is not allowed to directly access many, potentially unsafe, operations.
The Java applet sandbox, is the basis for a web browser's ability to control arbitrary, downloaded code. The sandbox is designed to contain applets and restrict an applet's access to critical system resources. The original Java sandbox, as implemented in JDK 1.0.2,1 provides a very strict access policy: applets are not allowed to access the local file system in any way, and they may only use network facilities to connect back to the host from which they originated. In later implementations of the JDK, an appropriately identified applet can be placed in a less restricted sandbox or even completely escape the sandbox.
Redefining the applet framework to take advantage of Alta's nested process features demonstrates two aspects of Alta. First, the Alta sandbox demonstrates that Alta provides sufficient mechanism to satisfy the containment requirements of the Java specification; second, the sandbox demonstrates application-level control over other applications. Additionally, the Alta sandbox provides a number of new features beyond the original. Applications running in the sandbox are only able to use the amount of memory granted to them. Each sandboxed application has its own separate static variable namespace. Sandboxed applications can contain sub-sandboxed child applications, and can use the java.lang.ClassLoader (for dynamically loading classes) and can create java.lang.ThreadGroups. These last two abilities are unavailable in the basic Java sandbox.
The Alta sandbox uses three mechanisms to contain and control processes. First, per-process static member data, per-process memory limits, and per-process class namespace control are directly supported by Alta. Second, simple limits are enabled by controlling the class namespace of the sandboxed process. For example, to completely deny access to the local file system, a sandbox could map the file access classes in its child to classes that throw an exception if any methods are invoked. The third mechanism for containing child processes is the Alta capability system. For example, to enforce a policy that denies network access after an application has accessed the local file system, a sandbox could use the class namespace controls to install replacement classes for the file and network access classes. These replacements would use capabilities and IPC, in place of native method calls, to perform the file or network operation. Thus a sandbox could make dynamic decisions about granting or denying access to the network based on the file access patterns of the application. (Section will explain this process in more detail.)
Chapter 2 details Fluke's nested process model and provides a short introduction to Java. The design of Alta is outlined in Chapter 3, followed by results and analysis in Chapter 4. Some future directions for Alta are listed in Chapter 5. Chapter 6 discusses other work that supports a process abstraction in Java and other language-based systems, along with work in safe languages and extensible systems in general. Chapter 7 presents conclusions that can be drawn from this work. Appendix A documents the Alta kernel API and Appendix B documents the interposable IPC-based API (IPCIF API).