Drawing the Red Line in Java

Summary. The presence of type-safety in systems based on type-safe languages has been erroneously coupled with an absence of a user/kernel boundary. Yet it is this user/kernel boundary which enables resource control, safe termination and safe interprocess communication. Although type-safety adequately fills the role of hardware in providing protection, without the red line, these other three services cannot be guaranteed.

In hw-based systems, the hw protection facilitates the creation of a kernel/user red-line; the authors show how to create such a red-line in a java system and explain why it is necessary.

More detail

In a system based on type-safe language, this red-line doesn't exist. The authors claim that with no red line, however, a system cannot ensure resource control, safe termination, or safe interprocess communication. Below are reasons why such services should be provided and how the authors created a red-line to provide such services in their KaffeOS system. The general recipe is:

1. Create the red line by separating Java code into user and kernel code. Kernel entry/exit implemented as function calls (language support to enable the compiler to automatically generate kernel entry/exit code being considered).
2. Create mechanisms to protect kernel code from terminating. A thread entering kernel code defers termination requests. Kernel code is trusted (empirically ensured) not to throw exceptions.

Resource control.

• Why.
  • Certain parts of the Java system must either have access to all the physical resources or not require memoery allocation (new, for example). The developers of Java failed to separate these unrestricted parts from those that are subjected to policy-defined resource limits.
  • It is difficult to properly account for garbage collection since the the whole GC system is considered "below the red line".
• How.
  • One possible solution for controlling memory is bytecode rewriting in which checks are inserted before memory allocations. If a violation is detected, a callback is invoked. Advantage: portable. Disadvantage: incomplete: callback can terminate and kernel code is not rewritten such that if memory is allocated on behalf of the user in kernel code, that memory is not accounted for.
  • In KaffeOS, kernel code structured such that is can back out of oom conditions. The runtime exception required by Java is created and thrown after the return to user code.
  • Fair scheduling is supported via multithreaded, preemptive scheduler with priority inheritance.
  • Garbage collection: each process receives it's own heap and per-process GC is above the red-line; cross-heap refs disallowed (how enforced?).

• Why. When a process is terminated, either by the kernel or self-terminated, no harm must come to the kernel or unrelated processes. Traditionally, can supply cleanup handlers (hard) or defer termination until out of critical area. Unix: whole kernel is "non-terminatable".
• How.
  • Anecdote: JDK originally specified that a thread could be terminated with a method that caused an exception to be thrown in the thread's context; the exception was handled like all runtime errors in that locks were released while unwinding the stack; this resulted in possibly inconsistent state if the thread were modifying some critical data structure. Thread.stop() has since been deprecated.
  • Later proposal also flawed since it suggested not releasing locks which would, of course, lead to deadlock.
  • Segments: a thread's execution path can be divided into segments and terminated only if not in a protected "server" segment. This solution only serves to "protect servers" and does not allow the termination of a malicious app.
  • KaffeOS:
    • threads executing in a process are destroyed and stack space reclaimed; this cleans up any stack-to-heap references.
    • Disallow cross-heap refs, so can simply merge the heap into the kernel heap and GC the kernel heap.

• Why.
  • Uncontrolled sharing in java causes resource control and termination problems: memory acct is difficult since it's unclear which process a shared object and termination is complicate if other processes hold live links to a shared object.
  • Java can currently distribute references to internal, possibly critical, resources (e.g., a thread control block).
• How.
  • Could disallow direct sharing -- require shared objects to be marshaled and unmarshaled over a serial connection. Inefficient.
  • KaffeOS provides a restricted form of direct sharing. Shared objects allocated in a shared heap which is subject to per-process limits; to prevent illegal cross-heap refs, control writes into the heap using write barriers, check code inserted before write instructions.