Experience with Charlotte: Simplicity and Function in a Distributed Operating System

This paper presents a retrospective view of the Charlotte distributed operating system, a testbed for developing techniques and tools to solve computation-intensive problems with large-grain parallelism. The final version of Charlotte runs on the Crystal multicomputer, a collection of VAX-11/750 computers connected by a local-area network. The kernel/process interface is unique in its support for symmetric, bidirectional communication paths (called links), and synchronous nonblocking communication. Our experience indicates that the goals of simplicity and function are not easily achieved. Simplicity in particular has dimensions that conflict with one another. Although our design decisions produced a high-quality environment for research in distributed applications, they also led to unexpected implementation costs and required highlevel language support. We learned several lessons from implementing Charlotte. Links have proven to be a useful abstraction, but our primitives do not seem to be at quite the right level of abstraction. Our implementation employed finite-state machines and a multitask kernel, both of which worked well. It also maintains absolute distributed information, which is more expensive than using hints. The development of highlevel tools, particularly the Lynx distributed programming language, has simplified the use of kernel primitives and helps to manage concurrency at the process level.