Search NIST's ICAT
By David Ferraiolo and Peter
One of the principal reasons that organizations continue to have security problems is that application software often contains numerous vulnerabilities. Many security systems (such as firewalls, intrusion detection systems, and virus checkers) attempt to protect these insecure applications by monitoring and filtering the application's interactions with users. Each security product provides a different monitoring or filtering technique and the use of multiple techniques can form a strong barrier against attack. However, ultimately, these barrier techniques are inadequate because users must be allowed to interface directly with vulnerable software applications. If such software contains a previously unknown vulnerability, then most likely an attacker can exploit it without being stopped. Despite this, our best defense (apart from building secure applications) is to install ever-stronger barriers around our software. One of the best places for such a barrier is as close to an application as possible: the operating system (OS).
An OS has direct control over applications and can provide strong security services to, and around, an application. However, many OSs allow applications too much control and thus vulnerabilities in applications often lead to complete compromises of computers. OSs themselves often have flaws; nevertheless, much of the public continues to purchase OSs known to be insecure even when given the option for more secure systems. Some people knowingly buy insecure systems because they prefer convenience to security. As computer security incidents become more widespread and dangerous, this line of reasoning may quickly change.
The purpose of this bulletin is twofold. First, it provides an overview of some security features that have often been neglected in mainstream OSs. It describes the extent to which these features have been implemented and how users can take full advantage of the available capabilities. Second, it warns users that OS security along with most other mainstream security mechanisms is imperfect and cannot stop all attacks. Despite this fact, using a combination of different security mechanisms can create a strong security barrier against attacks. Understanding the strengths and weaknesses of these techniques can aid one in the development of appropriate security policies, risk management plans, and in the purchase of security technology.
Important OS Security Features
Several OS security mechanisms are described here to demonstrate the importance of OS security in protecting application software. These are trusted paths, least privilege, non-discretionary access protection, and tokens. We cite these mechanisms as illustrative examples of the importance of OS security, not as a comprehensive or exhaustive list.
Trusted paths in OSs
Limited trusted-path capabilities exist in some widely used OSs. Windows NT, for example, uses a trusted-path mechanism to prevent Trojan horse programs from stealing logon passwords. When users log into a Windows NT system, they should first press control-alt-delete even if a logon window is already present. That keyboard signal causes an exception handler to run that suspends all non-OS processes and then presents a window to the user asking for a username and password. Thus, assuming that the OS has not been compromised, the user has reasonable confidence that the logon window is owned by the OS and not by malicious code. Similarly, the user also has reasonable assurance that no application apart from the OS is monitoring the user's keystrokes during logon.
While the current Windows NT trusted path is useful, OS vendors are working on further expanding this capability to allow trusted paths between programs, the OS, hardware devices (like smart cards), and users. The need for these expanded capabilities was highlighted by recently documented attacks in which a Trojan horse applet captured credit card numbers, PIN numbers, and passwords by emulating a window system dialog box.
Implementing Least Privilege
To prevent this problem, some OSs provide mechanisms by which one can assign programs only the specific privileges that they need. This is done by breaking the root privilege into a set of smaller privileges and thereby limiting the programs to a relatively small set of privileges. By giving programs the least number of privileges needed, a privileged program that is penetrated by a hacker will not give the hacker complete control of the host. For example, a program designed to eject a CD-ROM from a host has only the privilege to control the CD-ROM drive and not the ability to read the password file. A hacker then could not, as was recently done on a major OS, penetrate the eject program and gain the ability to read the password file.
Many versions of Unix are especially susceptible to this problem, since hackers can easily take advantage of the privileges of a penetrated program. When one program starts another, a newly created program runs with the user ID of the first program. This means that a malicious user who can exploit a bug in a root program may be able to start up an interactive root session. If a user is running as root, every program the user runs will have unlimited privileges on the system. The user can create any file, modify any file, and delete any file. The user can send and receive selected network packets, has the ability to intercept all packets on the network, and thus can view traffic between any two other hosts on the same network.
On many versions of Unix, the most a system administrator can do to implement the least-privilege principle is to give as few programs as possible root status. The granularity of implementing least privilege in most version of Unix is to choose whether or not a process should run as root. Some programs must be run as root and there is no way around giving them complete control of the machine. However, many administrators mistakenly run programs as root that could be run under user accounts. System administrators should carefully evaluate the privileged programs on critical servers and determine which, if any, could be run with user privileges instead of as root. Ideally, standard Unix versions would give system administrators the ability to implement the least-privilege principle with much finer granularity. Several vendors, however, do offer products that enhance the least-privilege capabilities in Unix.
Window NT has a finer grained, least-privilege mechanism for processes and users. Every Windows NT process runs as some user identity. Every user identity in Windows NT can be given a set of rights on that host. There exist at least 34 rights that can be given or denied to each user account. Example rights include changing the system time, managing security logs, accessing a computer over the network, taking ownership of files, and creating users. Despite this advanced least-privilege mechanism, all but two services (OS processes and programs that run in the background) included with Windows NT must be run as the system user which gives them complete control of the host. To fully take advantage of the least-privilege mechanism available in Windows NT, system administrators must create a separate user account for each service and give it only the privileges needed to run that service.
The use of a strong, least-privilege mechanism can eliminate many of the most commonly reported security problems with standard OSs, including many buffer overflow attacks. Even if a specific privileged program bug goes unfixed, use of the least-privilege principle prevents the bug from allowing a malicious user to bypass system security completely.
Non-discretionary access control can help ensure that system security features are enforced and tamperproof. With this technique, security administrators can ensure that critical files are properly write-protected and viewable by only a trusted set of people. Non-discretionary mechanisms can also be used to protect against inadvertent execution of untrustworthy applications since users can execute only those programs that they are allowed to execute. With discretionary access control, a user may have carefully defined a file protection policy but a virus could change that policy, making the user's files open to the world. This scenario is not possible in a non-discretionary access control environment.
Non-discretionary access control provides an organization with tighter security than is available with discretionary access control. However, this comes at a cost. Additionally, users may resist having file control policies specified by upper management. Also, it is time-consuming to manage what groups of users should have access to what files (although sophisticated software exists to make this easier). Despite these drawbacks, organizations requiring a high level of security, as well as organizations that cannot depend upon users' voluntary adherence to site security policy, should consider non-discretionary access control mechanisms. Non-discretionary access control capability can be added to many OSs with add-on software.
Integration of OS with Security
The Weaknesses of Barrier
However, no combination of these security barrier techniques is sufficient to guarantee resistance to determined attacks. This includes the use of OS security techniques. Each technique has a weakness that can be mitigated by the use of multiple barriers, but which ultimately will not be attack proof. Since users must interface directly with insecure software, security barriers are unable to stop attackers from exercising all application software flaws. In practice, however, a combination of these techniques provides a reasonably strong barrier against most attackers.
Using Patches to Enhance
However, firewalls do not, in most environments, adequately reduce the risk for applications that generate active content or implement transaction-oriented services. For example, firewalls do not typically have the processing power or ability to analyze downloaded JavaÔ applets. As the term implies, a firewall restricts overall access from an untrusted environment (the Internet) to a friendly environment (the local company network). The new paradigm of transaction-based Internet services makes these "perimeter" defenses less effective as the boundaries between friendly and unfriendly environments blur. A firewall controls broad access to all networks and resources that lie "inside" it. Once packets from a user have traversed the firewall and been authorized to enter the internal network, the firewall cannot prevent access to or modification of specific resources-in the worst case, the system security data itself. For Internet-based transaction systems, the security mechanisms must be able to provide or deny access to particular Web pages, applications, and databases on the basis of individual user profiles or server authentication. Firewalls are unable to provide such detailed security measures, important as they are to total systems security solutions.
Virus Detection Software
While the use of evaluated software is a necessary step in the direction of improved security, it does not guarantee the security of an organization. Product evaluations can find flaws and increase the level of trust in a product, but they do not guarantee the absence of all flaws. Even when using evaluated products, organizations must implement many other security mechanisms to create an in-depth defense strategy.
OS security technologies necessarily fit into any security plan because all of the applications one is trying to defend run on top of OSs. One should ensure that the security capabilities of an organization's OSs are fully utilized. Furthermore, one should plan future OS purchases based on the security of the OS itself as well as the security features it provides. More security-conscious organizations should consider purchasing add-on software that enhances the security of their OSs.
NOTE: Any mention of commercial products is for information only; it does not imply recommendation or endorsement by the National Institute of Standards and Technology nor does it imply that the products mentioned are necessarily the best available for the purpose.