Date: Fri, 19 Feb 1999 16:08:06 -0500 >From: "Simson L. Garfinkel" Subject: Process-table attack Wide-ranging attack works against almost any UNIX systems on the Internet ABSTRACT: The Process Table Attack is a [relatively] new kind of denial-of-service attack that can be waged against numerous network services on a variety of different UNIX systems. The attack is launched against network services which fork() or otherwise allocate a new process for each incoming TCP/IP connection. Although the standard UNIX operating system places limits on the number of processes that any one user may launch, there are no limits on the number of processes that the superuser can create other than the hard limits imposed by the operating system. Since incoming TCP/IP connections are usually handled by servers that run as root, it is possible to completely fill a target machine's process table with multiple instantiations of network servers. Properly executed, this attack prevents any other command from being executed on the target machine. DETAILS In the book Practical UNIX and Internet Security, Gene Spafford and I observed that the UNIX operating system originally contained few defenses to protect it from a denial-of-service attack. This is changing. With the growth of the Internet, there has been a concerted effort in recent years to strengthen the operating system and its network services to these attacks. Each time a network client makes a connection to a network server, a number of resources on the server are consumed. The most important resources consumed are memory, disk space, and CPU time. Some network services, such as sendmail, now monitor system resources and will not accept incoming network connections if accepting them would place the system in jeopardy. One system resource that has escaped monitoring is the number of processes that are currently running on a computer. Most versions of UNIX will only allow a certain number of processes to be running at one time. Each process takes up a slot in the system's process table. By filling this table, it is possible to prevent the operating system from creating new processes, even when other resources (such as memory, disk space, and CPU time) are widely available. The implementation of many network services leaves them open to a process table attack ? that is, an attack in which the attacker fills up the target computer's process table so that no new programs can be executed. The design of some network protocols actually leads the programmer into making these mistakes. An example of such a protocol is the finger protocol (TCP port 79). The protocol follows this sequence: 1. The client makes a connection to the server. 2. The server accepts the connection, and creates a process to service the request. 3. The client sends a single line to the server consisting of the name of the entity that the client wishes to finger. 4. The server performs the necessary database lookup and sends the information back to the client. 5. The server closes the connection. To launch a process table attack, the client need only open a connection to the server and not send any information. As long as the client holds the connection open, the server's process will occupy a slot in the server's process table. On most computers, finger is launched by inetd. The authors of inetd placed several checks into the program's source code which must be bypassed in order to initiate a successful process attack. If the inetd receives more than 40 connections to a particular service within 1 minute, that service is disabled for 10 minutes. The purpose of these checks was not to protect the server against a process table attack, but to protect the server against buggy code that might create many connections in rapid-fire sequence. To launch a successful process table attack against a computer running inetd and finger, the following sequence may be followed: 1. Open a connection to the target's finger port. 2. Wait for 4 seconds. 3. Repeat steps 1-2. The attack program is not without technical difficulty. Many systems limit the number TCP connections that may be initiated by a single process. Thus, it may be necessary to launch the attack from multiple processes, perhaps running on multiple computers. We have tested a variety of network services on a variety of operating systems. We believe that the UW imap and sendmail servers are also vulnerable. The UW imap contains no checks for rapid-fire connections. Thus, it is possible to shut down a computer by opening multiple connections to the imap server in rapid succession. With sendmail the situation is reversed. Normally, sendmail will not accept connections after the system load has jumped above a predefined level. Thus, to initiate a successful sendmail attack it is necessary to open the connections very slowly, so that the process table keeps growing in size while the system load remains more or less constant. We have also seen a variety of problems on BSD-based machines being used as the attacker. When the target machine freezes or crashes, the attacker machine sometimes crashes as well. Apparently the TCP stack does not gracefully handle hundreds of connections to the same port on the same machine simultaneously going into the FIN_WAIT_2 state. There are variants of this attack: 1. Use IP spoofing so that the incoming connections appear to come from many different locations on the Internet. This makes tracking considerably harder to do. 2. Begin the attack by sending 50 requests in rapid fire to the telnet, rlogin and rsh ports on the target machine. This will cause inetd to shut down those services on the target machine, which will deny administrative access during the attack. 3. Instead of initiating a new connection every 4 seconds, initiate one every minute or so. The attack slowly builds, making it more difficult to detect on packet traces. There are several ways to defend against the attack: 1. inetd and other programs should check to see the number of free slots in the process table before accepting new connections. If there is less than a predefined number of free slots, new connections should be accepted. 2. Alternatively, if there are more than a preset number of network daemons for the service running, incoming requests should be queued rather than serviced. 3. Network services (such as finger) should implement timeouts. For example, the statement alarm(30) could be inserted into the finger daemon source code so that the program would stop running after 30 seconds of execution. Simson L. Garfinkel, Sandstorm Enterprises, Inc. [Simson informed me over a year ago that he had discovered this attack and had notified many relevant operating system vendors. To the best of my knowledge, no one has addressed the problem in the intervening year. We thus include this item in the hopes of spurring some action, or at least awareness and public discussion. On the other hand, we of course do not recommend conducting experiments to demonstrate this flaw on other people's systems. PGN] -------------------------------------------------------------------------------- Date: Tue, 23 Feb 1999 01:03:25 -0500 From: Andrew Hobgood To: BUGTRAQ@netspace.org Subject: Re: Process table attack (from RISKS Digest) > Subject: Process-table attack > > The Process Table Attack is a [relatively] new kind of denial-of-service > attack that can be waged against numerous network services on a variety of > different UNIX systems. The attack is launched against network services This flaw isn't only limited to programs run from inetd (or other on-demand forking servers). Over a year ago, I reported a DoS attack present in the "comsat" daemon (used to notify users of incoming mail). That report can be found at: http://geek-girl.com/bugtraq/1997_3/0398.html Now, a simple way to avoid these kinds of denial of service attacks is to watch for multiple connections to a port (especially ones with no data) >from a single source (at an IDS or firewall level). You can then react with logging the attempts, firewalling the connections, or even spoofing connection resets to the local machine to clear out the connection table. The major problem with that approach, however, is that some programs, (the in.comsatd vulnerability, in particular) *look* like they're performing normal activity when a denial of service attack is in progress. Now, I'm sure that other programs exist that exhibit the same behavior, and these provide an even more worrisome issue than the normal forking-server family of daemons. I hope this gets the gears rolling in some of the brighter minds out there... /Andrew Hobgood [http://web.strange.net | Kha0S on EFnet IRC (#LinuxOS)] -------------------------------------------------------------------------------- Date: Mon, 22 Feb 1999 09:40:17 -0800 From: Jan B. Koum To: BUGTRAQ@netspace.org Subject: Re: Process table attack (from RISKS Digest) On Sat, Feb 20, 1999 at 01:42:53PM -0800, Mark Boolootian wrote: > Date: Fri, 19 Feb 1999 16:08:06 -0500 > >From: "Simson L. Garfinkel" > Subject: Process-table attack > > Wide-ranging attack works against almost any UNIX systems on the Internet > > ABSTRACT: > > The Process Table Attack is a [relatively] new kind of denial-of-service > attack that can be waged against numerous network services on a variety of > different UNIX systems. The attack is launched against network services > which fork() or otherwise allocate a new process for each incoming TCP/IP > connection. Although the standard UNIX operating system places limits on > the number of processes that any one user may launch, there are no limits on > the number of processes that the superuser can create other than the hard > limits imposed by the operating system. Since incoming TCP/IP connections > are usually handled by servers that run as root, it is possible to > completely fill a target machine's process table with multiple > instantiations of network servers. Properly executed, this attack prevents > any other command from being executed on the target machine. I have not tested this, but I don't think this is true for at least FreeBSD. You see, it has what is called login limits and you can indeed put limits on root login user. From /etc/login.conf: #root:\ # :cputime=infinity:\ # :datasize=infinity:\ # :stacksize=infinity:\ # :memorylocked=infinity:\ # :memoryuse=infinity:\ # :filesize=infinity:\ # :coredumpsize=infinity:\ # :openfiles=infinity:\ # :maxproc=infinity:\ # :memoryuse-cur=32M:\ # :maxproc-cur=64:\ # :openfiles-cur=1024:\ # :priority=0:\ # :requirehome@:\ # :umask=022:\ # :tc=auth-root-defaults: As far as I know (and I am sure 2829 peole will correct me if I am not), changing infinity to a numeric value should produce a desired result. AGAIN: I have not tested this yet for root user - but I know that the login limits do work for normal users. -- Yan -------------------------------------------------------------------------------- Date: Tue, 23 Feb 1999 00:11:50 +0100 From: Dirk Moerenhout To: BUGTRAQ@netspace.org Subject: Re: Process table attack (from RISKS Digest) While you are certainly right about the possible implications of this it looks as if you're lifting this higher than it is. You should not make an elephant from a fly. This is a possible problem but wise system-implementation can solve this easily enough. > ... > To launch a process table attack, the client need only open a connection to > the server and not send any information. As long as the client holds the > connection open, the server's process will occupy a slot in the server's > process table. This is true though a lot of services will time out after a while thus shutting down the proces and freeing the slot again. Services that wait hours without receiving a byte of data are badly coded. > ... > 1. Use IP spoofing so that the incoming connections appear to come from > many different locations on the Internet. This makes tracking > considerably harder to do. Tracking would be harder yes. But don't you think it's pretty hard to get a full spoofed 3WHS? AFAIK most Unices pass connects after the 3WHS. And while it's easy to spoof a SYN, it's a lot harder to spoof the rest. > 2. Begin the attack by sending 50 requests in rapid fire to the telnet, > rlogin and rsh ports on the target machine. This will cause inetd to > shut down those services on the target machine, which will deny > administrative access during the attack. The use of rlogin, rsh and telnet have long been depricated in favor of ssh. > 3. Instead of initiating a new connection every 4 seconds, initiate one > every minute or so. The attack slowly builds, making it more difficult > to detect on packet traces. Ok, but there aren't that many services that don't time out when receving either no data or invalid data. And if they don't they should. > There are several ways to defend against the attack: > > 1. inetd and other programs should check to see the number of free slots > in the process table before accepting new connections. If there is less > than a predefined number of free slots, new connections should be > accepted. > 2. Alternatively, if there are more than a preset number of network daemons > for the service running, incoming requests should be queued rather than > serviced. > 3. Network services (such as finger) should implement timeouts. For > example, the statement alarm(30) could be inserted into the finger > daemon source code so that the program would stop running after 30 > seconds of execution. Stuff like this already exists in many daemons. This isn't really a case of OS-problems but rather poorly implement services. Services that handle connects in a sane way should not be an aid in performing this attack. > Simson L. Garfinkel, Sandstorm Enterprises, Inc. Dirk Moerenhout -------------------------------------------------------------------------------- Date: Mon, 22 Feb 1999 14:33:23 -0800 From: James Lockwood To: BUGTRAQ@netspace.org Subject: Re: Process table attack (from RISKS Digest) On Sat, 20 Feb 1999, Mark Boolootian wrote: > The Process Table Attack is a [relatively] new kind of denial-of-service > attack that can be waged against numerous network services on a variety of > different UNIX systems. The attack is launched against network services Although not without quirks, I recommend that anyone interested in defending against this attack take a look at xinetd. It solves these problems and others, and has been available for a number of years (I started using it in 1993): (xinetd README) ] Q. Why should I use it ? ] A. Because it is a lot better (IMHO) than inetd. Here are the reasons: ] ] 4) It can prevent denial-of-access attacks by ] a. placing limits on the number of servers for each service ] (avoids process table overflows) ] b. placing an upper bound on the number of processes it will fork ] c. placing limits on the size of log files it creates > 2. Alternatively, if there are more than a preset number of network daemons > for the service running, incoming requests should be queued rather than > serviced. This is the approach that xinetd takes: (xinetd.conf manpage) ] instances determines the number of servers that can ] be simultaneously active for a service (the ] default is no limit). The value of this ] attribute can be either a number or UNLIM- ] ITED which means that there is no limit. > 3. Network services (such as finger) should implement timeouts. For > example, the statement alarm(30) could be inserted into the finger > daemon source code so that the program would stop running after 30 > seconds of execution. Or, better yet, a timeout option should be implemented in the inetd daemon. I have added this to my local copy of xinetd 2.2.1 with the following syntax: timeout = 30 # 30-second timeout timeoutsig = SIGINT # signal to send when timeout is hit Modifications are fairly trivial, does anyone know if Jan Wedekind and Chuck Murcko are still maintaining it? > [Simson informed me over a year ago that he had discovered this attack > and had notified many relevant operating system vendors. To the best of > my knowledge, no one has addressed the problem in the intervening year. Again, check out xinetd. It's not new and flashy (and it doesn't work with TLI services) but is does the job well. I have found it to be rock-solid. -James -- James D. Lockwood The (former) Getty Information Institute System Administrator 1200 Getty Center Drive, Suite 300 james@gii.getty.edu Los Angeles, CA 90049-1680 -------------------------------------------------------------------------------- Date: Mon, 22 Feb 1999 18:33:34 +0100 From: "Olle Segerdahl,D" To: BUGTRAQ@netspace.org Subject: Re: Process table attack (from RISKS Digest) On Sat, 20 Feb 1999, Mark Boolootian wrote: > The Process Table Attack is a [relatively] new kind of denial-of-service > attack that can be waged against numerous network services on a variety of > different UNIX systems. The attack is launched against network services > which fork() or otherwise allocate a new process for each incoming TCP/IP > connection. Although the standard UNIX operating system places limits on > the number of processes that any one user may launch, there are no limits on > the number of processes that the superuser can create other than the hard > limits imposed by the operating system. Since incoming TCP/IP connections > are usually handled by servers that run as root, it is possible to > completely fill a target machine's process table with multiple > instantiations of network servers. Properly executed, this attack prevents > any other command from being executed on the target machine. How is this DoS different from the Old "rescource exaustion" attacks? Anyone remember the "octopus" ? (keeping multiple sendmail-connections and depriving the machine of either memory or proc#:s, whichever came first.) I don't think it's fair to say it's "a [relatively] new kind of denial-of-service attack" /olle -- Above views are my own unless explicitly stated otherwise. God is real, until declared integer. -------------------------------------------------------------------------------- Date: Mon, 22 Feb 1999 16:06:35 -0500 From: Dug Song To: BUGTRAQ@netspace.org Subject: Re: Process table attack (from RISKS Digest) On Sat, 20 Feb 1999, Mark Boolootian forwarded: > >From: "Simson L. Garfinkel" > Subject: Process-table attack > > Wide-ranging attack works against almost any UNIX systems on the Internet interesting paper on a possible solution: http://www.scout.cs.arizona.edu/scout/Papers/TR98-10.ps see http://www.scout.cs.arizona.edu/scout/scout_scout.html for more info. -d. --- http://www.monkey.org/~dugsong/ -------------------------------------------------------------------------------- Date: Tue, 23 Feb 1999 08:43:24 -0000 From: John Conover To: BUGTRAQ@netspace.org Subject: Denial of service process table attacks The DNS process table attacks are mentioned at: http://lwn.net/daily/ptable.html I have been using tcpserver as a replacement for inetd for many months. It is at: ftp://koobera.math.uic.edu/www/ucspi-tcp.html and allows limiting the *_number_* of concurrent processes forked out of the inet facilities, as opposed to the *_rate_* of forked processes. Also, optionally, allows logging by port, with access control via a fast database, by IP, and user. I'm currently using it on httpd, ftp, faxd, smtp, telnet, pop3, and the qmail daemons. Seems viable. FWIW, FYI ... John -- John Conover, 631 Lamont Ct., Campbell, CA., 95008, USA. VOX 408.370.2688, FAX 408.379.9602 conover@inow.com, http://www2.inow.com/~conover/john.html