Operating system fingerprinting is and will forever be in the domain of gray hat hacking. What operating system is running where on a given network is a subject of curiosity for hackers and administrators alike. Every security patch adds to the complexity of vulnerability history. White Hat Administrators want to find and patch vulnerable systems. Black Hat Hackers want to find and exploit vulnerable systems. Code that helps to determine OS and Service version help both groups.

Outside of the security community, however, the ability to detect operating systems is an untapped resource. Intranet Sites could be tuned to give different content to different types of desktops. The possible uses for this type of system because are relatively unexplored, for until now the science of operating system detection has been accessible only to the most dedicated and patient researchers in the security industry.

This is a presentation of a new OS fingerprinting library libOS (name subject to coolness upgrades). Using this library both professional developers and weekend scripters will have access to the state of the art techniques. Further, they will be able to extend the system with tests of their own design.

The history of operating system fingerprinting begins with Queso, by Savage from Apostols.org. This was the first program to walk upright on its hind legs. Shortly after that came the largest single leap in OS fingerprinting evolution; nmap’s operating system fingerprinting system, by Fyodor. This system, incorporated into the port scanning functions of nmap, are the most complex and feature rich released to date. To discover the evolution before Queso, take a look at the history found in Fyodor’s excellent article http://www.insecure.org/nmap/nmap-fingerprinting-article.html

Last year, Ofir Arkin and Fyodor Yarochkin released the only other fingerprinting system which could be compared with nmap. This scanner, called X, used ICMP techniques to quickly delineate between operating systems. Although the new techniques were impressive, the scanner lacked the ability to distinguish more than a few OS types. There was no external database of operating system fingerprints that could be modified or upgraded easily. It was however the first use of logic to reduce the total number of packets sent. X only sent packets that would improve the OS guess.

The next generation of TCP/IP test will probably do more of this bending within the bounds of TCP. This type of test represents the best of active fingerprinting and passive fingerprinting. The excellent paper on the subject from intranode mentions that it is possible to foil the plot using a TCP/IP proxy. Ironically I believe that proxies are the future of fingerprinting. A fingerprinting proxy would take standard traffic sent to a host and bend the underlying TCP functions without damaging the data at the higher layers. Think of it as a kinder, gentler fragrouter. I would love it if my library could help somebody develop this!

A matter of minutes before libOS is released Xprobe2, the successor to X will be released. Based on the claimed features from the initial release at CanSecWest, many of the failings of X are addressed in Xprobe2. Potentially making it the most advanced fingerprinting system of its kind.

But which is better? Nmap, or some version of X? In truth this is an arbitrary question similar to what car is best, or whether one prefers blonde, brunette or bald. Just as a Hummer does not compare well with a Ferrari, X and nmap do not compare well.

The reason is simple, the meta-level logic in each system have very different goals. There are two portions to any remote OS fingerprinting system. The first is the Test Suite. The second is the Testing Logic.

A Test Suite is full of methods for detecting the differences between hosts. Often this means proactive tests, where you are analyzing the response to a carefully crafted communication. Some tests, on the other hand require no proactive communication, but rather the ability to analyze current traffic.

Test Logic is a method intelligently interpreting the results of tests and guiding further testing of the results.

In reality, Test Logic is what separates nmap and the original X. The goal of nmap is to get as much accurate and usable information to the user as possible. This means that it executes all of its tests no matter what, and when a test fails it will often repeat it. The database is further evidence of this. It contains far more individuated OS categories then X. For an exercise in excess, take a glance at the contents of nmap-os-fingerprints included in the source code for your copy of nmap.

The goal of X, on the other hand was to be as accurate as possible, with as few packets as possible. As a result, it includes a hard coded logic tree which makes its fingerprinting very effective. X does not knowingly run tests that give it information that it already has.

So which is better, X or nmap? Depends on what you want and what you need!

What if you want your operating system Test Logic to do something that neither nmap or X do? What if you want to see if a new Test is a useful addition to a Test suite? What if you want to include OS sensitivity to your code but forking a call to a current program is not elegant enough?

Then LibOS is for you!!

The goal of LibOS is to be threaded and OOP. This allows for parallel running of tests as well as greatly increased flexibility in Test Logic (where the fun really is). Further the Library will be replicated in several popular languages. Currently Java and C are supported, with C++ and Perl on the way. At the bottom level, most of the tests from nmap, queso and X are recoded using libnet instead of libpcap. The libnet version is the current beta 7b (more icmp support).

At this point you are thinking that LibOS is a beta library itself. This is very much true. It is based on beta libnet and the code needs a thorough going over by the community. This and the C++ and perl modules need to be finished. So there is still much to do. So much that I would appreciate help. There are a lot of design questions that are still open. Version 1.0 really does need to be solid code, so that serious applications can be designed on top of it. This means that I would like to get Architectural bugs worked out at the outset. I also want to lean towards features that the community needs, with a decided bias to feature that co-developers want. You know those "wouldn’t it be neat if" thoughts one has on occasion? If you have one about LibOS then email it to me… no really, I want to know.

Here is a diagram of the current architecture.

As you can see the core of the library is in C, but the Logic is re-implemented in each language. Also note that there is space for new tests written in any language. This means that if you really want a new test you can have your pick of languages to implement them in.

The C code forms the core of the Library. Queso, nmap and Xv1 tests are implemented at this level. The code to send each test is located in a separate .c file. Sometimes this is true for the Listener that grabs and parses the returning test. But where possible I have made a generalized listen that works for several tests.

All of the .java files are named beginning with a capital J, so that the Java portion of the library can be listed with ls J*.

The Java has a base class called JOStest from which the other Tests are derived. This class includes methods for executing a test as well as for storing the results of a test. This includes fields for storing the Family. Major Version, Minor Version and Architecture of the Operating System.

This is the main benefit of OOP. Fyodor(nmap) wrote me and told me that he did not want his database mixed with the other apps, he suggested that the other authors would feel the same way. I tend to agree with him, merely combining databases would dilute the power that each has individually. The abstraction of OS identity at an OOP level gets around this consider

Abstract Class JOStest

{

public String OSFamily;

public String OSMajorVersion;

public String OSMinorVersion;

public String OSOther

public String OSArch;

… constructor and functions

}

Every nmap, queso or x fingerprint can be it into these categories. Here is the coding procedure… OS Family is the major family of OS like Windows, Solaris, Macintosh or Linux. OS Major Version is for example Linux "2.4" or Windows "98". OS Minor Version is anything else about a version that could be considered well formed. What is well formed is very much an open design question, but I will tend to favor the method that the vendor uses. Like "sp2" for Windows 98 or "13-24" for linux 2.X. OSOther is anything about and OS that effects its fingerprint that is not well formed like "With X program installed" or something like that. OS Minor Verision is the method for catching fine distinctions between Operating Systems. Last there is OS arch, this will become more and more important as the Operating Systems continue to be ported to new architectures.

So an entry from nmap-os-fingerprints would be encoded thus…

# Linux Kernel 2.4.3-2.10.1smp (RedHat 7.0.98 Wolverine)

###OSFamily=Linux

###OSMajorVersion=2.4

###OSMinorVersion=3

###OSOther=(RedHat 7.0.98 Wolverine)

###OSFamily=Linux

###OSMajorVersion=2.10

###OSMinorVersion=smp

###OSOther=(RedHat 7.0.98 Wolverine)

###OSArch=smp

This, like many fingerprints in the database it uses one fingerprint for more than one operating system. Here I assume that print refers 2.4.3 is a separate system that 2.10.1smp. Unfortunately the prints in many of the databases are natural language and as a result it will take some time to correctly convert them into more formal descriptions.

Currently I am adding these ### comments to nmap-os-fingerprints and queso.conf. I am still undecided about how to handle X, I will probably wait to see what format X2 uses for its database and use that.

The file JOSimp.c contains the JNI to C tie in functions which make the calls to the C library

This code is not yet developed. This is because getting everything that I wanted from Java (read Threads) was easier. Also since the base functions were in C I knew that if you wanted to use them you could very easily. This is scheduled to be completed last.

Licensing is currently very complex but I hope to simplify it soon. Why you ask cant you just use GPL?? Well, not exactly. The first reason is that X is not released GPL. It contains the GNU header. But it also contains the following line

This line conflicts with the GPL which states

You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee.

Nmap and Queso however are both GPLed. Further I used actual nmap code as the base for the Options parsing code in the TX series of Tests. This means that for the time being I am bound by the GPL.

Ultimately I want the library code and any tests that I write to be distributed under the Lesser GPL. For obvious reasons, the functionality that I am presenting here is not new and the largest benefit is just that it is a library. So there is no benefit of releasing the code GPL. In order to facilitate this I have used only the algorithms from each of the programs. Algorithms are not copyrightable so I can release the library under LGPL, (with the exception of the options sequence which I should rewrite for clarity anyway)

The only hitch is that the nmap-os-fingerprints database and queso.conf are both released GPL and the functioning of my library depends on the contents of these files. When X2 comes out the same will be true of its database file

If the authors of these respective pieces of software so choose they can re-release these pieces of code to me under the LGPL. If they decide to do so then the entire contents of the Library will be released LGPL. Otherwise the Library portion (mine) of the software will be LPGL as well as new tests donated to the library. If one chooses to link through my software to the GPLed databases then your software will be bound by the GPL.

Initially I will release a GPL copy of the library until the original authors and I can meet and come to an agreement about how the library should be licensed.

There are several sub-projects and extensions which I would like to propose in the wake of this fingerprinting library.

1. Service Fingerprinting

The first is a service fingerprinting system. This will allow for an open-source OS version to service version database to be developed. This database will quickly become useful for guessing OS. In conjunction to this port database, a port patterns should be studied. What OS has what ports running on install. This is useful for detecting new installations (which are a dangerous source of attacks).

Hopefully this phase will build on the work of Winfingerprints SMB based OS detection scheme.

2. OS-Service to CVE Database map.

There already several different versions of databases like this. The difference is that this database will be focused on what the "detectable" version of the operating systems are. Consider the following hypothetical operating system history

1998 May OS IRUNWELL v1 patch release

1998 Dec OS IRUNWELL v1 patch b

1999 Jan OS IRUNWELL v1 patch c

1999 Feb OS IRUNWELL v2 patch release

1999 March OS IRUNWELL v1 patch d

1999 May OS IRUNWELL v2 patch b

1999 August OS IRUNWELL v2 patch c

1999 Dec Vulnerability OS IRUNWELL remote root

2000 Jan OS IRUNWELL v2 patch d -< includes fix

2000 Jan OS IRUNWELL v1 patch e -< includes fix

2000 May OS IRUNWELL v3 release

If we had perfect OS detection we would not need vulnerability scanners at all. We would just scan the network for "unpatched" versions. Unfortunately patches often do not change the networking software of the computer.

Programs like Nessus develop sophisticated tests in order to detect when a service is actually available.

3. Super Simple Operating System Scanner

This scanner would do nothing except create a up to the day catalog of an organizations Operating Systems and Services. With this when a vulnerability became public knowledge. A list of operating systems on your network that "could" be vulnerable could be printed. Then the Web Proxy could force any web traffic from that host to an Intranet server with a patch download.

This would create a very effective patch distribution system. In the weeks before the Nessus came out with a Test for the vulnerability. Nessus and other scanners will always be excellent for discovering where you are from a vulnerability standpoint. But if you know exactly what Operating Systems you have already on your network. Then you just need a fact mechanism for getting the right patches to the right people.

And you have a fast vulnerability scanner on the cheap.

4. LibOS-Apache Module

Just think when someone finds a vulnerability in "Windows – All Versions" and then Microsoft issues 10 patches, your patch download page would display only the patch that the user actually needed.