6. Defining your log files filters


Defining the log file filters is another place I took great care to make for flexibility and ease of use. To do this, you use the file filter.txt, which contains a definition of every field contained in every log file you are monitoring. Each line in the file contains the definition for one type of log file, and is defined as follows: log filenane, application generating this log file, and then the name of the various fields comprised in a log entry, all of this separated by commas. Note that if one of the log file you are monitoring is not comma-delimited, you can specify the delimiter by typing DELIM= followed by the delimiter character at the end of the line. Note that in this case, it is not recommended to use the LOGIP, LOGHOST, ... capabilities of LogAgent 4.0 to monitor it, because this data is comma-delimited, and it would create confusion in the system. This may be fixed in a future release. The log filename also allow for some flexibility, since the exact filename may not always be known in advance (if the filename contains the date, or in ComLog's case where the filename is random, for example). This is why this file allows for the wildcards * and ?, * matching any string and ? matching any character. Note that * can only be used alone, not in combination with a prefix or a suffix string. This may also be fixed in a later release.

Also note that some manipulation is done with the data when it gets reported to LogIDS. There are some predefined fields that you are encouraged to use, since the program uses them. Don't worry, the program still allows plenty of space for just any kind of field specification you may need. First of all, there are the fields logip, loghost, loguser, logdate and logtime, that corresponds to the values inserted by LogAgent with the configuration variables of the same name. These are equivalent to the fields ip, host, user, date and time, and if one of these field is not defined, then it will get the value of its associated field (for example, if you define only a logip field, then the ip field will get the same value, and can therefore be used as a field in rules.txt, more about this file in the next chapter). Note however that in the case you associate a value with both fields of the same category, then they will be both be available and not overwrite each other (this could be convenient when you have LOGIP on and the application generating the log also puts the IP, especially if the IP comes from a different network interface). Also, some applications can provide the date and time under the format date-time or time-date, instead of supplying two different fields delimited by a comma. You can handle this by associating these 2 formats to the fields date_time or time_date respectively, and LogIDS will break these values into date and time respectively. Note however that it is strongly recommended to use the logdate and logtime data provided by LogAgent if you plan to rely on a standard date and time format accross your logs (this could be useful to replay log sessions, something I plan to do in the near future).

Another data manipulation that happens automatically is with the fields source and dest. These can be used to define fields related to ip.ip.ip.ip:port formatted IP addresses. These fields will be automatically broken down into src_ip, src_port, dest_ip and dest_port respectively. Alternatively, if your data is already broken down into separate ip and port fields, use these later field names and they will automatically be transfered to source and dest accordingly, to give you greater flexibity and control over the fields you have available to define rules later on.

For the case scenario we started working on in this paper, I have prepared a basic sample filter.txt file so you can see what it looks like:

ZAlog.txt,ZoneAlarm,ip,host,user,date,time,type,date,time,source,dest,transport
antivirus.log,Antivirus,ip,host,user,date,time,message,username,infectedfile,virusname
appevent.log,Application Event,logip,loghost,loguser,logdate,logtime,line
sysevent.log,System Event,logip,loghost,loguser,logdate,logtime,line
secevent.log,Security Event,logip,loghost,loguser,logdate,logtime,line
alert.ids,Snort,logip,loghost,loguser,logdate,logtime,line
services.log,Running Services,logip,loghost,loguser,logdate,logtime,display,name,state,account,servfilename,startup
shares.log,Open Shares,logip,loghost,loguser,logdate,logtime,share,usergroup,rights,path,remark
startup.log,Startup config,logip,loghost,loguser,logdate,logtime,line
adsscan.log,ADS Scanner,logip,loghost,loguser,logdate,logtime,ads,size
integrity.log,Integrity checker,logip,loghost,loguser,logdate,logtime,line
dummy1.txt, dummy entry,logdate,logtime,line,date_time,this_delim_is_a_space,DELIM= 
dummy?.txt, dummy entry,logdate,logtime,line,time_date,this_delim_is_a_tab,DELIM=	
dummy3.*, dummy entry,line,this_delim_is_a_sqare,DELIM=#

This sample file contains the definitions for a variety of logs I played with while developping LogIDS. The first one is an entry for the personal firewall ZoneAlarm, the second line matches the log definition of an old McAfee version, which I had old logs to play with, Events from the Application, System and Security Event Viewer, Snort alerts, entries for the supplemental data provided by LogAgent 4.0 Pro (services, shares and startup conf), followed by filter definitions for LogAgent 4.0 companion tools adsscan and integcheck. The last 3 lines are dummy lines that I've put there as examples for alternate syntax as described above. Note that the use of fields ip, host, user, date and time in the ZoneAlarm and antivirus entries are equivalent to the fields logip, loghost, loguser, logdate and logtime in the other entries. Snort and Event Viewer entries are very basic, since these logs are not comma-delimited, or even single-character-delimited. This can be circumvented by specifying a single field that will contain the log line, and then perform string matching comparison in the rules to trigger events in LogIDS. LogIDS 1.0 Pro have additional features that allow for easier management of these specific log files (more about that later).

Note that this represents only a small fraction of the logs you may encounter out there, and that there may be a variety of applications (security-related or not) that could be benefical to keep an eye on their logs in order to increase the overall level of intrusion detection provided by LogIDS. If you have suggestions of applications not covered here, or you make filters for applications that are not listed here, and you want to share them with the comunity, send them to me and I'll put them on the website for people to download. Let's now move to the next and final step in configuring LogIDS before finally being able to try it, the rules definition.

5. Defining your network items
7. Defining your rules