Defining the log file filters is another place I took great care to make for flexibility and ease of use. To do this, you use the file filter.txt, which contains a definition of every field contained in every log file you are monitoring. Each line in the file contains the definition for one type of log file, and is defined as follows: log filename, application generating this log file, and then the name of the various fields comprised in a log entry, all of this separated by commas. Note that if one of the log file you are monitoring is not comma-delimited, you can specify the delimiter by typing DELIM= followed by the delimiter character at the end of the line. The log filename also allows for some flexibility, since the exact filename may not always be known in advance (if the filename contains the date, or in ComLog's case where the filename is random, for example). This is why this file allows for the wildcards * and ?, * matching any string and ? matching any character. Note that * can only be used alone, not in combination with a prefix or a suffix string.
Also note that some manipulation is done with the data when it gets reported to LogIDS. There are some predefined fields that you are encouraged to use, since the program uses them. Don't worry, the program still allows plenty of space for just any kind of field specification you may need. First of all, there are the fields logip, loghost, loguser, logdate and logtime, that corresponds to the values inserted by LogAgent with the configuration variables of the same name (please refer to LogAgent documentation for more details). These are equivalent to the fields ip, host, user, date and time, and if one of these field is not defined, then it will get the value of its associated field (for example, if you define only a logip field, then the ip field will get the same value, and can therefore be used as a field in rules.txt, more about this file in the next chapter). Note however that in the case you associate a value with both fields of the same category, then they will be both be available and not overwrite each other (this could be convenient when you have LOGIP on and the application generating the log also puts the IP, especially if the IP comes from a different network interface). Also, some applications can provide the date and time under the format date-time or time-date, instead of supplying two different fields delimited by a comma. You can handle this by associating these 2 formats to the fields date_time or time_date respectively, and LogIDS will break these values into date and time respectively. Note however that it is strongly recommended to use the logdate and logtime data provided by LogAgent if you plan to rely on a standard date and time format across your logs (this could be useful to replay log sessions, something I plan to do in the near future).
Another data manipulation that happens automatically is with the fields source and dest. These can be used to define fields related to ip.ip.ip.ip:port formatted IP addresses. These fields will be automatically broken down into src_ip, src_port, dest_ip and dest_port respectively. Alternatively, if your data is already broken down into separate ip and port fields, use these later field names and they will automatically be transferred to source and dest accordingly, to give you greater flexibility and control over the fields you have available to define rules later on.
Note that the fields 'filename' and 'application' are also available if you wish to use them, and they correspond to the logfiles filename, and the application associated with it in filter.txt.
For the case scenario we started working on in this paper, I have prepared a basic sample filter.txt file so you can see what it looks like:
ZAlog.txt,ZoneAlarm,ip,host,user,date,time,type,date,time,source,dest,transport
antivirus.log,Antivirus,ip,host,user,date,time,message,username,infectedfile,virusname
appevent.log,Application Event,logip,loghost,loguser,logdate,logtime,line
sysevent.log,System Event,logip,loghost,loguser,logdate,logtime,line
secevent.log,Security Event,logip,loghost,loguser,logdate,logtime,line
alert.ids,Snort,logip,loghost,loguser,logdate,logtime,line
services.log,Running Services,logip,loghost,loguser,logdate,logtime,display,name,state,account,servfilename,startup
shares.log,Open Shares,logip,loghost,loguser,logdate,logtime,share,usergroup,rights,path,remark
startup.log,Startup config,logip,loghost,loguser,logdate,logtime,line
adsscan.log,ADS Scanner,logip,loghost,loguser,logdate,logtime,ads,size
integrity.log,Integrity checker,logip,loghost,loguser,logdate,logtime,line
dummy1.txt, dummy entry,logdate,logtime,line,date_time,this_delim_is_a_space,DELIM=
dummy?.txt, dummy entry,logdate,logtime,line,time_date,this_delim_is_a_tab,DELIM=
dummy3.*, dummy entry,line,this_delim_is_a_sqare,DELIM=#
This sample file contains the definitions for a variety of logs I played with while developing LogIDS, and some dummy ones provided as syntax examples. The first one is an entry for the personal firewall ZoneAlarm, the second line matches the log definition of an old McAfee version, which I had old logs to play with, Events from the Application, System and Security Event Viewer, Snort alerts, and entries for some of the SIDTk 1.0 modules. The last 3 lines are dummy lines that I've put there as examples for alternate syntax as described above. Note that the use of fields ip, host, user, date and time in the ZoneAlarm and antivirus entries are equivalent to the fields logip, loghost, loguser, logdate and logtime in the other entries. Snort and Event Viewer entries are very basic, since these logs are not comma-delimited, or even single-character-delimited. This can be circumvented by specifying a single field that will contain the log line, and then perform string matching comparison in the rules to trigger events in LogIDS. LogIDS 2.0 Pro have additional features that allow for easier management of these specific log files (more about that later). LogAgent 5.0 Pro will also pre-analyse Snort logs and produce a single-lined log entry made of the most critical information contained in the alert.
Note that LogIDS 2.0 Pro offers some additional support for some of these log files, and give you with a set of easy-to-use pre-defined fields to build your rules on. They are:
label
: Name of the Event Viewer reporting (Application, System or Security), along with the record numbertimegenerated
: comes from the Event, self-explanatorytimewritten
: same as abovecomputer
: same as aboveeventtype
: can be any of : Error, Warning, Information, Audit Failure, Audit Success
*eventid
: Event numerical IDcategory
: comes from the Event, self-explanatorysource
: same as abovemessage
: same as above, if any, mostly seen in Security EventsAs for the Security Events, they also have an additional field, labelled 'description which contains a textual meaning to the Event ID field, which helps in interpreting the Event at hand. Look in the appendices for a complete list of the Security Events that LogIDS 2.0 Pro can handle automatically
* For a more complete description about the meaning of these values, please report to your Microsoft documentation regarding the Event Viewer.
Additional logfile support present in LogIDS 1.0 Pro is no longer support per se in LogIDS 2.0 Pro, with the exception of being handled like normal log files. The support for these log files have been moved in the SIDTk 1.0 for computer efficiency reasons (to reduce the amount of logs analyzed by the console), and the code related to it is completely available as Open Source. Snort support has been moved into LogAgent 5.0 Pro, who can backup the original Snort logs while producing a single-line, comma-delimited record of the most important fields.
Note that this represents only a small fraction of the logs you may encounter out there, and that there may be a variety of applications (security-related or not) that could be beneficial to keep an eye on their logs in order to increase the overall level of intrusion detection provided by LogIDS. If you have suggestions of applications not covered here, or you make filters for applications that are not listed here, and you want to share them with the community, there is now a user forum on the website where you can share it with the rest of the community. Let's now move to the next and final step in configuring LogIDS before finally being able to try it, the rules definition.
5. Defining your network items
7. Defining your rules