The agents are responsible for collecting all the data sent by the various devices existing on the network, in order to subsequently send it to the OSSIM server in a standardized way. The point of standardizing events prior to sending them to the server is so that the latter can deal with these events equally and so that storage and processing is simpler and more coherent.

The agents are installed on the sensor machines, normally one per machine although it is possible to install more than one if necessary. This will normally only occur in multi-level environments, where one machine with several agents can be sending information to various different servers, each from different devices.

Each of the events received by the OSSIM server has always been processed beforehand by an agent. Even in the case of a server to which only one other server is connected, designed to resend events, this other server, or the last in the chain of servers, will have received its data from an Agent.

The way in which the agent receives the data (which will then be converted into events for OSSIM) that it is going to send to the server is by means of reading a log file in most cases. There are plans to implement other types of data sources such as “socket” or DB in the future.

The ports to which the agent is connected are:

For any device from which one wishes to collect data a plugin has to have been created beforehand so that OSSIM is capable of processing it. This is achieved thanks to the creation of a plugin which basically consists of a series of regular expressions and a list that allows the event type being produced to be unambiguously identified, including Reliability evaluations.

Obviously, not all devices will be capable of saving data in a text file. It is therefore necessary for certain devices to use complementary tools and collection methods to ensure us that the data will be in a file within reach of the agent, so that the latter can read and process it.

When an agent is disconnected for any reason from the server to which it sends data, the events it is not capable of sending to the server in real time are not lost; it stacks the events and as soon as the connection is re-established, it sends them all, respecting the order in which they have arrived.

At this time each agent sends data to one server only.

Plugins are each of the elements defined in the Agent to analyze and standardize the information from a device. Once this has been standardised it is passed to the remaining functions of the Agent in order to be sent to the OSSIM server in the form of an event.

In general, each of the plugins can read and send data from a specific device, identified by its plugin_id, as well as each and every one of the event types belonging to that plugin, identified as its plugin_sid's, such as:

In OSSIM there are two types of plugin:

• Detectors: Their job is to read from the logs that store the devices and to standardize them so that the Agent can send them to the OSSIM server. Detector plugins passively read a file, socket or process and send events upon pattern-matching lines. Typical detector plugins are snort, p0f, arpwatch, pads, etc…
• Monitors: These receive a question from the OSSIM server. These plugins send a question to the corresponding tool, obtain the reply, and let the server know whether it agrees or not with what it has asked. Sample monitor plugins would be:
  • Nmap (The agent receives a monitor request, launches an nmap scan against a specific host:port pair and returns a message stating the open/filtered/closed status of the request host:port pair)
  • OSSIM C/A (After receiving a Compromise or Attack status request the agent watches for those values inside the ossim database, returning and event after having reached it or after the timeout expires)
  • Tcptrack (OSSIM Server asks the agent for specific tcp session information such as duration and bytes sent/received)

These two types of plugin are those which will then be used in the correlation in order to generate alarms.

We are going to illustrate an example of each of the types of plugin:

As we have already seen, Detectors are not activated as a result of a question from the server, as is the case with Monitors. The question received may be something like:

watch-rule plugin_id="2006" plugin_sid="i" condition="gt" value="2" from="192.168.1.10" to="192.168.1.11"

The agent sends and normalizes the events in the same way, except with 4 special types that generate events which, although also normalized, have been normalized with different characteristics.

This is due to the fact that we wanted a greater amount of information for certain events which the general standardisation was not supplying us. Therefore, a more specific standardisation had to be applied. It was also necessary to process these events in a particular way as they are stored in different DB tables.

The events of 3 of these 4 special types (MAC event, OS event, Service Event), deal with the information in a slightly different way from the rest. The plugin sids indicate the different statuses that the events can adopt once they reach the OSSIM server; thus it does not make sense for the agent to send them (this is kept for historical reasons). As the agent does not save status it cannot know whether or not the event has been seen before. The different plugin sids that the server will insert subsequently in place of those which the agent sends would be:

• (1) New: The event is new and has never been seen
• (2) Change: The event has been seen before, but one of its major fields has changed
• (3) Deleted: Reserved
• (4) Same: The event has already been seen
• (5) Unknown: Reserved

For specific plugins, there can be reserved fields. These are fields which are not generated or sent in the message from the agent, but which are reserved for use in the OSSIM server. This field or fields must not be used by the agent or overwritten with other data. For the time being, they are used for MAC, O.S. and Service events. In the OSSIM server the event data is copied in these fields, this being used for correlation.

Any event it receives from different plugins or devices. An example would be:

event type="detector" date="2006-08-09 12:12:11" plugin_id="4002" plugin_sid="1" sensor="192.168.1.10" interface="eth1" priority="1" src_ip="192.168.1.8" dst_ip="192.168.1.8" data="user1" log="Aug  9 12:12:11 ossim-sensor sshd[6466]: (pam_unix) authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=user1"

The fields of which the standardised event consists are:

• type: Type of event, Detector or Monitor.
• date: date on which the event is received from the device.
• sensor: IP address of the sensor generating the event
• interface: deprecated
• plugin_id: Identifier of the type of event generated
• plugin_sid: Class of event within the type specified in plugin_id
• priority: Possible deprecated (agent can't decide priority, just server)
• protocol: Three types of protocol are permitted in these events. Should a different one reach the server, the event will be rejected::
  • TCP: Once the event reaches the server, it is converted internally to the nº 6
  • UDP: Once the event reaches the server, it is converted internally to the nº 17
  • ICMP: Once the event reaches the server, it is converted internally to the nº 1
• src_ip: IP which the device generating the original event identifies as the source of this event
• src_port: Source port
• dst_ip: IP which the device generating the original event identifies as the destination of this event
• dst_port: Destination port
• log: Event data that the specific plugin considers as part of the log and which is not accommodated in the other fields. Due to the Userdata* fields, it is used increasingly less.
• data: Normally stores the event payload, although the plugin may use this field for anything else
• username: User who has generated the event or user with whom it is identifying, mainly used in HIDS events
• password: Password used in an event
• filename: File used in an event, mainly used in HIDS
• userdata1: These fields can be defined by the user from the plugin. They can contain any alphanumeric information, and on choosing one or another, the type of display they have in the event viewer will change. Up to 9 fields can be defined for each plugin.
• userdata2
• …
• userdata9

Events which inform of changes in MAC address for specific IPs. This could be used in order to create directives sensitive to ARP Spoofing, for example. These events are generated by arpwatch software, integrated into OSSIM. An event of this type can also be generated in the server thanks to Service event messages coming from the agent. These internal messages can be of the MAC change type; when it reaches the OSSIM server, the latter redoes it and converts it into a MAC type event, treating it in the same way as these.

The MAC type event generated may be like this, for example:

host-mac-event host="183.127.115.4" interface="eth1" mac="0:4:23:80:fb:ha" vendor="Intel Corporation" date="2006-03-17 11:30:09" sensor="163.117.131.11" plugin_id="1512" plugin_sid="1" log="ip address: 163.117.155.2 interface: eth1 ethernet address: 0:4:23:88:fb:8a ethernet vendor: Intel Corporation  timestamp: Friday, March 17, 2006 11:30:09 +0100"

We can see that part of the information which is included in the different event fields is also included in the log.

This event has a series of specific fields for it. To simplify the explanation of the fields, although we list all those existing in the event, we are only commenting on the specifics of this event type.

• host: Host IP which has changed MAC
• mac: MAC address in hexadecimal
• vendor: Card manufacturer
• sensor
• interface
• date
• plugin_id: In this case it will always be 1512
• plugin_sid: Without importance, the OSSIM server will assign the correct one
• log

• userdata1: Copy of the MAC address
• userdata2: copy of vendor

Events which inform of a change in the operating system of a machine. Care must be taken to ensure that these changes are not due to load balancers or high-availability systems as in that case it would not be a real change (they can be disabled from the frontend web).

The event generated by the agent after reading the information is of this type:

host-os-event host="192.168.1.81" os="Windows" date="2006-12-23 22:56:13" sensor="192.168.1.10" plugin_id="1511" plugin_sid="1" log="Windows XP" interface="eth1"

The fields of this event type will be:

• host: IP to which the O.S. has been shown
• os: Operating System displayed for the host indicated
• sensor
• interface: deprecated
• date
• plugin_id: In this case, it will always be 1511 (p0f)
• plugin_sid: Assigned by the OSSIM server
• log

• userdata1: In this case it is used to maintain the O.S. for correlation

We use these events in order to keep an inventory of the systems existing on the network, new active applications and open ports being detected. They are also used in cross-correlation, together with the OSVDB DB.

An example event would be:

host-service-event host="192.168.1.77" sensor="192.168.1.10" interface="eth1" port="80" protocol="6" service="www" application="CCO/4.0.3 (Unix) tomcat" date="2006-03-27 07:59:54" plugin_id="1516" plugin_sid="1" log="blablablablabla"

The fields are:

• host
• sensor
• interface: deprecated
• port: Open port displayed in the Host machine
• protocol:
• service: Type of service existing in the specified port (www, ssh, ftp…)
• application: Specific application which executes the displayed service
• date
• plugin_id: This will generally be 1516 (pads)
• plugin_sid: Assigned by the OSSIM server
• log

• userdata1: *Copy of the application field
• userdata2: *Copy of the service field

A sample ossim sensor has got at least the following files:

[DIR]/etc/ossim/agent/

/etc/ossim/agent/aliases.cfg
/etc/ossim/agent/config.cfg

[DIR]/etc/ossim/agent/plugins/

/etc/ossim/agent/plugins/$plugin_name.cfg

OSSIM is capable of collecting information in various ways. In general, all products on the market usually emit or store logs in one of these ways:

The device from which the logs wish to be extracted can inject information directly into the syslog of an OSSIM sensor. An agent will be active in this sensor to read from this syslog, and will standardize the events so that they can be sent to the server on which it depends..

Obviously, in order to add a new plugin it is not sufficient to inject data into the syslog. Instead the steps mentioned in create a new plugin must be followed.

Let us take the NetScreen Manager plugin as an example. This device sends the information on its events to the remote syslog of a machine (we would configure it so that it went to the ossim sensor). In the file /etc/ossim/agent/plugins/netscreen-manager.cfg we can see the special characteristics which enable this plugin to read from the syslog:

[config]
type=detector
enable=yes
source=log
location=/var/log/netscreen.log

Let us look at the last line. In location we can see the place from which the plugin from the agent responsible for dealing with it, is going to read the information. The syslog will need to be configured so that the specific NetScreen Manager events are stored in this file. If we want to keep saving them in the standard file (/var/log/syslog for example) we would just need to modify the location to coincide with it.

Given that the plugin does not mind whether it reads from one file or another and that the lines it reads just need to agree with the established format, it is not actually important whether the log has arrived thanks to the syslog or another element. The format in this case would be the one belonging to the netscreen-manager, which is shown together with its regular expression which agrees with it:

# Log Day Id, Log Record Id, Time Received (UTC), Time Generated (UTC),
# Device Domain, Device Domain Version, Device Name, Category, Sub-Category,
# Src Zone, Src Intf, Src Addr, Src Port, NAT Src Addr, NAT Src Port, Dst Zone,
# Dst Intf, Dst Addr, Dst Port, NAT Dst Addr, NAT Dst Port, Protocol,
# Policy Domain, Policy Domain Version, Policy, Rulebase, Rule Number, Action,
# Severity, Is Alert, Details, User, App, URI, Elapsed Secs, Bytes In,
# Bytes Out, Bytes Total, Packets In, Packets Out, Packets Total, Repeat Count,
# Has Packet Data, Var Data Enum

regexp=([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)

Only the lines which coincide with this regular expression will be sent to the OSSIM server, and that is why it does not matter from which file the information is read.

In order to accept data from another machine, the sensor's syslogd daemon will need to be configured in order to accept connections entering from it to UDP port 514. To this end, the daemon must be started up with the parameter -r:

# syslogd -r

Of course, any firewall configuration must be adjusted so that the events can pass through this port.

We must remember to modify the start files if we want it to always start up in the same way whenever the machine is restarted.

We now need to go to the machine that will be emitting the syslog message. In general, it will be an appliance which has an option for resending data to another place. If it is a typical UNIX machine, the file which will need to be modified is /etc/syslog.conf:

*.*     @OSSIM_sensor_IP

The syslogd daemon is restarted on both machines so that the changes can take effect and we are now receiving syslog information.

If for security reasons we do not want the interface which will be receiving the syslog data to possess IP, the passlogd tool may be used: http://nixbit.com/cat/system/networking/passlogd/ . This tool allows the syslog messages appearing on the network to be collected even though the interface through which they are collected does not have an IP address, irrespective of where the messages are going.

Given that the syslog messages are sent in UDP format and without being encrypted, we believe it is important to configure OSSIM so that these messages are sent through some kind of tunnel. This tunnel can be provided by stunnel, ssh, openVPN, or any other remote encryption method. We shall see here how to do this with ssh, which is one of the most straightforward ways. Obviously this method can only be used if the data source can be manipulated for encryption (the OSSIM sensor will always allow it, although some devices emitting syslog events will not)

If we are going to use an ssh tunnel we will not need to specify to the daemon that allows other machines to access it, as the syslogd daemon will be under the impression that everything is being done in local mode.

1. The first thing will be to create the tunnel between the logging server and the machine emitting the event. It does not need to be root if the port is greater than 1024:

$ ssh -f -N -g -R 41000:localhost:41000 root@IP_remote_syslog_data_sender

1. We then simply need to use netcat to convert the UDP data stream which syslog is using within a TCP to be able to tunnel it through the ssh. This way we also make sure that the messages are sent more securely, as UDP messages do not require receipt notification and can be lost. We would execute the following in the machine emitting the logs:

# nc -l -u -p syslog | nc localhost 41000

1. And finally in the sensor, from where the logs are going to be read once they arrive:

# nc -l -p 41000 | nc localhost -u syslog

This can all of course be put in a script and executed in background, or included in a daemon so that a console does not have to be kept open

In order to receive traps in SNMP format from a device, it will be necessary to install in the sensor which is going to receive the data the program snmptrapd, the daemon controlling it, and the program snmptrapfmt (or simply snmptrap, depending on the software used).

The program can be obtained in several ways:

• If we are in a distribution that uses apt we can simply do something like:

# apt-get install snmptrapfmt

• If we want to install the program in order to install it in any UNIX, we must obtain the Net-SNMP tools (http://net-snmp.sourceforge.net/)

Snmptrapfmt gives format to the trap received as indicated in the configuration file, located in /etc/snmp/snmptrapfmt.conf. By default, all the data will be stored on the file specified in the configuration, in snmptrapfmt.conf. If no log file is specified, the data is normally injected directly into the syslog.

No special configuration is required for receiving the data. The daemon just needs to be uploaded.

To collect the logs subsequently, we will just need to create a new plugin which points towards the file in which the logs are stored, in this case for example it would be in /var/log/snmptrapfmt.log

Use of sockets is restricted at the momment to monitor-type plugins, so it can't be used for detector-type plugins. It's planned to accept log inyections through sockets in the future.

The configuration of the Monitor plugin corresponds to its creation, in any case we specify here the variables which tell us that it is being read from a socket:

[config]
type=monitor
enable=yes
source=unix_socket
location=localhost:40010

In just the same way as with Syslog and SNMP, in order to read from any log file a plugin needs to be created to read from it. The entries defining the file from which it will be read are in:

[config]
source=log
location=/var/log/mylog.log

We can parse data from any file in this way.

As is said in his web page, Osiris is a Host Integrity Monitoring System that periodically monitors one or more hosts for change. It maintains detailed logs of changes to the file system, user and group lists, resident kernel modules, and more. It’s recommended to also validate osiris’s binary itself.

Although this is not a collection method itself (because this is more like a plugin type integrated in OSSIM) we explain it here because it's one possible collection method to extract information from UNIX machines. This collection method uses the usual file logging like was explained above.

You can get Osiris from several places (like his homepage, http://osiris.shmoo.com/index.html), and to install it in a system with apt, you just need to do the following:

• On the Ossim agent do:

# apt-get install osiris osirismd

• On every monitored host:

# apt-get install osirisd

One important thing after you have configured Osiris, is to setup the OSSIM plugin to fit in the exact osirisd configuration. You need to go to the file /etc/ossim/agent/plugins/osiris.cfg and edit the following stanza:

[translation]
# How did you call the host on osiris setup ?
test_host1=192.168.1.10

After installing and starting, type “osiris” and log in as admin. You’ll get instructions on how to change the admin password, it’s recommended to do so.

<note> If you're in a windows machine, you need to adjust program files and windows path information to your setup/language. </note>

Remember to check your firewall settings: Sensor initiated connections towards the monitored hosts must be allowed against port 2265.

In order to get osiris to watch what we need, create a new scanning policy:

# osiris
Osiris Shell Interface - version 4.2.0-release
 >>> authenticating to (127.0.0.1)

User: admin
Password:

connected to management console, code version (4.2.0-release).
hello.

osiris-4.2.0-release: new-host

[ new host ]

  > name this host [127.0.0.1]:  dmz
  > hostname/IP address [127]: 127.0.0.1
  > description []: dmz computer
  > agent port [2265]:
  > enable log files for this host? (yes/no) [no]: yes  

Scan Databases:

    => keep archives of scan databases?  Enabling this option means that the
       database generated with each scan is saved, even if there are no changes
       detected.  Because of disk space, this option is not recommended
       unless your security policy requires it. (yes/no) [no]:

    => auto-accept changes?  Enabling this option means that detected
       changes are reported only once, and the baseline database is
       automatically set when changes are detected. (yes/no) [yes]:

    => purge database store?  Enabling this option means that none
       of the scan databases are saved.  That is, whenever the baseline
       database is set, the previous one is deleted. (yes/no): [yes]:

Notifications:

    => enable email notification for this host? (yes/no) [no]:
    => send notification on scheduled scans failures? (yes/no) [no]:
    => send scan notification, even when no changes detected  (yes/no) [no]:
    => send notification when agent has lost session key  (yes/no) [no]:
    => notification email (default uses mhost address) []:

Scheduling:

  > configure scan scheduling information? (yes/no) [no]:
  > enable this host? (yes/no) [yes]: yes

host                  => dmz
hostname/IP address   => 127.0.0.1
description           => dmz computer
agent port            => 2265
host type             => generic
log enabled           => yes
archive scans         => no
auto accept           => yes
purge databases       => yes
notifications enabled => no
notifications always  => no
notify on rekey       => no
notify on scan fail   => no
notify email          => (management config)
scans starting on     => (not configured)
scan frequency        => daily (every 1440 minutes).
enabled               => yes

Is this correct (y/n)? y
 >>> new host (dmz) has been created.
Initialize this host? (yes/no):yes

You need to remind that you need to activate the host, and enable log files for it. All the other variables may be ok by default.

Snare is one of the possible methods to extract information from a windows box. There are other tools to do this (NtSyslog for example) but we think that at this moment Snare fits in the OSSIM needs perfectly.

To collect Events generated in a windows box we need to install in it Snare (http://www.intersectalliance.com/projects/SnareWindows/index.html). Snare get’s itself installed as a windows Service which is started by default on boot. Then, configure snare to forward this events to a machine with an OSSIM Agent installed in it. As the messages are sent using syslog facilities, you can refer to syslog to check how to configure it. The agent will read the data from syslog file (or whatever file you specified in syslog)

Here at OSSIM we decided that the field delimiter for the Snare events will be ”;”, so you need to change it in the Snare configuration to get it working.

Each Windows host with snare agent installed must be able to send UDP port 514 data towards the sensor. All the Windows events are normalized into the OSSIM nomenclature and sent to the ossim-server.

Collecting data from Checkpoint is a very typical work for the security technical people. You need to download and configure a specific tool to download the logs from the FW-1. That tool need to be installed in the sensor that is going to receive the logs.

Those logs, once downloaded and stored in the sensor hard drive, will be readed from a plugin, exactly equal as the other plugins in the Agent

You need to download fw1-loggrabber, a tool designed to remotely extract log data from a Checkpoint FW-1 firewall thanks to the Checkpoint proprietary LEA protocol (part of their OPSEC API).

There are multiple ways to configure this. You can use it with or without authentication in the FW-1. You'll need to edit two files and configure it:

• lea.conf
• fw1-loggrabber.conf

Once started, if you do a 'ps' you should get something like this:

# ps xa |grep loggrabber
14353 ?        S     22:27 /opt/fw1-loggrabber-1.11.1-linux/fw1-loggrabber -c /opt/fw1-loggrabber-1.11.1-linux/fw1-loggrabber.conf -l /opt/fw1-loggrabber-1.11.1-linux/lea.conf
 2165 pts/0    R+     0:00 grep loggrabber

To configure it you'll need to modify the connection data to connect to the FW-1, and modify the place where it stores the logs. In the fw1-loggrabber.conf config file you need also to change this statement to left it where the plugin can reads:

OUTPUT_FILE_PREFIX="/var/log/ossim/fw-1"

If you use unauthenticated access to the FW-1, you can configure the lea.conf file just with your FW-1 IP.

lea_server ip 192.168.1.1    
lea_server port 50003

The following lines will show how easy it is to create a new plugin. We'll create two different sample plugins, one that reads the agent's own output, one that monitors the system and checks for a running process.

The plugins consist of two basic files, one with its configuration, and another with the information that the OSSIM server needs in order to correlate the events subsequently. In order to create a new plugin, it will only be necessary to create these two files as specified in the documentation. One of the most important parts is to create the regular expression which must correspond to the one in the log file of the device for which we are creating the plugin. Both the server and the plugin have to agree on what each plugin_id and plugin_sid of each event means; both files are therefore inseparable and it is essential to have both in order for the plugin to work.

Sample plugins:

The main steps are:

1. Add plugin entry into config.cfg
2. Create plugin file
3. Create regexp
4. Create plugin_sid sql file
5. Populate database

1. Add the plugin name and config location file to our main config file, by default at ”/etc/ossim/agent/config.cfg”

my_network_device=/etc/ossim/agent/plugins/my_network_device.cfg

2. Create a sample, empty plugins/my_network_device.cfg

This file will have (at least) three main sections:

• [DEFAULT]

Contains information which can be referenced and fetched from everywhere.

• [config]

Contains local information which can only be referenced from outside the file or within it's own section.

• [ossim-agent-format]

Contains actual event parsing code, you can have multiple sections for multiple events. Additionally, another section can be used:

• [translation]

There are plugins where the unique, numeric “plugin_sid” can't be extracted from the log message and has to be compared against a list. This list goes in here.

2.1. [DEFAULT]

This section will only contain our unique plugin identificator, 50001.

[DEFAULT]
plugin_id=50001

2.2. [config]

Here we'll specify some configuration options such as the processes name, where to find it, how to start/stop it and so on. Everything is pretty much self-explaining.

[config]
type=detector
enable=yes
process=ossim-agent
start=no   ; launch plugin process when agent starts
stop=no     ; shutdown plugin process when agent stops
startup=/etc/init.d/%(process)s start
shutdown=/etc/init.d/%(process)s stop
source=log
location=/tmp/sample
# create log file if it does not exists, 
# otherwise stop processing this plugin
create_file=true

As a side note, source can only be “log” for detectors and one of “socket, unix_socket, database, command, file and log” for monitors. We also see how variables can be used within it's own section, as in %(process)s

2.3. [translation]

As we can encounter any kind of data in the logs which are going to be analyzed by the plugins, it is necessary to have the capacity to define each of the fields to which it belongs and its format. This is achieved using the following types of functions:

{translate($action)}

These functions enable us to modify various parameters which may exist in the event we are going to read from the log file that the device has generated, so that it is easier to send them to the OSSIM server afterwards with the information standardized.

Here we'll enumerate some plugin_sids for our new plugin.

[translation]
INFO=1
ERROR=2
WARNING=3

2.4. [ossim-agent-format]

Now for the last part. Here you need to know a bit about regular expressions, and know some important things about how plugins are built.

• You can reference matched chunks either by number ({$1}, {$2}, …) or by name ({$section}, {$severity}, …)
• You can use functions on the matched data which are defined inside ParserUtil.py.

• You can translate() strings into sids that have been previously defined inside the [translation] section.
• You can use aliases defined inside aliases.cfg, such as:

• Aliases are written as \ALIAS. For example, a tcp connection could be written as:

regexp='(\IPV4):(\PORT)->(\IPV4):(\PORT)'

[ossim-agent-format]
# 2007-01-22 10:49:28,191 Output [INFO]: Added Plain output
event_type=event
regexp='^(?P<date>\d+-\d+-\d+\s+\d\d:\d\d:\d\d,\d+)\s+(?P<section>\w+)\s+\[(?P<severity>\w+)\]:\s+(?P<message>.*)$'
date={normalize_date($date)}
src_ip="127.0.0.1"
plugin_sid={translate($severity)}
userdata1={$section}
userdata2={$message}

3. Creating the regexp

There are many good sources on regular expressions, it is strongly recommended to check one of those. For the lazy, here a quick intro:

        pattern = '(\S+) (\S+) (\S+) \[(\d\d)\/(\w\w\w)\/(\d\d\d\d):(\d\d):(\d\d):(\d\d).+"(.+)" (\d+) (\S+)'

Here a sample apache log line:

66.194.xx.xx - - [09/Dec/2005:17:43:55 +0100] “GET / HTTP/1.1” 200 0 ”-” “Mozilla/5.0 (compatible; Konqueror/3.0-rc2; i686 Linux; 20020112)” ”-”

Recalling what we know about regular expressions this one says:

1. Match and store anything that’s not a space (66.194.xx.xx)
2. Followed by a space
3. Match anything not a space again. Store it. (-)
4. Space.
5. Match no-space. (-)
6. Space and escape a “[“ sign
7. Match 2 digits exactly(09)
8. Escape a “/”
9. Match 3 letters exactly (Dec)
10. Escape another “/”
11. Match 4 digits exactly(2005)
12. Ignore “:”
13. Match 2 digits (17)
14. Ignore “:”
15. Match another 2 digits (43)
16. Ignore another “:”
17. Match the last 2 digits (55)
18. Ignore anything up to the next double quote
19. Capture anything before the next double quote
20. Ignore a space
21. Capture any number of digits (200)
22. Ignore another space
23. Capture everything up to the next space (0)

4. Creating the plugin_sid SQL file.

Among the last things left to do is identifying all the possible plugin_sid values our plugin can handle. In our sample plugin we only have three cases, located inside the [translation] section:

[translation]
INFO=1
ERROR=2
WARNING=3

<note> Of course, sometimes you won't need to do a translation. May be that in the logline appears some kind of identifier, and you c an take the number from it. For example, the osiris plugin can match the following plugin sid in the line:

#Feb 28 18:16:15 Gestalt osirismd[1949]: [102][*][err] user authentication failure: admin

regexp="(?P<date>\SYSLOG_DATE)\s+(?P<sensor>[^\s]*)\s+osirismd\[(?P<pid>\d+)\]:\s+\[(?P<plugin_sid>[^\]]*)\]\[(?P<src_ip>[^\]]*)\]\[(?P<event_type>[^\]]*)\]\s+user authentication failure:\s+(?P<username>\S+)"

In this example, the plugin_sid will be 102:

plugin_sid={$plugin_sid}

</note>

So, this leaves us with four SQL lines:

INSERT INTO plugin (id, type, name, description) VALUES (50001, 1, 'my_network_device', 'My Network machine');

INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (50001, 1, NULL, NULL, 'My-net-device: INFO', 1, 3);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (50001, 2, NULL, NULL, 'My-net-device: ERROR', 2, 3);
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, name, priority, reliability) VALUES (50001, 3, NULL, NULL, 'My-net-device: WARNING', 3, 3);

5. Populate the DB

Let's say we called our sql file ossim-agent.sql. The easiest way to get this information into the database is issuing the following command on the server host:

cat my_network_device.sql | mysql -uUSER -pPASSWORD ossim

Restart the OSSIM server so that it is aware of the new plugin information.

/etc/init.d/ossim-server restart

6. Complete config file

And once again, all the sections together. (This would be my_network_device.cfg).

[DEFAULT]
plugin_id=50001

[config]
type=detector
enable=yes
process=my_network_device
start=no   ; launch plugin process when agent starts
stop=no     ; shutdown plugin process when agent stops
startup=/etc/init.d/%(process)s start
shutdown=/etc/init.d/%(process)s stop
source=log
location=/tmp/sample
# create log file if it does not exists, 
# otherwise stop processing this plugin
create_file=true

[translation]
INFO=1
ERROR=2
WARNING=3

[ossim-agent-format]
# 2007-01-22 10:49:28,191 Output [INFO]: Added Plain output
event_type=event
regexp='^(?P<date>\d+-\d+-\d+\s+\d\d:\d\d:\d\d,\d+)\s+(?P<section>\w+)\s+\[(?P<severity>\w+)\]:\s+(?P<message>.*)$'
date={normalize_date($date)}
src_ip="127.0.0.1"
plugin_sid={translate($severity)}
userdata1={$section}
userdata2={$message}

Once that we have created a plugin, we have to test it. In this example, our my_network_device will send syslog data to the ossim-agent. So you can trigger events using a small python script that sends a message to syslog. This process can be used to verify that the agent and server are functioning and that the agent can send information to the server.

For this test, we'll use the following different plugin:

DELETE FROM plugin WHERE id = "20000";
DELETE FROM plugin_sid where plugin_id = "20000";


INSERT INTO plugin (id, type, name, description) VALUES (20000, 1, 'foobar', 'Foobar demo detector');


INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, reliability, priority, name) VALUES (20000, 1, NULL, NULL, 6, 4, 'foobar: new foo found on (DST_IP)');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, reliability, priority, name) VALUES (20000, 2, NULL, NULL, 6, 1, 'foobar: foo the same on (DST_IP)');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, reliability, priority, name) VALUES (20000, 3, NULL, NULL, 10, 2, 'foobar: foo changed on (DST_IP)');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, reliability, priority, name) VALUES (20000, 4, NULL, NULL, 8, 3, 'foobar: foo deleted on (DST_IP)');
INSERT INTO plugin_sid (plugin_id, sid, category_id, class_id, reliability, priority, name) VALUES (20000, 5, NULL, NULL, 10, 5, 'foobar: alien foo on (DST_IP)');
__END__

First, check the OSSIM web interface and verify the plugin exists in Configuration→Plugins.

If for some reason you modify the values of reliability and priority for each plugin_sid, you will need to restart the OSSIM server in order for it to take effect.

For the test network, I recommend that you configure something similar to the following diagram. In the following case, the (OSSIM server and OSSIM framework) run on the host trinidad. The OSSIM Agent runs on the host cienfuegos. The network has three other hosts: mojito, eldedo, and guane. The three hosts send their syslog data to cienfuegos, where we have the foobar plugin monitoring the syslog output from /var/log/user.log .

For this test setup, I assume you are running Debian. Modify cienfuegos so that it can receive remote syslog. Modify the file /etc/default/syslogd and make sure it contains the following line.

SYSLOGD="-r"

Now on each of the hosts (eldedo, guane, and mojito) configure them to send syslog data for user.log to cienfuegos. You will have to edit /etc/syslog.conf .

user.*                         @cienfuegos

Now restart your syslog daemon.

/etc/init.d/syslogd restart

This is the sample python script that will send a message to syslog. Run this program on the host on which you want to generate an event. If you configured the option setup, you will want to run the following script on each of the remote hosts. Otherwise, you can run it on the sensor.

#! /usr/bin/python
import syslog
import sys

syslog.openlog("FOOBAR", syslog.LOG_PID , syslog.LOG_USER )
for arg in sys.argv:
        if arg == "1":
                syslog.syslog(syslog.LOG_WARNING, "new foo found")
        elif arg == "2":
                syslog.syslog(syslog.LOG_WARNING, "foo the same")
        elif arg == "3":
                syslog.syslog(syslog.LOG_WARNING, "foo changed")
        elif arg == "4":
                syslog.syslog(syslog.LOG_WARNING, "foo deleted")
        elif arg == "5":
                syslog.syslog(syslog.LOG_WARNING, "alien foo")

syslog.closelog()

Run the program. Run this program on the server for which you want to generate the event. The following will send the first syslog message.

testfoobar.py 1

The second will send the 5th syslog message.

testfoobar.py 5

Check your syslog output. Depending upon how your syslog is configured, it will go to one of the syslog files. Tail all of them and see if you see a message. You should see information regarding foobar.

tail /var/log/{syslog,user.log,messages}

Now check the agent.log file and see that it matched.

tail /var/log/ossim/agent.log

Now check the OSSIM _server_ and see if it made it there.

tail /var/log/ossim/server.log

Check the Web Interface for the event under 'Control Panel → Event Viewer'. You should see an event similar to the below. If the event made it, you successfully created a plugin.

[foobar] foobar: a new foo detected

Q. I don't see any information in /var/log/syslog

A. Check /var/log/{messages,auth.log,syslog} and other files. You can use the tail command.

 tail /var/log/{messages,auth.log,syslog}