11/26/2014

DataPower: How to send logs to a remote syslog server

Well, here's my first post to this blog. From time to time, I'll try to share some things I learned over the past few years I'm enjoying working with DataPower. Thanks to Bruno Neves for inviting me to colaborate to this blog!

I will start with a little thing, so simple, but I consider it so important: to send DataPower log information to a remote server. Because we don't want to keep DataPower system logs in DataPower filesystem only, right? We never know when an appliance will crash!

DataPower offers a variety of options to send logs elsewhere. We can send them to a SOAP client, via email (SMTP), to a NFS system... Here, we will send log information to a syslog server. The key object to achive this is the Log Target object. It's located at default domain (of course, your user account must have the right privileges to deal with it).

Configure a syslog server


First things first! We need a syslog server, right? Here, I will show how to set up a syslog server in a machine running Ubuntu. It's quite simple using rsyslog service. First, you need to allow rsyslog to accept incoming syslog information from clients. For that, you have to change the rsyslog.conf file, as showed bellow:

$> sudo vi /etc/rsyslog.conf

Uncomment the lines that allow incoming traffic over UDP or TCP protocols. I chose TCP, so I remove comments from these lines below to allow traffic over TCP using port 514:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Now, we will define the file name pattern for our log files. I decided to store the files at /var/log/rsyslog/<datapower_name>, and they must be named with the date (year-month-day) plus DataPower name. So, I had to add the following lines to the end of the file:

$template DailyPerHostLogs, "/var/log/rsyslog/%HOSTNAME%/%$YEAR%%$MONTH%%$DAY%-%HOSTNAME%.log
dtp* -?DailyPerHostLogs

If you want to change the file name pattern, that's fine. Go to http://www.rsyslog.com/doc/master/configuration/properties.html and look for other options.

Now we have to allow the user syslog to write on our log directory:

$> sudo chown syslog:syslog rsyslog

Save these changes and restart rsyslog service:

$> sudo service rsyslog restart

Configure a Log Target on DataPower


Now it's time to setup a syslog client on DataPower, by creating a new Log Target object. I will explain how to do that via Web GUI. Later you can play around and try to do that with SSH or XML.
  • On default domain, look for "Log Target" or go to Objects > Logging Configuration > Log Target;
  •  A list of Log Targets will be displayed. You must have at least one, the default-log object, which throws log entries to the default system log. Don't change it, unless you really know what you're doing! Click Add to create a new object;
  • Change the following properties:
    • Name: it's the name of the object. I named it "Syslog-LogTarget" (you can go with anything else);
    • Target Type: select "syslog-tcp";
    • Local IP Address: the IP address of your DataPower device. In my case, it's "192.168.75.128" (which is set up in a Host Alias object);
    • Local Identifier: identifies who's sending the log info for the syslog server. I went with"dtp6";
    • Remote Host: the IP address where the syslog server is running. For me, it's "192.168.75.133";
    • Remote Port: the port where the syslog server is running. It should be "514", unless you set up a different port on rsyslog.conf file;
Log Target configuration with syslog-tcp as target type
  • We also have to define what type of info will be sent to our syslog server. Go to the Event subscriptions tab. A list of events must be provided. Here, we will send all types of information, with a log level of "notice". You may want to have a more filtered log. If you do, select different events;
Adding event subscriptions to Log Target

  • Save these changes (by clicking Apply).

At this point, you should start seeing your logs being sent to your syslog server. You will notice that the file is named with the date stamp and your DataPower identifier.

$> cd /var/log/rsyslog/dtp6
$> ls
20141125-dtp6.log

Now, use tail -F * while interacting with DataPower and watch the magic happen.

I hope this helps. Let me know your thoughts. Feel free to comment here.

Cheers!

12 comments:

  1. Very helpful info, thanks!

    ReplyDelete
  2. Hello,

    is there a way to force datapower to use a single consistent source port for its syslog messages ?
    It seems it uses dynamic source ports, what is very confusing for our firewalls (as it creates a session for each record).

    BTW this is not the way recommended by the RFC 3164:
    "It is RECOMMENDED that the source port also be 514 to indicate that the message is from the syslog process of the sender, but there have been cases seen where valid syslog messages have come from a sender with a source port other than 514. If the sender uses a source port other than 514 then it is RECOMMENDED and has been considered to be good form that subsequent messages are from a single consistent port."

    ReplyDelete
    Replies
    1. Hey Stéphane,

      I doubt there is a way to force DataPower to do that... I suppose that even the SPI feature of modern firewalls wouldn't help with that as, as you said, it creates a new session for each syslog message.

      I believe that the only way to make it work flawlessly is by creating a broader firewall rule to allow all messages from all DataPower ports and with destination equal to Syslog server on port 514.

      By the way, if you have proof that DataPower is doing that, let's say by having some packets captured, I would suggest sending it to IBM in a form of PMR stating that their code is not complying with the syslog RFC. That should be faster than submitting it as a Request for Enhancement.

      Delete
  3. This is very helpful information and Thanks for sharing.

    If you want to log entire request message in the Rsyslog How can we do that? Is there any restriction at Rsyslog server to log >2kb messages.

    Thanks

    ReplyDelete
  4. Thank you for such a informative and easy to understand post on configuring syslog for datapower.

    ReplyDelete
  5. Hi,
    Thank you for the information.
    Please tell me In what pattern logs will store in remote server? as well as how will get the logs directly from remote server?

    ReplyDelete
  6. Hello,

    I want to send custom logs from my policy (under WSP), to a remote (syslog) location.
    I tried creating log category and log target to write the data in category to the location. I was successful by DP limits you to write data upto 2048 bytes only.
    What can be the alternate solution?

    I'm new to DP, so could you please help with description how can this be done.
    Thanks!

    ReplyDelete