How to setup SNMP v3 on DataPower

I'm creating this entry because I needed to setup SNMP to my current client and didn't find much information on the internet, so, as I always say: "human´s knowledge belongs to the world". :)

The environment consist of 2 IDGs devices and one Linux Red Hat server as a consumer. The current DataPower firmware is IDG.7.6.0.4.

The only page for SNMP is the one below. The most important part is the user configuration which I will show you right after:

See that the "Local IP Address" is my outbound interface. Port is the default one, 161, SNMP version is the v3 and security level is "Authentication, No privacy". Those are the values I used as an example, you can change it if needed. Attention, this is only to make sure DataPower is configured correctly. This is not a SNMP tutorial. :)

Now go to your server (Linux Red Hat with snmp-net) and run:

 #snmpgetnext -v 3 -u snmp-pooling -a MD5 -A "snmppooling" -x DES -X "snmppooling" -l authNoPriv 10.136.51.214 .1.3.6.1.4.1.14685.3.2.15.1.2

#SNMPv2-SMI::enterprises.14685.3.2.15.1.2.1 = STRING: "admin"

See that the password: snmppooling was setup in the previous page of DataPower. Replace also the IP address from the command below by your DataPower IP address. The .1.3.6.1.4.1.14685.3.2.16.1.2 is the ObjectID value, you can get it opening the .txt files from SNMP Setting > Enterprise MIBs tab. See screenshot:

You can open such file in the MibBrowser, you can get more details about this tool here: http://integrationyay.blogspot.de/2014/04/datapower-testing-snmp-using.html, loading the file in this free tool, you will see the Object ID values in a more friendly way.

See an example screenshot:

DataPower: How to duplicate, rename, and bulk delete objects?

Introduction

Did you know that all the objects that exist in a domain are there just because of a single configuration file? Did you know that you can wipe out all the objects by simply deleting the content of this file? Did you know that you can manipulate all the objects using your favorite text editor?

If you answered yes to any of the above questions, you are done and don't need to spend any more time at this article. If not, take a seat, relax, and enjoy these tips as you become a DataPower cheater.

Understanding the case

While booting, DataPower will look for all the .cfg files spread over all the domains. These .cfg files contain instructions on what objects and configuration should be loaded to the memory for execution time. Actually, the only reason why you have the objects in place when you restart your domain or recycle the entire box is because of these .cfg files that are storing details of almost everything for you.

Take a moment to analyze any .cfg file you may have in the default or any other existing domain. Usually they are named after the domain name, so if you have a domain called Sandbox, you will have one config:///Sandbox.cfg file. The .cfg file of the default domain falls into an exception and will be called autoconfig.cfg.

Take a moment there, start reading some lines, you will soon realize that all the objects you have created using any DataPower interface will be there. Created a new Crypto Profile? It will be there. Created a new Deployment Policy? It will also be there. Defined a password for a local user? Nah, that will not be there, and honestly, I don't know where this information is stored. :-)

Usage

I particularly use this a lot. Duplicating, triplicating and sometimes quadruplicating Deployment Policies used to kill me back in the day. That was because I used to do all that using the WebGUI. If you have already done that you know what I am talking about. Imagine a scenario which you need to create one Deployment Policy for each environment, let's say one for UNIT, one for INTG, one for PERF, and one (or more) for PROD. I would imagine you would keep the one for UNIT that you supposedly created first open in one browser tab and visually refer to it when creating the others. That means a lot of eyes work from left to right and MANY clicks to add the Modified Configurations. So after dying a few times, I decided to investigate a less time consuming way to do this, and that was when I had this idea to do this through the .cfg file.

My UNIT deployment policy was the following:

deployment-policy "Service-UNIT-DeploymentPolicy"

 modify "*/*/wsm/wsm-endpointrewrite?Name=.*&Property=WSEndpointRemoteRewriteRule/RemoteEndpointHostname&Value=.*" "change" "" "unit-server.datapower.com"

 modify "*/*/wsm/wsm-endpointrewrite?Name=.*&Property=WSEndpointRemoteRewriteRule/RemoteEndpointPort&Value=.*" "change" "" "80"

 modify "*/*/services/multiprotocol-gateway?Property=DebugMode" "change" "" "off"

 modify "*/*/services/ws-proxy?Property=DebugMode" "change" "" "off"

exit

So to duplicate something similar for my INTG environment, I just copied it and pasted it on the line below, changing the parameters pertaining to the new environment, such as name, hostname, port, etc:

deployment-policy "Service-UNIT-DeploymentPolicy"

 modify "*/*/wsm/wsm-endpointrewrite?Name=.*&Property=WSEndpointRemoteRewriteRule/RemoteEndpointHostname&Value=.*" "change" "" "unit-server.datapower.com"

 modify "*/*/wsm/wsm-endpointrewrite?Name=.*&Property=WSEndpointRemoteRewriteRule/RemoteEndpointPort&Value=.*" "change" "" "80"

 modify "*/*/services/multiprotocol-gateway?Property=DebugMode" "change" "" "off"

 modify "*/*/services/ws-proxy?Property=DebugMode" "change" "" "off"

exit

deployment-policy "Service-INTG-DeploymentPolicy"

 modify "*/*/wsm/wsm-endpointrewrite?Name=.*&Property=WSEndpointRemoteRewriteRule/RemoteEndpointHostname&Value=.*" "change" "" "intg-server.datapower.com"

 modify "*/*/wsm/wsm-endpointrewrite?Name=.*&Property=WSEndpointRemoteRewriteRule/RemoteEndpointPort&Value=.*" "change" "" "80"

 modify "*/*/services/multiprotocol-gateway?Property=DebugMode" "change" "" "off"

 modify "*/*/services/ws-proxy?Property=DebugMode" "change" "" "off"

exit

Once done, it was just a matter of restarting the domain and the Deployment Policies were replicated.

Other related tips...

Use it to find out the right CLI command to perform a specific action

Don't remember exactly the CLI command to create a crypto object? Just go to the .cfg file and try to find a reference for it, when you find it, that is the command you are looking for. That is because the .cfg is nothing more, nothing less than CLI commands grouped in a single place.

Use it to delete a massive amount of objects

For example, you created a bunch of Web Service Proxies or Multi Protocol Gateway. By default, when you create these objects you are rewarded with a lot of child objects that will not go away even after you remove their parents, such as matches, processing actions, slm policies, etc. Good news is that they are named after their parents, so you can sort them out easily and quickly select them for removal.

Changing object names

In general, you cannot change object names in DataPower. Gave it the wrong name? Delete it and recreate it. As an alternative to this, go to the .cfg file and rename it there. Once done, restart the domain and voilà!

Recommendations

• Be careful. Make sure you know what you are doing before playing with this file. Rule of thumb, always have a backup of it in case something goes wrong.
• Some words are reserved, and the default interfaces won't be there to tell you that, so be careful. Rule of thumb, avoid naming objects that can be confused with an existing command, for example when naming an Alias object, use "xmlAlias" instead of only "xml", as it may be parsed incorrectly by the interpreter when loading the objects to the memory.
• And last but not least, BE CAREFUL, not sure if I have already said that.

Conclusion

The possibilities are endless, you can optimize a lot of your time after mastering on this. Duplicating, bulk deletion, searching for commands, are just the obvious things you can do while playing with the .cfg files. The limit here is your imagination, so if you think of something cool that the knowledge present in this article enabled you to do, don't hesitate to share it with us in the comments section below. Happy .cfg'ing!

DataPower: Keeping your appliances less prone to attacks

Chances are if you have your DataPower appliances facing the Internet, you have suffered or are suffering right now brute force attacks. Don't you think so? Check your logs...

This problem becomes more evident when you have default ports open to the Internet, let's say ports 22, 80, 443, 5550, 9090, etc...

This happens because there are thousands of robots out there scanning for all IPs and ports open all around the world. They use something called brute force technique.

Brute force is a type of attack that tries to get access to servers by repeating different combinations of credentials. It may also make use of a dictionary with the most commonly used passwords out there, like 123456, password, qwerty, abc123, etc.

Let's take a look at this real case scenario:

20150213T043514Z [network][error] : tid(2): TCP connection attempt refused from 159.226.43.96 to X.X.X.X port 80

20150213T043515Z [network][error] : tid(2): TCP connection attempt refused from 159.226.43.96 to X.X.X.X port 80

20150213T063841Z [auth][error] : User '/etc/init.d/iptables stop' failed to log in.

20150213T063841Z [auth][error] : User 'service iptables stop' failed to log in.

20150213T063841Z [auth][error] : User '/tmp/init.d/iptables stop' failed to log in.

20150213T072620Z [auth][error] : User 'service iptables stop' failed to log in.

20150213T072628Z [auth][error] : User 'chmod 777 148080' failed to log in.

20150213T072636Z [auth][error] : User 'cd /tmp' failed to log in.

20150213T073442Z [network][error] : tid(2): TCP connection attempt refused from 78.101.49.77 to X.X.X.X port 80

20150213T073443Z [network][error] : tid(2): TCP connection attempt refused from 78.101.49.77 to X.X.X.X port 80

20150213T084505Z [auth][error] : User 'service iptables stop' failed to log in.

20150213T084513Z [auth][error] : User 'chmod 777 148080' failed to log in.

20150213T084521Z [auth][error] : User 'cd /tmp' failed to log in.

20150213T120039Z [network][error] : tid(2): TCP connection attempt refused from 199.217.118.79 to X.X.X.X port 10000

20150213T120350Z [network][error] : tid(2): TCP connection attempt refused from 112.221.251.221 to X.X.X.X port 80

20150213T124135Z [auth][error] : User '/etc/init.d/iptables stop' failed to log in.

20150213T124135Z [auth][error] : User 'service iptables stop' failed to log in.

20150213T124135Z [auth][error] : User '/tmp/init.d/iptables stop' failed to log in.

Someone trying to login with the user id "service iptables stop" or "chmod 777 148080", looks suspicious, right? LOL

By observing this behavior for a few months now, I noticed that most IPs are from China. On the connection attempts I received today however, just one is from China (159.226.43.96), then we have one from Qatar (78.101.49.77), one from United States (199.217.118.79), and one from Korea (112.221.251.221). Before you are start blaming these countries, keep in mind that it is really simple to fake an IP like that in order to hide the real source of the attacks.

What I want to say here is that blocking IPs with ACLs will barely provide a solution, even if you block huge ranges of IPs of a given country. It may work for the short term, but will fail on the long term as other IPs will rise. Trust me, I tried! :-)

The truth is you are never 100% safe, what you can do is to try to understand how these robots work and come up with a strategy to deceive or avoid them. Some robots are more intelligent than others, so they will spend some time performing a full port scan in a certain IP in order to determine what the open ports are, and then direct an specific attack for specific services. For example, if port 22 is open, it will start with user id and password, and only after it succeeds it will try actual OS commands. If port 80 is open, it will probably try to exploit some recently discovered vulnerabilities pertaining to Web Servers.

Your best bet to avoid this kind of attack is to just eliminate the default ports from your configuration, as well as limiting the number of ports open to the Internet, for example, there is hardly the need to leave the administration related services, such as SSH, SOMA, and WebGUI, open to the Internet, when most of the companies have VPN services that would allow administrators to access those interfaces from a more secured environment.

EDIT: Dan Zrobok also wrote a very interesting article about security titled Five Common Security Issues Found in DataPower Environments. We highly recommend this reading as he mentions other very important topics on Firmware Currency, Administrative Accounts, TLS Cipher Suites and Protocol Versions, and Exception Information Leaks (when you give more information in the error message than needed).

By having all this information in mind, you are not only keeping your device safer, but you are also implementing best practices that are used in the most secure enterprises around the globe.

Have other ideas to prevent attacks, share with us!

DataPower: How to send logs to a remote syslog server

Well, here's my first post to this blog. From time to time, I'll try to share some things I learned over the past few years I'm enjoying working with DataPower. Thanks to Bruno Neves for inviting me to colaborate to this blog!

I will start with a little thing, so simple, but I consider it so important: to send DataPower log information to a remote server. Because we don't want to keep DataPower system logs in DataPower filesystem only, right? We never know when an appliance will crash!

DataPower offers a variety of options to send logs elsewhere. We can send them to a SOAP client, via email (SMTP), to a NFS system... Here, we will send log information to a syslog server. The key object to achive this is the Log Target object. It's located at default domain (of course, your user account must have the right privileges to deal with it).

Configure a syslog server

First things first! We need a syslog server, right? Here, I will show how to set up a syslog server in a machine running Ubuntu. It's quite simple using rsyslog service. First, you need to allow rsyslog to accept incoming syslog information from clients. For that, you have to change the rsyslog.conf file, as showed bellow:

$> sudo vi /etc/rsyslog.conf

Uncomment the lines that allow incoming traffic over UDP or TCP protocols. I chose TCP, so I remove comments from these lines below to allow traffic over TCP using port 514:

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Now, we will define the file name pattern for our log files. I decided to store the files at /var/log/rsyslog/<datapower_name>, and they must be named with the date (year-month-day) plus DataPower name. So, I had to add the following lines to the end of the file:

$template DailyPerHostLogs, "/var/log/rsyslog/%HOSTNAME%/%$YEAR%%$MONTH%%$DAY%-%HOSTNAME%.log
dtp* -?DailyPerHostLogs

If you want to change the file name pattern, that's fine. Go to http://www.rsyslog.com/doc/master/configuration/properties.html and look for other options.

Now we have to allow the user syslog to write on our log directory:

$> sudo chown syslog:syslog rsyslog

Save these changes and restart rsyslog service:

$> sudo service rsyslog restart

Configure a Log Target on DataPower

Now it's time to setup a syslog client on DataPower, by creating a new Log Target object. I will explain how to do that via Web GUI. Later you can play around and try to do that with SSH or XML.

• On default domain, look for "Log Target" or go to Objects > Logging Configuration > Log Target;
•  A list of Log Targets will be displayed. You must have at least one, the default-log object, which throws log entries to the default system log. Don't change it, unless you really know what you're doing! Click Add to create a new object;
• Change the following properties:
• Name: it's the name of the object. I named it "Syslog-LogTarget" (you can go with anything else);
• Target Type: select "syslog-tcp";
• Local IP Address: the IP address of your DataPower device. In my case, it's "192.168.75.128" (which is set up in a Host Alias object);
• Local Identifier: identifies who's sending the log info for the syslog server. I went with"dtp6";
• Remote Host: the IP address where the syslog server is running. For me, it's "192.168.75.133";
• Remote Port: the port where the syslog server is running. It should be "514", unless you set up a different port on rsyslog.conf file;

Log Target configuration with syslog-tcp as target type

• We also have to define what type of info will be sent to our syslog server. Go to the Event subscriptions tab. A list of events must be provided. Here, we will send all types of information, with a log level of "notice". You may want to have a more filtered log. If you do, select different events;

Adding event subscriptions to Log Target

• Save these changes (by clicking Apply).

At this point, you should start seeing your logs being sent to your syslog server. You will notice that the file is named with the date stamp and your DataPower identifier.

$> cd /var/log/rsyslog/dtp6

$> ls

20141125-dtp6.log

Now, use tail -F * while interacting with DataPower and watch the magic happen.

I hope this helps. Let me know your thoughts. Feel free to comment here.

Cheers!

DataPower: The Interoperability Test Service

Hate having to create an XML Firewall service configured as loopback every time you want to test a single stylesheet using DataPower extension functions? Well, you may have more options available to you…

Since version 5.0, IBM released a capability called Interoperability Test Service (IOP). One of the features this mode supports is the ability to listen to requests containing a stylesheet file along with its equivalent request, then it processes the instructions present on the stylesheet against its request, and finally sends the output of the stylesheet back as the result.

To use the IOP, you first need to enable the service at Objects > Device Management > Interoperability Test Service.

For the second step, you will need a client to send requests to the service you have just enabled. IBM thinking about on making our lives easier, made available two clients that can be found at Resource Kit 5.0.

After downloading it, just extract it to your local file system, and then refer to the files DPInteropClient.jar or dp-interop-client.sh (depending on your preference, operating, system, etc).

The Resource Kit 5.0 comes with some samples that can be used for testing. The first scenario we will cover here will be the conversion of an XML file into the base 64 by using the dp:encode extension function:

The XML message to be converted to base 64
$ cat message.xml
<msg>Hello World!</msg>

The XSLT used to encode the message into base 64
$ cat toBase64.xsl
<xsl:stylesheet version="1.0" 
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:dp="http://www.datapower.com/extensions"
  extension-element-prefixes="dp">

  <xsl:template match="/">
    <xsl:value-of select="dp:encode(., 'base-64')"/>
  </xsl:template>
</xsl:stylesheet>

The IOP client sending both the XSLT and message to be encoded along with the result from DataPower:
$ ../../clients/dp-interop-client.sh -x toBase64.xsl -i message.xml -h [dphostname] -p [port]
>> Creating the XSLT request...
>> Sending the request to http://dphostname:port/
>> No basic authentication is provided.
SGVsbG8gV29ybGQh

Another interesting feature available is the ability to test XPath syntaxes. For example, take into account the following XML (not included in the Resource Kit):

$ cat bookstore.xml 
<?xml version="1.0" encoding="UTF-8"?>
<bookstore>
<book category="COOKING">
<title lang="en">Everyday Italian</title>
<author>Giada De Laurentiis</author>
<year>2005</year>
<price>30.00</price>
</book>
<book category="CHILDREN">
<title lang="en">Harry Potter</title>
<author>J K. Rowling</author>
<year>2005</year>
<price>29.99</price>
</book>
<book category="WEB">
<title lang="en">XQuery Kick Start</title>
<author>James McGovern</author>
<author>Per Bothner</author>
<author>Kurt Cagle</author>
<author>James Linn</author>
<author>Vaidyanathan Nagarajan</author>
<year>2003</year>
<price>49.99</price>
</book>
<book category="WEB">
<title lang="en">Learning XML</title>
<author>Erik T. Ray</author>
<year>2003</year>
<price>39.95</price>
</book>
</bookstore>

Now use the following XPath expression against it (this expression will select the title and price of all books on the category "WEB" that are priced above 39.95:

/bookstore/book[@category=\"WEB\" and price>39.95]/(title|price)

To apply the above XPath expression to the IOP client command, use the following:

$ ../../clients/dp-interop-client.sh -t xpath -e "/bookstore/book[@category=\"WEB\" and price>39.95]/(title|price)" -i bookstore.xml -h [dphostname] -p [port]
>> Creating the XPath request...
>> Sending the request to http://dphostname:port/
>> No basic authentication is provided.
Found 2 nodes:
-- NODE --
<title lang="en">XQuery Kick Start</title>
-- NODE --
<price>49.99</price>

Cool, huh?

Besides XSLT and XPath testing, you can also use this tool to test FFD requests and perform schema validation. For more details, refer to the Interoperability Test Service page at the DataPower Information Center.

I remember that in the past, there used to be a DataPower plugin for Eclipse that enabled you to do practically the same stuff, but for some reason I think it was discontinued by IBM. Have you ever heard of it? Had a chance to use it? How do you compare it with this new method available?

DataPower: No more XG45, XI52, XB62 and XI50B

You read it correctly folks, as of November 21st, 2014, there will be no more DataPower Service Gateway XG45, no more DataPower Integration Appliance XI52, and no more DataPower B2B Appliance XB62. No reason to be scared though, even with that scary headline I put on purpose (I am sorry about that).

As usual, every October IBM releases some news regarding the next firmware version that becomes available every November. This time IBM announced the firmware version 7.1 and with it, its intention to unify the family of appliances into a single product, the IBM DataPower Gateway.

The IBM DataPower Gateway will come equipped with the same functions the current XG45 comes with. If advanced functions are needed, such as B2B features available only on the current XB62, a software module can be installed to satisfy the requirement.

This is actually good news for companies that already have XG45 and/or XI52 appliances and were planning to acquire more appliances to support B2B functions. With the version 7.1, it will now be possible to install the new software module on the current appliances and take advantage of the B2B functions previously available only on the XB62 appliances.

The DataPower family of appliances is downsizing again, in both models and colors (remember the good times?). From the green XA35, the yellow XS40, the blue XI50, the purple XB60, and the silver XM70, passing through the XG45, XI52, XB62, all black, to only one Gateway Appliance now (color TBD, but I can bet it will be black as well). In spite of that, no feature was left behind, every thing that could be done a few years back can still be done on the current offerings, and that is what really matters for us, DataPower lovers.

Please refer to the official IBM Announcement describing the new offerings in details.

IIB: WebSphere MQ no longer required to run IBM Integration Bus

As you may know, the IBM Integration Bus is the successor of the WebSphere Message Broker. You may also know that to install the WebSphere Message Broker you also need to install a copy of the WebSphere MQ. And this is also true for the latest final version of the IBM Integration Bus, the version 9.

What you may not know is that the upcoming version of the IBM Integration Bus, the version 10, will no longer require a pre-installation of the WebSphere MQ, based on the current version of the IBM Integration Bus, the V10 Open Beta.

WebSphere MQ license is still part of the IBM Integration Bus package, but its installation will be required in only a few cases, such as when you want to use an MQ Input or MQ Output nodes in one of your flows.

To see what else is new in this fairly new version, you can refer to the latest release notes of the product published by IBM back in May.

DataPower: IBM technotes are now providing GatewayScript sample code in addition to XSLT

With the recent release of the GatewayScript for the DataPower family of appliances, IBM is now writing technotes that, in addition to XSLT, also contain a sample of code for the new GatewayScript format.

An example of this, would be a technote published last week that describes how to remove the Content-Type header that DataPower automatically adds to the responses. For the XSLT example they provided the following solution:

<xsl:template match="/">
 <dp:remove-http-response-header name="Content-Type"/>
</xsl:template>

Now for the GatewayScript, the following code was provided:

var hm = require('header-metadata');
hm.current.remove('Content-Type');

For experienced DataPower developers, the first solution seems simpler, as it requires only one line of code to perform that action (ignoring of course the opening and closing of the <xsl:template> element).

The GatewayScript solution though, may look simpler for those used to the JavaScript language, which the GatewayScript is based on.

What would have been your choice if you were assigned to solve this issue? XSLT or GatewayScript?

DataPower: Meet the new features that will be available on firmware V7.0

We are exactly one month away from the general availability of the new DataPower firmware V7.0. Although we have already mentioned the capability that will change the course of our lives forever (just kidding), the new firmware will also bring to the table many other cool features that many clients have been asking as enhancements, like the following:

• A dedicated virtual appliance edition for developers
• SFTP support for XG45
• GatewayScript (JavaScript enablement in the processing policy)
• Improved API Management
• Network Link Aggregation for redundancy and increased throughput
• WebSocket Proxy for low-latency communication
• Support for Sterling Multi-Enterprise Integration Gateway (MEIG)
• GTID or Global Transaction ID to ease troubleshooting of chained services
• Citrix XenServer support for additional deployment flexibility

Yesterday IBM released a new slide deck detailing each of the new features. Highly recommended if you/your company have been waiting for any of these features to be available.

DataPower: 5 secrets that you may not have known

Okay, let's go straight to the point, starting with a few questions:

1. Is it possible to disable a Multi-Protocol Gateway or a Web Service Proxy?
2. Is it possible to export a certificate from the cert: directory?
3. Is it possible to increase the number of transactions in the probe history?
4. Can a Front Side Handler object be created automatically when creating a Web Service Proxy?
5. Can a service have higher priority over others?

You might have answered yes to at least one of these questions, but would you be surprised if I told you that the answer for every single one is yes?

Follow the link below to see more details about each of the secrets above.