Monitor WSO2 Servers with Wireshark

I recently had to look in to how to use Wireshark to monitor the traffic between set of WSO2 products, I found some useful configuration steps which was bit difficult to find in a single place in internet. We had a requirement of monitoring all the HTTP traffic and we wanted to monitor complete payload even though some of the transactions were HTTPS, so I will be describing how to configure Wireshark for a scenario like this. Since I was new to Wireshark I found the initial instructions were not very clear to start using Wireshark, so that is one of another motivation for writing this blog post. This blog post will be a very quick one which doesn't contains any screenshots, I hope to write a detail one but I need to write this before I forget these steps...

My Environment is Ubuntu so when you install it using Ubuntu Software Center you will get a package which is compiled with GnuTLS so to monitor HTTPS traffic you don't have to compile Wireshark again. If you are on Non-Unix system you have to compile it with GnuTLS (there are plenty of documentation on how to do this).

I assume you have a service running on WSO2 WSAS which is secured and you have a client to invoke that Service, and you have another client invoke another unsecured Service in WSAS, with this setup you can monitor HTTP and HTTPS traffic goes to both services.

1. Start Wireshark and see whether you see set of Interfaces where you can monitor, if you do not see any interface, it's because you do not have enough rights to monitor any of the interfaces. So please start as a root/sudo mode if you did not see any interface.
2. Now go to Edit -> Preferences -> Protocols -> HTTP there you will see two entries like "TCP/Ports" and "SSL/Ports". Add your Servers (WSO2 WSAS) HTTP port in to TCP/Ports list with a comma if you have number of ports to configure in a scenario where you want to monitor traffic among few hops add them as comma separated values. (Ex: HTTP - 9763).
3. Now add your server HTTPS port in to SSL/Ports entry, by default it's having 443 to monitor browser HTTPS traffic (Ex:, HTTPS - 9443).
4. Now you are in a position to monitor the traffic goes to WSAS by applying http filter. But you will not see the encrypted messages going to the Secured Service. To decrypt the messages which goes to WSAS you need to export the private key of the server (WSO2 WSAS). To do that you can use a simple tool like Keytool IUI[2].
5. Download the keytool IUI [2], extract it and run the script (I got some errors in the tool when I was trying to run the shell script, so if you get the same just run the following command). $JAVA_HOME/bin/java -Xms128m -Xmx196m -jar rc15ktl.jar
6. This will open the Keytool and go to Export -> KeyStores Entry -> Private Key There you will see four fields to be filled. Now what we are going to do is simply get the private key of the WSO2 WSAS in unencrypted format and add that in to Wireshark which will help to decrypt the messages.
7. Give the keystore file path which is WSAS_HOME/resources/security/wso2carbon.jks, and the password will be wso2carbon, The add private Key and Certificate Chain file locations to the place where you want to store the private key. And make sure you put the file type as PEM in last two fields.
8. After pressing OK you will ask to select the select Keypair trust among a list, select the entry you have "wso2carbon" as the alias. Lets assume the output private key file path is /home/lahiru/client/fhlb/wso2is-3.0.1/resources/security/private.pem.
9. Now you have have the private key of the server in unencrypted format, now go to Edit -> Preferences -> Protocols -> SSL there you will see a field (empty) RSA Key list for that add the following entry. 127.0.0.1,9443,http,/home/lahiru/client/fhlb/wso2is-3.0.1/resources/security/private.pem ,,,private-key-path-of-the-server
10. Since we are going to monitor traffic in the same machine we are selecting the interface 127.0.0.1 (If your operation system is Windows things will not work out of the box with loop-back address so for window users please refer this[1]). To select the interface go to Capture -> Interfaces there you will see 127.0.0.1 press the start button in 127.0.0.1 interface.
11. Now you can send two messages to secure and un-secure services then after you get the responses go to Wireshark and add Filter (from the top field with the name filters) http.
12. Now ideally you should see four entries, request-response messages for two transactions and you will be able to view the content of the secure request-response messages too.

Please refer[3] for further detail about how to decrypt the messages using Wireshark.

[1]http://wiki.wireshark.org/CaptureSetup/Loopback

[2]http://www.softpedia.com/get/Security/Security-Related/KeyTool-IUI.shtml

[3]http://htluo.blogspot.com/2009/01/decrypt-https-traffic-with-wireshark.html

A Great mahindian sets down

Even though I thought of writing a blog post about Monath ayya who left us few days back I am not in a good position to have creative ideas, I have been thinking about him hundred times last few days. For me accepting he left us is really painful. I am sorry that I couldn't make it to Monath ayyas funeral because I am out of the country but no matter where we are you are always at the bottom of our hearts.

I remember the way we played cricket in Sangamitta ground, the way we went to school together long time back and the time I met you during last Mahinda Night in April you were still Monath ayya .... the humble simple man. From the school days Monath ayya was a genius who were there in our school and he was a model which we always used to talk as a senior student. He was the active guy and he was the actor and he was the best English Literature expert and He was the College western band Leader. There was a math class we all used to go (Mrs Gange) and I still remember he used to be given gifts from the teacher for getting 100 marks for the exams.

During the time we had the war we were very strong as Mahindians because we all knew that you are a hero in Air Force, we never needed any one else. We were proud about you when we heard that you were the best flying officer, we were proud about you when we see your name in the paper and we were proud to see your pictures with Kafirs but finally we were not lucky to be with you. You are always our pride Monath ayya...

Simply it was like my brother is fighting in the war and I believe it's same for any mahindian. We are always greatful to you the job you have done for sake all of us. You stayed little time with us and did amazing thing for us and you left us. Still I cannot imaging that I won't be able to see you or talk to you again... though this is quit painful I am personally trying to accept the fact that we lost you ... We all love you Monath ayya ... you are the unforgettable Mahindian in our time. We all proud of you... I am sorry again that I couldn't pay my respect to you but I will make sure I will do my job for you according to our religion....

Rest in Peace Monath ayya .... You are a great Mahindian ... you are a great Hero .... You are a great Sri Lankan you are the Humble good man we always call Monath ayya ...our pride !

Installing Apache Qpid with persistency on Enterprice SUSE Linux

This document describes how to install Apache Qpid C++ broker 0.8 with persistence on Enterprise SUSE Linux.

1. Checkout the 0.8 source code from here[1] and checkout the separate persistence module for 0.8 release from here[2]. And Download the Qpid C++ broker & client distribution from here[3].

• Note: You need to checkout the complete source of Qpid because, to configure the persistency module you need some other parts of the Qpid distribution, only C++ source will not be enough.

2. Assume QPID-DIR is the directory where you have checked out the sources.

Installing Qpid C++ broker

1. Extract qpid-cpp-0.8.tar.gz and assume Qpid C++ code will be found in $QPID-DIR/qpidc-0.8 directory.

You will see following directories inside QPID-DIR, these directories will be used in below instructions,
QPID-DIR
- qpidc-0.8 - qpid source we are going to build
- qpid - complete qpid source code to be used during building of persistence module
- cpp - persistence module source code

2. Before start building from the source, please install following required libraries using yast (Put the SUSE DVD in to your drive and install from yast). You need to have an internet connection to install some components, because some of these libraries cannot be found in the SUSE distribution DVD pack. For them please download RPMs with dependencies and install them.

1. boost (1.35)(*)
2. e2fsprogs (1.39)
3. pkgconfig (0.21)

4. openais (0.80.3)
5. corosync (1.0.0.rc1)
6. xqilla (2.0.0)
7. xerces-c (2.7.0)
8. nss
9. nspr

• Qpid has been built using the GNU C++ compiler:

10. gcc (3.4.6)
11. GNU make (3.8.0)
12. autoconf (2.61)
13. automake (1.9.6)
14. help2man (1.36.4)
15. libtool (1.5.22)
16. doxygen (1.5.1)
17. graphviz (2.12)
18. ruby 1.8 (1.8.4)
19. ruby-devel
20. python-devel
21. swig (1.3.35)

3. After successfully installing the required packages run the following commands inside $QPID-DIR/qpidc-0.8

• Note: Please make sure you are not on root during configuring and making the project, you need to be on root only when running make install command.
• ./configure --without-sasl=no --prefix=$QPID-DIR/qpidc-0.8/build
• make
• su
• make install

• Note: during the configuring and build process if you get an error telling cannot find a particular library try to find out the required package to install with the use of the error and yum search command, install them and try to build again from the beginning.

4. After successfullying running above commands you can see a directory "build" inside $QPID-DIR/qpidc-0.8. To start Qpid C++ broker run the script $QPID-DIR/qpidc-0.8/build/sbin/qpidd. (Make sure you switch back to the normal user mode from root login)

5. If you need any configuration changes, edit the qpidc.conf file inside $QPID-DIR/qpidc-0.8/build/etc.

Installing Qpid persistency module

1. Now shutdown the started server and install the following packages using yum.

• libaio-devel
• libdb-devel
• With Enterprise SUSE Linux we encountered some issues with version in-compality during installation of libdb-devel so please follow the below steps.

1. Run yast and search for the text “bdb” (berkley database) and you will see libdb-4.5 packages remove all the 4.5 libdb packages, during this process yast will show you the dependent components for these packages and ignore removing them during the un-installation process.

2. Remove if there are libraries with the name libdb_cxx-4.3 in /usr/lib/ directory for that you can simply run a find command inside /usr/lib like below
3. find -name ‘*db_cxx*’ , if there are files (.so files) remove them from /usr/lib/

4. Now download libdb-4.6-devel and libdb-4.6 rpms and install them.

5. After installing this required header files will get saved in to /usr/include/db/include/ directory copy two header files in /usr/include/db/include (db_cxx.h and db.h) to /usr/include directory.

6. Now you are ready to do the configure, make and make install follow the steps below to proceed with building the message store module.

2. After successfully installing above packages run the following commands to build the persistence module inside QPID-DIR/cpp/ to build the persistency module. (Assume with the first step you have checked out the message store module to QPID-DIR/cpp).

Note: In the very first step we have checked out the complete qpid source from [1] so we are going to use that location ($QPID-DIR/cpp) during the build of the persistence module.

./bootstrap

./configure --with-qpid-checkout=$QPID-DIR/qpid

make

su root

make install

3. Add the path $QPID-DIR/cpp/lib/.lib/ to you LD_LIBRARY_PATH.

Running Qpid C++ broker with persistency

1. Now you are ready to start Qpid C++ broker with persistency module. Run the following command to start Qpid C++ broker.

cd $QPID-DIR/qpidc-0.8/build/sbin

./qpidd --load-module msgstore.so

2. To view the message store module configurations run the qpidd like this.

• ./qpidd --load-module msgstore.so -h

3. Finalize your configuration in the broker and start Qpid broker using step 4.1 and start the Listeners first, so broker will create durable queues for all the listeners.

[1]https://svn.apache.org/repos/asf/qpid/tags/0.8/qpid/
[2]http://anonsvn.jboss.org/repos/rhmessaging/store/tags/qpid-0.8-release/cpp/
[3]http://qpid.apache.org/download.cgi

My stay at Dhamma Kuta for 11 days

I thought of writing a blog post what I have done during my year end vacation which is the best few days i have spend in my life. I am writing this post not to share an intellectual property, but if at least one person get benefit from this it will be the best thing for me. I will be keep on updating more information time to time so this post will be changed in coming days.I have applied the Vipassana 10 day meditation course (if you are a Sri Lankan when you hear this words meditation course you might get bored, but they simply use this in the western pattern so that western people will also understand things , just forget about the word...) in May and hopefully I was able to start it on 18th of December to 29th of December at Dhamma Kuta which is one of the Vipassana meditation center in Sri Lanka. For more information about the courses you can follow up them from their website[1], course schedules and how the things are rules etc. I will be posting something which does not exist in the web site.

My experience

When it comes to the experience best part will be what I have gain from doing Vipassana meditation, but it's not the correct time for this, I am continuing meditation at home and I will be posting on it in future, this part will be about my experience during the 10 days. I had a very strange, interesting and peaceful feeling during my stay at Dhamma Kuta which I haven't got in my life before. It was a very simple ten days I spend. Students are not suppose to talk to anybody (if you need something you simply have to talk to Dhamma helpers and you can talk to the teacher few word) and I consider that as one of the best experience. You will be given a simply cell which contains a bed, cloth rack and a small table. So whole few days were spend alone, simple food, and keep on meditating for more than 10 hours per day (don't get scared this can be done and thousands and thousands of people are attending these courses all over the world). So it was peaceful calm and quite 10 days in my life. But since it was away from the home and I was alone I felt like running away back home on the second day but hopefully I was strong enough to stay with a good determination.

What you should bring

I think this part will be useful if you are going for the first time, if you are a foreigner who is going to travel to Sri Lanka to attend to this course this part will be useful. This list is if you are going to Dhamma Kuta, but there will be slight changes if you are going to one of the other two centers in Sri Lanka. If you bring everything you want you will hardly speak to the management so you can keep the noble silence very effectively.

• You need to bring cloths which is enough to survive 10 days and I would say simple easy to wear cloths would be fine. Since this part of the country is pretty cold you need to bring some other stuff which protect you from the cold environment. But the management is providing blankets for you, if it's difficult to bring blankets, still you will be good.
• Slippers, bed sheet, pillow cover, torch will be very helpful.
• Soap, brush, tooth paste etc.
• I will update the list time to time when things get remind me.

This post is not completed yet.

[1]www.dhamma.org

WSO2 product cluster with registry replication in Oracle

I recently notice an obvious mistake users doing when they configure WSO2 product cluster and use WSO2 G-Reg as the central repository for the replication among the cluster nodes. You can find a guide on how to set this up in an WSO2 Oxygen Tank tutorial [1] and there we are using Oracle as the database.

If you go through this you will see we are doing two registry mountings in all the cluster nodes, so every cluster node is having two database configuration, one for the local registry database configuration and one for the mount registry database configuration (configuration of the central registry database).

When it comes to Oracle, normally users are using a same database URL and differentiate among different databases by changing the user name and password. This works without any issue, but when it comes to mounting we are referring the URL of the database configuration and we expect you will put different URL values for two database configuration (local registry and config and governance mounting). So with Oracle users will simply not aware of this and use different username password and they get following error.

java.sql.SQLException: Total number of available connections are less than the total number of committed connections

at org.wso2.carbon.registry.core.jdbc.utils.Transaction$ManagedRegistryConnection.commit(Transaction.java:474)

at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.commitTransaction(EmbeddedRegistry.java:403)

at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.resourceExists(EmbeddedRegistry.java:594)

at org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:545)

at org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent.setupMounts(RegistryCoreServiceComponent.

java:250)

So this error comes when someone try to mount from it's own database, if you use a configuration like below obviously we are throwing this exception.

<dbConfig name="wso2registry">

<url>jdbc:oracle:thin@qcdvcn1001-vip.dev.rsft.net:15000:DEV541</url>

<userName>wso2_esb</userName>

<password>wso2_esb</password>

<driverName>oracle.jdbc.driver.OracleDriver</driverName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<minIdle>5</minIdle>

</dbConfig>

<dbConfig name="esbMountRegistry">

<url>jdbc:oracle:thin@qcdvcn1001-vip.dev.rsft.net:15000:DEV541</url>

<userName>wso2_greg</userName>

<password>wso2_greg</password>

<driverName>oracle.jdbc.driver.OracleDriver</driverName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<minIdle>5</minIdle>

</dbConfig>

You can see above two URL's are same but username password are different.

So the solution for this issue is differentiate the URL by giving username in the database URL before the "@" sign. So the dbconfig elements of the cluster node registry.xml will looks like this.

<dbConfig name="wso2registry">

<url>jdbc:oracle:thin:wso2_esb@qcdvcn1001-vip.dev.rsft.net:15000:DEV541</url>

<userName>wso2_esb</userName>

<password>wso2_esb</password>

<driverName>oracle.jdbc.driver.OracleDriver</driverName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<minIdle>5</minIdle>

</dbConfig>

<dbConfig name="esbMountRegistry">

<url>jdbc:oracle:thin:wso2_greg@qcdvcn1001-vip.dev.rsft.net:15000:DEV541</url>

<userName>wso2_greg</userName>

<password>wso2_greg</password>

<driverName>oracle.jdbc.driver.OracleDriver</driverName>

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

<minIdle>5</minIdle>

</dbConfig>

[1]http://wso2.org/library/tutorials/2010/11/setup-wso2-esb-cluster-wso2-greg

How to monitor and manage Apache Qpid C++ broker

In this blog post I am describing how to setup the Qpid C++ broker monitoring. First I have to say you that to do this you need to checkout the complete qpid source code which include Java/python/C++ etc. After successfully checking out make sure you have installed python in your machine.

Go to python directory and run the build command for python like this

python setup.py build

Go to command directory inside python directory and set the environment variable PYTHONPATH to the python directory (not to command directory).

Inside the command directory you can see set of python scripts which you can used to manage running Apache Qpid C++ broker.

Successfully doing all of these you can try running all the scripts with --help option and see all the possible options. With these scripts I have done Qpid monitoring and queue create/deleting topic creation/deletion like operation and i will be writing more about these things in few more blog posts.

Apache Qpid C++ broker persistency

I have been doing some Apache Qpid C++ broker work during last few weeks and I thought I should write some important things I learned during Apache Qpid configuration process. In this blog post I will be explaining how to configure persistent with Apache Qpid C++ broker.

To add persistence to Apache Qpid you have to use another project which maintained by RedHat and you first need to check out[1] the appropriate release tag which sync with the Apache Qpid release you are using.

First you should have a working Apache Qpid C++ broker build in your machine with a full checkout of the source of the release. If you are using a release please get a check out of the appropriate tag.

Now you have two checkouts Qpid complete checkout from the tag and message-store checkout from redhat sub version repository for the appropriate Qpid release.

Now go through the README file of the message store and install required softwares to build the message store successfully by running below commands.

cd

./bootstrap

./configure --with-qpid-checkout=

make

make install

Now point your LD_LIBRARY_PATH to /lib/.libs and go to the place where you have build Apache Qpid C++ broker and run the qpidd like below.

./qpidd --load-module msgstore.so

After you start this successfully you can use persistent queues with Apache Qpid C++ broker. You can run ./qpidd --load-module msgstore.so -h to see the available options with the loaded module msgstore.so.

To use the queue persistence please create durable queues, I will be writing few more blog posts on how to manage Apache Qpid C++ broker which will describe how to monitor C++ broker and how to create queues, topic and exchanges using python scripts provided in Qpid.

[1]http://anonsvn.jboss.org/repos/rhmessaging/store/tags/