Saturday, 24 June 2017

How to use nested UDTs with WSO2 DSS

WSO2 Data Services Server(DSS) is a platform for integrating data stores, creating composite data views and hosting data in different sources such as REST style web resources.

This blog guides you through the process of extracting the data using a data services when nested User Defined Types (UDT) used in a function.

Lets take the following oracle package that returns a nested UDT. When a nested UDT (A UDT that use standard data types and other UDT in it) exists in the oracle package, oracle package should be written in a way that it returns a single ref cursor, as DSS do not support nested UDTs out of the box.

Lets take the following oracle package that includes a nested UDT called 'dType4'. In this example I have used Oracle DUAL Table to represent the results of multiple types included in the 'dType4'.

Sample Oracle Package


create or replace TYPE dType1 IS Object (City VARCHAR2(100 CHAR) ,Country VARCHAR2(2000 CHAR));
/
create or replace TYPE dType2 IS TABLE OF VARCHAR2(1000);
/
create or replace TYPE dType3 IS TABLE OF dType1;
/
create or replace TYPE dType4 is Object(
Region VARCHAR2(50),
CountryDetails dType3,
Currency dType2);
/

create or replace PACKAGE myPackage IS
FUNCTION getData RETURN sys_refcursor;
end myPackage;
/
create or replace PACKAGE Body myPackage as FUNCTION getData
RETURN SYS_REFCURSOR is
    tt  dType4;
    t3  dType3;
    t1  dType1;
    t11 dType1;
    t2  dType2;
    cur sys_refcursor;
  begin
    t1  := dType1('Colombo', 'Sri Lanka');
    t11 := dType1('Delihi', 'India');
    t2  := dType2('Sri Lankan Rupee', 'Indian Rupee');
    t3  := dType3(t1, t11);
    tt  := dType4('Asia continent', t3, t2);
    open cur for
      SELECT tt.Region, tt.CountryDetails, tt.Currency from dual;
    return cur;
  end;
end myPackage;
/

Lets see how we can access this Oracle package using the WSO2 Data Services Server.

Creating the Data Service

1. Download WSO2 Data Services Server
2. Start the server and go to "Create DataService" option
3. Create a data service using given sample data source.

In this data service I have created an input mapping to get the results of the oracle cursor using 'ORACLE_REF_CURSOR' sql type. The given output mapping is used to present the  results returned by the oracle package.


<data name="NestedUDT" transports="http https local">
   <config enableOData="false" id="oracleds">
      <property name="driverClassName">oracle.jdbc.driver.OracleDriver</property>
      <property name="url">jdbc:oracle:thin:@XXXX</property>
      <property name="username">XXX</property>
      <property name="password">XXX</property>
   </config>
   <query id="qDetails" useConfig="oracleds">
      <sql>{call ?:=mypackage.getData()}</sql>
      <result element="MYDetailResponse" rowName="Details" useColumnNumbers="true">
         <element column="1" name="Region" xsdType="string"/>
         <element arrayName="myarray" column="2" name="CountryDetails" xsdType="string"/>
         <element column="3" name="Currency" xsdType="string"/>
      </result>
      <param name="cur" ordinal="1" sqlType="ORACLE_REF_CURSOR" type="OUT"/>
   </query>
   <resource method="GET" path="data">
      <call-query href="qDetails"/>
   </resource>
</data>

Response of the data service invocation is as follows

<MYDetailResponse xmlns="http://ws.wso2.org/dataservice">
   <Details>
      <Region>Asia continent</Region>
      <CountryDetails>{Colombo,Sri Lanka}</CountryDetails>
      <CountryDetails>{Delihi,India}</CountryDetails>
      <Currency>Sri Lankan RupeeIndian Rupee</Currency>
   </Details>
</MYDetailResponse>


Saturday, 28 January 2017

Use ZAP tool to intercept HTTP Traffic

ZAP Tool

Zed Attack Proxy is one of the most popular security tool that used to find security vulnerabilities in applications.

This blog discuss how we can use the ZAP tool to intercept and modify the HTTP and HTTPS traffic.

Intercepting the traffic using the ZAP tool


Before we start, lets download and install the ZAP Tool.

1) Start the ZAP tool using / zap.sh

2) Configure local proxy settings
 To configure the Local Proxy settings in the ZAP tool go to Tools -> Options -> Local Proxy and provide the port to listen.


3) Configure the browser
 Now open your preferred browser and set up the proxy to listen to above configured port.

For example: If you are using FireFox browser browser proxy can be configured by navigating to "Edit -> Preferences -> Advanced -> Setting -> Manual Proxy Configuration" and providing the same port configured in the ZAP proxy


4) Recording the scenario

Open the website that you want to intercept using the browser and verify the site is listed in the site list. Now record the scenario that you want to intercept by executing the steps in your browser.


5) Intercepting the requests

Now you have the request response flow recorded in the ZAP tool. To view the request response information you have to select a request from the left side panel and get the information via the right side "Request" and "Response" tabs.

Next step is to add a break point to the request to stop it to modify the content.

Adding a Break Point

Right click on the request  that you want to add a break point, and then select "Break" to add a break point



After adding the breakpoint. Record the same scenario that you recorded above. You will notice that, when the browser reached to the intercepted request it will open up a new tab called 'Break'.

Use the "Break" tab to modify the request  headers and body. Then click the "Submit and step to next request or response" icon to submit the request.




Then ZAP will return the request to the server with the changes applied to it.

Sunday, 31 July 2016

Docker makes your life easy !!!

Most of the time we have come across situations to set up a cluster for WSO2 products. With in a product QA cycle it is a very common thing. But as you all know it consumes considerable amount of time to set up the cluster and troubleshoot.

Now, with the use of dockers we can set up a cluster within few seconds and it makes your life easy :)

So let me give a basic knowledge on what is "docker"

What is Docker


In most simplest terms, docker is a platform to containerize software images.

Install Docker  : https://docs.docker.com/engine/installation/linux/ubuntulinux/

What is Docker Compose


Docker compose is used to compose several applications and run those using one single command to initialize in multiple containers.

Install Docker Compose : https://docs.docker.com/compose/install/

For some of the wso2 products there are docker compose images already exists in a private repository.

Main purpose of this blog is to highlight some of the useful docker commands you have to use while working with docker compose images.

To explain some of the usages I will be using ESB 4.9.0 docker compose image.
You can get a clone of the git repository where the docker compose image for ESB 4.9.0 is available. Follow the instructions in the READ.ME to setup the ESB docker cluster.

Start Docker container

docker-compose up

Build the changes and up the docker compose image

docker-compose up --build

Stop docker containers

docker-compose down 

Start docker in demon mode

 Docker-compose up -d

List docker images

 docker images 

List running docker containers

 docker ps 

Login to an active container

docker exec -i -t <container_id> /bin/bash 

Delete/Kill existing containers

 docker rm -f $(docker ps -aq) 

View container logs 

 docker logs <container_id> 

Insert a delay between docker containers

Sample Scenario: When running ESB cluster, first we want to ensure that DB is up and running, Therefore we can introduce a delay and start the ESB nodes. To configure this, you can add below property to the docker-compose.yml file
 environment:
      - SLEEP=50

Add additional host names

Sample Scenario: Lets assume you want to use a backend service hosted in Application Server in another instance. Host name of the Application Server is "as.wso2.org". Docker can not resolve the host name unless you defined the host name in docker-compose.yml file as below.
 extra_hosts:
      - "as.wso2.org:192.168.48.131"

Enable additional ports

Sample Scenario: Each of the ports used for the docker compose should be exposed through the docker-compose.yml file. If you are using inbound HTTP endpoint with port 7373, this port should be exposed as below.
   ports:
      - "443:443"
      - "80:80"
      - "7373:7373"

Friday, 17 June 2016

Configuring an email notification to define user password

This blog will discuss how to configure an auto generate email to define the password when creating a user via management console. In WSO2 Identity Server there is an inbuilt feature called 'Ask Password' to fulfill this requirement. Lets look at how to implement this in other wso2 products.

'Ask Password' is a feature that comes with wso2 Identity Server. The purpose of this feature is to allow the users to decide there own password rather than defining a password by the server administration and allow the user to change the defined password.

So let me move on to the purpose of writing this blog.

While I was working with WSO2 API Manager, I got a requirement that the APIM administrator wants to create users via APIM management console, but the administrator wants to allow the users to define a password by the user itself. This requirement can be fulfilled using the 'Ask Password' feature available in wso2 Identity Server.

Scenario

APIM Administrator creates a user by providing a username and a user email through the management console. Then an email will be sent to the defined email address with a redirection URL to define a password for the user account.

I will use APIM 1.10.0 product to explain this.


Steps to configure 'Ask Password' feature in APIM 1.10.0


1. Download APIM server

2. Log in to APIM server as the administrator

When you go to 'Add User' option you can see a window like below.



Now lets look at how to configure auto-email to set user password.

3. Install 'Account Recovery and Credential Management' feature in APIM

Due to some of the limitations in identify server feature activation, you have to install 'Account Recovery and Credential Management' feature in APIM 1.10.0. ( Steps to install a feature in wso2 product can be found from [1]).

4. As the next step, do the configuration changes mentioned here in APIM server.

These configurations are required to enable 'Ask Password' feature.

5. Restart the server after above changes.

When you navigate to 'Add User' option you can see that 'Ask password' feature is installed in UI as below.



6. Now create a user from APIM management console by defining the user email address.

You can verify whether auto generate email is received to the defined user email address and the user can define a password through the redirection screen provided in the auto-generated mail. Then check whether the user can successfully log in to APIM server.



Now APIM administrator can add the users via management console and allow the users to define a password they prefer.





[1] https://docs.wso2.com/display/Carbon440/Installing+Features+via+the+UI

Tuesday, 14 June 2016

Encrypting sensitive information in configuration files


Encrypting information 

I thought to start from basics before dig in to the target topic. So lets look at what is "encrypting".

Encrypting information is converting information in to another format, which is hard to understood. As we all know encrypting information is really useful to secure sensitive data.

In wso2 products, there is an inbuilt 'Secure Vault' implementation to encrypt plain text information in the configuration files to provide more security.


In this post I will not discuss about the secure vault implementation in details. You can refer 'secure vault implementation' to get more insight about it.
In wso2 products based on carbon 4.4.0 or later visions, 'Ciper Tool' feature is installed by default, therefore you can easily use that to encrypt sensitive information in the configuration file. 

Lets move on to the main purpose of this blog.

We already know that we can use ciper tool encrypt the information in configuration files. But can we encrypt the sensitive information in properties files or .json files ??

How to encrypt information when we can't use xpath notation?


Using the ciper tool we can encrypt any information if we can specify the xpath location of the property correctly. So basically if xpath notation can be defined for a certain property we can encrypt that using the ciper tool without much effort. Detailed steps to encrypt information based on an xpath can be found from here.

But in the properties file or .json files we can not define a xpath. Now you might be thinking how can we encrypt the information in these files !!!

To overcome this, we can manually encrypt the sensitive information using the ciper tool. You can refer the detailed steps provided here to manually encrypt the sensitive information in properties file and .json files.

However, I want to point you out to a very important fact. When you encrypt a sensitive information in a properties file or .json file, the product component which reading the encrypted property should have written in a way to call the secure vault to decrypt the value correctly.



Thursday, 21 April 2016

Information filtering using grep commands

While I was working on monitoring the long running test issues, I thought it would be useful to write a post on the usage of 'grep' commands in Linux.

In this article I will be discussing few real examples of using "grep" commands and how to execute grep commands as a shell script.

Search for a given string 

This command is use to search for specific string in a given file.
grep "<Search String>" <File Name>

Ex: In the below example, it will search for the string "HazelcastException" within wso2carbon.log file.
grep "HazelcastException" wso2carbon.log 

Search for a given string and write the results to a text file

This command is use to search for a given string and write the search results to a text file.
grep "<Search String>" <File Name> > <Text File Name>

Ex: In the below example, it will search for the string "HazelcastException" within wso2carbon.log file and write the search results to "hazelcastexception.txt" file.
grep "HazelcastException" wso2carbon.log > hazelcastexceptions.txt

Execute grep commands as a shell script

In some situations it will be useful to execute grep commands as a shell script.
Ex: While I was monitoring the long running test for certain exceptions, I used to search all the target exceptions from wso2carbon.log files and write those to specific files for further reference.

Follow below steps to execute multiple search strings and write those to text files using a shell script.

1) Create a file and add the grep commands to that file as given below and save it as a shell script. (Here I will name this file as "hazelcastIssues.sh")

#!/bin/bash
grep "HazelcastException" wso2carbon.log* > hazelcastexception.txt
grep "Hazelcast instance is not active" wso2carbon.log* > hazelcastnotactive.txt

2) Now add the shell script to the <Product_HOME>/<repository>/<logs> folder

3) Execute the script file using below command
./hazelcastIssues.sh
After you execute the shell script, it will grep all wso2carbon.log files for the given search string and write those to separate text files.


Sunday, 10 January 2016

Allowing empty charachters using regular expression

This post will guide you to configure regular expression, to allow empty characters (spaces) for properties like user name and role name.

Validations for User Name, Role Name and Password are done using the regular expressions provided in <Product_Home>/repository/conf/user-mgt.xml file.

I will be taking EMM product as the example. By default empty characters are not allowed for role names in management console. If you enter a role with empty character (ex: Device Manager) you will get a message as in below image.

https://picasaweb.google.com/lh/photo/eNmEGi4R214dCwa0zZk09I3xd-zHfRG5vyi8Cg2gSIE?feat=directlink

 Follow below steps to allow empty characters for role name.

1. Go to <EMM_HOME>/repository/conf/user-mgt.xml file and open the file. Then change <RolenameJavaRegEx> property and <RolenameJavaScriptRegEx> proerty as given below
Property name="RolenameJavaRegEx">[a-zA-Z0-9\s._-|//]{3,30}$</Property>

Property name="RolenameJavaScriptRegEx">^\w+( \w+)*$</Property>

Note -
  • <RolenameJavaScriptRegEx> is used by the front-end componenet for role name validation
  •  <RolenameJavaScriptRegEx> is used for back-end validation

2. Then restart the server

Now you will be able to add role names with empty spaces (ex: Device Manager).