Setting up Apache Storm Topology for CEP distributed mode deployment


This post will guide you through the storm part of the configuration for the wso2 Complex Event Processor(CEP) deployment in distributed mode. Please refer: https://docs.wso2.com/display/CLUSTER44x/Clustering+CEP+4.0.0 for the CEP part of the configuration. 

  

Here I will be illustrating a basic storm setup, with three instances running as below diagram.


Their details are as below
  1. Nimbus host - 192.168.22.231
  2. Zookeeper host -192.168.22.235
  3. Supervisor host - 192.168.22.232

Configuring Zookeeper

I will be referring the location of zookeeper folder  as <Zookeeper_HOME>.

1. Download  zookeeper  from here
2. Extract the zookeeper.zip file to 192.168.22.235 instance
3. Go to <Zookeeper_HOME> directory and create a folder called 'data'.
4. Zookeeper needs a configuration file to start. You can create a file called zoo.conf inside <Zookeeper_HOME>/conf directory

tickTime=2000
dataDir=<Zookeeper_HOME>/zookeeper-3.3.6/data
clientPort=2181

5. Open a terminal and go to <Zookeeper_HOME>/bin and enter below command to start zookeeper instance
./zkServer.sh start
6. You can check whether zookeeper start properly using below command
echo ruok | nc localhost 2181
if zookeeper is running properly it will respond with "imok"

Configuring Storm Nimbus

 

I will be referring the location of nimbus folder  as <Nimbus_HOME>.

1. Download apache storm from here
2. Extract the file in to relevant instance
3. Go to <Nimbus_HOME>/conf folder and add below configuration details to storm.yaml file
  • Provide the host name of zookeeper node
storm.zookeeper.servers:
     - "192.168.22.235"
  • Configure zookeeper port (you can use the default port)
storm.zookeeper.port: 2181
  • Create a local directory called "localdir" in <Nimbus_HOME>/bin location, and give the path to the created directory as below. This localdir is used to manage the temporary files of nimbus node. 
storm.local.dir: "/home/ubuntu/storm_cluster/storm_manager/apache-storm-0.9.5/bin/localdir" 
Add below configrations to storm.yaml
nimbus.host : "192.168.22.231"
nimbus.seeds : ["localhost"]
nimbus.thrift.port: 6627
nimbus.thrift.threads: 64
nimbus.thrift.max_buffer_size: 1048576
nimbus.childopts: "-Xmx1024m"
nimbus.task.timeout.secs: 30
nimbus.supervisor.timeout.secs: 60
nimbus.monitor.freq.secs: 10
nimbus.cleanup.inbox.freq.secs: 600
nimbus.inbox.jar.expiration.secs: 3600
nimbus.code.sync.freq.secs: 300
nimbus.task.launch.secs: 120
nimbus.reassign: true
nimbus.file.copy.expiration.secs: 600
nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator"
topology.min.replication.count: 1
topology.max.replication.wait.time.sec: 60
nimbus.credential.renewers.freq.secs: 600
4. Go to <Nimbus_HOME>/bin location in terminal and type below command to start nimbus node
./storm nimbus

Configuring Storm Supervisor

 

I will be referring the location of supervisor folder  as <Supervisor_HOME>.

1. Download apache storm from here
2. Extract the file in to relevant instance
3. Go to <Supervisor_HOME>/conf folder and add below configuration details to storm.yaml file
  • Provide the host name of zookeeper node
storm.zookeeper.servers:
     - "192.168.22.235"
  • Configure zookeeper port (you can use the default port)
storm.zookeeper.port: 2181
  • Provide the host name and port of storm nimbus
nimbus.host: "192.168.22.231"

storm.nimbus.port: 6627
  • Create a local directory called "localdir" in <Supervisor_HOME>/bin location, and give the path to the created directory as below. 
storm.local.dir: "/home/ubuntu/storm_supervisor/apache-storm-0.9.5/localdir" 
  • supervisor.slots.ports – this is used to define the number of workers a user wants to create in storm supervisor. For ports are configured here, you can increase the number of ports depending requirements. (ex – if user defined 5 ports, then storm will allocate up to 5 workers to run on the machine )
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
4. Go to <Supervisor_HOME>/bin location in terminal and type below command to start supervisor node
./storm supervisor

Access Storm web interface

1. Go to nimbus host and open a new terminal
2. Go to <Nimbus_HOST>/bin and type below command to start storm UI
./storm ui
3. Open a web browser and go to "http://<nimbus_host_ip>:8080" to access the web interface

  
Now you have successfully configured the storm cluster. Then configure CEP distributed mode cluster by referring to "Distributed mode deployment" section in https://docs.wso2.com/display/CLUSTER44x/Clustering+CEP+4.0.0

Comments

  1. Nice and attractive post you shared on Apache Storm

    Thanks

    ReplyDelete
  2. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Storm , kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Sangita Mohanty
    MaxMunus
    E-mail: sangita@maxmunus.com
    Skype id: training_maxmunus
    Ph:(0) 9738075708 / 080 - 41103383
    http://www.maxmunus.com/

    ReplyDelete
  3. It's a Nice post! Thank you for sharing your knowledge to others, it was very informative and in depth one.
    Apache Storm Training in Electronic City

    ReplyDelete

Post a Comment

Popular posts from this blog

Use ZAP tool to intercept HTTP Traffic

Working with WSO2 carbon Admin Services

The 100% Open Source ESB