Pages - Menu

Weblogic Server - Coherence Configuration Example

Friday, July 26, 2013

Weblogic Server - Coherence Configuration

For theoretical knowledge on coherence read my this post - http://weblogicserveradministration.blogspot.com/2011/05/oracle-coherence-37-coherenceweb.html

In this post I am going to cover practical example on Coherence


I have used Weblogic Server 10.3.5 + Coherence 3.6

High Level Steps


1.  Install weblogic server 10.3.5 with coherence 3.6 or 3.7

2.  Create a domain
3.  Create a machine, start node manager and check status from console if it's accessible
4.  Create a a simple cluster with two managed servers, and assign both managed servers to 
      machine
5.  Create a Coherence cluster and assign your above defined cluster to coherence cluster
      [ OR instead of above step2 and step3, you can just create two managed servers, then  
         create a coherence cluster and assign both managed servers to this coherence cluster ]
6.  Start a cache server
7.  Start both managed servers
8.  Deploy shared library files
9.  Create a testing application and deploy on cluster
10.Test 

Steps


1. For installation see my other post 
http://learn-weblogic.blogspot.in/2013/07/weblogic-server-installation-console.html

2. For domain creation see my other post

http://learn-weblogic.blogspot.in/2013/07/create-domain-in-gui-mode.html
http://learn-weblogic.blogspot.in/2013/07/weblogic-domain-creation-in-console.html
http://learn-weblogic.blogspot.in/2013/07/domain-creation-in-console-mode-on.html

3. For machine,node manager configuration see my other post
http://learn-weblogic.blogspot.in/2013/07/configure-start-stop-node-manager-step.html


4. To create a simple cluster and assign servers to cluster see my other post
http://learn-weblogic.blogspot.in/2013/07/vertical-clustering-step-by-step.html

5. Create a coherence cluster and assign your above created cluster to this coherence cluster

a. Login to admin console
b. Click on Coherence Clusters option on left hand side panel
c. Do the configuration like below and save

Unicast Listen Address - your machine hostname or i.p
Unicast Listen Post - 8085 ( or any other available port of your machine )
Multicat Listen Address - 231.1.1.1 ( or any multicast range available i.p but make sure this need to update in coherence server startup script which i will explain later )
Multicast Listen Port - 7777 ( or you can use any other available port of your machine but make sure this need to update in coherence server startup script which i will explain later )




After creating coherence cluster, click on Target tab and assign your previously create simple cluster to this coherence cluster

Now you can see like below under coherence cluster option



6. Start Cohrence Server

Copy the below contents, paste in a text file and rename it as .bat ( on windows )
( if you are on unix based systems then change scripts variable settings, path etc accordingly )

Start-Coherence.bat

setlocal

set COHERENCE_HOME=C:\Oracle\Middleware2\coherence_3.6
set COH_OPTS=%COH_OPTS% -server -cp %COHERENCE_HOME%\lib\coherence.jar;%COHERENCE_HOME%\lib\coherence-web-spi.war;
set COH_OPTS=%COH_OPTS% -Dtangosol.coherence.management.remote=true -Dtangosol.coherence.cacheconfig=/WEB-INF/classes/session-cache-config.xml -Dtangosol.coherence.distributed.localstorage=true -Dtangosol.coherence.clusterport=7777 -Dtangosol.coherence.clusteraddress=231.1.1.1 -Dtangosol.coherence.session.localstorage=true
java %COH_OPTS% -Xms256m -Xmx256m -XX:MaxPermSize=256m com.tangosol.net.DefaultCacheServer
:exit

Now start above create script

Output should be like below



7. Start both managed servers from console


8.  Deploy below shared libraries on your cluster ( During deployment make sure to choose 
     option deploy this file as library not application for both files )
coherence-web-spi.war from  ORACLE_HOME\coherence_3.6\lib and  active-cache-1.0.jar  
  from  ORACLE_HOME\wlserver_10.3\common\deployable-libraries

9.  Create a testing application like below

[1] Create a folder CoherenceTest
[2] Inside CoherenceTest folder create 
      a. META-INF folder, Inside it a file with name manifest.mf with below contents


Extension-List: active-cache
active-cache-Extension-Name: active-cache
active-cache-Specification-Version: 1.0
active-cache-Implementation-Version: 1.0


 [3] WEB-INF folder
        
        3.1 inside it, create a file web.xml with below contents

               <?xml version = '1.0' encoding = 'windows-1252'?>
               <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web- 
               app_2_4.xsd" 
               xmlns="http://java.sun.com/xml/ns/j2ee" version="2.5">
              <description>Empty web.xml file for Web Application</description>       
             </web-app>
        
         3.2 inside it, create a file weblogic.xml with below contents

         <weblogic-web-app xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app"  
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
         xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app 
         http://www.oracle.com/technology/weblogic/weblogic-web-app/1.1/weblogic-web-app.xsd">
         <library-ref>
           <library-name>coherence-web-spi</library-name>
         </library-ref>
         <coherence-cluster-ref>
            <coherence-cluster-name>CoherenceCluster</coherence-cluster-name>
         </coherence-cluster-ref> 
           </weblogic-web-app>

( Make sure here cluster name exactly the same you have given to your coherence cluster )

        3.3 inside it, create a folder with name lib and copy coherence.jar file from 
               COHERENCE_HOME\lib to here ( ORACLE_HOME\coherence_3.6\lib in my case )
        
 [4]. A jsp file with name counter.jsp ( copy and paste below code inside it )


<html>
    <body>
<h3>
      Counter :
      <%
         Integer counter = new Integer(1);
         HttpSession httpsession = request.getSession(true);
         if (httpsession.isNew()) {
                httpsession.setAttribute("count", counter);
                out.println(counter);
         } else {
                int count = ((Integer) httpsession.getAttribute("count")).intValue();
                httpsession.setAttribute("count", new Integer(++count));
                out.println(count);
         }
      %>
      </h3>

    </body>
</html>


Below is the structure


10. Deploy this CoherenceTest application as in exploded form on your cluster

11. Test

Now, first access you application using first managed server URL

http://host:first_managed_server_port/CoherenceTest/counter.jsp

it will show you as counter : 1


After that access your application using second managed server URL

http://host:second_managed_server_port/CoherenceTest/counter.jsp

it will show you counter : 2



So, here session shared on your external coherence server and shared across servers on your cluster..

you are done...cheers!!!!!!!

Rotation of .out log files created by node manager

Unfortunately there is no way provided with node manager or weblogic to rotate your .out log files created by node manager , you have to manually cleanup them or you can use  any of the below method to auto cleanup the same.

1st Method
----------
-----

You can redirect .out logs to the .log file and managed .log via admin console.
[ console > your_server > under logging tab ]

Fot this - 

Go to to YOUR_DOMAIN/bin/nodemanager
Take a backup of your existing wlscontrol.sh
Edit it and change OutFile=$ServerDir/logs/$ServerName.out to OutFile=$ServerDir/logs/$ServerName.log




2nd Method
-----------
------

on Linux, You can change OS parameters in /etc/logrotate.conf file 

Append below lines at the end

<out_log_file_dir_path>/*.out
{
copytruncate
rotate 4
size=500Mb
}

OR

Use command "logrotate" using cron to achieve, You can use it with copytruncate to rotate logs.


3rd Method
-----------
-----

Add option -Dweblogic.log.RedirectStdoutToServerLogEnabled=true to redirect .out logs to .log log file and manage it via admin console

[ console > your_server > under logging tab ]

Weblogic Enroll Domain Node Manager : Weblogic Machine Domain Enrollment With nmEnroll()

Weblogic Machine OR Domain enrollment with nmEnroll()

Weblogic Enroll Domain
Weblogic Enroll Node Manager

There are lots of WLST command though I feel it's very important and saw lot's of engineers struggling with this So, though to publish a separate post for this.

nmEnroll()

NmEnroll is used to enroll a machine or you can say to enroll a domain with the node manager because if you have multiple domains ( of different installer ) running on a host then you can handle all with a single node manager but you have have to enroll each domain with the same node manager.

This is applicable if you have domains from different oracle_home or you can say from different installer on same host ( means you have more than one installation of weblogic on same host with each installer having its different host ).

If you are creating multiple domains from single installer then no need to enroll because it will enroll automatically. 


So to enroll a machine or domain with a particular node manager, follow below steps

1. Start Admin server of domain which you want to enroll with node manager

2. Go to wlst prompt ( go to your_domain, run setDomainEnv and then "java weblogic.WLST")

3. Connect with admin server
    connect("user","password","host:admin_port")

4. Run nmEnroll command
    nmEnroll('C:/Oracle/Middleware/wlserver_12.1/common/nodemanager')

   Where "C:/Oracle/Middleware/wlserver_12.1/common/nodemanager" is the node manager  
   home of node manager with which you want to enroll your domain

5. Update nodemanager.domains under node manager home and add your domain path in    
    same format as earlier domains path there, like below i have added domain in red

     JJTU_Domain1=C\:\\Oracle\\Middleware2\\user_projects\\domains\\JJTU_Domain1
     base_domain=C\:\\Oracle\\Middleware2\\user_projects\\domains\\base_domain   

6. Update node manager password ( username also if not correct there ) from admin console
    ( login to admin console > click on your domain name on left hand side > click on security tab
     on right hand side > click on Advance link > update node manager password ( username
     also if not correct there )


Now you can able to connect with node manager


 wls:/base_domain/serverConfig> nmConnect("weblogic","weblogic123","localhost","40046","base_domain","C:/Oracle/Middleware2/user_projects/domains/base_domain","plain")
Connecting to Node Manager ...
Successfully Connected to Node Manager.

Note 1 - 

If you are getting below error during connect with node manager, then it means you haven't changed username,password for nodemanager from admin console ( above step 6 )

WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Access to domain 'base_domain' for user 'weblogic' denied

Note 2 - 

If you are getting below error during connect of node manager then it mean you haven't followed above defined steps OR haven't added your domain on nodemanager.domains


WLSTException: Error occured while performing nmConnect : Cannot connect to Node Manager. : Configuration error while reading domain directory

SSL Exceptions in Admin Server and Node Manager

SSL EXCEPTIONS IN ADMIN SERVER AND NODE MANAGER.

javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from oracle.test.com – xx.xxx.xx.xx. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.
The above exception is one of the most common exception encountered during the setup of Weblogic Server in an environment. The stack does suggest what could be the reasons but the diagnostics are not mentioned.
To debug this issue -

 First we need to check the certificates used by Admin Server and the Node Manager.

1) If we have Admin and the Node Manager using demo certificates -

1->Then the issue can be due to improper DNS mapping. We can use the nslookup to check the DNS entry.
2-> For testing purpose we can provide the ip address as the listen address for the admin server and the node
       manager and see if the issue is still occurring.
3-> Also we will have to turn of host name verification and the basic validation check of the certificates. We
      can do it by specifying the following flag in startWeblogic.sh
     -Dssl.debug=true -Dweblogic.security.SSL.ignoreHostnameVerification=true  \
     -Dweblogic.security.SSL.enforceConstraints=off
       And the following flag in startNodeManager.sh
      -Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false \
      -Dweblogic.security.SSL.enforceConstraints=off

2) If the Admin Server is using Custom Identity and Custom trust, then its better to configure the node manger with custom identity and custom trust as well.
By default the Node Manager is configured with Demo Identity and Demo Trust. To change it to custom identity and custom trust, we need to specify the following values in the nodemanager.properties file present in nodemanager home
Keystores=CustomIdentityandCustomTrust
CustomIdentityAlias=
CustomIdentityKeyStoreFileName=
CustomIdentityKeyStorePassPhrase = xxxxxx
CustomIdentityKeyStoreType = JKS
CustomIdentityPrivateKeyPassPhrase = xxxxxxx
Apply the same flags as above in the startup script of Admin Server and Node Manger.
Check from the console whether Node Manager is reachable or not.
3) Another option can be to use PLAIN communication between Admin Server and Node Manager.
We can change the Listen Type to PLAIN for the Node Manager from the console and set the secureListener=false in the nodemanager.properties file present in nodemanager home.

Weblogic Verify Node Manager Status From Admin Console If It's Reachable For Weblogic Or Not

Weblogic Verify Node Manager Status From Admin Console If It's Reachable For Weblogic Or Not

How to verify configured Node manager status from console

1. Login to admin console

2. On left hand side, select Machines

3. Select your machine on right hand side

4. Select Monitoring -> Node manager status tab on right hand side and see if its Reachable or Inactive



Note :- If you see status as Inactive it doesn't mean your node manager is not running, there may be your configuration issues like you have configured wrong port ot type ( ssl, plain etc ) from weblogic console.

Node Manager : Configurations and Important considerations

Node Manager : Configurations and Important considerations

 Important Note :

I have noticed most of the time a very common mistake done by most of the engineers is, If you are changing node manager port from Weblogic console that doesn’t mean you are changing the port of node manager. Node manager run on a separate port defined in node manager home nodemanager.properties file and if you are changing port from admin console then it means you are telling to Weblogic that ok my node manager is listening on this port.

How to change node manager port ?

First time when you start the node manager it configured and start on port 5556. Later you can change the port in nodemanager.properties file under node manage home directory.
If you have installed Weblogic under /oracle/middleware and wlserver_103. Is your WL_HOME product directory under this then your node manager home is
Node Manager Home = /oracle/middleware/wlserver_10.3/common/nodemanager

Important note about nodemanager.properties file –

1)      Nodemanager.properties file

You will not able to see this file under nodemanager home until you will start node manager at least one time after fresh installation, or you can say nodemanager.properties created after first start of your node manager.
After first start of  node manager, go to node manager home and update the nodemanager.proerties for  ListenPort parameter.
After change node manager port in nodemanager.properties file, make sure to change the port from admin console also ( “admin console > machine > select your node manager > configuration > node manager” ), otherwise you will see node manager status as inactive from console like below 



2)   javax.net.ssl.SSLHandshakeException status from console or in log node manager file

During first fresh installation of Weblogic, node manager is configured for secure ssl mode and when you will start the node manager first time it will start listening on secure connection, and also the node manager type is set as ssl.

 if you will see from admin console

In nodemanager.properties file you will see   -

SecureListener= true

And from admin console you will see node manager status as below  with SSLHandshakeException  error even after starting node manager


So, after fresh Weblogic installation and after starting node manager first time, make sure to change port in nodemanager.properties file as well as from admin console ( if you really want your nodemanager to run on other port apart from default 5556 ), change SecureListener = false in noemanager.properties file and change Type as “Plain” from tha admin cosole ( like screenshot below ) , restart node manager and check status from console and now you will find it in reachable state.



How to Start and Stop Node Manager

To Start Node Manager

Go to your nodemanager bin directory

 $ cd $NM_BIN\server\bin where NM_BIN =$WL_HOME\server\bin

Where WL_HOME =  /Oracle/Middleware/wlserver_10.3

Where /Oracle/Middleware/  is your Weblogic installation directory

Note :   Make sure to replace this installation directory as well as main product directory ( wlserver_10.3   here ) according  to your environment and Weblogic version

./ startNodeManager.sh  ( on unix/linux)

startNodeManager.cmd ( on window )

To Stop Node Manager

There is no specific command to  stop node manager, you have to kill the process manually to stop the node manager.

On window

You can go to “services” option from  “control panel > administrative tools” and stop the node manager process by right click on it or identify the process id from task manager by enabling PID option under view tab and use window process kill utility to kill the process.

On Linux/Unix

Identify the process of node manager and kill that.

ps -ef|grep -i nodemanager.javahome|grep -v grep

You will get the outout like below

mcnegi  9796  9773  0 Aug25 ?        00:40:53 /weblogic/appsvr/product/wls_103/jrockit_160_05/jre/bin/java -jrockit -Xms32m -Xmx200m -Xverify:none -Xverify:none -Djava.security.policy=/weblogic/appsvr/product/wls_103/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.nodemanager.javaHome=/weblogic/appsvr/product/wls_103/jrockit_160_05/jre weblogic.NodeManager -v

kill -9 9796

OR

you can use below single command to kill node manager

kill -9 `ps -ef | grep -i nodemanager.javahome | grep -v grep | awk {'print $2'} | head -1`

Tag : Command to kill node manager, kill node manager, stop node manager, stop Weblogic node manager
Note:   This command will not work if you have more than one node manager running on same machine, you have to modify command further to some more strings to grep like path location.

 Node Manager  Status

You can check status of node manager either from admin console or you can grep the process for the same
1)      To check status from console, login to admin console > machine > select your machine > monitoring tab, make sure status should be “Reachable”

2)      Grep the process like below
ps -ef|grep -i nodemanager.javahome|grep -v grep


Tags : node manager, Weblogic node manager, sslhandshake exception, node manager inactive state, node manager error, start stop node manager, kill node manager, grep node manager process

Node Manager Concept, Important Files

Server instances in a WebLogic Server production environment are often distributed across multiple domains, machines, and geographic locations. Node Manager is a WebLogic Server utility that enables you to start, shut down, and restart Administration Server and Managed Server instances from a remote location
A Node Manager process is not associated with a specific WebLogic domain but with a machine
You can use the same Node Manager process to control server instances in any WebLogic Server domain, as long as the server instances reside on the same machine as the Node Manager process. 
Node Manager must run on each computer that hosts WebLogic Server instances whether Administration Server or Managed Server that you want to control with Node Manager.

WebLogic Server provides two versions of Node Manager, Java-based and script-based, with similar functionality. However, each version has different configuration and security considerations.

Java-based Node Manager
  • Java-based Node Manager runs within a Java Virtual Machine (JVM) process. 
  • This version of Node Manager determines its configuration from the nodemanager.properties file.
  • Java-based Node Manager provides more security than the script-based version.


Script-based Node Manager

  • For UNIX and Linux systems, WebLogic Server provides a script-based version of Node Manager. 
  • This script is based on UNIX shell scripts, but uses SSH for increased security. SSH uses user-id based security
  • This version does not provide as much security as the Java-based version. 
  • The advantage of the script-based Node Manager is that it can remotely manage servers over a network that has been configured to use SSH. No additional server installation is required. The scripts merely have to be copied to the remote machine.

What You Can Do with Node Manager
  • Start, Shut Down, and Restart an Administration Server
  • Start, Shut Down, Suspend, and Restart Managed Servers
  • Monitor Servers and View Log Data

Starting Java-based Node Manager
Start scripts for Node Manager are installed in the WL_HOME\server\bin directory, where WL_HOME is the top-level installation directory for WebLogic Server. 
Use startNodeManager.cmd on Windows systems and startNodeManager.sh on UNIX systems.
The scripts set the required environment variables and start Node Manager in WL_HOME/common/nodemanager. Node Manager uses this directory as a working directory for output and log files. 

Important Node Manager Configuration Files
nodemanager.properties
This is the configuration file used by the Java-based version of Node Manager.
This file is located in WL_HOME/common/nodemanager.
nodemanager.hosts
This file contains a list of all the trusted hosts that can issue commands to Node Manager.
This file is located in WL_HOME/common/nodemanager.
nodemanager.domains
This file contains mappings between the names of domains managed by Node Manager and their corresponding directories. 
This file is located in WL_HOME/common/nodemanager.
nm_data.properties
This file stores the encryption data the Node Manager uses a symmetric encryption key. The data is stored in encrypted form.
This file is located in WL_HOME/common/nodemanager.
nm_password.properties
This file stores a username/password pair specific to the Node Manager server that is managing this domain. This is known as the Node Manager secret. The username and password are appended to a salt value (obtained from the SerializedSystemIni.dat of the domain) and SHA-hashed.
This file is located in DOMAIN_HOME/config/nodemanager.
boot.properties
Node Manager uses this file to specify a boot identity when starting a server. This file is located in domain-name/servers/server_name/data/nodemanager.
startup.properties
Each Managed Server instance has its own startup.properties file with properties that control how Node Manager starts up and controls the server. Node Manager automatically creates this file by using properties passed to Node Manager when the Administrative Server was last used to start the server. This allows a Node Manager client or startup scripts to restart a Managed Server using the same properties last used by the Administrative Server.
These properties correspond to the server startup attributes contained in ServerStartMBean and the health monitoring attributes in ServerStartMBean.
This file is located in domain-name/servers/server_name/data/nodemanager.
server_name.addr
server_name.addr stores the IP address added when a server starts or is migrated. This file is generated after the server IP address is successfully brought online during migration. server_name.addr is deleted when the IP address is brought offline. The server IP address is used to validate remove requests to prevent addresses being erroneously removed while shutting down the server.
This file is located in domain-name/servers/server_name/data/nodemanager.
server_name.lck
server_name.lck is generated by each server and contains an internally used lock ID.
This file is located in domain-name/servers/server_name/data/nodemanager
server_name.pid
server_name.pid is generated by each server and contains the process ID of the server. Node Manager checks the process ID generated by the server during crash recovery.
This file is located in domain-name/servers/server_name/data/nodemanager
server_name.state
server_name.state is generated by the server and contains the server's current state. Node Manager monitors the contents of this file to determine the current state of the server.
Note: Do not delete or alter this file. Without this file Node Manager cannot determine the current state of the server.
This file is located in domain-name/servers/server_name/data/nodemanager.
 

Blogger news

Blogroll

Most Reading