Load Balancing Hardware Evaluation

I was recently tasked with evaluating Content Sensitive Switching/Load Balancing solution for a client’s PeopleSoft infrastructure. The client is upgrading its current Peopletools 8.3 to 8.9 (HCM). As part of the upgrade, there has been expressed desire by management to build a fault-tolerant architecture. The growing need to increase availability requires building redundancy across every tier of the architecture. Deploying a solution that meets that goal requires a good understanding of Load balancing architecture and algorithms, and most importantly PeopleSoft Weblogic hardware requirements for load balancing. The dynamics of Load balancing a.k.a Content Sensitive Switching a.k.a Application Switching is to present a Virtual IP (VIP) address to end-users that send incoming requests to one or more back-end web servers. Traffics are then re-routed by a pre-defined algorithm (round robin) to these back-end web servers.

In addition to routing traffic, load balancing tools are typically required to perform several functions related to the routing decision -

• Monitor server loads and distribute incoming requests to balance the load across servers
• Examine the client requests to determine which server is appropriate to handle the request
• Identify the client to maintain session affinity with a particular server for e-business applications
• Maintain session persistence to enable failover to a hot standby to improve availabilityDetection and avoidance of many common denial-of-service attacks
• SSL acceleration to improve the performance of secure applications
• Simplified configuration and management
• Health Monitoring

Weblogic Load Balancer Requirements

Weblogic requires that load balancing hardware support compatible active or passive cookie persistence and SSL persistence mechanism.
I. Passive Cookie Persistence
Passive cookie persistence enables Weblogic Server to write a cookie containing a session parameter information through the load balancer to the client.
II. Active Cookie Persistence
Active cookie persistence enable Weblogic server to write a cookie containing a session parameter information through the load balancer to the client provided the load balancer does not modify Weblogic Server cookie. Load balancers that alter active cookie sessions are not supported by Weblogic.
III. SSL Persistence
When SSL is used, the load balancer performs all encryption and decryption of data between clients and the Weblogic server cluster. The load balancer then uses the plain text cookie that Weblogic Server inserts on the client to maintain an association between the client and a particular server in the cluster.


Stateless Vs Stateful Failover

Another thing that played a huge role in our decision making was the Stateful Vs Stateless failover issues. How much data loss are we able to handle if a server fails ?. Which vendor hardware provide these kind of failover solution. Invariably in our case, transaction loss was not an option.
I. Stateless Failover
Under stateless failover, if a server fails user session shall be restarted with another functioning server. All unsaved data on the failed server shall be lost but the user login will be preserved.
II. Stateful Failover
With stateful failover, user session are maintained on another machine in cluster. Weblogic Server creates a primary session state on the server to which the client first connects, and a secondary replica on another Weblogic Server instance in the cluster. The replica is kept up-to-date so that it may be used if the server that hosts the servlet fails. Stateful failover enables session resiliency during critical failure.

With these in mind, I set out to review a number of load balancing hardware offerings in the market that best fit our requirements. The following vendor offerings were reviewed – Zeus Traffic Manager (ZXTM), NetScaler, Foundry ServerIron and BigIP (f5). While most the aforementioned products offers the basic load balancing requirements – Content Sensitive Switching, Advanced Traffic Routing, Session persistence and so on. BigIP (f5) stood from the pack – Read Ben Rockwood’s blog entry on NetScaler Vs f5. Having the largest market share in the load balancing hardware market, BigIP (f5) was a familiar ground for me. I’ve worked with it in the past and it worked well. My preference for BigIP (f5) were in the following space –

Scalability - BigIP provides the most scalable hardware solution in the market. If one service is nearing capacity, scaling it is as simple as adding another instance of the service to your network and then to the BigIP load balancing pool without disruption of service. This seamless scalability feature gives it an edge over its competitor.
SSL Acceleration and Consolidation – With BigIP, you can handle SSL decryption on the load balancing hardware thereby eliminating those CPU-intensive activities at the server level. Another thing I like about BigIP is the ability to consolidate SSL certificates thereby saving money per certificate. Instead of having SSL certificates on a per server basis, you can now having 1 certificate on your load balancing hardware that handles all secured service requests

In conclusion, we went with BigIP because of its ability to scale easily and it's simplified configuration/management.

Tuxedo Administrative Console (Web GUI)

Without much ado, I'll kick this blog off with configuring and administering the Tuxedo Administrative Console a.k.a Tuxedo Web GUI. A lot of PeopleSoft Administrators aren't aware of the full capability of the Tuxedo Web GUI, hence they don't take advantage of its functionalities. What I like most about this utility is the ability to make dynamic changes to PSTUXCFG file, which doesn't exist in PSAPPSRV.CFG file. In addition, logs are more intuitive and laid out better than on the host server. I use it mostly when we are running a Performance load test or during peak performance period.

The Tuxedo Administrative Console has evolved over the years. Since its inception in Tuxedo 5.1, a lot has been added to enhance the functionality and looks. Back then, the GUI can only run on Windows. From versions 6.1 - 8.1, the GUI can be configured on both Unix and Windows platforms. I recently configured the GUI for one of my client running Tuxedo 8.1 SP3 and here is primer for those of you interested.


Tuxedo Web GUI consists of the following:

• A simple web server, HTML files and Java applets
• A Listener process for Applets to connect back to
• Configuration Files
  
  The Web server is a Tuxedo Supplied web server called “tuxwsvr”. It is not a commercial grade web server as it can only handle up to twenty (20) simultaneous connections. The web server configuration file is located on the $TUXDIR/udataobj/tuxwsvr.ini. The primary purpose of the web server is to obtain information from the Tuxedo BBL.
  
  The Listener is also a Tuxedo process that listens for the web server request at a well-known port. The listener process is called “wlisten” and it enables communication between the web server “tuxwsvr” and Tuxedo processes. The wlisten configuration file is also located on $TUXDIR/udataobj/webgui/webgui.ini.
  
  Connect Procedure
  
  1. Browser contact the web server at a well-known address via an http string
  2. Web server contacts the wlisten process at a well-known address established via the configuration files
  3. Wlisten contact/spawns handlers to deal with client
  4. GUI begins to communicate via applets
  
  Starting Tuxedo Admin Console
  
  To start Tuxedo Web GUI, you need to start the web server (tuxwsvr) and the listener process (wlisten) via commandine. Here’s how you start the GUI :
  
• psoft@hrm33: /opt/JDC/ps/hms820/devel $ wlisten –I /$TUXIDIR/udataobj/webgui/webgui.ini
• psoft@hrm33: /opt/JD/ps/hms820/devel $ tuxwsvr –l //: -I /$TUXDIR/udataobj/tuxwsvr.ini -L /$PS_HOME/appsvr/demo/LOGS (where l =network address of the web server; I = name/location of the initialization file; L= Name and location of logfile for web requests and error)

To verify the processes are running , grep for “wlisten” and/or “tuxwsvr” i.e ps –ef grep tuxwsvr. Once started, enter the following connection string on your browser to enter the GUI – http://[machine_name]:[tuxwsrv_port]/webguitop.html


Login/Password

There are 2 reserved login into the console :

• tpsysadm (Tuxedo Administrator)
• tpsysopr (Tuxedo Operator – limited to admin rights. Cannot Start/Shutdown domain) Default password for tpsysadm/tpsysopr is ‘password’. This can be changed on the tlisten.pw file located on $TUXDIR/udataobj/tlisten.pw
  
  Adding Domains
  
  From the Web GUI, Navigate the following path – Domain > Open. Add the domain name and PSTUXCFG path for that domain.

Quick Guide

Current Number of Client : You can view the current number of clients connected to the domain from the machine level via the Machine drop down on the left. Select the Machine name and and tab over to “Statistics 2”.

Current Number of Client (Client drop down): The Tree display on the left also provides you with information on current connected clients. At this level provides an additional layer of administration. Right-click on a client to view options.

Adding/Removing Server Processes : From the Tree display on the left, expand GROUPS, expand APPSRV, expand SERVERS. Right-click on the server process you want to add/remove.


Changing JSL Parameters : From the Tree display on the left, expand JSLGRP, expand SERVERS and select JSL. Tab over to the ARGS tab and make changes to the MIN/MAX value of the JSL.

You can also make dynamic changes to JSL parameters like port number, min/max client per handler, init timeout, client cleanup timeout and so on via the Command tab.

Modifying Server Process Parameters: From the Tree display on the left, drill down to server process and the service you want to change. This particular examples shows you how to change Service Time out for a PSQRYSRV process.


Boot/Shutdown a domain : From the tool bar, select ACTIVATE (This can boot all or a selected portion of a domain.). The activate menu will appear with series of tabs. Select the desired tab. Check the ‘only activate/deactivate administrative servers’ check box and click the activate/deactivate button depending on the server status at the time this action is performed.

The activate/deactivate function only start processes that don't need a database connection. To activate/deactivate individual server processes, you need to add these 2 variables to the psappsrv.ubx file - PS_SERVDIR={ $PS_SERVDIR}
PS_SERVER_CFG= {$PS_SERVDIR} {FS}{CFGFILE}