TAFC Initialization Check (TIC)

User Guide

Page 1
Information in this document is subject to change without notice.
No part of this document may be reproduced or transmitted in any form or by any means,
electronic or mechanical, for any purpose, without the express written permission of TEMENOS Holdings NV.

Copyright 2002-2006 TEMENOS Holdings NV. All rights reserved.

 Document revisions
Date Author Changes Made
27/04/2010 Jon Tricker Draft
29/04/2010 Jon Tricker Add config URL section
06/05/2010 Jon Tricker Add monitor section.
18/05/2010 Jon Tricker Add warning about firerwalls.
15/12/2010 Balamurugan T Update with Tafc Service usage
15/12/2010 Satya Ravipati Updated Spooler.tic section.

Page 2
Contents
1 Introduction.........................................................................................................................................5
2 The Context Configuration file...........................................................................................................6
2.1 Sections.........................................................................................................................................6
2.2 Parameters.....................................................................................................................................6
2.3 The [common] section..................................................................................................................6
2.4 Meta-sections................................................................................................................................7
2.5 Configuration file as a URL..........................................................................................................7
2.6 A sample configuration file..........................................................................................................7
3 Querying the configuration file...........................................................................................................8
3.1 The ‘tafc_properties’ utility..........................................................................................................8
3.2 The ‘confchk’ utility.....................................................................................................................9
3.2.1 Broadcast mechanism............................................................................................................9
3.2.2 TTL value..............................................................................................................................9
3.2.3 How long to wait for replies?................................................................................................9
3.2.4 Firewalls................................................................................................................................9
4 TAFC initialization checks (TICs)....................................................................................................10
4.1 Enabling/Disabling TICs............................................................................................................10
4.2 Fine tuning TICs.........................................................................................................................10
5 TAFC utility......................................................................................................................................11
5.1 Operation....................................................................................................................................11
5.2 UUIDs.........................................................................................................................................12
5.3 TAFC monitor events.................................................................................................................12
5.4 Future development....................................................................................................................12
6 TAFC services...................................................................................................................................13
6.1 Service Name..............................................................................................................................13
6.2 TAFC service commands............................................................................................................13
6.3 TAFC service and TICs..............................................................................................................14
7 Debugging.........................................................................................................................................14
8 Appendix 1 - Known TICs................................................................................................................15
8.1 core.tic.........................................................................................................................................15
8.2 diskfree.tic...................................................................................................................................15
8.3 fsize.tic........................................................................................................................................15
8.4 jdls.tic..........................................................................................................................................15
8.5 nofiles.tic.....................................................................................................................................15
8.6 numcpu.tic...................................................................................................................................15
8.7 oslevels.tic...................................................................................................................................15
8.8 portno.tic.....................................................................................................................................15
8.9 spooler.tic....................................................................................................................................15
8.10 umask.tic.................................................................................................................................16

Page 3
Page 4
1 Introduction
This user guide provides detailed instructions on how to use TAFC Initialization Checks (TICs). It also
covers the, closely related, topic of TAFC configuration files.

In a multi application server architecture T24 is deployed as an application cluster with one database
server and multiple TAFC application servers. Before a server goes on line it can make a number of
initialization checks. These confirm that the server’s internal configuration is sufficient to successfully
run TAFC.

A server can also pole for other TAFC server instances and confirm that their configuration is
compatible with its own.
Each cluster is identified by a ‘context name’ set in the TAFC_CONTEXT environment variable.
Servers only attempt to communicate with other servers from the same cluster.

Generally each server will be run on a separate machine. However there is no reason why more than one
instance (for the same or different contexts) should not be run on a given machine. In this document the
term ‘remote’ refers to ‘another’ server instance even if happens to be co-located.

Page 5
Information in this document is subject to change without notice.
No part of this document may be reproduced or transmitted in any form or by any means,
electronic or mechanical, for any purpose, without the express written permission of TEMENOS Holdings NV.

Copyright 2002-2006 TEMENOS Holdings NV. All rights reserved.

2 The Context Configuration file
On each server there can be a number of TAFC configuration files. Each file contains setup information
for one context on that server. The pool of configuration files (one on each server) for a given context
act as a decentralized database configuring behaviour of the cluster associated with that context name.

If the configuration file is present it is located at:-

$TAFC_HOME/config/$TAFC_CONTEXT/tafc.ini

For example, if TAFC_HOME="/home/tafc" and "TAFC_CONTEXT=default" then the file is:-

/home/tafc/config/default/tafc.ini

2.1 Sections
The config file is made up of a number of ‘sections’. Each section starts with a ‘label’:-

[sectionName]

2.2 Parameters

Each section contains a number of named parameters as follows:-

paramName=paranValue

Parameters are textual strings that can be compared, for equality/inequality, with other parameters. At
the time of writing numeric parameter types are not supported. Although a parameter may have
numerical meaning to it’s local application it cannot be compared as ‘greater’ or ‘less’ than another
parameter.

2.3 The [common] section

Section name [common] is reserved. Parameters in this section are, from a configuration management
perspective, equivalent to environment variables. For example:-

[common]
aParam=123

On a remote server will match a local environment variable set as:-

export aParam=123

Page 6
2.4 Meta-sections
Some sections, for example [OSLevels], are generated ‘on the fly’ when the configuration file is
queried. This enables the same comparison mechanism to be used for meta-sections and regular
sections.

2.5 Configuration file as a URL

If environment variable TAFC_INI_URL is set then the configuration file will be loaded from this
location instead of the local system. This mechanism enables several servers to share a configuration
file. Be aware:-

1. This introduces a single point of failure. If the location becomes unavailable all servers will
fail.
2. If all servers use the same configuration all equality TICs will pass. All inequality TICs will
fail. Where differing parameters are required they should be put into environment variables
rather than the file.

2.6 A sample configuration file

A typical configuration file is:-

[common]
anEnvVariableName=aValue
[aSectionName]
aParameterName=aParameterValue
anotherParameterName=anotherValue

Page 7
3 Querying the configuration file
Several utilities that can be used to query the configuration file.

3.1 The ‘tafc_properties’ utility

‘tafc_properties’ displays the contents of the local configuration file. This is useful when determining if
the file has been found and if it is correctly formatted.

‘tafc_properties’ displays sections and meta-section from the file but, unlike ‘confchk’, it does not
display environment variables.

Page 8
3.2 The ‘confchk’ utility
‘confchk’ compares parameters from the local configuration file with their equivalents on remote
servers. For detailed syntax see usage page:-

confchk -?

Equality or inequality is checked for individual parameters or all parameters in a section.

3.2.1 Broadcast mechanism

‘confchk’ discovers remote servers using a ‘broadcast’ mechanism (actually UDP multicast). When a
remote ‘tafc’ instance (see below) receives the broadcast it returns a reply containing the requested
parameters.

3.2.2 TTL value

When a broadcast message is sent it contains a ‘time to live’ (TTL) value. A detailed description of is
beyond the scope of this document. Essentially this tells the network how many routers to pass the
message through before it is discarded. If all your servers are on the same subnet then a TTL of 0 is
sufficient.

TTL can be specified via the ‘confchk’ command line.

Note : The higher TTL is set the more hops across the network multicast packets will be make i.e. the
more machines will potentially try to become part of your named cluster. Therefore keep TTL as low as
possible. Whatever value TTL is set to broadcasts are not generally forwarded onto a WAN. It is not
envisaged that clusters will span more that one site. If this is to be done then some form of tunnelling
will be required such that both sites are effectively part of the same LAN.

3.2.3 How long to wait for replies?

Following a broadcast there is the question of how long to wait for replies. Have all the remote servers
replied or are the requests/replies just travelling slowly across the network, hung up in router queues,
being processed by the remote system etc.?

By default confchk waits for 5 seconds and then assumes all replies have arrived. This value is
configurable via the command line.

3.2.4 Firewalls
A likely cause of confchk failure is the presence of a firewall on either the client or server. It this is
suspected enable the multicast ports (currently 20003 and 20007) for UDP traffic.

Page 9
4 TAFC initialization checks (TICs)
Each TICs is a test that can be run to check a given aspect of system behaviour. The test is run and returns a
pass/fail indication. Each TIC is implemented as a ‘.tic’ file located at:-

$TAFC_HOME/config/$TAFC_CONTEXT/<name>.tic

e.g. :-

$TAFC_HOME/config/default/oslevel.tic

Each TIC has a description tag which is printed when run. Additionally it may have ‘frequency’
defining how often it is to be run.

Appendix 1 contains a list of currently known TICs.

4.1 Enabling/Disabling TICs

If a TIC is not required remove the .tic file from the context directory.

If an extra TIC is required copy the .tic file into the context directory.

4.2 Fine tuning TICs

‘.tic’ files are actually ‘zip’ files. These contain a tic.xml file describing how the TIC is to be run and, optionally,
any executables required to run the TIC.

In general there is no reason to modify TICs. If you wish to fine tune their behaviour de-archive the zip into a
directory with the same name. Then and modify the expanded tic.xml file.

Page 10
5 TAFC utility
The ‘tafc’ utility checks that all of a cluster’s servers are configured such that the cluster will start
successfully. At present the cluster administrator must manually invoke ‘tafc’ on all servers. Although
all instances do not need to be started at exactly the same moment they must be allowed to run in
parallel. If all instances operate for a reasonable time (several minutes), without displaying failure or
returning to the command line, then the cluster is correctly configured. ‘tafc’ can then be stopped and
the server instances started.

Initially ‘tafc’ runs all the TICs for a given context. Some TICs check behavior of the local system.
Some use ‘confchk’ to compare local parameter with equivalent parameters on remote ‘tafc’ instances.
Since several calls to ‘confchk’ are made this may take a few seconds.

If all TICs pass ‘tafc’ starts a ‘reporter service’. This receives and replies to parameter requests from
remote ‘confchk’ calls. Periodically ‘tafc checks’ and reruns any TICs that have a repeat ‘frequency’.

If at any point a TIC fails then instability in the cluster exists. Tafc takes its reporter service off line and
exits.

For detailed syntax see the usage page:-

tafc -?

5.1 Operation

The tafc mechanism is not 100% deterministic. A slow, but misconfigured, server could be missed in the
initial check. Also if misconfigured server started first it can freeze out good servers on the same
context. With a tafc instance running on each server any misconfiguration should become apparent
within a few cycles of running the TICs.

This flags up the problem to the cluster administrator who can then diagnose and resolve the issue.

Tafc should rapidly find most misconfiguration issues most of the time.

Page 11
5.2 UUIDs

‘confchk’ broadcast are also received by the local ‘reporter service’. If this were to respond, to its own
request, then any check for ‘unequal’ parameters would always fail. To prevent this each instance of
‘tafc’ generated a unique, time stamp based, UUID and writes it into the ‘TAFC_UUID_ENV_VAR’
environment variable. Tafc then can identify and ignore its own broadcasts.

If preferred a UUID can manually be generated for each server and hard coded into the environment
variable e.g. :-

set TAFC_UUID_ENV_VAR=7bc17c6a-520d-11df-8002-0016175d8499

If this environment variable is not set then each confchk instance generates its own UUID. I.e. all ‘tafc’
instances will reply to it.

5.3 TAFC monitor events

If monitoring is enabled ‘tafc’ also sends TIC pass/fail events to the monitor. The event category is
“TAFC” and failures are classified as errors. The monitor can thus collate information from several
‘tafc’ instances running on different servers.

For more information about monitor configuring see the monitor user guild.

5.4 Future development

Currently ‘tafc’ is a diagnostic utility. At some point in the future it may be more closely integrated with
the server. ‘tafc’ will run automatically at server start time and if TICs fail the server will be
automatically shut down.

Page 12
6 TAFC services
In windows platform, the ‘tafc’ utility can be run as a service in background. There could be
independent services started based on different TAFC context. These “tafc” services support basic
operations like install, start, stop and remove.

By default, tafc service will be running with “default” context. The user has to pass “-c context” option
to start “tafc service” for a specific TAFC context. The user should able to run parallel “tafc” services
with a limitation that only one service for each context.

6.1 Service Name

The name of the “tafc“ will be formed with the help of the context used. The service name for any
context type is :
“Tafc Context – (<context>)”
At any time the service name can be checked in Task manager->services.

6.2 TAFC service commands

Install a “Tafc” service:
The user can install the default “tafc” service or the service by mentioning the TAFC context name.
1. Run “tafc install” in windows command prompt to install “Tafc” service
2. Run “tafc -c <context> start” in windows command prompt to install “Tafc” service for a
specific context
3. The installed services can be viewed in “Windows Task Manager”->services

Start a “Tafc” service:

Only installed services can be started using the commands below. A try to start unavailable service will
be throwing error while issuing “tafc service”. The TICs for the corresponding context will be executed
by the “tafc “ service started.
1. Run “tafc start” in windows command prompt to start default “Tafc” service
2. Run “tafc -c <context> start” in windows command prompt to start “Tafc” service for a
specific context
3. Check the service status in Windows Task manager -> services

Stop a “tafc” service:

The services in “RUNNING” state can be stopped by the below command. Incomplete service
installation or already terminated services could through error.

1. Run “tafc stop” in windows command prompt to stop default “Tafc” service
2. Run “tafc -c <context> start” in windows command prompt to stop “Tafc” service for a
specific context
3. Check the service status in Windows Task manager -> services

Page 13
Uninstall a “tafc” service:

1. Run “tafc start” in windows command prompt to remove default “Tafc” service
2. Run “tafc -c <context> start” in windows command prompt to remove “Tafc” service for a
specific context
3. Check the service status in Windows Task manager -> services

6.3 TAFC service and TICs

On successful installation and start, the “tafc” service runs the TICs configured for the specific context
in background. So, the display messages from TICs will not be printed in screen while running “tafc”
service.

The user can configure or customize the TICs in the same way as TAFC utility. To stop the execution of
TICs, a “tafc stop” can be issued in command prompt to stop the “tafc” service.

7 Debugging
A useful sequence for debugging is:-

Check that configuration files are set up on both servers:-

1. Run ‘tafc_properties’ on the local server. Confirm that the local tafc.ini file is found and
contains the right parameters.
2. Log onto the remote server and repeat step 1.

Check that ‘confchk’ can compare values on the both machines:-

1. Start ‘tafc’ on the remote server. Confirm that it gets as far as “Listening for configuration
requests…”. If not fix any failing TICs on the remote machine.
2. Run “confchk –v –s OSLevels”. Confirm that the remote system replies.

If the remote system fails to reply consider:-

1. Is there basic network functionality between the machines? Run ping.

2. If the value of TTL value specified in your confchk calls high enough? Check how many
routers are between two systems. Check that they support UDP multicast forwarding.
3. Check that any firewalls have the confchk UDP port (default 20003) open.

Check that configuration files are compatible:-

1. Also start tafc on the local server. Confirm that it also gets as far as “Listening for
configuration requests…”. If not fix any failing TICs on the local machine.

Page 14
8 Appendix 1 - Known TICs
This section describes currently known TICs.

8.1 core.tic
Checks for adequate core dump space on the local server.

8.2 diskfree.tic
Checks for sufficient free disk space on the local server.

8.3 fsize.tic
Checks for correct file size ulimit setting on the local server.

8.4 jdls.tic
Checks that JDLS configuration is the same on all servers.

8.5 nofiles.tic
Checks for a usable number of OS file handles on the local server

8.6 numcpu.tic
Checks for required number of CPUs on the local server … i.e. is it powerful enough?

8.7 oslevels.tic
Checks that OS type and revision are the same on all servers.

8.8 portno.tic
jBASE port numbers (the JBCPORTNO environment variable) for remote servers are compared with
those for the local server. Fails if any are the same.

Note because parameters are simple strings overlapping port ranges will not be detected.

8.9 spooler.tic
Checks that the spooler is running at respective JBCSPOOLERDIR location.
spooler.tic looks for the QUEUE’s in environment variable JBCSPOOLERDIR directory and checks
their status. It will report WARNING/FAILURE according to our spooler set-up.

Page 15
Usage:-
tic_spooler {options} {Spooler-Queue List}

Valid Options:
-h Usage Help text.
-v verbose mode.
-f Consider Warnings as Failures.

Arguments:
Queue1 Queue2 ... If all Spooler-Queues are Good Return SUCCESS else Return WARN/FAIL
based on argument -f.
Examples (Sample Commands):-
 tic_spooler
Run a spooler check on all the spooler queues in its respective JBCSPOOLERDIR.
 tic_spooler –f
Run a spooler check on all the spooler queues in its respective JBCSPOOLERDIR and if any of
the Queue Status is at Warning Level it promotes to Failure and reports the same.
 tic_spooler –f –v
Run a spooler check on all the spooler queues in its respective JBCSPOOLERDIR and if any of
the Queue Status is at Warning Level it promotes to Failure and reports the same. Apart from
these it runs everything in verbose mode where you get some detailed output on screen.
 tic_spooler MyQueue1 MyQueue2
Run a spooler check only on the Queues specified in the argument list of spooler in its respective
JBCSPOOLERDIR.
 tic_spooler –f MyQueue1 MyQueue2
Run a spooler check only on the Queues specified in the argument list of spooler in its respective
JBCSPOOLERDIR and if any of the Queue Status is at Warning Level it promotes to Failure
and reports the same.

8.10 umask.tic
Checks that sufficient user mask privileges are set on the local server.

Page 16