Slide 1

WW TSS-04: Advanced
Troubleshooting for Wonderware
Application Server

Rich Liddell Javier Aldán

Technical Account Manager Technical Account Manager
Global Customer Support Global Customer Support
rich.liddell@invensys.com javier.aldan@invensys.com

© 2013 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of Invensys or its subsidiaries.
All third party trademarks and service marks are the proprietary marks of their respective owners.
Agenda

• Tools & Technique

• Install
• Deploy\undeploy
• Multi-Galaxy
Communication
• Tech Notes & Tech
Alerts

Slide 3
Common tools

SMC Logger
Platform Manager
Object Viewer
Task Manager
MiniDump
Windows Event System Files
Wonderware Developer Network (WDN)

Slide 4
Troubleshooting 101 - What Did You
Change?

Slide 5
System Management Console (SMC)

Slide 6
Object Viewer – Locate Process ID

Slide 7
Object Viewer

Find off scan or quarantined objects:

• Uncheck Search by Name
• Check only show objects

Slide 8
Object Viewer

Find Object ID.

Slide 9
Secret Dialog Menu

Slide 10
How can I tell if someone deployed
something?

Did you
Are Nope….
•Gobject_Change_Log
Deploy?
Sure?!?

•Objects affected
•Operation performed
•User Comment
•User Logged on

Slide 11
Return all operations for past 24 hours

SELECT Change.change_date, Change.user_profile_name,

Oper.operation_name, user_comment, gObj.Tag_name
FROM gobject_change_log Change
JOIN lookup_operation Oper
ON change.operation_id = Oper.operation_id
JOIN Gobject GObj
ON GObj.gobject_Id = Change.gobject_Id
WHERE Change.change_date > DateAdd(hour,-24,getdate())
--and Tag_Name = 'UserDefined_001'
--and Oper.operation_name like '%Deploy%'
--or Oper.operation_name = 'ModifiedAutomationObjectOnly'
ORDER BY change.change_date desc

Slide 12
Engine Attributes

Slide 13
Automatic MiniDump Generation

• MiniDump will enable any process from ArchestrA to

dump its process information to a dump file if it
ever terminates abnormally or hits an exception
error.
• If a Minidump file is generated it will be created
automatically at the default path of:
<drive>:\Program
Files\ArchestrA\Framework\minidump
• The minidump can be quite large depending on the
process
(200 - 800mb)

Slide 14
Enable MiniDump

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\ArchestrA\Fram
ework\Debug]
"MinidumpEnabled"=dword:00000001
"MaximumDumpFilesAllowed"=dword:0000000a
"MinidumpType"=dword:00000003

Slide 15
Tech Note 726
Capturing a Memory Dump File Using the Microsoft®
Debug Diagnostic Tool (32bit)

Slide 16
Wonderware Developer Network

https://wdn.wonderware.com

Slide 17
Contact Wonderware

Via Email:PremiumSupport@Invensys.com

Via Phone (US):1-800-WONDER1 (or 1-800-966-3371)

(international): 1-949-639-8500

• You will need to have a UserID.

Slide 18
Install issues…..

Slide 19
Unable to Modify install

• You have successfully

installed
• Go to add modify the install
via add\remove programs
• You then see a quick flash
and nothing happens.
• You uninstall & reinstall and
yet the same behavior

Slide 20
Missing Install folder
C:\Program Files (x86)\Common Files\ArchestrA\Install\{F7D0430C-1E73-4546-BB8C-3E23DF991668}\External

Slide 21
We need to copy it from the Root of the install:

Slide 22
 Disable UAC during install

Must disable User Account Control (UAC)

before installing
•Failing to do this can result in functional
issues
–Which can lead to re-install

*NOTE: UAC also must be disabled on IDE and GR nodes.

ArchestrA System Platform 2012 R2 supports UAC-enabled operations on run-time nodes

Slide 23
Combating Deploy issues

Slide 24
Deploy issues
Failed to deploy <PcName>: Failed to get the
bootstrap's version

•The security policies in the customer domain

are blocking some unauthenticated RPC calls
with anonymous impersonation level.

•Static cloaking has been enabled for Bootstrap

and GR and added code to impersonate the
ArchestrA user before deploy/undeploy.

Slide 25
Failed to get the bootstrap's version

Slide 26
Failed to get the bootstrap's version

Slide 27
Known Issue with Windows 7 Client

3.1 SP3
• L00124127
3.1 SP3 p01
• L00122194
3.5
• L00122265
3.6
• L00123983

Slide 28
Deployment issues: What can go wrong?

•DCOM
•NMX Local mode
•Versions
•NIC Binding order
•aaBootstrap not responding
•aaLogger hanging
•Global Data Cache
•Platforms still deployed but removed from GR

Slide 29
Deployment Troubleshooting Checklist

Check Time synchronization between Platform

Nodes
Configure network binding order when using
multiple Networks
Disable TCP Offload Engine (TOE)
Setup ArchestrA Admin Account on all
Platforms
OSConfiguration utility – 3.1sp3p1 on 2008
sometimes requires the version from 2012
R2

Slide 30
Deployment Troubleshooting Checklist

Make sure all Platforms have the same

Version and Hotfixes

Check Firewall Settings (required Ports are

documented in the ReadMe of the Product)

Check Tech Note 461, Troubleshooting

Bootstrap communication

Slide 31
Deployment Logflags

Category packages responsible for

deployment:

PlatformCategory Package – responsible for deploying

platform engine

EngineCategory Package – responsible for deploying

redundant or non redundant application engines.

ApplicationCategory Packages – responsible for

deploying Areas, DI Objects, and Application Objects

Slide 32
Deployment Logflags

Components involved in deploy/undeploy process :

WWpackageManager – component used by aaGR
clients (IDE, GRAccess) to interact with
WWPackageServer.

WWPackageServer – component running under

aaGR (service running on Galaxy Repository
node), which is used for interacting with
database, validation and sorting of the objects
that has to be deployed etc.

Slide 33
Deployment Logflags

Components involved in deploy/undeploy process :

Bootstrap – service that has to be installed on
every IAS node, which among other
functionalities is used during platform
deployment/un-deployment.

Platform Install Manager – responsible for

installing all code modules on local or remote
nodes using MSI.

Slide 34
Deployment Logflags

Components involved in deploy/undeploy process :

File Copy Service – Responsible for copying the
files to remote nodes.

DCOMTransport – This is the underlying transport

used by the File Copy Service to transfer files
between nodes.

Slide 35
GPO enabled

Unable to deploy/undeploy or configure objects

in AppServer v3.6 with customer GPO enabled;
access denied due to insufficient permissions in
objects' "...\CheckedIn" and "...\CheckedOut"
folders
Hot Fix
•L00126108 (3.6)

Slide 36
Known Issue

Deploy of a redundant engine without cascade

causes all running objects to be lost.
•Hot Fix
–L00126469(3.6)

Slide 37
Global Data Cache distribution

aaGlobalDataCacheMonitorSvr
• ArchestrA GlobalDataCacheMonitorServices. This
service will appear in the Task Manager once a
platform is deployed to the machine. This service
hands information for the Areas and alarms via the
XML, also handles security calls.

Slide 38
Global Data Cache

Slide 39
Overview Global Data Cache

GR Node Remote Platform

aaBootstrap.exe aaBootstrap.exe

aaGlobalDataCacheMonitorSvr

Slide 40
Global Data Cache Issue
Couldn't get platform name - maybe the platform is
not available at this time.
IPlatformInformationClerk2::GetPlatformIdentity(Plat
formID=xx), hr = 80040405

Platform or Engine mismatch occurred because of

non functional Data Cache distribution between the
Platforms
To resolve the mismatch Problem redeploy the
remote Platform
•Hotfix
–L00125442 (3.1 SP3 p01)
–Addressed in 3.6 p01 release
Slide 41
Global Data Cache Issue

GlobalDataCache folders do not sync if the

aaGlobalDataCacheMonitor service is crashed or
restarted.
Hotfix
• L00125643 (3.6)
• Addressed in 3.6 p01 release

Slide 42
Orphaned platforms

Connection accepted from address <nodename1>,

which differs from existing entry , address
<nodename2>. New connection will be denied

•Root cause is an orphan platform which was

removed from the galaxy improperly and is still
trying to connect to the Galaxy
•Identify the Node where the platform is running
and remove it by using platform remover

Slide 43
Platform Exceed Maximum Heartbeats

Slide 44
Platform Exceed Maximum Heartbeats

Solution:
Setting the proper value in your Platform and AppEngine Configuration
Editor

Slide 45
Platform Remover (Killer)

Run as Administrator
Clear out Checkpoint files
• C:\Program Files
(x86)\ArchestrA\Framework\Bin\CheckPointer
Clear out Cache folder
• <RootDrive>\ProgramData\ArchestrA\Cache

Slide 46
Platform Remover (Killer)
Fails to run when there
are more than 100
platforms.

Slide 47
 Scripting Considerations

•Using the right script

•Debugging
•Logmessage()
•What is Async for
•Script Timeout/Error

S
©
Invensys proprietary &
Slide 48

Inve
liconfidential
nsy
d
Engineering Efficiency
2014

• Script Editor
• Auto complete function
• Me
• MyContainer
• Scripts
• Multi level Undo-Redo
• Line Numbering
• Consistent color coding
• Syntax Error Indication

Slide 49
Engineering Efficiency
2014

• Scripting: Exception Handling

Trap Exception
Handle Exception

Slide 50
Let the Engine / Object Relax While
First Loading
Use a while true script instead of a
On true for large tasks (such as IO
set reference).

Delay with
If Script.ExecutionCnt == 2

Slide 51
Use LogMessage()

Why have needless Logmessages going to the logger

unless required. Always block them in with an IF
statement:

If me.Debug then
Logmessage(me.msg);
Endif;

Slide 52
Async Scripts

• SQL scripts should always be Async

• Engine.AsynScriptMaxThread default size is 5

•Engine.AsyncScriptsWaitingCnt
•use this for sizing AsynScriptMaxThread

Slide 53
Keep it Clean

Slide 54
Keep it Clean

WAS Clean-up Guide:

Improves time to open templates and objects.
Improves time to check-in objects and templates.
Deploying the InTouch app is faster.
Restoring a Galaxy is faster.
Backup was faster and smaller

Slide 55
Keep it Clean

Tech Note 930

https://wdnresource.wonderware.com/support/k
bcd/html/1/t002746.htm

Slide 56
Multi-Galaxy Communication?

Slide 57
Remote data

Symptom:
View does not show remote Galaxy data

Possible reasons:
MxData Service is not deployed
Discovery Services are not configured correctly
Platform is not deployed on the node where MxDataService is running
Remote node is not reachable

Slide 58
Secure Write

Symptom:
Writes do not work from InTouchView when security is enabled

Possible reasons:
Security mode of Galaxy is set to “Galaxy Security”
Security mode of InTouch is not set to “ArchestrA”
User has not logged into the remote Galaxy at least once
Default User Authentication service is not deployed on GR node
Security mode of local and remote Galaxies does not match
User does not have sufficient permissions to perform the write
Remote node is not reachable
Slide 59
ASBService OS Account

1. What if the ASBService OS account is not permitted? What

account can be used to start the service?

2. Can the ASBService OS account be disabled?

Slide 60
ASBService related warnings

3. ASBSecurity Proxy: Connect null FindResponse finding

IManageASBSecurity on the SR node”

• The ArchestrA Watchdog service needs to be started before

creating a new Galaxy
• Once the ArchestrA Watchdog service is fixed, the platforms
had to be redeployed

Slide 61
ASBService related warnings

4. aaServicesDeployAgentHost -:- ASBSecurity Proxy: CallDisconnect

delegate caught exception The communication object,
System.ServiceModel.Channels.ServiceChannel, cannot be used for
communication because it is in the Faulted state.

• Tech Alert 173

• Uninstall / Reinstall product

Slide 62
ASBService OS Account

Tech Alert 173

Cannot Create a Galaxy or Connect to Any Existing Galaxy
After Renaming a Computer if Wonderware Application
Server 2012 R2 (Version 3.6) is already installed on the
Computer

Slide 63
Failed to UnpairWithGR…

If one of the Galaxies used as a Galaxy Pair in a Multi-Galaxy

Configuration is unavailable, the pair cannot be "unpaired."

Slide 64
Failed to UnpairWithGR…

• System Platform requires that both

paired Galaxies must be present for
unpairing to occur cleanly. Outside of
seeing the orphaned Galaxy pair in the
paired Galaxy list, there is no adverse
impact to the system's operation. To
reduce orphaned unpaired Galaxies,
unpair galaxies before disconnecting
from the network.

Slide 65
Hotfix

When using FSGateway in a multi-galaxy

configuration and adding a large number tags to
FSGateway using an OPC Client the tags get stuck in
an initializing state.
Hotfix
• L00124824

Slide 66
Questions?

Slide 67
Latest issues

Slide 68
100% CPU on aaEngine.exe

Engines get stuck at 100% CPU

•NmxSvc is modified to ensure that it doesn't
send incorrect disconnect message to the
remote platforms.
–Hotfix
•L00124013 (3.5 p01)
•L00127549 (3.6)
*Addressed in 3.6 p01 release.

Slide 69
RDI object

Bad items that do not exist in the PLC causes RDI to

take the AppEngine down over time.
• Hotfix
–L00128094 (3.6)

Slide 70
Old Alarms

Old Alarms showing in Alarm Control

• They cannot be Acknowledged
Hotfix
• L00127843 (3.6)

Slide 71
Tech Alerts

TA # 173
• Cannot Create a Galaxy or Connect to Any Existing Galaxy
After Renaming a Computer if Wonderware Application Server
2012 R2 (Version 3.6) is Already Installed on the Computer

Slide 72
Tech Alerts

Tech Alert 174

System Corruption Can Result when Importing
Object Files (aaPKG) Created in a Higher
Application Server Version

Cannot deploy objects after importing objects

developed in 3.1 SP3 P01 to 3.1 SP3 (exists in all
version of Application Server up to 2012 R2)

Slide 73
Slide 74
Tech Alerts

Tech Alert 180

Silenced Alarms are not Logged in the
WWAlmDB Database

Tech Alert 181

Platform Fails to Deploy on Server 2003 SP2 or
XP SP3 Nodes When Using App Server 3.6 P01

Slide 75
Wonderware Developer Network

https://wdn.wonderware.com

© Invensys 2009
Slide 76
Invensys proprietary & Slide 76
confidential
Contact Wonderware

Via Email:PremiumSupport@Invensys.com

Via Phone (US):1-800-WONDER1 (or 1-800-966-3371)

(international): 1-949-639-8500

• You will need to have a UserID.

Slide 77
Questions?

Slide 78
Slide 79