Java Projects

Tools and techniques

Mike Godfrey Information Technology Services / The University of Texas at Austin

Outline
Look at three java projects Different architectures adopted Problems encountered and lessons learnt Tools and other useful techniques

Who are we?

Remains of centralized data processing Part of the unit providing IT services across campus Traditionally mainframe based applications with web front-ends scripted in a simple in-house language Changing mission as our administrative application development role is done by departments Project list that requires a much more powerful language pushing adoption of Java Changes are forcing developer to acquire new skills and ways of working java camp.

The Systems
Clips a web-based file system for distributing course materials Jwebagent a java runtime for our in-house scripting language Tech lounge a web-based developer documentation store

Clips file system

Clips Requirements
A simple to use, intuitive interface Handle all file types Accessible from any location Date controlled display of materials Easy transfer of materials from class to class Work well with existing mainframe class management system Work within the UTDirect portal framework

Basic Design Choices

File system metaphor for user interface Browser access for both students and faculty Use native XML database to model the file system in a recursive schema Store binary files in same XML database JSP with scriptlets for display clips interface

Use of XML for Meta Data

File system metaphor seems ideally suited to the tree structure of XML Meta data structure based on two nodes types; directorynodes and binary-nodes One XML document contains all meta information needed to build file structure for a given class Easy to re-build hierarchy of resources from XML tree Database uses schema to specify stored document structure

Meta-Data Details
Directory nodes store information about a directory: name, creation time, description, etc Binary nodes store information about a binary file: name, foreign key, size, content type, etc Directory nodes can contain either binary nodes or other directory nodes (hierarchy) clips meta data

Clips Architecture
XML database (binaries and Meta-data)

Tamino api Helper Classes Broker api

JSP presentation layer

User

Authorization and class info

mainframe Adabas

Broker Natural
Existing class management system

New Issues for us

Handling and auditing file uploads from web pages (multipart encoded forms) Building and manipulating XML records How to best make large scale use of JSPs in a complex Java system How to simplify communication with mainframe

Useful Tools
Log4j used for debugging and recording production exceptions (system written before Java 1.4) JDOM very user friendly api for manipulating the XML cos.jar used to parse the multipart encoded forms Entire-X broker ACI for Java Java stub generator for mainframe modules Tamino - native XML database

Lessons Learnt
Be careful of scriplets in JSP; they expand to fill the available space! Better to use a controller servlet to organize navigation through JSPs XP and pair-programming works A Java developers best friend is Goggle

Alternative JSP architecture

View jsp Maintain Update Copy jsp jsp jsp View jsp Maintain jsp Update jsp Copy jsp

Controller servlet

Auth Task

Form Resource Parser List

Copy Task Update Task

Auth Task

Form Resource Parser List

Copy Task Update Task

mainframe

XML store

mainframe

XML store

Jwebagent

.WBX file

Jwebagent

HTML file

What is Webagent?
A scripting language with a syntax like Natural (mainframe language) Scripts transformed into C++ objects forming a virtual machine which is serialized to a .WBX file Runtime reads .WBX file, rebuilds the C++ virtual machine and executes it. Pro: easy and productive for creating web front ends to mainframe applications. Con: limited functionality

Why Jwebagent?
Same as Webagent but the rebuilding of the runtime is done in Java plus Enable java classes to run existing Webagent scripts Enable Webagent scripts to use java classes Improve execution speed no process forking and script buffering

Main Runtime Elements

Servlet handles all execution requests for a WBX file Executor coordinates loading file, running the VM, and handling errors for an individual file request File loader re-builds the serialized script objects Context objects handle data specific to that particular request VmOpCodes functional elements of the VM Data objects variables, literals

Runtime
vmOp broker vmOp broker vmOp broker vmOp vmOp vmOp Serial code object

file request

Servlet

Executor

Builds then executes

Script context http req

env http resp

Data buffer

Request context

WBX file

Useful Tools
Junit unit testing framework (junit.org) JProbe application analyzer (Quest Software) JMeter load testing suite (Apache.org) HttpExplorer gui http request and response tool (in-house) Jedit text editor (Jedit.org) IntelliJ IDEA Java IDE (Jet Brains) CVS (Tortoise, WinCVS, Putty for SSH) Xvi32 hex editor

Junit unit testing

Provides easy way to set and teardown test situations Automated running of many tests Organize tests in suites and suites into metasuites Text or gui UI for test reporting Refactor with confidence! (very valuable) Trick is to know what tests to write Good with Jwebagent as it is not database driven

JProbe code analyser

Threadalizer finds deadlocks and conflicts (did not show any problems for Jwebagent) Memory Debugger shows memory use by classes and can find hanging objects (useful strings to byte arrays) Profiler tracks objects and the call pattern, records method time and number of calls (most useful tool identify prime objects to optimize)

JProbe Code Optimizations

Strings to byte arrays saved memory on string generation io to nio gave 25% increase in through-put In-lining of frequently used methods to reduce calling overhead Combining of related classes when frequently constructed Un-doing OOP improved performance!

JMeter load testing

Build and store test plans Control number of threads, ramp-up times and thread delay to simulate different testing scenarios Test http apps, SOAP and web services, LDAP, JDBC, etc Display test results in graphs, tables, files, etc Build tests with logical flows to suit application Apply pre and post requests during tests Very valuable for performance tuning (with JProbe)

JVM Tuning
Changed default of client mode to server mode (20% improvement) Other settings have given some improvements Use incremental GC rather than stop the world style Uses the flags that control survivor ratio for the GC generations we have short lived objects Increased default memory allocation

Did we make it?

First tests showed consistently slower times than the C++ version ( up to 33* slower) JProbe analysis and JVM tuning reduced this to about 2* slower on average Through-put is slightly lower JMeter shows a mean that is much faster than C++, but and average that is much higher Problem appears to be in the multi-treading of Broker Tried JNI against C++ broker and see same results

Lessons learned
Always build a test suite before refactoring Never optimize until you know what to optimize Do not assume GC defaults are optimal Java needs lots of memory! Allow for distractions in your project estimates (estimated time 9 months, actual time start to finish 2.5 years, actual time spent on project 10 months)

Tech Lounge

Project Goals
Nice, easy to use documentation store Replace mainframe based system Provide better editing and display options Allow limited access to subsets of articles Control the life-time of articles to ensure relevancy Provide good search capabilities and crossreferencing options Store multiple document formats

Design Choices
Store articles in native XML database Article content stored against a subset of XHTML strict DTD easy to display and construct Each article has a meta-data XML file for owners, dates, related articles, number of reads, etc Article display via XSLT for HTML and pdf Article maintenance via jsps and custom tag libraries tech lounge

Article Maintenance Architecture

New Article jsp Maintain Article jsp

Controller servlet

Auth Task

Form Parser

Article Bean

mainframe

XML store

Useful Tools
Many things from Apache DHTML Graphic designer Created Tools/utilities Tamino Wrappers abstraction layer UID (QUID-like foreign key generator)

Problems/Mistakes
Not using CVS well during first half of project Not writing unit tests Not using logging for debug messages

Evolution
Display of XML moving towards all XSLT JSP usage moving away from interlinked pages towards controller server style JSP structure scriplets bad, custom tags good(?) Building up institutional tools/utilities and standards that are increasing productivity Individual experts towards wide spread Java use through in-house training courses

Review of our Java Experience

Pros: More flexible and capable Greater abstraction possible, increased elegance Many very powerful tools/utilities freely available Cons: Requires higher competency of developers Less productive for our typical applications Needs larger machines

Other projects
PdaParser code generation to ease communication with mainframe modules Fatcookie authentication Virtual File System extension to JEdit to work with our unix server script editing proceedure Current Jar management database (entirely XML/XSLT) Web service wrapper for mainframe modules Dynamic pdf generation from mainframe data

Questions?

Useful Project Management Tip

Remember the Bullfighter!