Topics in EOne Applications Performance

Topics in EnterpriseOne
Applications Performance
An Oracle JD Edwards EnterpriseOne
Performance and Tuning Paper
November 2009
PURPOSE STATEMENT
This document provides considerations when reviewing system performance relative to

your JD Edwards EnterpriseOne system. Many factors can influence performance and
your results may differ depending on many different variables.
This document is intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing decisions. The development,
release, and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle. Updates and enhancements are provided in
accordance with Oracle’s Technical Support Policies at:
www.oracle.com/support/collateral/oracle-technical-support-policies.pdf
Performance and Tuning Title Page i

Topics in EnterpriseOne Applications Performance
EnterpriseOne Performance / Introduction................................................. 3

EnterpriseOne Performance / Topics in Perfomance ................................ 3
Software Performance First Principles ...................................................... 3
UBEs on the client: a BAD idea................................................................. 7
Use C code Business Functions – NOT Event Rules or Named Event
Rules ............................................................................................................. 11
Indexes are NOT the answer to everything............................................ 13
Loop Invariants........................................................................................... 16
Memory Leaks versus memory consumption......................................... 21
Buffered Inserts........................................................................................... 28
JDE Cache APIs ......................................................................................... 30
EnterpriseOne Performance / Conclusions ............................................... 32
Topics in EnterpriseOne Performance Page 2

ENTERPRISEONE PERFORMANCE / INTRODUCTION

This paper is intended to enumerate the most important software
performance principles to consider when coding JD Edwards EnterpriseOne
(EOne) applications. It is targeted primarily at EOne developers; namely,
those who will create EOne applications or modify existing ones.
Assumed is a basic knowledge of EOne, relational database concepts, the

operating system platforms including iSeries, and the C language.
The topics are mostly application related, but there are some tools and
operating system material covered.
ENTERPRISEONE PERFORMANCE / TOPICS IN PERFOMANCE

The basics of software performance are covered, followed by several of the
most useful and ubiquitous performance related topics specifically relevant to
EnterpriseOne.
Software Performance First Principles

Any discussion of EnterpriseOne software performance must begin with an
understanding of the basic concepts software performance. Everything
reduces to First Principles:
WHERE IS THE CODE SPENDING ITS TIME?
This breaks down into two basic sub-principles:

• What gets called too many times? A BSFN or API may be taking
very little time per call, but get called a very large amount of times.
• What is taking too long? A BSFN or API may be called only once,
but take 90% of the runtime
You are looking first for where the time is spent, not to apply rules. The
Performance Analysis process is much like pruning a large tree; one looks
first to trim entire branches, not small leaves. The first goal of that analysis
process is to find the big branches. See the figure below:

Performance is non-linear thinking; it is NOT first a “top ten list”, nor is it a set
of tools, anecdotes, or rules. The latter are consequences which follow from
First Principles.
Studying the user’s manuals for circular saws, nail guns, and a host of other
tools does not prepare one to build a house. A list of the “top ten” household
electrical problems does not enable one to become an electrician. In the
former case, one needs to understand how to properly frame a house, how to
install plumbing, electricity, sheet rock, etc. In the latter case, the
fundamental principles of household wiring need to be studied and mastered.
Similarly, a “top ten” list of common performance problems does not prepare
one to analyze performance. Handing a tool to developers is NOT
tantamount to preparing them for performance analysis.
The purpose of the “top ten” lists is to give structure and specific direction
to the performance analysis process for people who already understand it.
It gives starting points to which the theoretical process can be practically
applied. It can allow one to short circuit a more lengthy analysis by
leveraging problems that others have already solved.
Of course – if the solution to a given problem is NOT on an existing “top ten”

list, then the analyst who truly understands performance can forge new
ground and add item #11 to the “top ten” list.
You assume NOTHING a priori about where the time is being spent, nor how
a given change will impact a particular application or use case. Rather, the
process in question should be profiled and the time spent should be
determined. An example of a common misconception is that all performance

problems are caused by bad SQL, while another is that adding indexes will
solve any problem. The analysis process is what gives these answers.
Yet a further analogy of the performance analysis process: A flu virus and a
bacterial infection have many of the same symptoms…but you would not
treat both with antibiotics. Rather - you conduct tests to determine which
sickness is actually present. THEN, you consult a list of medications to treat
that particular illness. The “medication” in this parable would be a SAR or a
code fix; the profiling data should point to the application of a code fix, not a
superficial set of symptoms.
While a detailed and rigorous treatment of the process is beyond the scope
of this paper, here is a brief summary of the salient truths of the performance
analysis process. Collectively they should form a reasonably complete picture
of performance analysis to the experienced EOne software developer.
Static Analysis of source code files is NOT performance analysis. While
certain classes of “low hanging fruit” opportunities can be preemptively
spotted in this manner, there is simply no way of accurately predicting
where time is spent without running the code and capturing a runtime
profile. Merely reading the code WILL NOT WORK.
Identify and define the problem – You can’t find a solution if you are
“If you don't know where you're going,
guessing at what the problem truly is. You will end up with a solution to a
chances are you will end up somewhere
non-existent problem.
else.”
- Yogi Berra Use a TOP DOWN analysis method – This means that you start at the
Application level, where the problem actually manifests itself. Then – you
work backwards from there, determining where the time is being spent.
An example of a faulty BOTTOMS UP technique would be collecting

database statistics as the first step, and trying to fix individual SQL. You do
NOT know that the time spent in the EOne application code is primarily spent
in SQL operations. This is a common mistake made by DBMS experts
whose knowledge is limited to the database, when they are asked to help
with EOne application code performance.
Do not take wild guesses at the solution - … by applying SARs and ESUs
which may be time consuming and difficult to apply, but in which you have
only minimal confidence as a potential solution to the problem.
Profile the code – This answers the question “Where is the time being spent?”
Make proper use of logs – EOne debug logs can be generated on every
platform and provide the profiling data. The Performance Workbench tools
should be used.
There is a red paper on UBE performance on My Oracle Support (Doc ID
748333.1):

https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=
748333.1
It is critical to generate debug This paper has detailed instructions on how to properly generate debug logs
logs properly which can be used for performance analysis and code profiling. It is critical
to follow these instructions precisely and accurately in order to generate
debug logs which produce suitable results and valid profiles.
Use Performance Workbench – It is available on My Oracle Support (Doc
ID 747328.1), and is a standard tool for EOne performance analysis with a
long track record of success in solving problems. Latest version available
here:
https://support.us.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=
747328.1
Always look at the jde.log first. Always use jde.logs – Look at them FIRST, before any other more complex
analysis, and before collecting large and time consuming debug logs. There
may be easy and obvious answers in the jde.log. In any case, any anomalies
in the jde.log should be explained or fixed before proceeding.
Do NOT Undersample – A two minute run of a four hour UBE will NOT
“The power of statistical analysis depends on
provide an accurate profile of the UBE. A code profile is a collection of
sample size: the larger the pile of data the analyst
has to work with, the more confidently he can draw
statistics. Performance analysis is at its heart a statistical exercise, and all
specific conclusions about it. A right handed statistics are only as good as the sample size.
hitter who has gone two for ten against left
Do not run E1 code on a fat client – Bad for many reasons. A lengthy
handed pitching cannot as reliably be predicted
to hit .200 against lefties as a hitter who has
section on this follows…
gone 200 for 1000.”
Your solutions should be driven by analyzing and finding where the time is
going, not by taking a solution from a top ten list. The “top ten” list merely
- Micheal Lewis, Moneyball
streamlines the analysis process. It does not replace it.
As an example, consider analysis of a UBE which you know INSERTs a large

amount of records into a transaction table.
• From a blind reading of a “top ten” list, you would enter a SAR to get
Buffered INSERTs applied to the code. This may entail complex
recoding and/or refactoring, which would take time and testing.
• However – armed with the knowledge of Performance First Principles,

you first run the offending code and get a profile of the UBE using the
debug logs. This shows that the Business Function which performs the
INSERTs accounts for only 5% of the program’s runtime. This means
that the INSERTs themselves account for less than 5% of the runtime.
Even if Buffered INSERTs can be expected to cut this time by 60%,
that’s less than 3% of the total UBE’s runtime. Applying this code
change may be a theoretically a good idea on paper – but it yields only
marginal improvement in practice for this particular application.

• So in the above example – a push-button use of the “top ten list” leads to
wasted time…but application of First Principles leads you to first answer
the question: “WHERE IS THE CODE SPENDING ITS TIME?” and
avoids lengthy “rabbit hole” excursions chasing non-existent problems.
“WHERE IS THE CODE This remainder of this paper only highlights some of the most important
SPENDING ITS TIME?” “anecdotes” in EOne. This should NOT be used as an all inclusive guide to
performance and should only be used in the framework of First Principles
analysis. It does not cover every existing scenario, only common ones
which may be of general use.
You should use the principles not as guessing or rolling the dice, but more
like “Counting cards” in blackjack. If you have knowledge about a piece of
code based on First Principles and profiling, then the lists and anecdotes
(such as those in this paper) provide a means of systematic “guessing” at a
solution. You will be applying a solution which has worked in the past for
other situations which fit a similar statistical profile.
UBEs on the client: a BAD idea

The reasons why UBEs should NOT be run on the fat client are summarized
in a short anecdote:
A man is looking for his lost keys under the street lamp.
His companion tells him that he's probably dropped them near his car.
The man replies, "I know, but the light is better over here."
To put it another way:
If I have a sore throat ….

I can’t send my brother to the doctor for a diagnosis just because
we’re related (yes - even if it’s an identical twin)
People continue to commonly capture UBE logs on the client for a number of
reasons:
• Because it’s easiest
• Because they know how to work in a client environment, but not a

server’s
• Because they think they can’t disrupt the server
• Because they think they’d have to bounce E1

But the bottom line that one must learn how to generate server logs!!!! This
means one has to do the analysis work in a timeframe which allows the
creation of logs on the server with minimal disruption…but understanding that
one needs to break eggs to make an omelet. It is crucial to remember the
following:

UBE logging can be set at runtime - ONLY for the UBE, not impacting any
other EOne Processes.
• There is no need to modify the jde.ini
• There is no need to bounce EOne services
Here are the reasons:
1. Many features unique to UBEs are DISABLED in a fat client

environment
• Buffered INSERTs
• JDB cache APIs: JDB_AddTableToDBCache()
• Transaction Processing
These factors alone are a showstopper for many UBEs…these are very
basic pieces of functionality which have a huge impact on UBE
performance.
EONe client side code is In a recent scan of EOne Tools source code, the IAMASERVER
fundamentally different preprocessor directive appears the following numbers of times:
than server side code.
• UBE system code: 68 places
• JDEKRNL code: 1099 places
• JDVDRV code: 64 places
• JDELIBE code: 28 places
The point is that the EOne client side code stream is different in many
and sundried ways from the server. Externally things may look the same,
but “under the hood” there are very different mechanisms involved.
2. All database I/O will now be issued across the network…rather than
close to the data, as on a server
• This disrupts the code profile by making I/O appear much slower
and less efficient than if the UBE were run on the server
• This makes it easy to falsely conclude that I/O latency is a

problem…when in a server environment the problem simply
does not exist. The real problem may be hidden when running
in a client environment.
• Complex UBEs containing large Master Business Functions

contain a lot of database I/O – and are very, very chatty when
running on the client.
• Even mapping BSFNs to the server to mitigate the DB I/O issue

results in a very, very chatty process.
• The above is especially true for iSeries

• Some tables may even be mapped to a local MSDE database in
the client environment…which can actually mask potential
performance problems caused by database latency.
3. Database I/O is executed differently on the fat client due to the
different drivers on the fat client than on the server
• SQL statements can be built and executed differently on the fat

client than on the server
• Driver version mismatches can also occur

4. Fat client is usually a much slower machine with fewer resources
• It takes much longer to run the desired data selection and/or use
case and get the desired profile …so both the I/O AND the code
itself is much slower.
• The UBE can consume all the resources on the client machine
5. Fat client is not even supported starting in 8.11
• Analysis should NOT occur against unsupported

configurations
The fat client environment • In addition to the JDB features mentioned earlier (#1) – any
is not even supported number of other areas may not function properly in a fat client
starting in 8.11 environment.
• Transaction Processing in particular is risky
• Note that “Not supported” means no testing has occurred. Quality

Assurance does not include any client-side testing because this is NOT a
supported configuration.
• The fact that UBEs will even still run on the fat client is observed, not
supported behavior.
6. Debug logs are cluttered with fat client / win32 / interactive GUI
code, since UBEs run as a thread.
• You will NOT get the neat, beginning to end profile containing
only UBE code.
• You will get a highly confusing, obfuscated profile, with plenty of

red herrings – see below:

You got four calls to
VersionListLaunchUBE()
taking all the time
Or not….
7. The multiple threads on the win32 client environment make the

debug logs difficult to read and parse
8. Since UBEs run as a thread in the fat client, they share resources
with interactive applications, such as JDB cache settings, OCM
mappings etc…
• This can lead to unintended side effects which are unique to the
client
9. The operating system is entirely different on a fat client than on any
of the server platforms.
• Any factors rooted in operating system specifics will be missed if

analysis takes place on a fat client environment.
• Even if the server is also a Windows machine – it is still very different
from the fat client.
• The Windows multi-user server environment differs from the single-

user client environment in many fundamental ways
10. In PrintQueue directory – debug log contains only Event Rules
logging with no timestamps
• On server – if ER debugging is enabled (UBEDebugLevel > 0) – the

ER logging is placed in the same file as the standard, time-stamped
debug log entries

• On client - the debug log in PrintQueue only contains ER entries
with NO timestamps:
Use C code Business Functions – NOT Event

Rules or Named Event Rules
Event Rules (“ER”) are interpreted code. C code Business Functions are
compiled code. Compiled code is much faster and more efficient than
interpreted code. Whenever a significant fraction of time is spent in complex
ER logic, that code should likely be considered as a candidate for conversion
to C code. This is not to say that there is anything inherently wrong or
defective about ER. Rather – its intended use is the implementation of
simple business logic in a user-friendly manner which does not require
knowledge of C programming. When used within these constraints, it does
its job well. This is no different than interpreted Fourth Generation Language
(4GL) products on other products; typically these proprietary tools are not
nearly as efficient as 3GL languages (e.g. C / C++).
Implement as much business logic as possible in C-code Business
functions. Complex business logic should be written as hand-coded C
Business functions. Instead of calling Business Functions in a tight loop from
Event Rules, convert that ER loop into a single “wrapper” C-business function
which in turn calls the other Business Functions.

“While”-loops in particular are • “While”-loops in particular are notoriously slow in ER. ER while
notoriously slow in ER. loops should be viewed as prime candidates for conversion to C code in
UBEs. The presence of While loops should be one of the key
determining factors as to whether a given piece of logic is “too complex”
for ER.
But what about Named Event Rules? Does the Named Event Rules
(“NER”) mechanism provided by EOne provide a means to deliver the best of
both worlds: namely, the performance of compiled C code, while only
requiring knowledge of the user-friendly ER GUI interface?
The short answer is that NER mechanism works well for simple pieces of
Event Rules code, but for more complex logic, a hand-coded C function is
usually a better choice. Here are the upshots:
• Named Event Rules code (NER) is machine generated C-code
• Note that there is nothing inherently “slow” about NER code versus
hand-coded C business functions……they are both complied C
functions.
• BUT – NER functionality and maintainability is limited to what is

available and can be done in Event Rules….
Many pieces of important The following are examples of key mechanisms not available or exposed via
EOne functionality which can the ER interface:
be leveraged in C Business
• JDE Cache APIs
Functions are not available in
ER. • Linked Lists
• Arrays
Additionally, many EOne system APIs have a more limited set of parameters
when called via the ER interface than are available in their “C” counterparts.
• The ER SELECT interface (“Table I/O”) can only generate a

“SELECT *” – a limited column list cannot be specified. The C APIs
allow the developer to specify a limited column list (as an array of
NEXTID values) using the JDB_OpenTable() API, which can have
a significant impact on performance
Note that a possible alternative to this is the use of EOne VIEWs.

Creating a limited column list VIEW against a table can avoid the
need to specify a column list; the SELECT * against the VIEW will
indeed return only the VIEW columns in the fetched rows. A
limitation of this approach is that it is not very dynamic; to add or
delete columns would require modifying the VIEW, a change
requiring modification and rebuilding of specs. The regression
impact must also be considered if the VIEW is used in other places.
Changing a column list, conversely, requires only C-code

modifications. The impact is then limited to the Business Function in
question. This makes it less painful to make changes if the initial
column list was not quite correct.
• The Batch Interconnect interface is also more limited in ER than via
the C API. For example, the ability to specify and modify UBE input
variables does not exist in ER, while this is possible in hand coded C.
Finally, the machine generated NER C code accomplishes certain tasks less
efficiently than would be possible in hand-coded C.
• Population of large data structures is one example. The NER
mechanism churns out code which uses loops and APIs, rather than
efficient direct assignments. The latter would be possible in hand-
coded C Business Functions.
While NER C-code is easily generated from the ER, it cannot be fine tuned
for performance at the C code level.
• Even though the result of the NER generation process is

Even though the result of the
NER generation process is complied C code, it can only be modified via the much more
complied C code, it can only limited ER interface. If the NER C code is modified directly, the
be modified via the much changes would be overwritten upon the next package build.
more limited ER interface.
Note that the base EOne product does contain many NER functions (the
NER source files begin with an “N” instead of a “B”). Note also, however,
that most of the very complex Master Business Functions are written as
hand-coded C functions, even though the NER mechanism was available.
The message is that NER functionality is intended for power users to

create compiled C code from simple ER tasks. Skilled developers who
are literate in the “C” language should use hand-coded C functions for
complex business logic, particularly for code which requires a good
deal of looping. Whenever possible, human-authored C business functions
should be used to develop critical customizations of core business logic for
the E1 product. If C expertise is not available – or if performance is not a
priority – then the NER tool could be used.
Indexes are NOT the answer to everything

The motto of those not familiar with software performance analysis seems to
be: “When in doubt, add an index. When really confused, add a whole
bunch of indexes”

Indexes should NEVER be added in bulk, as if 100 new indexes will help
things in some mystical way just by existing….even if adding indexes makes
people feel good. Much like drugs…they seem to make your problems go
away, but they solve nothing in the long run. JUST SAY NO.
Indexes should ALWAYS be added one at a time and deliberately – one

index should tune one very specific SELECT generated by a specific
identifiable use case.
You should be absolutely sure the performance problem is really and truly
being caused by a long running SELECT or SELECTs. IF THE TIME IS
NOT BEING SPENT IN SPECIFIC IDENTIFIABLE SELECT STATEMENTS
– THEN FORGET ADDING INDEXES….
An iSeries DBMON trace or the iSeries job logs may contain many index
IF THE TIME IS NOT BEING
SPENT IN SPECIFIC
recommendations….but many may only reduce a single 20ms query down to
IDENTIFIABLE SELECT 10ms, for example.
STATEMENTS – THEN FORGET
On the other hand – these are examples of reasons why you WOULD add a new
ADDING INDEXES….
index:
• Reducing a 20 minute query to 5 seconds…
• Reducing a 50ms query down to 5ms, if said query gets called a million
times…..
• If the above cases had a significant impact on the runtime or response times
processes involved.
It is very easy to test indexes:
• Run each offending SELECT before adding the index
• Capture an explain plan of the SELECT in question
• If iSeries is the platform, DELETE SQL packages
• Add the new index to the database
• Capture a second explain plan of the SELECT in question
• If the newly added index gets used as per the second plan – and
reduces the runtime – then add that index
Then – document the new index very clearly, including the reason it
was added. Otherwise, some overzealous DBA may want to delete it two
years from now…to save disk space.

Adding unused indexes is NOT a harmless vice:
An index is really a re-ordered • One trouble with indexes is that they use valuable disk
copy of the base table. space, and become almost impossible to remove. This is
. because no one ever seems to remember why a given index was
added…so no one is certain if an obscure application or use
case actually needs the index. For large tables such as F0911,
the indexes can be a very significant fraction of the disk space
used by the database. You MUST think of each index as being a
re-ordered copy of the base table.
Indexes create a penalty in • Another major concern is that while indexes on a table help
INSERT, UPDATE, and the performance of SELECT statements, they create a
DELETE operatons.
penalty on INSERTs, UPDATEs, and DELETEs. When rows
. are added, altered, or deleted from a table, maintenance has to
be done on all these indexes to add, alter, or delete all
references to these rows in the indexes. For very large tables,
this can be significant.
• Finally, too many indexes on a table can confuse the DBMS

optimizer, causing it to make suboptimal choices, or slowing
down the optimization process by giving too many options.
• The upshot here is that adding a potentially unused index is NOT

a harmless mistake without downside.
The iSeries DBMON results and/or job log messages should NEVER be
used blindly to add indexes. Very broad heuristics are used to generate
these suggestions, and need to be “sanity checked” very closely.
Note that just because a SELECT statement takes a long time to process -
that does NOT mean it needs an index. There are several other reasons why
a SELECT is slow and/or inefficient
• Returns a very large rowset – If the SELECT returns a very large

number of rows, it will take a long time to process. If it returns more
than 10% of the rows in a table, then a table scan will almost always
be used, because this is most efficient.
• Poorly tuned DBMS – The database configuration and/or

maintenance tasks can impact a SELECTs execution plan. An

example of this is Oracle Statistics; if they are not run regularly, a
performance penalty will result against many queries. Additional
examples are factors such as disk layout, rollback space size, etc…
• Poorly written SQL – The SELECT itself could be poorly written,

causing slow parse and execution time. This is not common in EOne,
which is middleware based and does not allow users to directly edit
SQL.
The bottom line: You should always verify the need for the index by cause
and effect …. and first principles: “WHERE IS THE CODE SPENDING ITS
TIME”
Loop Invariants
The crux of this issue is: why do something many times which need only
be done once?
Many performance issues have their roots in configuration and settings, so

knowledge of the EOne installation and/or operating system is required. But
research and mitigation of the Loop Invariant issue involves analyzing,
understanding, and modifying code. Skills in reading and analyzing code
are key to identifying inefficiencies such as loop invariants.
The salient points are the following:
Do not repeatedly recalculate • Do not repeatedly recalculate values which do not change
values which do not change
• Calculate the value ONCE and reuse the result
.
• Recalculate only when the value changes
• This issue impacts customized EOne financial reports in particular
Here is a case study from a customer performance problem which was

caused by loop invariant inefficiency.
After having captured a debug log of a problematic custom financial report,

Performance Workbench was run to generate a statistical profile. This
showed a large number of calls to a Business Function (DTDateTitle) taking
a significant faction of the runtime:

Upon examination of the Event Rules for this UBE, it was confirmed that this
function was being called for each and every row the UBE processed, yet the
inputs were not changing. Therefore, the outputs were not changing…and
hence the function only needed to be called ONCE.
The solution was to move the call to DTDateTitle to the “Init” section of the
UBE and simply save the results for all further use. This resulted in the
function being called ONCE for the entire UBE, and the runtime of the UBE
being reduced by over 40%.

Note that this is a textbook example of the application of First Principles –
the problem was found by getting a profile and determining where the code
was spending its time.
The code profile also showed significant time spent in the Business Function
PAPeriodActivity. Drilling down into this function, we see its time in turn
is spent in CalculatePeriodFiscalYearOffsets:

Drilling into CalculatePeriodFiscalYearOffsets shows time spent in
GetCompanyInfo:
So GetCompanyInfo is called from a standard E1 code path:
PAPeriodActivity calls CalculatePeriodFiscalYearOffsets

which calls GetCompanyInfo. The latter is a rote, “boring” function which
just computes constant values – but it consumes a very significant portion of
this program’s time. There are at least four calls to GetCompanyInfo per
record processed. This amounts 100’s of thousands of times in the report.
There are FAR fewer companies than the number of times this gets called…
Looking at the data selection for this report (extracted from the debug log), it
is clear that one ONE single company is involved:

The C source code for CalculatePeriodFiscalYearOffsets was
examined to determine what can be done to reduce the calls to
GetCompanyInfo.
Based on this, the proposed change was to conditionalize the call to

GetCompanyInfo so that it gets called only when the company has actually
changed.

The result was that GetCompanyInfo was called only ONCE for the entire
report, reducing its runtime by a further 28%.
Once again, this Loop Invariants exercise illustrates the methodology

involved in profiling to determine where the code is spending its time, and
drilling down to determine:
“What is taking too long” and “What is called too many times”
Memory Leaks versus memory consumption
“Memory Leak” has become a catch-all term for any memory related
problem. BUT – it is NOT always an accurate description of the true problem.
Too often, it is used by those who “know enough to be dangerous”
A true memory leak is when heap memory has gone out of scope and has
not been freed by the programmer. Heap memory is dynamically allocated at
runtime by the process, and it is what causes the memory footprint of the
process to rise and fall at runtime
In C-code: at the lowest code level, pointers to heap memory get initiated by
malloc(), and released by free(). The JDE wrappers for these functions are:
jdeAlloc() / jdeFree(). Failure to call the free() API when memory
allocated by malloc() is no longer needed by the program results in a true
“leak”.
Stack / code memory, conversely, is statically allocated by the process at

initiation, and its size does not change at runtime. Arrays and other declared
variables are examples of this; their size remains static.
The way in which memory problems manifest themselves in EOne code is

usually an “Out of Memory” or other similar error in the jde.log. This means
the available process memory has been exceeded. This will appear in
manner similar to this:
A drop in the program’s throughput, which may happen in a sudden, “fall of a

cliff” fashion, will sometimes occur.

If the cause of the memory depletion is a true leak, it usually easier to fix: just
add the missing API call to free the offending memory.
But just because the process memory has been depleted does NOT mean
there is a true memory leak…instead, it may be an issue of excessive
Memory Consumption.
When memory which is built up and used by the program and meets the
following criteria:
• Remains in program scope
• Is a purposeful, “by-design” aspect of the program.

…then we do not have a true memory leak. The memory is not released
because it is legitimately being used by the program.
Note that this does NOT mean the situation is acceptable…just that it’s not a
leak. In this case, code changes or re-factoring may indeed be necessary to
reduce the memory footprint. The most common source of memory
consumption in EOne application code is the use of jdeCache APIs, which
are most commonly used for storing the contents of a database table in
memory. Details are in the jdeCache section below.
Just because an EOne job – Call Object Kernel, UBE, etc - has increased its
memory footprint after running for a period of time does NOT mean that there
is a memory leak…or even a problem! ANY E1 process will store many
values in memory: environment handles, system data, commonly used
tables. This consumes memory. The tradeoff is performance for memory
footprint: It is more efficient to retrieve values from memory than to re-fetch
them from the database or to recalculate them.
Transactional MBFs also store large amounts of data in the form of cache and
other handles & pointers, e.g.: F4211FSBeginDoc(),
F4211FSEditDoc(), etc
A memory leak will yield a memory profile which never stops increasing
Memory consumption will yield a memory profile which levels off…or

alternatively rises and falls around a steady state. BUT – the process could
still run out of memory before the steady state…and appear to be a leak. See
the picture below:

Memory
usage
“Ski Slope”
Possible Memory Leak –
Memory keeps increasing
“Mesa”
Memory Consumption –
Memory usage increases after
startup, but levels off or varies
around a rough steady state
Time
“Tools such as Visual Quantify give Finding the cause of leaks and consumption at the EOne application code
you tons of data – but JDE debug level does NOT usually require cumbersome 3rd party tools. The output from
logs give you information …. The these tools is often not for the faint of heart. They typically require re-
latter is highly underrated as a compiling with special switches and options; they often can’t be run against
means of solving problems” EOne release code
- “The Performance Whisperer”
The results are also easily misinterpreted…. the output will show possible
sources of leaks. The data has to be read and interpreted properly. The
upshot is that memory tools require some degree of skill to use properly.
.
Performance Workbench records several metrics at the beginning of the
Summary file output to assist in finding the most common sources of memory
leaks and memory consumption:
• JDB_InitBhvr() not matched by JDB_FreeBhvr()
• JDB_OpenTable() not matched by JDB_CloseTable()
• jdeCache operations:
A large number of calls to jdeCacheAdd()is the single biggest indication
of memory consumption
jdeCacheInit() not matched by jdeCacheTerminate() or
jdeCacheTerminateAll()
See the figure below an example of the Performance Workbench output

You do NOT need to run the offending program to the point of actually exposing
the failure message in the jde.log in order to determine the cause of the problem.
You need only search for the patterns which would lead to the failure
If the source of the leak / consumption issue is in EOne Tools code, it can be
more subtle, and likely to require 3rd party tools to track down. These issues
will fly under the radar of debug logs, since low-level Tools diagnostics are
not available in the EOne logs.
However, serious and disruptive problems truly originating at the EOne tools
code level are much less common. Most true memory problems which are
disruptive will be found at the application code level.
And – there are a very finite number of places in the EOne application code
which consume or leak large amounts of memory.
Note that the failure point in the code for a leak or consumption problem is
almost certainly NOT related to the cause of the problem. The first piece of
code which fails to allocate memory is a “drive by” victim; it is the shrapnel,
not the bomb.
The memory leak or consumption silently does its damage “behind the
scenes”…until another unrelated area of the code unfortunate enough to
require memory is eventually impacted. See the example below:

The cause of the leak / consumption is NOT
related to the F3002 SELECT…This is merely
the “random victim” of excessive memory
usage somewhere else…DO NOT enter a SAR
against the code which failed to allocate the
memory
Another cause of excessive memory consumption in EOne code is JDB

caching of certain tables. JDB Caching is the method of caching via the
F98613 table, using Primary Key SELECTs and the JDB_FetchKeyed
API. Normally, programs will access a small number of records in a
cached table a very large number of times. This is refereed to as Low
Cardinality.
However – some applications access a very large number of records in a

cached table. This is a High Cardinality situation.
Consider cached tables which can be LARGE, and would consume

significant program memory if cached via F98613, such as the Account
Master (F0901).
This is a bit elusive of a problem. To track it down, If none of the

following are evident in the log analysis:
• Large jdeCache build up indicated by calls to jdeCacheAdd()
• Missing
JDB_CloseTable()/JDB_FreeBhvr()/jdeCacheTermina
te()
..then a possible culprit is JDB Cache
• View the F98613 via the P98613 application

• Determine which of the tables listed in P98613 are very large
AND coincide with very heavy usage as shown by the
Performance Workbench output.
• Does the process call JDB_AddTableToDBCache() to add
additional tables? As an example, the MRP UBE caches F4801,
F4101, and F4102 via this method. This has caused problems
with memory consumption in the past, as the contents of these
tables can be several hundred megabytes in size, and a complex
MRP run can access nearly every row in all three tables.
Some possible mitigations include:
• Remove the table from cache – if performance is still acceptable
• Increase per process memory limit – if possible. More on this

below…
• Use JDB_ClearTableDBCache() to periodically flush JDB
cache – if feasible
A bit on per-process memory limits….when a “memory allocation failure”
occurs, what exactly has run out?
The short answer is that the “out of memory” condition is usually driven by a
per-process limit; the process has exceeded limits inherent to the operating
system. There is a fundamental cap on the amount of memory a process is
capable of addressing on a given operating system.
The answer is usually NOT to add more memory to the machine. The
per process limit puts a cap on how much physical memory an individual
process may use… regardless of how much memory is physically on the
machine in question
For a 32-bit operating system, a process can theoretically address up to

4GB of memory….but the devil is in the details…per each platform
supported by EOne.
AIX – per process limits
• The default per process limit for heap memory is 256 MB
• This can be increase to 2GB in increments of 256MB segments via an

environment variable (no re-boot / re-build of UNIX kernel necessary):
LDR_CNTRL
Total Number of Segments Process Memory Limit
LDR_CNTRL=MAXDATA=0x100000001 256 MB
LDR_CNTRL=MAXDATA=0x400000004 1 GB

LDR_CNTRL=MAXDATA=0x500000005 1.24 GB
LDR_CNTRL=MAXDATA=0x800000008 2 GB
• It is defined in the parent process of the process that is to be affected.

For example, the following example defines one additional data segment:
export LDR_CNTRL=MAXDATA=0x10000000
start_process
unset LDR_CNTRL
HP-UX – per process limits
• It is more difficult to tune pre-process memory limits in the HPUX

operating system.
• For 32-bit processes, HP-UX divides the 4GB of addressable memory

into 4 quadrants that can only be used for specific purposes.
quadrant 1 -- Program Text 1 Gigabyte
quadrant 2 -- Program Data 1 Gigabyte
quadrant 3 -- Global Space 1 Gigabyte
quadrant 4 -- Global Space .75 Gigabyte
• The ‘chatr’ UNIX command makes it possible to increase the amount of

Program Data space available to a process. This might be useful if the
UBE needs a little more space than the defaults.
• However, the space from the other quadrants that would be reallocated
to private data space is originally allocated to shared memory. This may
have unintended consequences for shared memory, which is used
heavily by EOne server processes.
Solaris and Windows – per process limits
For Solaris, the amount of per process heap memory is dynamic up to the
4GB 32-bit limit, so there are no parameters to adjust.
Windows had a hard limit of 2GB of memory space available to each

process.
iSeries
The iSeries uses a different memory model from the other platforms.
It employs a model knows as Single-level storage, and does not have the
4GB limit. There is a much larger limit on addressable memory.

Process (known as “jobs” on the iSeries) get their physical memory from
Memory Pools. Jobs do not have an address space as on other platforms,
and “Out of memory” errors are less common on iSeries.
Buffered Inserts
SQL INSERTs can be costly operations when performed thousands of times.
Each operation is a network roundtrip to the database
Buffered INSERTs allow For UBEs which perform bulk-loading operations, JDB middleware provides
many INSERT operations to the Buffered INSERT functionality to improve performance. Many INSERT
be delivered to the database operations can be issued to the database with a single operation. This can
in a single operation. have a very significant impact on UBEs which spent a large percentage of
their time INSERTing rows.
.
It can be a fairly simple change to implement:
• Via C-code in an EONe Business Function, there is the
JDB_SetRequestOption() API
o Use the following option:

JDB_REQUEST_OPTION_BUFFER_INSERTS
o Call this API once at the beginning of the UBE for the table in
question, most likely in the “Init” section. Then, Buffered
Inserts will be activated for the duration of the UBE process.
• Via Event Rules: there is a check-box option for buffered inserts in
ER Table I/O via the Open Table operation: “Advanced Options” :
All subsequent calls to JDB_InsertTable() for the table in question will

write INSERT rows into a buffer in memory rather than write them to the
database.
The buffer is flushed to the database when one of the following occurs:
• JDB_CloseTable() is called against the table in question
• The table is closed in Event Rules

• The buffer is “FULL”…This occurs:
When 100 rows are stored
OR
When 64KB of data is in the buffer (AS/400: 32KB)

• The JDB_DBFlushBufferInserts() API is called.
This API is usually NOT necessary. Use it to manually flush the buffer if
necessary in your algorithm
• JDB_CommitUser() API is called
Here are some caveats regarding Buffered INSERTs
• If a record must be accessed in the program after it is INSERTed,

Buffered Inserts may be of little use. This is because any given INSERT
operation will NOT write the record to the database – only to a buffer. The
record is only written to the database when the buffer is flushed. The
JDB_DBFlushBufferInserts() API may be used to make INSERTed
records available by manually forcing a buffer flush.
• Since the buffer is flushed to the database upon every

JDB_CloseTable(), it is crucial to structure the code so that a
maximum number of INSERTs occur between the Open and Close of
the table. If a table is Open()’ed and Close()’ed once for every INSERT
operation, then applying Buffered Inserts will actually hinder
performance – sometimes considerably.
The best case is to Open() and Close() the table only once:
1. JDB_OpenTable() the table
2. Perform all INSERT operations in a loop

3. JDB_CloseTable() the table
• Tables which contain BLOB records will NOT use Buffered Inserts.
Buffered INSERTs will simply be a “no-op”, even if enabled.
• Tables which contain POST INSERT Trigger functions will NOT use
Buffered Inserts – including:
F0018, F00150, F03B11, F05100, F05138, F060116, F08001, F08101,
F08102, F32943, F32944, F32945, F38111, F4211, F4818, F49631
Buffered INSERTs will simply be a “no-op”, even if enabled.

• Tables whose rows are greater than 64KB in total size are bad
candidates for Buffered Inserts. However, most EnterpriseOne tables
are significantly smaller than this.

• Error Detection and recovery can be more complex with Buffered
Inserts. This is because a failed or Duplicate Key INSERT will not be
flagged immediately; the record gets INSERTed into a buffer and does
not get written to the database.
JDE Cache APIs

The biggest single source of Memory issues at the application level of EOne usually are related to the use
memory consumption in of the jdeCache APIs. JdeCache is very practical within its intended
EOne application code is the purpose: it is a tool to store indexed data who size does not exceed a few
use of jdeCache APIs tens of thousands of rows. Within these constraints, it can provide a
response time ten times faster than the use of a work table. It is indeed used
.
quite effectively within these limits for many EOne applications. Diminishing
returns occur at larger levels (hundreds of thousands or millions of records),
so that a DBMS work table would likely yield no worse of a response time
than the cache.
This benefit does come at a memory price: jdeCache APIs are the single
biggest source of significant memory consumption in EONe applications.
There is significant memory overhead involved in the use of these APIs well
above and beyond the data being stored.
• ~850 bytes of data is stored per record per index…

in addition to the record data
• This is typically larger than the data in the record itself!
• References: an 886 byte data structure is stored for every call to

jdeCacheInit()
To improve and optimize scalability:
Remove all unneeded • Remove unneeded indexes

indexes and terminate calls to
• Terminate ALL calls jdeCacheInit() and
jdeCacheInit() and
jdeCacheInitMultipleIndex() after the first one
jdeCacheInitMultipleIn
dex() Note that the size of the record is less important than the number of
indexes in determining the memory cost of jdecache. Eliminating unneeded
.
indexes can reduce memory overhead significantly.
Consider the following:

• jdeCacheInit()/ jdeCacheInitMultipleIndex()
These APIs hard-code the number of cursors per cache to 100. Most
EOne application code only requires two or three cursors; almost none
require more than ten.

The continuous product improvement in EOne code provides a significant
mitigation for this issue in 8.96.
New jdeCache APIs are • jdeCacheInitEx()/ jdeCacheInitMultipleIndexEx()

available in 8.96:
These are new cache creation APIs available in 8.96. They
jdeCacheInitEx()/
dramatically reduce the memory overhead of jdeCache by allowing
jdeCacheInitMultipleIndexEx()
developers to specify how many cursors are needed per cache. They
. differ only by a single parameter: the cursor count. (See the figure
below) The cursor count (which cannot be changed once initialized) can
be specified to be between 1 and 100
This reduced number of cursors per cache provides:
• Significantly lower memory overhead costs
• Ability to store more records per cache
EVERY piece of new code in EVERY piece of code using 8.96 and beyond should use these new
tools release in 8.96 and cache APIs. Write any and all new code using these new cache APIs.
beyond should use these new You should be sure to have an understanding of how many cursors are
cache APIs. used – or really needed - by each specific cache.
Fixing existing code could be as simple as a “find and replace exercise”.

Specifying the number of cursors may require a bit of thought, but
consider the risks of “shooting from the hip” and just using a low value
such as ‘5’ or ’10’. In most cases – there’s probably very little danger of
destabilizing the code.
If you use the new APIs, but hardcode a ’50’ in the cursor parameter, this
only decreases the memory consumption by a factor of 2, leaving far too
much memory reduction on the table. The new APIs will not provide any
memory reduction benefits - unless this ability to specify a greatly
reduced number of cursors is actually used.
This is the single biggest thing that can be done to reduce memory
consumption – and improve performance – across a large number of
UBEs and interactive apps.
Pass the number of

cursors as a
paremeter:
nSizeCursors
ENTERPRISEONE PERFORMANCE / Conclusions
This paper has given an introduction to the notion of Performance First
Principles and its importance to the software performance analysis process.
It has also enumerated several specific performance items relevant to

EnterpriseOne applications:
• UBEs on the client: A BAD Idea
• Use C code Business Functions – NOT Event Rules or Named Event

Rules
• Indexes are NOT the answer to everything
• Loop Invariants
• Memory Leaks versus memory consumption
• Buffered Inserts
• JDE Cache APIs
These items must be incorporated into the First Principles analysis process,
not applied in a vacuum. NEVER make a fix (or enter a SAR) which will
involve a great deal of time and effort without having evidence that it will
impact the time being spent in the application in question.
The process is part Art, part Science, IS NOT “Plug and Chug”, IS NOT
reading data from a report, matching it up to a set of axioms, and tossing it
over a wall.
It IS knowing the principles and applying them. It is a thinking activity of

interpreting / analyzing the data with the help of developers who wrote the
code. (or BY the developer who wrote the code)
It IS understanding that a particular “pattern” might be a problem in one

application but not another
It IS an iterative process. You get better at it by doing it more.

[November] 2009
Author: [OPTIONAL]
Contributing Authors: [OPTIONAL]
Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
Copyright © 2009, Oracle. All rights reserved.

This document is provided for information purposes only and the
contents hereof are subject to change without notice.
This document is not warranted to be error-free, nor subject to any
other warranties or conditions, whether expressed orally or implied
in law, including implied warranties and conditions of merchantability
or fitness for a particular purpose. We specifically disclaim any
liability with respect to this document and no contractual obligations
are formed either directly or indirectly by this document. This document
may not be reproduced or transmitted in any form or by any means,
electronic or mechanical, for any purpose, without our prior written permission.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Topics in EOne Applications Performance

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Topics in EOne Applications Performance

Uploaded by

Copyright:

Available Formats

Topics in EnterpriseOne

This document provides considerations when reviewing system performance relative to

Performance and Tuning Title Page i

EnterpriseOne Performance / Introduction................................................. 3

Topics in EnterpriseOne Performance Page 2

ENTERPRISEONE PERFORMANCE / INTRODUCTION

Assumed is a basic knowledge of EOne, relational database concepts, the

ENTERPRISEONE PERFORMANCE / TOPICS IN PERFOMANCE

Software Performance First Principles

WHERE IS THE CODE SPENDING ITS TIME?

This breaks down into two basic sub-principles:

Topics in EnterpriseOne Performance Page 3

Of course – if the solution to a given problem is NOT on an existing “top ten”

Topics in EnterpriseOne Performance Page 4

An example of a faulty BOTTOMS UP technique would be collecting

Topics in EnterpriseOne Performance Page 5

As an example, consider analysis of a UBE which you know INSERTs a large

• However – armed with the knowledge of Performance First Principles,

Topics in EnterpriseOne Performance Page 6

UBEs on the client: a BAD idea

To put it another way:

If I have a sore throat ….

• Because it’s easiest

• Because they know how to work in a client environment, but not a

• Because they think they can’t disrupt the server

• Because they think they’d have to bounce E1

Topics in EnterpriseOne Performance Page 7

• There is no need to modify the jde.ini

• There is no need to bounce EOne services

Here are the reasons:

1. Many features unique to UBEs are DISABLED in a fat client

• JDEKRNL code: 1099 places

• JDVDRV code: 64 places

• JDELIBE code: 28 places

• This makes it easy to falsely conclude that I/O latency is a

• Complex UBEs containing large Master Business Functions

• Even mapping BSFNs to the server to mitigate the DB I/O issue

• The above is especially true for iSeries

Topics in EnterpriseOne Performance Page 8

• SQL statements can be built and executed differently on the fat

• Driver version mismatches can also occur

5. Fat client is not even supported starting in 8.11

• Analysis should NOT occur against unsupported

• Note that “Not supported” means no testing has occurred. Quality

• You will get a highly confusing, obfuscated profile, with plenty of

Topics in EnterpriseOne Performance Page 9

7. The multiple threads on the win32 client environment make the

• Any factors rooted in operating system specifics will be missed if

• The Windows multi-user server environment differs from the single-

• On server – if ER debugging is enabled (UBEDebugLevel > 0) – the

Topics in EnterpriseOne Performance Page 10

Use C code Business Functions – NOT Event

Topics in EnterpriseOne Performance Page 11

• BUT – NER functionality and maintainability is limited to what is

• The ER SELECT interface (“Table I/O”) can only generate a

Note that a possible alternative to this is the use of EOne VIEWs.

Topics in EnterpriseOne Performance Page 12

• Even though the result of the NER generation process is

The message is that NER functionality is intended for power users to

Indexes are NOT the answer to everything

Topics in EnterpriseOne Performance Page 13

Indexes should ALWAYS be added one at a time and deliberately – one

• Reducing a 20 minute query to 5 seconds…