C‐Based External Procedures

In the beginning, PL/SQL, and only PL/SQL, was used as a language with which to
program the Oracle server, and Oracle client applications, within the server
itself. Starting
with version 8.0 of Oracle, the ability to implement stored procedures in other
languages
was introduced. This feature is called external procedures and covers both C (or
anything
that is C callable) and Java‐based stored procedures. In this chapter, we will
focus
exclusively on C; the next chapter covers Java.
This chapter will cover external procedures from an architectural perspective,
showing
you how they have been implemented by the Oracle kernel developers. Additionally,
we
will see how you must configure your server to support them, and how you should
configure your server for safety reasons. We will demonstrate how to write an
external
procedure using Oracleʹs Pro*C. This external procedure will be used to write the
contents
of any LOB to the serverʹs file system. Before we get to that example though, we
will work
through a ʹprimerʹ example that demonstrates how to send and receive all
interesting
datatypes to and from PL/SQL and C. This primer will also develop a generic
template
useful for developing all subsequent C based external procedures quickly. We will
cover
how the C code should be implemented, with a mind towards being able to support
this
server‐side code. We will also look at the implementation of the SQL binding we
create for
this external procedure. We will learn how we make it callable from SQL and PL/SQL,
and
how we should code the binding layer for ease of use on part of the people who will
actually use our developed code. Lastly, we will cover the pros and cons of
external
procedures, and cover any miscellaneous server errors (ORA‐XXXX errors) you might
encounter.
The Pro*C example that will be developed in this chapter supplies a missing piece
of
server functionality. Oracle supplies the package DBMS_LOB to manipulate large
objects
(LOB). Within this package is a procedure, loadfromfile, which reads an arbitrary
file from
the OS file system, into the database and stores it there. However, they do not
supply a
function, writetofile, to take the contents of a LOB and write it to the OS ‐ a
common
requirement. We will rectify this issue, and create for ourselves a LOB_IO package.
This
package allows us to write any BLOB, CLOB, or BFILE object to a separate file
outside of
the database (so for BFILEs we have effectively given ourselves a copy command
since the
source BFILE is already located outside of the database).
When Are They Used?

A single language or environment is not capable of providing every single feature

and
function you could ever want. Every language has shortcomings, things it cannot do,
which you really want to do ‐ there are features missing, things overlooked. When
Iʹm
writing in C, sometimes Assembler comes in handy. When Iʹm writing in Java,
sometimes
PL/SQL comes in handy. The point being that you donʹt always drop down to a ʹlower
levelʹ language ‐ sometimes you take a step up to a higher level. External
procedures
would be considered a ʹstep downʹ to a lower level language. You will typically use
them
to integrate existing C callable code you already have (a DLL ‐ Dynamic Link
Library on
Windows; supplied by some third party you would like to call from the database), or
to
extend the functionality of some existing packages, as we are. This is the same
technology
Oracle itself uses to extend the serverʹs capabilities ‐ for example we saw how
interMedia
makes use of this feature in the previous chapter, and in Chapter 13 on
Materialized Views
we saw how DBMS_OLAP makes use of this capability.
The first external procedure I wrote was a ʹsimple TCP/IPʹ client. With it, in
version 8.0.3 of
the database, I gave PL/SQL the ability to open a TCP/IP socket to an existing
server and
send and receive messages. The server I could connect to could have been a Net News
Transport Protocol (NNTP) server, an Internet Message Access Protocol (IMAP)
server, a
Simple Mail Transfer Protocol (SMTP) server, a Post Office Protocol (POP) server, a
web
server, or so on. By ʹteachingʹ PL/SQL how to use a socket, I opened a whole new
spectrum
of opportunities. I could now:
• Send e‐mail from a trigger using SMTP
• Incorporate e‐mail into the database using POP
• Index a newsgroup through interMedia text using NNTP
• Access virtually any network based service available to me
Instead of thinking of the server as a server, I started thinking of it as a client
‐ a client of all
of these other servers. Once I got their data into my database, I could do a lot of
things
with it (index it, search it, serve it up in a different way, and so on).
Over time, this became such a frequently used tool that it is now an integrated
feature of
the database. Starting with release 8.1.6 of the Oracle server, all of the services
that had
been provided in the simple TCP/IP client are now implemented in the UTL_TCP
package.
Since that time, I have written a couple of other external procedures. Some to get
the
system clock down to a finer resolution than the built‐in SYSDATE function would
return,
others to execute operating system commands, or get the systemʹs timezone, or to
list the
contents of a directory. The most recent one that I have developed, and the one
that we
will explore here, is a function to take the contents of any LOB, be it a Character
LOB
(CLOB), Binary LOB (BLOB), or BFILE, and write it to a named file. Another nice
side
effect of this package will be functionality similar to UTL_FILE, but for binary
files (which
UTL_FILE cannot generate). Since the server supports the concept of a Temporary
LOB,

and has functions to WRITE to a Temporary LOB, this new package we are going to
implement will give us the ability to write an arbitrary binary file from PL/SQL.
So in
short, what we will get from this package is:
• The capability to export any LOB to an external file on the server.
• The capability to write any binary file of up to virtually any size with any data
(this
is similar to UTL_FILE which works with text data, but fails with arbitrary binary
data).
As you can see by some of the examples above, the reasons for using an external
procedure can be many and varied. Typically, you are using it to:
• Supply missing functionality.
• Integrate existing code, perhaps from a legacy system that performs data
validation.
• Speed up your processing. Compiled C will be able to perform some
computationally expensive operation faster than interpreted PL/SQL or Java.
As always, the choice to use something like external procedures comes with certain
cost.
There is the cost of developing the code in C, which is more complex , in my
opinion, than
developing in PL/SQL. There is the cost of portability ‐ or the potential inability
to be
portable. If you develop a DLL on Windows, there is no guarantee that source code
you
wrote would function on a UNIX machine, or vice versa. Iʹm of the opinion that you
should only use an external procedure when the language (PL/SQL) gives you no other
opportunity.
How Are They Implemented?
External procedures run in a process physically separate from the database. This is
for
reasons of safety. While it would be technically possible for the existing database
server
processes to dynamically load your DLL (on Windows) or .so (Shared Object code on
Solaris) at run‐time, it would expose the server to unnecessary risk. Your code
would have
access to the same memory space as the server processes do, and this might include
areas
of memory such as the Oracle SGA. This could allow developed code to accidentally
corrupt, or otherwise damage, kernel data structures, possibly leading to loss of
data, or a
crashed database instance. In order to avoid this, external processes are executed
as a
separate process that does not share those memory areas with the server.
In most cases, the separate process would be configured to execute as a user other
than the
Oracle software account. The reason for this is much the same as why they are run
in a
separate process ‐ safety. For example, we are going to create an external
procedure that is
capable of writing files to disk (as the example we will develop below does). Letʹs
say you
are on UNIX, and the external procedure is executing as the Oracle software owner.
Someone calls your new function and asks to write a BLOB to
/d01/oracle/data/system.dbf.

Since the Oracle software owner is the user ID executing this code, we will be able
to do
this, thus inadvertently overwriting our system tablespace with the contents of
some
BLOB. We might not even notice that this happened until we shutdown and restarted
our
database (many days later). If we had run the external procedure as some less
privileged
user, this could not happen (that user would not have WRITE on the system.dbf
file). So,
for this reason, when we get to the section on configuring the server for external
procedures, weʹll find out how to set up a ʹsafeʹ EXTPROC (EXTernal PROCedure)
listener
that runs under a different OS account. This is very similar to why web servers
typically
execute as the user nobody on UNIX, or in some low‐privileged account in Windows.
So, when you invoke an external procedure, Oracle will create an OS process called
EXTPROC for you. It does this by contacting the Net8 listener. The Net8 listener
will create
the EXTPROC process for us in much the same way as it spawns dedicated servers or
shared servers. This can be seen in Windows NT by using the NT Resource Toolkit
utility
tlist to print a tree of processes and subprocesses. For example, I started a
session that
accessed an external procedure, and then issued tlist ‐t, and I see the following:
C:\bin>tlist ‐t
System Process (0)
System (8)
smss.exe (140)
csrss.exe (164)
winlogon.exe (160)
services.exe (212)
svchost.exe (384)
SPOOLSV.EXE (412)
svchost.exe (444)
regsvc.exe (512)
stisvc.exe (600)
ORACLE.EXE (1024)
ORADIM.EXE (1264)
TNSLSNR.EXE (1188)
EXTPROC.EXE (972)
lsass.exe (224)
...
This shows that the TNSLSNR.EXE process is the parent process of the EXTPROC.EXE.
The EXTPROC process and your server process can now communicate. More importantly,
the EXTPROC process is able to dynamically load your developed DLL (or .so/.sl/.a
file on
UNIX).
So, architecturally it looks like this:

Where:
1. You have your initial connection to the database. You are running in either your
dedicated server, or some shared server process.
2. You make a call to an external procedure. Since this is your first call, your
process
will get in touch with the TNSLISTENER (Net8 listener) process.
3. The Net8 listener will start (or find in a pool of already started) an external
procedure process for you. This external procedure will load the requisite DLL
(or .so/.sl/.a file on UNIX).
4. You can now ʹtalkʹ to the external procedure process, which will marshal your
data
between SQL and C.
Configuring Your Server
We will now cover the requisite setup we must do to enable an external procedure to
execute. This will involve setting up the LISTENER.ORA and TNSNAMES.ORA files on
the server, not on the client machine. Upon install, these files should have
automatically
been configured for you with the external procedure (EXTPROC) services. If so, your
LISTENER.ORA configuration file will resemble:
# LISTENER.ORA Network Configuration File:
C:\oracle\network\admin\LISTENER.ORA
# Generated by Oracle configuration tools.

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = tkyte‐del)(PORT = 1521))
)
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
)

)
)

SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = C:\oracle)
(PROGRAM = extproc)
)
(SID_DESC =
(GLOBAL_DBNAME = tkyte816)
(ORACLE_HOME = C:\oracle)
(SID_NAME = tkyte816)
)
)
These are the important things in the listener file for external procedures:
• (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1)) ‐ Set up an IPC‐based
address. Remember the value of the KEY. You can make it whatever you want; just
remember what it is. On various operating systems, the KEY is case sensitive as
well keep that in mind.
• (SID_DESC = (SID_NAME = PLSExtProc,) ‐ Remember the SID_NAME, call it
PLSExtProc or something similar. By default, this SID will be configured to be
PLSExtProc.
You may configure your LISTENER.ORA file manually using a plain text editor, or
using
the Net8 Assistant. It is strongly recommended that you use the Net8 assistant as
the
slightest configuration file error ‐ such as a mis‐matched parenthesis ‐ will
render these
configuration files useless. If using the Net8 Assistant, follow the steps outlined
in the
online help under NetAssistantHelp, Local, Listeners, How To.., and finally
Configure
External Procedures for the Listener.
After modifying the LISTENER.ORA file remember to stop and start your listener
using
the commands Lsnrctl stop and Lsnrctl start on the command line.
The next file is the TNSNAMES.ORA file. This file needs to be in the directory that
the
server will use to resolve names. Typically, a TNSNAMES.ORA file is found on a
client as
it is used to find the server. This is one case where the server needs it to find a
service itself.
The TNSNAMES.ORA file will have an entry similar to:

# TNSNAMES.ORA Network Configuration

File:C:\oracle\network\admin\TNSNAMES.ORA
# Generated by Oracle configuration tools.

EXTPROC_CONNECTION_DATA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
)
(CONNECT_DATA =
(SID = PLSExtProc)
(PRESENTATION = RO)
)
)
These are the important things in this configuration:
• EXTPROC_CONNECTION_DATA ‐ This is the service name the database will be
looking for. You must use this name. See the caveat below with regards to the
names.default_domain setting in your SQLNET.ORA configuration file.
• (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1)) ‐ This should be the same as
in the LISTENER.ORA file. In particular, the KEY = component must match.
• (CONNECT_DATA =(SID = PLSExtProc), ‐ The SID = must match the SID in the
(SID_DESC = (SID_NAME = PLSExtProc) from the LISTENER.ORA.
The following is a caveat on the EXTPROC_CONNECTION_DATA name. If your
SQLNET.ORA specifies some default domain, it needs to be on the TNSNAMES entry. So
if you have a SQLNET.ORA with a setting such as:
names.default_domain = world
you would need to specify EXTPROC_CONNECTION_DATA.world, not just
EXTPROC_CONNECTION_DATA in the TNSNAMES.ORA file.
Any errors in the above configuration files will almost certainly lead to the error
ORA‐
28575 shown below:
declare
*
ERROR at line 1:
ORA‐28575: unable to open RPC connection to external procedure agent
ORA‐06512: at ʺUSERNAME.PROCEDURE_NAMEʺ, line 0
ORA‐06512: at line 5

Upon receiving this error, the steps that might be useful to resolve this error are
as follows:
• Verify the extproc program is available and executable.
• Verify the database environment is properly set up for extprocs.
• Verify the listener is correctly configured.
Weʹll take an in depth look at each of these steps now. They assume you have
actually hit
this error for some reason.
Verify the extproc Program
The first sanity check after youʹve configured extprocs and received the ORA‐28575
is to
verify the existence and execution of the extproc program itself. This is easily
done from
the command line in either Windows or UNIX. You should do this when logged in with
the credentials of the user that will be starting the listener (since that is the
process that will
execute the extproc), to verify that execute permissions for this user are in
place. You will
simply do the following:
C:\oracle\BIN>.\extproc.exe

Oracle Corporation ‐‐‐ SATURDAY AUG 05 2000 14:57:19.851

Heterogeneous Agent based on the following module(s):

‐ External Procedure Module

C:\oracle\BIN>
You are looking for output similar to the above. Note that I have run this from the
[ORACLE_HOME]\bin directory as this is where we would find the extproc.exe program.
If you cannot run this program, that would be an indication of a corrupted
installation or
some operating system configuration that must be corrected.
Verify the Database Environment
There are a couple of things to check in the database environment. First and
foremost is to
verify that the correct TNSNAMES.ORA is being used, and that it is configured
correctly.
For example, on my UNIX machine, using truss I can see that:
$ setenv TNS_ADMIN /tmp

$ truss sqlplus /@ora8i.us.oracle.com |& grep TNSNAMES

access(ʺ/export/home/tkyte/.TNSNAMES.ORAʺ, 0) Err#2 ENOENT

access(ʺ/tmp/TNSNAMES.ORAʺ, 0) Err#2 ENOENT

access(ʺ/var/opt/oracle/TNSNAMES.ORAʺ, 0) Err#2 ENOENT
access(ʺ/export/home/oracle8i/network/admin/TNSNAMES.ORAʺ, 0) = 0
...
So, Oracle looked in:
• My home directory for a TNSNAMES.ORA file.
• The directory specified by the environment variable TNS_ADMIN.
• The /var/opt/oracle directory.
• And lastly $ORACLE_HOME/network/admin/.
for a TNSNAMES.ORA file. A common mistake is to configure the TNSNAMES.ORA file
in the [ORACLE_HOME]/network/admin directory, but have a TNS_ADMIN
environment variable that causes Oracle to find a different TNSNAMES.ORA elsewhere.
So, make sure you have configured the correct TNSNAMES.ORA (if there is any
confusion,
you can simply set the TNS_ADMIN environment variable before starting the server;
this
will ensure the copy you want Oracle to use is in fact being used).
Once you have verified that the correct TNSNAMES.ORA is being used, we can look for
configuration errors in that file. You would follow the steps outlined above to
make sure
the (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1)) and (CONNECT_DATA =(SID
= PLSExtProc)) components are correctly set up. You do this by comparing them to
the
LISTENER.ORA configuration. If you use the Net8 Assistant to set this up, you will
not
have to concern yourself with matching parentheses. If you do it manually, be very
careful.
One mismatched, or out of place, parenthesis will make it impossible to use an
entry.
Once you have verified these settings appear correct, we will look at the
TNSNAMES.ORA entry name used. It must be EXTPROC_CONNECTION_DATA. It
cannot be anything else, although it may have a domain appended to it. Verify the
spelling
of this entry. Check your SQLNET.ORA configuration file at this time. Oracle looks
for the
SQLNET.ORA file in the same way it looks for the TNSNAMES.ORA file. Beware that it
does not have to be in the same directory as the TNSNAMES.ORA file ‐ it may be in
one of
the other locations. If it either of:
• names.directory_path
• names.default_domain
are set, we must verify that we are in compliance with them.
If we have the names.default_domain set for example to (WORLD), we must ensure this
domain appears in the TNSNAMES.ORA entry. Instead of
EXTPROC_CONNECTION_DATA, it must be
EXTPROC_CONNECTION_DATA.WORLD.