Environmental Modelling & Software 26 (2011) 434e443

Contents lists available at ScienceDirect

Environmental Modelling & Software

journal homepage: www.elsevier.com/locate/envsoft

Overview of the atmospheric model evaluation tool (AMET) v1.1 for evaluating
meteorological and air quality models
K. Wyat Appel a, *, Robert C. Gilliam a, Neil Davis b, Alexis Zubrow b, Steven C. Howard a
a
Atmospheric Modeling and Analysis Division, National Exposure Research Laboratory, Office of Research and Development, U.S. Environmental Protection Agency, RTP,
NC 27711, USA
b
University of North Carolina - Institute for the Environment, Chapel Hill, NC 27599, USA

a r t i c l e i n f o a b s t r a c t

Article history: This paper describes the details of the Atmospheric Model Evaluation Tool (AMET) v1.1 created by
Received 26 May 2010 scientists in the Atmospheric Modeling and Analysis Division (AMAD) of the U.S. Environmental
Received in revised form Protection Agency (EPA). AMET was first developed to evaluate the performance of the 5th Generation
7 September 2010
Mesoscale Model (MM5) and the Weather Research and Forecasting (WRF) meteorological model output
Accepted 30 September 2010
Available online 28 October 2010
and was later extended to include capabilities for evaluating output data from the Community Multiscale
Air Quality (CMAQ) model as well. AMET is designed to leverage several open-source software packages
that are used in combination to 1) pair the modeled and observed values in time and space, 2) store these
Keywords:
Air quality model evaluation
paired datasets in an easily accessible and searchable database and 3) access and analyze these data
Meteorological model evaluation using a statistical package. Through this process, AMET is able to provide a convenient method for
CMAQ evaluating meteorological and air quality model predictions. The use of a searchable, relational database
WRF allows the entire dataset to be quickly subset into only those data that are of the most interest for the
MM5 current analysis, a process that is often tedious and time consuming without the use of a database. In
addition to common summary statistics (e.g. RMSE, bias, and correlation), several of the many analysis
features available in AMET include scatter plots, time series plots, box plot and spatial plots as part of
operational model evaluation. Additionally, several unique analysis functions are also available in AMET,
and the system provides a framework within which users may extend the current functionality for their
own custom analyses. While AMET was designed to work specifically with MM5, WRF and CMAQ model
output, it could easily be modified to work with output data from other meteorological and air quality
models.
Published by Elsevier Ltd.

Software Availability Year first available: 2007

Software name: Atmospheric Model Evaluation Tool (AMET)
Developers: Robert C. Gilliam and K. Wyat Appel, Neil Davis and
Alexis Zubrow
Address: 109 T.W. Alexander Drive, Research Triangle Park, NC
27711
Email Addresses: appel.wyat@epa.gov, gilliam.robert@epa.gov
Software Required: MySQL Database Software (http://www.mysql.
com) R statistical software (http://www.r-project.org)
Program Language: Perl; Fortran;
Availability: Available for download through the Community Model
and Analysis System (CMAS) website (http://www.
cmascenter.org)
1. Introduction

Evaluation of model simulations, which usually includes

Abbreviations: Atmospheric Model Evaluation Tool, AMET; Weather Research
and Forecasting, WRF; 5th Generation Mesoscale Model, MM5; Community Mul-
tiscale Air Quality, CMAQ; Atmospheric Modeling and Analysis Division, AMAD;
Environmental Protection Agency, EPA; Community Modeling and Analysis System,
CMAS; Meteorological Assimilation Data Ingest System, MADIS; Meteor-
ologyeChemistry Interface Processor, MCIP.
* Corresponding author. US EPA, 109 TW Alexander Dr., Research Triangle Park,
NC 27711. Tel.: þ1 91 954 10757; fax: þ1 91 954 11379.
E-mail address: appel.wyat@epa.gov (K. Wyat Appel).
comparing the model predicted values against the available
observed values, is critical to establishing the model’s credibility
(Dennis et al., 2010). For the development of the Community
Multiscale Air Quality (CMAQ; Byun and Schere, 2006; Foley et al.,
2010), evaluation of model predicted concentrations for many
different atmospheric pollutants is required since the model is used
to make regulatory decisions by the U.S. Federal and State

1364-8152/$ e see front matter Published by Elsevier Ltd.

doi:10.1016/j.envsoft.2010.09.007
 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443 435

The AMET Flow Chart

Observations Model Output Model

Evaluation R Scripts

• Meteorological • MM5/WRF (MET) Database

Domain Statistics
• Air Quality • CMAQ (AQ)
MySQL relational Diurnal Statistics
database stores all
Observation-Model model-observation Time Series Plots
Synchronization pairs
Spatial Plots
Match observations with model Convienient method
predictions in time and space to store data from a Box Plots
(Fortran) large number of
model simulations Scatter Plots

Generate database records Bar Plots

(Fortran)
Other, User-developed tools “Soccer Goal”
Plots
Open source format allows for
Populate Database user community development “Bugle” Plots
(Perl) of new tools and analysis
techniques Histograms

Fig. 1. Flow chart showing the basic structure of the AMET meteorological and air quality modules.

Table 1
List of R scripts currently available in the AMET meteorological and air quality modules.

Script Name AMET Module Description

run_met_aq Meteorological/Air Quality Compares air quality and meteorological data. Creates a spatial plot of correlation and
a time series plot of bias.
run_spatial_surface Meteorological Creates a series of maps comparing surface observations and model values.
run_summary Meteorological Creates a summary plot of a series of meteorological variables. Includes time series,
scatter and box plots.
run_time series Meteorological Creates a time series plot of temperature, mixing ratio, wind speed and direction for one
or more sites.
run_daily_barplot Meteorological Creates a series of daily bar plots for all sites in the domain.
run_plot_prof Meteorological Creates time-height plots of wind speed and direction profile statistics.
run_plot_raob Meteorological Creates vertical profile plots of rawinsonde sites.
run_wind_prof Meteorological Creates wind arrow profiles for a set of specific monitoring sites.
run_boxplot Air Quality Creates a box plot of observed and modeled quartiles.
run_boxplot_hourly Air Quality Creates a 24-h diurnal box plot of hourly data.
run_bugleplot Air Quality Creates a “bugle plot”, which plot bias and error as a function of concentration and includes
curves representing performance goals.
run_pave_overlay Air Quality Creates a PAVE overlay file of observed values.
run_plot_spatial Air Quality Creates spatial plots of the observed values, model values, and the difference between the two.
run_scatterplot Air Quality Creates a scatter plot of observed versus modeled values and includes select summary statistics
on the plot.
run_scatterplot_MtoM Air Quality Creates a model-to-model scatter plot at the selected network sites.
run_scatterplot_single Air Quality Creates a scatter plot for sites from a single network and includes many statisics on the plot.
run_scatterplot_skill Air Quality Creates a scatter plot that includes forecast skill metrics for ozone.
run_soccerplot Air Quality Creates a “soccer goal” plot, which plots bias verses error with performance lines in the shape of a
soccer goal.
run_stacked_barplot Air Quality Creates a stacked bar plot of total PM2.5 mass from CSN data and color codes the contribution from sulfate,
nitrate, ammonium, total carbon and other PM mass.
run_stats_plots Air Quality Creates plots of NMB, NME, FB, FE and correlation, along with a comma delimited file with various statistics.
run_time series Air Quality Create a time series plot for a site or multiple sites (averaged) for a select time period.
436 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443

Fig. 2. Sample plots from the run_summary script. The plot on the left is for temperature, while the plot on the right is for wind direction.

Fig. 3. Sample plot from the run_plot_prof script from the AMET meteorological module. The color shading and white lines indicates model-predicted wind speed, while the black
dots indicate the model-predicted planetary boundary layer height.
 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443 437

Fig. 4. Sample bias plots from a) the run_spatial_surface script from the AMET meteorological module and b) the run_stats_plots script from the AMET air quality module.
438 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443
Fig. 5. Sample plots from the a) run_scatterplot script and b) run_scatterplot_skill
script available in the AMET air quality module. The run_scatterplot script can be used
for any species, while the run_scatterplot_skill script was designed specifically for
assessing model performance for ozone.
governments. The evaluation is further complicated by the
requirement of evaluating the meteorological predictions that are
used to provide gridded meteorological data to the CMAQ model.
Less than a decade ago, CMAQ model simulations typically
spanned only several weeks to a month (in some rare cases several
months) and were limited spatially (e.g. eastern half of the United
States). The increase in computing power over time has resulted in
larger model domains, longer simulation time periods and a greater
number of model simulations (e.g. model sensitivities). Today,
CMAQ simulations routinely span several months to a year and in
some cases multiple years, and have expanded spatially to cover the
continental United States and the Northern Hemisphere. The result
are terabytes of data available for analysis against observed data
which conventional methods of data analysis (e.g. spreadsheets) are
not well suited. In addition to longer duration simulations and larger
domains, the shorter wall-clock model run times have resulted in
many more model simulations being performed. The increase in the
duration, spatial size and number of model simulations all require
greater organization and efficiency to systematically evaluate the
large amounts of data being produced. While some model evalua-
tion software tools already exist (e.g. the Meteorological Evaluation
Tool available for evaluating WRF simulations), there are limited
free, open-source software tools available to perform comprehen-
sive meteorological and air quality model evaluations.
The Atmospheric Model Evaluation Tool (AMET) was developed by
the United States Environmental Protection Agency’s (EPA) Atmo-
spheric Modeling and Analysis Division (AMAD) to aid in its evalua-
tion of the meteorological and air quality models within the CMAQ
modeling system. Since its initial development, AMET has been
updated and maintained through the Institute of the Environment at
the University of North Carolina at Chapel Hill. AMET organizes,
provides consistency and speeds-up the evaluation process for
operational meteorological and air quality model simulations.
Although it was developed specifically to aid in the evaluation of the
CMAQ modeling system, the AMET software can be easily modified to
work with other modeling systems.
This paper describes the structure and functionality of the AMET
software. A broad description of the AMET software will be
provided, followed by a detailed description of how the AMET
software works, including a description of process by which
observed and modeled values are paired, how those paired data are
organized and stored in the database, and an overview of the
analysis scripts currently available in AMET. Finally, examples of the
analysis products available from the AMET software will be pre-
sented and described.
2. Tool description
AMET is used to pair observations and gridded model output in
space and time and to evaluate the model performance for a set of
predicted or diagnosed atmospheric fields. There are separate
modules in AMET for evaluating meteorological and air quality model
output. This separation is necessary because both the observed and
predicted meteorological and air quality data are quite different,
utilizing different file formats for both the observed and model data.
In addition, the observed meteorological and air quality data are often
obtained from networks that use different sampling protocols, which
can make pairing meteorological and air quality data difficult. One
advantage of separate meteorological and air quality modules in
AMET is that the modules can be installed individually, allowing
a user to reduce installation time and complexity if only meteoro-
logical or air quality analysis is required. Although the meteorological
and air quality evaluations use separate modules, there is an analysis
tool available in the software that allows for cross-comparison of the
meteorological and air quality data.
AMET is built on several open-source software packages and
uses the MySQL (http://www.mysql.com) database software to
store the paired observation and model output along with various
metadata. The pairing is performed by several scripts, which also
insert the paired data into the database using Fortran and Perl
programs. Finally, the analysis of the data utilizes the R statistical
software package (http://www.r-project.org/). The AMET software
includes cshell scripts which help streamline the installation
process, including setting up the database structure for use with
AMET. Fig. 1 presents a flow chart of the basic structure of the AMET
software.
 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443
Fig. 6. Sample plots from the run_timeseries script available in the AMET air quality module. The top plot shows the observed (black open circles; from the IMPROVE network in
this example) and CMAQ predicted (red open triangles) concentrations. The bottom plot shows the time series of model bias (blue open circles) and error (red open triangles). The
dashed line represents the average error for the period.
2.1. Pairing observations and modeled outputs
The first step required as part of the AMET software is to pair the
observed and modeled values in space and time. In the air quality
module of AMET, this is accomplished through the Site Compare
software, which is a Fortran program released as a tool along with
the CMAQ model and is available for download through the
Community Modeling and Analysis System (CMAS; http://www.
cmascenter.org/) website (packaged as part of AMET). Site
Compare performs matching using the nearest neighbor approach,
and applies temporal averaging to the modeled data if required (e.g.
if an hourly CMAQ output file and daily average observation are
provided Site Compare will compute a daily averaged CMAQ value
for the correct location to match with the observational data). The
software provides a means of matching modeled and observed
species with different names and combining the modeled or
observed species to create derived species (e.g. total carbon aerosol
from elemental and organic carbon). In addition to the Site Compare
software, text files containing the site information (site name,
location, time-zone, etc.) for each network are required and are
provided for several U.S. networks. The Site Compare software is run
separately for each observation network and individual comma
separated files are created for each network which contain the
paired observed and model data for any or all (user specified) of the
species available from that network. The paired data are then
inserted into the MySQL database using a Perl script.
439
In most cases, the Site Compare program is able to read the
observation data files that are provided directly from the major air
quality networks in the United States. The air quality networks that
Site Compare/AMET currently supports are the Chemical Speciation
Network (CSN; previously STN), the Interagency Monitoring of
PROtected Visual Environments (IMPROVE; http://vista.cira.
colostate.edu/improve/) network, the Clean Air Status and Trends
Network (CASTNET; http://www.epa.gov/castnet/), the National
Atmospheric Deposition Program (NADP; http://nadp.sws.uiuc.edu/)
networks and the SouthEastern Aerosol Research and CHaracter-
ization Study experiment (SEARCH; http://www.atmospheric-
research.com/studies/SEARCH/index.html). Some data obtained
through the EPA’s Air Quality System (AQS) may require a pre-
processor that is included with the Site Compare software. Users
should obtain the observation data directly from the various network
resources and then refer to the AMET user’s guide for instructions on
how use those data with AMET.
For the meteorology module of AMET, all pairing of observations
with model data is done using Perl. The model output file is probed
for time/date information and grid projection. The Perl script cycles
through each model output time, extracts corresponding observa-
tions, determines the location of the observation sites relative to the
model grid, pairs the observations with the model values and inserts
the paired data into the MySQL database. The meteorological
matching is performed using either bilinear interpolation or nearest
neighbor. Additionally, AMET attempts to match near-coastal
440 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443
Fig. 7. Sample plots from the a) run_boxplot script and b) run_boxplot_hourly script
available in the AMET air quality module. The shading represents the inter-quartile
range of the observations and model predictions, while the lines represent the median
of the observations and model predictions.
monitors with land based grid-cells as opposed to oceanic grid-cells,
and both the model output and observations are considered to be on
identical temporal resolutions (i.e. instantaneous values represent-
ing 1 h). The meteorological module is designed to use observations
from the Meteorological Assimilation Data Ingest System (MADIS;
http://madis.noaa.gov). The MADIS is a widely used, comprehensive
meteorological observational dataset, and requires a free subscrip-
tion to access the observation data. Once a subscription to the MADIS
has been established, AMET can either automatically download the
observation data for the period of the model simulation, or the data
can be retrieved prior to the execution of the script.
Currently, the meteorological model of AMET is designed to work
with meteorological output from either the 5th Generation Mesocale
Model (MM5; Grell et al., 1994), the Weather Research and Fore-
casting (WRF; Skamarock et al., 2008) model or the Meteor-
ologyeChemistry Interface Processor (MCIP; Otte and Pleim, 2010). A
pre-processor is required for MM5 model output to be used with
AMET, however no pre-processor is required for the WRF or MCIP
output as those data can be read directly by the Perl module. The
AMET meteorological module supports evaluation of both surface
and upper-air meteorological fields (surface only for MCIP data). The
surface fields currently available in the AMET meteorological module
include temperature, water vapor mixing ratio, wind speed, and wind
direction. The upper-air fields available for analysis are wind speed
and direction from vertical wind profilers; wind speed and direction
and potential temperature from aircraft profile (ACARS) data; and
potential temperature, relative humidity, wind speed and direction
from the National Weather Service balloon soundings.
2.2. The AMET database structure
MySQL is an open-source relational database software that can
used to store and quickly access large amounts of data. Aside from
providing a convenient method for storing the observationemodel
pairs, the use of a relational database allows the entire set of data to
be quickly subset based on any criteria available in the database
(e.g. by specific date, location or location metadata). The ability to
subset the data quickly is extremely useful since model perfor-
mance often varies spatially and temporally, and the spatial and
temporal subsets are often different for each pollutant or chemical
species. The MySQL software is free to non-commercial users and is
available for many different computing platforms.
In AMET, the MySQL database consists of project tables that
contain all the paired data for a particular model simulation. Each
project table consists of individual “records”. A record consists of the
paired observed/modeled data for a particular site at a particular
time, along with additional information indicating the date and time
of the record and the location of the site given in lat/lon coordinates.
A single project may consist of hundreds to thousands to millions of
records depending on the length and spatial size of the simulation.
For any given record in the AMET database, several important
data fields are required. The record must include a site name or site
identification number, location information in the form of latitude
and longitude in order to plot the site data in space, and the start
and end date for the observation.
While it should be obvious that the start and end times of obser-
vations vary, it is also possible that the duration of the observation can
vary as well, ranging from very short durations on the order of an
hour, to longer averaging time periods (e.g. a week). As such, the start
and end time data are used to plot the records in time and are also
used for temporal averaging of the data. Finally, the record should
contain all the observational and modeled data for that particular site
and time, which is generated through the data pairing programs. The
number of fields depends on the number of different variables
measured at a particular site, with some observational networks
measuring only a few different variables, while other networks can
have a large number of different measurements available.
In addition to the individual project tables, the database also
contains separate project log tables for the meteorological and air
quality modules. The project log tables store information about each
project, including when the project was created, who created it, and
specific information about the model simulation. The database also
has site tables for the two modules that store metadata about each
observation site, data which can then be used in database queries to
subset the records.
 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443 441

Fig. 8. Sample plot from the run_barplot script from the AMET air quality module for two different CMAQ model simulations. The height of the bars represent the total PM2.5 mass,
 þ
while the color shading represents the contributions of sulfate (SO2
4 ), nitrate (NO3 ), ammonium (NH4 ), total carbon TC), and other PM2.5 mass (Other) to the total PM2.5 mass.

2.3. R statistical package scripts
The R project for Statistical Computing, “R”, is an open-source
software for statistical computing and graphics based on the “S”
language, and is available for many different computer platforms.
The R software can be used through a command line interface or
with the use of scripts, and contains a large number of functions for
computing common statistical metrics and generating graphics (e.g.
box plots and scatter plots). Additionally, the R software contains
a library for interfacing with the MySQL database software. All these
features make R an attractive computational and plotting software
package for AMET.
The information stored in the database in each project table is
queried through individual cshell scripts for each R analysis script
(the R analysis scripts are describing in detail in the next section).
The R software has available a MySQL library which contains
commands for interfacing with the MySQL database. Through the
various commands included in the R MySQL library, the database
can be queried based on any of the information stored in the data-
base tables. For example, one could limit their data analysis to
a single day or week, or to a single state or collection of states. The
AMET scripts include instructions for performing common queries
(e.g. queries based on a date range or location). However, anyone
familiar with basic database queries could easily formulate their
own query criteria to subset the data in any way they choose.
The AMET software already includes a collection of R scripts that
perform various analyses such as computing standard statistical
metrics such as bias, error and correlation; creating common
statistical plots such as scatter plots and time series plots; and
creating unique plots designed specifically for AMET. Table 1 lists the
R scripts currently available for the AMET meteorological and air
quality modules, along with a brief description of the script func-
tions. These scripts are used to generate statistics and plot the
results using the various plotting functions available in the R soft-
ware. Many of the output products available in AMET are frequently
used in the air quality modeling community. In addition to using the
provided R scripts, the more advanced user could use these scripts as
a template for creating new R scripts tailored to their particular
application and analysis.
3. Examples of tool applications
In 2007, AMETv1.1 was released as a tool by CMAS as part of the
CMAQv4.6 model release. The AMET installation and users guide
(http://www.cmascenter.org/help/documentation.cfm?
MODEL¼amet&VERSION¼1.0) provide detailed information on
how to set-up and run AMET with the CMAQ modeling system.
Since its release in the fall of 2007, the AMET has been downloaded
over 550 times, indicating the tool has garnered significant interest
in the meteorological and air quality communities.
For the past five years, AMET has been used extensively by
AMAD for research using the CMAQ modeling system (Appel et al.,
2007, 2008, 2010; Pleim and Gilliam, 2009; Gilliam and Pleim,
2010; Otte, 2008a,b). Additionally, AMET has been used by EPA’s
Office of Air Quality Planning and Standards (OAQPS) to aid in the
evaluation of their air quality simulations for federal rule making
applications and by other groups to evaluate MM5 and WRF model
simulations for efforts not necessarily associated with CMAQ model
applications. AMET is currently under active development at the
Institute for the Environment and at the EPA.
The following (Figs. 2e9) highlight some key analysis capabil-
ities of AMET (refer to Table 1 for additional information). Fig. 2
442 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443
Fig. 9. Sample plots from the a) run_soccerplot script and b) the run_bugleplot script
from the AMET air quality module.
presents examples of the run_summary script available in the
AMET meteorological module. The plot on the left is for near-
surface temperature and includes a scatter and box plot, while the
plot on the right is for wind direction and includes a directional
scatter plot and histogram. Wind speed and moisture analyses are
also available, but are not shown here. The summary plots available
in AMET are particularly useful for quickly gauging the overall
performance of a meteorological simulation. The scatter plot and
wind direction error plot allow for quick assessment of where the
largest bias and error are in a simulation, while the summary
statistics provide an overall assessment of the simulation perfor-
mance in reproducing near-surface observations.
Fig. 3 presents an example of one of the time-height plots available
in the AMET meteorological module. The color shading represents the
model predicted wind speed, while the black dots represent the
height of the model predicted boundary layer. The observed wind
speed, along with statistical metrics such as Root Mean Square Error
(RMSE), bias and the index of agreement are also provided by this
analysis script. This type of script is useful for determining how well
the meteorological model simulates the boundary layer characteris-
tics at different times of the day. Correct simulation of the boundary
layer is particularly important for accurate predictions of atmospheric
pollutants from air quality models.
Fig. 4a provides an example of a spatial plot of model bias from
the run_spatial_surface script available in the AMET meteorological
module, while Fig. 4b is a similar plot of Normalized Mean Bias
(NMB) available from the run_stats_plots script in the air quality
module. These plots are intended to provide a spatial assessment of
the model performance. Both scripts have the capability of plotting
other statistics, including error (absolute or normalized), Root
Mean Square Error (RMSE) and correlation. The run_stats_plots
script also includes a comma separated file containing numerous
statistics in addition to those plotted by default. These scripts are
often used early in a model assessment effort to identify regions
where model performance may be deficient and where further
investigation using the other analysis scripts available in AMET
would be most useful.
Fig. 5 shows examples of the run_scatterplot and run_scatter-
plot_skill scripts, two of the scatter plot scripts available in the
AMET air quality module. Along with the run_stats_plots, the
scatter plot is often used early in a model assessment effort to
gauge the overall model performance and identify any large model
biases that may be present. The run_scatterplot script allows
multiple observation networks and model simulations to be
plotted, along with up to five summary statistics (which can be
changed by the user). The run_scatterplot_skill script is designed
specifically for assessing ozone model performance based on the
ambient air quality standard for ozone (currently 75 ppb). Along
with standard summary statistics, the plot also provides several
forecast skill metrics based on the ozone standard, including
accuracy, probability of detection (POD) and false alarm rate (FAR).
A sample plot from the plot_time series script available in the air
quality module is shown in Fig. 6. The script creates two plots by
default, the top plot showing the observed and model predicted
values, and second plot below showing the corresponding model bias
and error. The plot can be used to plot individual sites, or in the case of
multiple sites, the values are averages. A similar time series plot is
available for the meteorological module, which also allows the user to
group multiple sites into an averaged time series plot. The time series
plots are particularly useful for examining model performance over
time and for analyzing the model performance at individual sites.
Fig. 7 presents examples of the run_boxplot script for the CSN
daily sulfate and the run_boxplot_hourly script for hourly ozone
from the AQS available in the AMET air quality module. Plots like
these are useful for assessing how well the air quality model
simulates both the average atmospheric concentration (by
comparison of the median values) and the distribution of the inter-
quartile range (25%e75%) of concentrations.
Fig. 8 illustrates an example of the run_barplot script from the
AMET air quality module, showing the total fine particulate mass
(PM2.5) concentration (indicated by the height of the bar) from the
observations and two different CMAQ model simulations. The
different colors represent the contribution to the total mass from
sulfate, nitrate, ammonium, total carbon and other PM2.5. The
stacked bar plot is extremely useful for determining which species
contribute the largest amount to the overall concentration of PM2.5
mass, while also determining how well each individual species is
simulated by the model.
Fig. 9 presents examples of the run_soccerplot and run_bugle-
plot scripts available in the AMET air quality module. The soccer
goal plot (Fig. 9a) is essentially a scatter plot, with a normalized bias
metric plotted on the x-axis and a normalized error metric plotted
on the y-axis. Lines are plotted at pre-determined bias and error
values, which together form a box in a shape similar to a soccer
goal. Any number of species from any number of networks can be
plotted on the soccer plot, allowing for a quick assessment of the
overall performance of the air quality simulation.
The bugle plot is designed to be used with a single species from
any number of networks, and plots either a normalized bias metric
(Fig. 9b) or normalized error metric (not shown) against the
 K.W. Appel et al. / Environmental Modelling & Software 26 (2011) 434e443
average species concentration. Criteria and goal lines, which vary as
a function of concentration, are also included on the plot and form
a shape similar to that of a bugle. The idea behind the bugle plot is
that at lower concentrations the acceptable range for bias and error
is larger than at higher concentrations.
Beyond the examples described above, the structure of the AMET
software allows for various other types of analyses. While most of
the examples above utilize surface observations, AMET is capable of
analyzing just about any type of data, including aloft observations
(e.g. radiosondes, ozonesondes). Additionally, while the existing
analysis scripts in AMET are based on data paired in time and space,
through the development of new R scripts AMET can easily be
extended to include analysis of unpaired data such as cumulative
distributions, spectral density analysis and principle component
analysis. In its most basic form, the AMET software provides
a convenient way to store, organize and access observation and
model data for any type of analysis that a user may wish to perform.
4. Conclusions
AMET is a comprehensive software package that is used to pair
observations with gridded model data for meteorological and air
quality related applications. AMET performs statistical calculations
and creates a variety of output products to enable researchers to
better understand and evaluate model predictions and improve the
science within the models. AMET output includes tabular statistics,
graphical representations of statistics, and spatial plots. AMET is built
entirely on open-source software, and while currently set-up to work
with MM5, WRF and CMAQ model data, the software could easily be
extended to work with other models. For example, the AMET soft-
ware has been used to evaluate Comprehensive Air Quality Model
with Extensions (CAMx; ENVIRON, 2010) model output with the use
of file conversion software available for that model.
The AMET software is designed to organize observed and pre-
dicted data into a relational database that can be easily accessed to
help improve the efficiency in which a model’s performance can be
assessed. AMET has been used by various groups in the meteoro-
logical and air quality communities to synthesize model results and
enhance model development. The use of a common model evaluation
tool allows for better understanding of the model evaluation process
and easier interpretation of the evaluation results.
AMET will continue to grow and improve through internal
development within AMAD, external development through
collaborating partners, and through contributions and suggestions
from the user community. Several signinficant improvements to
the AMET software are planned for the next release of the tool. One
improvement will be to combine the meteorological and air quality
modules into a single streamlined module that will allow for
greater cross-comparability of the meteorological and air quality
data. A second improvement focuses on greatly expanding the
types of datasets that can be used for model evaluation. These
changes will allow the user to include non-standard observational
datasets, such as field campaign data or observations from outside
of the U.S. with no changes to the underlying AMET code. Addi-
tionally, the use of satellite data has been streamlined with the
development of tools to regrid satellite data to regular grids, (e.g.
regrid swath data to the CMAQ lambert conformal grid), as well as
to calculate comparable variables (e.g. column aggregated aerosol
optical depth values from model output).
Additional future enhancements to the AMET software will
include implementing new analysis scripts (e.g. Taylor plots); adding
non-parametric analysis techniques such as cumulative distribution
frequency plots, QeQ plots and histograms; adding temporal
decomposition capabilities such as Kolmogorov-Zurbenko filtering
(Zurbenko, 1986) and spectral density plots; adding the ability to
443
easily compute common indicator ratios (e.g. O3/NOY); improving
existing scripts to add flexibility and improve the quality of the plots;
improving the organization of the database to increase overall speed;
and improving the AMET interface. One possibility for improving the
AMET interface would be the development of a JAVA-based interface
that will enable AMET to be easily used on multiple computing
platforms. Many of these improvements are currently in develop-
ment and will be included with the next release of the AMET software.
Acknowledgements
The authors would like to thank Sarav Arunachalam with the
University of North Carolina’s Institute for the Environment for
helping develop the AMET release code and the AMET installation
and users guides. The authors would also like to thank Sharon
Phillips of the EPA’s Office of Air Quality Planning and Standards,
Air Quality Analysis Division for her help with the testing and
evaluation of the AMET software.
Appendix
Disclaimer e The United States Environmental Protection
Agency through its Office of Research and Development funded and
managed the research described here. It has been subjected to
Agency review and approved for publication.
References
Appel, K.W., Gilliland, A.B., Sarwar, G., Gilliam, R.C., 2007. Evaluation of the
Community Multiscale air quality (CMAQ) model version 4.5: sensitivities
impacting model performance: Part IeOzone. Atmos. Environ. 41, 9603e9615.
Appel, K.W., Bhave, P.V., Gilliland, A.B., Sarwar, G., Roselle, S.J., 2008. Evaluation of
the Community Multiscale Air Quality (CMAQ) model version 4.5: sensitivities
impacting model performance; Part IIeparticulate matter. Atmos. Environ. 42,
6057e6066.
Appel, K.W., Roselle, S.J., Gilliam, R.C., Pleim, J.E., 2010. Sensitivity of the Community
Multiscale Air Quality (CMAQ) model v4.7 results for the eastern United States
to MM5 and WRF meteorological drivers. Geosci. Model. Dev. 3, 169e188.
Byun, D.W., Schere, K.L., 2006. Review of the governing equations, computational
algorithms, and other components of the Models-3 Community Multiscale Air
Quality (CMAQ) modeling system. Appl. Mech. Rev. 55, 51e77.
Dennis, R., Fox, T., Fuentes, M., Gilliland, A., Hanna, S., Hogrefe, C., Irwin, J., Rao, S.T.,
Scheffe, R., Schere, K., Steyn, D., Venkatram, A., 2010. A framework for evalu-
ating regional-scale numerical photochemical modeling systems. Environ. Fluid
Mech.. doi:10.1007/s10652-009-9163-2.
ENVIRON, 2010. User’s Guide to the Comprehensive Air Quality Model with
Extensions (CAMx) Version 5.20. Available at. ENVIRON International Corpo-
ration, 773 San Marin Drive, Suite 2115Novato, California 94998. http://www.
camx.com/files/CAMxUsersGuide_v5.20.pdf.
Foley, K.M., Roselle, S.J., Appel, K.W., Bhave, P.V., Pleim, J.E., Otte, T.L., Mathur, R.,
Sarwar, G., Young, J.O., Gilliam, R.C., Nolte, C.G., Kelly, J.T., Gilliland, A.B.,
Bash, J.O., 2010. Incremental testing of the Community Multiscale Air Quality
(CMAQ) modeling system version 4.7. Geosci. Model. Dev. 3, 205e226.
Gilliam, R.C., Pleim, J.E., 2010. Performance assessment of new land-surface and
planetary boundary layer physics in the WRF-ARW. J. Appl. Meteor. Climatol.
doi:10.1175/2009JAMC2126.1.
Grell, G.A., Dudhia, A.J., Stauffer, D.R., 1994. A description of the Fifth-Generation
PennState/NCAR Mesoscale model (MM5). NCAR Technical Note NCAR/TN-
398þSTR. Available at. http://www.mmm.ucar.edu/mm5/doc1.html.
Otte, T.L., 2008a. The impact of nudging in the meteorological model for retro-
spective air quality simulations. Part I: evaluation against national observation
networks. J. Appl. Meteor. Climatol. 47, 1853e1867.
Otte, T.L., 2008b. The impact of nudging in the meteorological model for retro-
spective air quality simulations. Part II: evaluating collocated meteorological
and air quality observations. J. Appl. Meteor. Climatol. 47, 1868e1887.
Otte, T.L., Pleim, J.E., 2010. The meteorology-Chemistry interface processor (MCIP)
for the CMAQ modeling system. Geosci. Model. Dev. 3, 243e256.
Pleim, J.E., Gilliam, R.C., 2009. An Indirect data Assimilation Scheme for Deep Soil
temperature in the PleimeXiuLand surface model. J. Appl. Meteor. Climatol. 48,
1362e1376.
Skamarock, W.C., Klemp, J.B., Dudhia, J., Gill, D.O., Barker, D.M., Duda, M.G., Huang,
X-Y, Wang, W., and Powers, J.G., 2008. A description of the advanced research
WRF version 3. NCAR Tech Note NCAR/TN 475 STR, 125 pp, Available from UCAR
Communications, P.O. Box 3000, Boulder, CO 80307.
Zurbenko, I.G., 1986. The spectral analysis of time series. North-Holland Series in
Statistics and Probability.