Numeric Computation and Statistical

Data Analysis on the Java Platform

Advanced Information and Knowledge
Processing Chekanov Sergei V
Visit to download the full and correct content document:
https://textbookfull.com/product/numeric-computation-and-statistical-data-analysis-on-
the-java-platform-advanced-information-and-knowledge-processing-chekanov-sergei-
v/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Scientific Data Analysis using Jython Scripting and

Java Advanced Information and Knowledge Processing
2010th Edition Chekanov Sergei V

https://textbookfull.com/product/scientific-data-analysis-using-
jython-scripting-and-java-advanced-information-and-knowledge-
processing-2010th-edition-chekanov-sergei-v/

Advanced R Statistical Programming and Data Models:

Analysis, Machine Learning, and Visualization 1st
Edition Matt Wiley

https://textbookfull.com/product/advanced-r-statistical-
programming-and-data-models-analysis-machine-learning-and-
visualization-1st-edition-matt-wiley/

Metaprogramming in R: Advanced Statistical Programming

for Data Science, Analysis and Finance 1st Edition
Thomas Mailund

https://textbookfull.com/product/metaprogramming-in-r-advanced-
statistical-programming-for-data-science-analysis-and-
finance-1st-edition-thomas-mailund/

Real Estate Analysis in the Information Age Techniques

for Big Data and Statistical Modeling 1st Edition
Winson-Geideman

https://textbookfull.com/product/real-estate-analysis-in-the-
information-age-techniques-for-big-data-and-statistical-
modeling-1st-edition-winson-geideman/
Functional Programming in R: Advanced Statistical
Programming for Data Science, Analysis and Finance 1st
Edition Thomas Mailund

https://textbookfull.com/product/functional-programming-in-r-
advanced-statistical-programming-for-data-science-analysis-and-
finance-1st-edition-thomas-mailund/

Advanced Object-Oriented Programming in R: Statistical

Programming for Data Science, Analysis and Finance 1st
Edition Thomas Mailund

https://textbookfull.com/product/advanced-object-oriented-
programming-in-r-statistical-programming-for-data-science-
analysis-and-finance-1st-edition-thomas-mailund/

Statistical Data Analysis using SAS Intermediate

Statistical Methods Mervyn G. Marasinghe

https://textbookfull.com/product/statistical-data-analysis-using-
sas-intermediate-statistical-methods-mervyn-g-marasinghe/

Advanced linear modeling statistical learning and

dependent data 3rd Edition Christensen R

https://textbookfull.com/product/advanced-linear-modeling-
statistical-learning-and-dependent-data-3rd-edition-
christensen-r/

Advanced Information Networking and Applications:

Proceedings of the 33rd International Conference on
Advanced Information Networking and Applications
(AINA-2019) Leonard Barolli
https://textbookfull.com/product/advanced-information-networking-
and-applications-proceedings-of-the-33rd-international-
conference-on-advanced-information-networking-and-applications-
Advanced Information and Knowledge Processing

Sergei V. Chekanov

Numeric
Computation and
Statistical Data
Analysis on the
Java Platform
Advanced Information and Knowledge
Processing

Series editors
Lakhmi C. Jain
Bournemouth University, Poole, UK and
University of South Australia, Adelaide, Australia

Xindong Wu
University of Vermont
Information systems and intelligent knowledge processing are playing an increasing
role in business, science and technology. Recently, advanced information systems
have evolved to facilitate the co-evolution of human and information networks
within communities. These advanced information systems use various paradigms
including artificial intelligence, knowledge management, and neural science as well
as conventional information processing paradigms. The aim of this series is to
publish books on new designs and applications of advanced information and
knowledge processing paradigms in areas including but not limited to aviation,
business, security, education, engineering, health, management, and science. Books
in the series should have a strong focus on information processing—preferably
combined with, or extended by, new results from adjacent sciences. Proposals for
research monographs, reference books, coherently integrated multi-author edited
books, and handbooks will be considered for the series and each proposal will be
reviewed by the Series Editors, with additional reviews from the editorial board and
independent reviewers where appropriate. Titles published within the Advanced
Information and Knowledge Processing series are included in Thomson Reuters’
Book Citation Index.

More information about this series at http://www.springer.com/series/4738

Sergei V. Chekanov

Numeric Computation
and Statistical Data Analysis
on the Java Platform

123
Sergei V. Chekanov
HEP Division
Argonne National Laboratory
Lemont, IL
USA

Additional material to this book can be downloaded from http://extras.springer.com/.

ISSN 1610-3947 ISSN 2197-8441 (electronic)

Advanced Information and Knowledge Processing
ISBN 978-3-319-28529-0 ISBN 978-3-319-28531-3 (eBook)
DOI 10.1007/978-3-319-28531-3

Library of Congress Control Number: 2016932001

© Springer International Publishing Switzerland 2016

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained herein or
for any errors or omissions that may have been made.

Printed on acid-free paper

This Springer imprint is published by SpringerNature

The registered company is Springer International Publishing AG Switzerland
This book is dedicated to my family
Preface

Numerical and statistical algorithms are typically confined within a specific pro-
gramming language. For example, the R open-source data-analysis software uses a
specialized scripting language, which is an implementation of the “S” programming
language. Many commercial mathematical programs follow this trend. This book is
about a platform for statistical calculations using algorithms that are not confined by
a chosen language. For example, this platform allows mixing Python and Java
numerical libraries, or using them on their own. Or, one can use this book to
program statistical code using other languages, such as Groovy, Ruby, and
BeanShell. This book is about an approach to scientific programming and visual-
ization that does not set strict requirements on specific programming languages, nor
on operating systems where such calculations are performed.
There are many books written about Java—one of the most popular program-
ming languages. There are many books written about Python, which is another very
popular programming language. This book explains how to mix them, bringing
incredible algorithmic power and cutting-edge numeric libraries to scientific com-
putations and data visualization.
In this book I did not go deep inside particular scientific research area, since the
aim was to give concrete examples which illustrate which Java libraries should be
used to perform computations. In the cases when I could not cover the subject in
detail, a sufficient number of relevant references was given, so the reader can easily
find necessary information for each chapter using external sources.
Thus this book presents practical approaches to numerical computations, data
analysis, and knowledge discovery, focusing on programming techniques. Each
chapter describes the conceptual underpinning for numerical and statistical calcu-
lations using Java libraries, covering many aspects from simple multidimensional
arrays and histograms to clustering analysis, curve fitting, neural networks, and
symbolic calculations. To make the examples as simple as possible from the
computational point of view, I fully embrace the scripting approach in the course of
this book. This leads to short and clear analysis codes, so you could concentrate on
the logic of analysis flow rather than on language-specific details.

vii
viii Preface

This book uses Python as the main programming language, since it is elegant and
easy to learn. It is a great language for teaching scientific computation. For devel-
opers, this is an ideal language for fast prototyping and debugging. The book dis-
cusses how to design code snippets for numeric computation and statistics on the
Java platform. To be more exact, we will use Jython (Python implemented in Java), a
language that uses not only native Python modules, but can also access very com-
prehensive Java classes. The reader will learn how to write analysis codes, while
numerous code snippets will give you some ideas on numeric algorithms which can
easily be incorporated into realistic research application. The book includes more
than 300 code snippets to produce data-visualization plots in 2D and 3D.
I am almost convinced myself that this book is self-contained and does not
depend on detailed knowledge of computing language, although knowledge of
Python and Java is desirable. However, the reader may still need some programming
background in order to use this book with other languages, such as Groovy,
BeanShell, and Ruby, since I did not give very detailed coverage of these languages.

Who Is This Book for

This book is intended for general audiences, for those who use computing to make
sense of data surrounding us. It can be used as a source of knowledge on data
analysis and statistical calculations for students and professionals of all disciplines.
This book was written for undergraduate and graduate students, academics, pro-
fessors, and professionals of any field and any age. The book could be used as a
textbook for students.
We also hope that this book will be useful for those who study financial markets,
since the numeric algorithms discussed in this book are undoubtedly common to any
knowledge discovery research. This book equips readers with the description of a
computational platform for statistical calculations which can be viewed as an inex-
pensive alternative to costly commercial products used by financial-market analysts.
I assume the readers are not familiar with Python/Jython, the main programming
language used for code snippets in this book. But some basic understanding of
statistics and mathematics would be very helpful to understand the material of this
book.
All example codes of this book can easily be transformed to Java, Groovy,
Ruby/JRuby, or BeanShell codes. You are presumed to have knowledge of pro-
gramming in Java, if you will choose the path of moving the examples to Java, or if
you will decide to create Java libraries to be deployed as jar files for a new project.
The book will discuss how to do this, and a few Java examples will be provided.
Transformations of the example snippets to scripting languages, such as Groovy,
Ruby/JRuby, or BeanShell, may require some knowledge of these scripting
languages. The good thing is that the analysis algorithms and numerical libraries
will be exactly the same, so a little effort is required to move to other languages.
Again, we will show you how to convert Jython codes to these languages. In most
Preface ix

cases, our examples should be sufficient to get started with a new language. The
more knowledge about Groovy and Ruby/JRuby you can bring, the more you will
get out of this book.

Books You May Read Before

The material of this book is self-contained. However, to understand the material

deeper, you may need to look at other sources. First of all, there are plenty of good
books [1–5] on Python and Jython, which are more complete for language-specific
topics than the information given in this book. If you program in Java that forms the
backbone of numerical and graphical libraries discussed in this book, a great deal of
supplementary information can be found in Java books [6–10].
Secondly, there are several books on Groovy, a popular scripting language that
can be used to work with the Java numerical libraries discussed in this book [11,
12]. If your choice is JRuby, the Ruby programming language on the Java platform,
look at these books [13, 14] to get started.
Thirdly, as you read, you may need to look at external sources to understand the
material better, especially when we come to statistical interpretations of data. We will
supply the reader with the necessary references, so he or she can choose the most
appropriate (and affordable) books to discover the world of data analysis and data
mining.

References

1. Pilgrim M (2004), Dive into Python, Apress

2. Guzdial M (2005) Introduction to computing and programming in Python, a multimedia
approach, Prentice Hall
3. Martelli A (2006) Python in a nutshell (in a nutshell (O’Reilly)), O’Reilly Media, Inc.
4. Lutz M (2007) Learning Python, 3rd edn. O’Reilly Media, Inc.
5. Langtangen H (2008) Python scripting for computational science. Springer-Verlag, Berlin,
Heidelberg
6. Richardson C, Avondolio D, Vitale J et al (2005) Professional Java, JDK 5 Edition, Wrox
7. Arnold K, Gosling J, Holmes D (2005) Java(TM) programming language. In: The Java Series,
4th edn. Addison-Wesley Professional
8. Flanagan D (2005) Java in a nutshell, 5th edn. O’Reilly Media, Inc.
9. Eckel B (2006) Thinking in Java, 4th edn. Prentice Hall PTR
10. Bloch J (2008) Effective Java. In: The Java Series, 2nd edn. Prentice Hall PTR
11. Subramaniam V (2013) Programming Groovy 2: dynamic productivity for the Java
Developer, Pragmatic Programmers, LLC
12. King D, Glover A (2013) Groovy in action, Manning Publications, 2007, https://www.
manning.com/books/groovy-in-action
13. Edelson J, Liu H (2008) JRuby cookbook, O’Reilly Media
14. Bini O (2007) Practical JRuby on Rails Web 2.0 Projects: Bringing Ruby on Rails to Java,
Apress
Acknowledgements

This book describes a software which is a collective work of many developers who
have dedicated themselves to scientific computing. The author is grateful to all
people who contributed to scientific software, and for their inspiration and dedi-
cation to science and knowledge-discovery software.
Many numeric and graphic libraries discussed in this book were released as
open-source projects. I am grateful to the authors of such open-source programs for
their enthusiasm to share their work, and for making their software publicly
available.
You can find a list of contributions to the software packages described in this
book on the jWork.ORG web page (http://jwork.org/dmelt/). A special note of
thanks to those of you who reported bugs in a constructive way, helped with
solutions, and shared your knowledge and experience with others.
Much of this project grew out of fruitful collaboration with many of my col-
leagues who devoted themselves to high energy physics. Over the course of the past
twenty-five years I have learned a lot about programming aspects of scientific
research. I would like to thank my colleagues for checking and debugging the
examples shown in this book, and here the list will be endless.
I would like to thank everyone at Springer for their help with the production
process. In particular, managing editors H. Desmond and J. Robinson, who helped
start this book in its present form.
Not least, personal thanks go to my dear wife, Tania, and my sons, Alexey
(Alosha) and Roman, for their love and patience to a husband and father who was
only half (mentally) present after coming from his work. Without their patience and
understanding, this book would not have been possible. Finally, I also thank my
parents and sister for their support of my interests in all aspects of science.

Chicago Sergei V. Chekanov

January 2016

xi
Contents

1 Java Computational Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Programming in Java. . . . . . . . . . . . . . . . . . . . . . 1
1.1.2 The DMelt Software Platform. . . . . . . . . . . . . . . . 3
1.1.3 Some Warnings . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.4 Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 Scripting with DMelt . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.1 Learning by Example . . . . . . . . . . . . . . . . . . . . . 7
1.2.2 Using Jython for Code Examples . . . . . . . . . . . . . 8
1.2.3 Differences with Other Math Software . . . . . . . . . . 9
1.3 Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 DMelt Workbench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.1 Source Code Editor . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.2 DMelt Libraries . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.3 Jython and BeanShell Consoles . . . . . . . . . . . . . . 15
1.4.4 Accessing Methods of Instances . . . . . . . . . . . . . . 17
1.4.5 Editing Jython Scripts . . . . . . . . . . . . . . . . . . . . . 17
1.4.6 Running Jython Scripts . . . . . . . . . . . . . . . . . . . . 17
1.4.7 Macro Files of the DMelt IDE . . . . . . . . . . . . . . . 18
1.4.8 Running BeanShell Scripts . . . . . . . . . . . . . . . . . . 19
1.4.9 Compiling and Running Java Code . . . . . . . . . . . . 19
1.4.10 DMelt Code Assist . . . . . . . . . . . . . . . . . . . . . . . 20
1.4.11 Other Features . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.4.12 Working with Images . . . . . . . . . . . . . . . . . . . . . 23
1.4.13 DMelt License . . . . . . . . . . . . . . . . . . . . . . . . . . 24
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

xiii
xiv Contents

2 Introduction to Jython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.1 Code Structure and Jython Objects . . . . . . . . . . . . . . . . . . . 27
2.1.1 Numbers as Objects . . . . . . . . . . . . . . . . . . . . . . 30
2.1.2 Formatted Output . . . . . . . . . . . . . . . . . . . . . . . . 32
2.1.3 Mathematical Functions . . . . . . . . . . . . . . . . . . . . 32
2.2 Complex Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.3 Strings as Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4 Import Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.4.1 Executing Native Applications . . . . . . . . . . . . . . . 37
2.5 Comparison Tests and Loops . . . . . . . . . . . . . . . . . . . . . . . 38
2.5.1 The “if-else” Statement . . . . . . . . . . . . . . . . . . . . 38
2.5.2 Loops. The “for” Statement . . . . . . . . . . . . . . . . . 39
2.5.3 The “continue” and “break” Statements . . . . . . . . . 39
2.5.4 Loops. The “while” Statement . . . . . . . . . . . . . . . 40
2.6 Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.6.1 Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.6.2 Tuples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.6.3 Dictionaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.6.4 Functional Programming . . . . . . . . . . . . . . . . . . . 48
2.7 Java Collections in Jython . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.7.1 List. An Ordered Collection . . . . . . . . . . . . . . . . . 50
2.7.2 Set. A Collection Without Duplicate Elements . . . . 53
2.7.3 SortedSet. Sorted Unique Elements . . . . . . . . . . . . 54
2.7.4 Map. Mapping Keys to Values . . . . . . . . . . . . . . . 55
2.7.5 Java Map with Sorted Elements . . . . . . . . . . . . . . 55
2.7.6 Real-Life Example: Sorting and Removing
Duplicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.8 Random Numbers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.9 Time Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.9.1 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.10 Python Functions and Modules. . . . . . . . . . . . . . . . . . . . . . 60
2.11 Python Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
2.11.1 Initializing a Class . . . . . . . . . . . . . . . . . . . . . . . 65
2.11.2 Classes Inherited from Other Classes. . . . . . . . . . . 65
2.11.3 Java Classes in Jython . . . . . . . . . . . . . . . . . . . . . 66
2.11.4 Not Covered Topics . . . . . . . . . . . . . . . . . . . . . . 66
2.12 Parallel Computing and Threads . . . . . . . . . . . . . . . . . . . . . 67
2.13 Arrays in Jython. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
2.13.1 Array Conversion and Transformations . . . . . . . . . 69
2.13.2 Performance Issues . . . . . . . . . . . . . . . . . . . . . . . 69
2.13.3 Used Memory . . . . . . . . . . . . . . . . . . . . . . . . . . 70
2.14 Exceptions in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Contents xv

2.15 Input and Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

2.15.1 User Interaction . . . . . . . . . . . . . . . . . . . . . . . . . 72
2.15.2 Reading and Writing Files . . . . . . . . . . . . . . . . . . 72
2.15.3 Input and Output for Arrays . . . . . . . . . . . . . . . . . 74
2.15.4 Working with CSV Python Module. . . . . . . . . . . . 74
2.15.5 Saving Objects in a Serialized File . . . . . . . . . . . . 76
2.15.6 Storing Multiple Objects . . . . . . . . . . . . . . . . . . . 77
2.15.7 Using Java for I/O . . . . . . . . . . . . . . . . . . . . . . . 78
2.15.8 Reading Data from the Network . . . . . . . . . . . . . . 79
2.16 Real-Life Example. Collecting Data Files . . . . . . . . . . . . . . 80
2.17 Using Java for GUI Programming. . . . . . . . . . . . . . . . . . . . 82
2.18 Concluding Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3 Mathematical Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.1 Python Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.2 Functions in DMelt. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.2.1 Java Implementation of F1D . . . . . . . . . . . . . . . . 89
3.2.2 Manipulations with 1D Functions . . . . . . . . . . . . . 90
3.3 Plotting 1D Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.3.1 Building a Graphical Canvas . . . . . . . . . . . . . . . . 93
3.3.2 Drawing 1D Functions . . . . . . . . . . . . . . . . . . . . 97
3.3.3 Plotting Functions on Different Pads . . . . . . . . . . . 99
3.3.4 Short Summary of HPlot Methods . . . . . . . . . . . . 99
3.3.5 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
3.4 2D Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3.4.1 Functions in Two Dimensions . . . . . . . . . . . . . . . 102
3.4.2 Displaying 2D Functions . . . . . . . . . . . . . . . . . . . 103
3.4.3 Using a Contour Plot. . . . . . . . . . . . . . . . . . . . . . 105
3.5 3D Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
3.5.1 Functions in Three Dimensions. . . . . . . . . . . . . . . 106
3.6 Functions in Many Dimensions . . . . . . . . . . . . . . . . . . . . . 107
3.6.1 FND Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 107
3.6.2 Drawing FND Functions . . . . . . . . . . . . . . . . . . . 108
3.7 Custom Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
3.7.1 Custom Functions and Their Methods . . . . . . . . . . 109
3.7.2 Custom Functions Using Expression Builder . . . . . 112
3.7.3 Custom Functions in Jython . . . . . . . . . . . . . . . . . 114
3.8 Parametric Surfaces in 3D . . . . . . . . . . . . . . . . . . . . . . . . . 119
3.8.1 FPR Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 119
3.8.2 3D Mathematical Objects . . . . . . . . . . . . . . . . . . . 122
xvi Contents

3.9 Function Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

3.9.1 Minimization of Multidimensional Functions . . . . . 125
3.9.2 Calling Migrad Directly . . . . . . . . . . . . . . . . . . . . 126
3.10 File Input and Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
4 Data Arrays. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.1 1D Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.1.1 P0D Data Container . . . . . . . . . . . . . . . . . . . . . . 132
4.1.2 P0D Transformations. . . . . . . . . . . . . . . . . . . . . . 135
4.1.3 Statistical Summary. . . . . . . . . . . . . . . . . . . . . . . 136
4.1.4 Displaying P0D Data. . . . . . . . . . . . . . . . . . . . . . 138
4.1.5 File Input and Output . . . . . . . . . . . . . . . . . . . . . 139
4.2 Arrays for Two Dimensions . . . . . . . . . . . . . . . . . . . . . . . . 143
4.2.1 Data with Errors . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.2.2 Viewing P1D Data . . . . . . . . . . . . . . . . . . . . . . . 150
4.2.3 Plotting P1D Data. . . . . . . . . . . . . . . . . . . . . . . . 151
4.2.4 Contour Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
4.2.5 Manipulations with P1D Data. . . . . . . . . . . . . . . . 153
4.2.6 Advanced P1D Operations . . . . . . . . . . . . . . . . . . 155
4.2.7 Weighted Average and Systematical
Uncertainties . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
4.2.8 File Input and Output . . . . . . . . . . . . . . . . . . . . . 159
4.2.9 Example I: Henon Attractor . . . . . . . . . . . . . . . . . 162
4.2.10 Example II. Weighted Average . . . . . . . . . . . . . . . 163
4.3 Other Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
4.3.1 P2D Data Container . . . . . . . . . . . . . . . . . . . . . . 166
4.3.2 P3D Data Container . . . . . . . . . . . . . . . . . . . . . . 169
4.3.3 PND Data Container . . . . . . . . . . . . . . . . . . . . . . 171
4.3.4 File Input and Output . . . . . . . . . . . . . . . . . . . . . 174
4.4 Third-Party Data Containers. . . . . . . . . . . . . . . . . . . . . . . . 174
4.4.1 Math Arrays. . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
4.4.2 Jaida Data Containers . . . . . . . . . . . . . . . . . . . . . 175
4.4.3 jMathTools Arrays . . . . . . . . . . . . . . . . . . . . . . . 178
4.4.4 Colt Data Containers . . . . . . . . . . . . . . . . . . . . . . 181
4.4.5 Lorentz Vector . . . . . . . . . . . . . . . . . . . . . . . . . . 182
4.5 Multidimensional Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . 185
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5 Linear Algebra and Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 187
5.1 Vector and Matrix Packages . . . . . . . . . . . . . . . . . . . . . . . 187
5.1.1 Basic Matrix Arithmetic. . . . . . . . . . . . . . . . . . . . 189
5.1.2 Elements of Linear Algebra . . . . . . . . . . . . . . . . . 190
5.1.3 Jampack Matrix Computations . . . . . . . . . . . . . . . 191
5.1.4 La4J Library . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Contents xvii

5.1.5 EJML Matrix Library . . . . . . . . . . . . . . . . . . . . . 195

5.1.6 Multithreaded Matrix Computations . . . . . . . . . . . 196
5.1.7 JBlas and Other Matrix Packages . . . . . . . . . . . . . 199
5.1.8 Python Vector and Matrix Operations . . . . . . . . . . 199
5.1.9 Matrix Operations in SymPy . . . . . . . . . . . . . . . . 201
5.2 Algebraic Manipulations with Tensors. . . . . . . . . . . . . . . . . 202
5.3 Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
5.3.1 Polynomial Equations . . . . . . . . . . . . . . . . . . . . . 203
5.3.2 Linear Systems of Equations . . . . . . . . . . . . . . . . 204
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
6 Symbolic Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
6.1 Using the Octave Language . . . . . . . . . . . . . . . . . . . . . . . . 207
6.2 Java Symbolic Computing Library . . . . . . . . . . . . . . . . . . . 208
6.2.1 Conversion to Elementary Functions . . . . . . . . . . . 210
6.2.2 Numeric Calculations . . . . . . . . . . . . . . . . . . . . . 210
6.2.3 Simplify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
6.2.4 Substitutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
6.2.5 Differentiate . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
6.2.6 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
6.2.7 Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
6.2.8 MathML Output . . . . . . . . . . . . . . . . . . . . . . . . . 213
6.2.9 Integration with DMelt Plotting Canvases . . . . . . . 213
6.3 Using SymPy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
7 Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
7.1 One-Dimensional Histogram . . . . . . . . . . . . . . . . . . . . . . . 220
7.1.1 Probability Distribution and Probability Density . . . 224
7.1.2 Histogram Characteristics. . . . . . . . . . . . . . . . . . . 225
7.1.3 Initialization and Filling Methods . . . . . . . . . . . . . 225
7.1.4 Accessing Histogram Values . . . . . . . . . . . . . . . . 228
7.1.5 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
7.1.6 Histogram Operations . . . . . . . . . . . . . . . . . . . . . 230
7.1.7 Accessing Low-Level Jaida Classes. . . . . . . . . . . . 231
7.1.8 Graphical Attributes . . . . . . . . . . . . . . . . . . . . . . 232
7.2 Histogram in 2D. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
7.2.1 Histogram Operations . . . . . . . . . . . . . . . . . . . . . 236
7.2.2 Graphical Representation . . . . . . . . . . . . . . . . . . . 236
7.3 Histograms in Jaida . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
7.4 Histogram in 3D. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
7.5 Profile Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
xviii Contents

7.6 Histogram Input and Output. . . . . . . . . . . . . . . . . . . . . . . . 244

7.6.1 External Programs for Histograms. . . . . . . . . . . . . 246
7.7 Analyzing Histograms from Multiple Files. . . . . . . . . . . . . . 247
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
8 Scientific visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
8.1 Graphical Canvases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
8.2 HPlot Canvas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
8.2.1 Working with the HPlot Canvas . . . . . . . . . . . . . . 255
8.2.2 Saving Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
8.2.3 Reading Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
8.2.4 Axes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
8.2.5 Summary of the HPlot Methods . . . . . . . . . . . . . . 259
8.2.6 Exporting to Image Files . . . . . . . . . . . . . . . . . . . 261
8.2.7 Labels and Keys . . . . . . . . . . . . . . . . . . . . . . . . . 262
8.2.8 Geometrical Primitives. . . . . . . . . . . . . . . . . . . . . 265
8.2.9 Text Strings and Symbols . . . . . . . . . . . . . . . . . . 266
8.3 Interconnected Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
8.4 Showing Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
8.5 Lightweight Canvases . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
8.5.1 Henon Attractor Again . . . . . . . . . . . . . . . . . . . . 275
8.6 Canvas for Interactive Drawing . . . . . . . . . . . . . . . . . . . . . 276
8.6.1 Drawing Diagrams . . . . . . . . . . . . . . . . . . . . . . . 277
8.6.2 SHPlotJa Class . . . . . . . . . . . . . . . . . . . . . . . . . . 279
8.7 Custom Plotting in XY . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
8.7.1 HPlotXY Canvas . . . . . . . . . . . . . . . . . . . . . . . . 279
8.7.2 WPlot Canvas . . . . . . . . . . . . . . . . . . . . . . . . . . 280
8.7.3 HPlotJas Canvas . . . . . . . . . . . . . . . . . . . . . . . . . 281
8.8 HPlot2D Canvas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
8.9 Visualization in 3D. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
8.9.1 HPlot3D Canvas . . . . . . . . . . . . . . . . . . . . . . . . . 285
8.9.2 HPlot3DP Canvas . . . . . . . . . . . . . . . . . . . . . . . . 285
8.9.3 Mathematical Objects in 3D . . . . . . . . . . . . . . . . . 289
8.10 Plotting Real-Time Data . . . . . . . . . . . . . . . . . . . . . . . . . . 291
8.10.1 Real-Time Data Using SPlot . . . . . . . . . . . . . . . . 292
8.10.2 Real-Time Data Using HPlotRT . . . . . . . . . . . . . . 293
8.11 Graphs and Java GUI Components . . . . . . . . . . . . . . . . . . . 293
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
9 File Input and Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
9.1 Nonpersistent Data: Memory-Based Data. . . . . . . . . . . . . . . 297
9.2 Object Serialization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
9.3 Persistent Event Records . . . . . . . . . . . . . . . . . . . . . . . . . . 300
9.3.1 Sequential Input and Output . . . . . . . . . . . . . . . . . 300
9.3.2 Opening Data in a Browser . . . . . . . . . . . . . . . . . 302
1.1 Introduction 5

conventions and code layout. We even admit that some parts were not designed
with the highest possible performance for code execution in mind. The reason is
simple: it was not written by professional programmers. The numerical libraries
were written by many people at different times, most of them were students and
scientists who had to develop numerical and data visualization algorithms for their
own research programs, since commercial software companies either could not offer
similar programs or their products were too expensive. Many contributed packages
have been discontinued many years ago, but have been brought to life after their
inclusion into DMelt. In addition, some packages were written using Java 1.1, and
this had also some impact on the coding style of certain libraries.
Thus, a professional programmer may immediately find some parts of the code
that look unprofessionally written. This is true even for some examples shown in
this book. The reason for this was not because we were not aware of such coding
issues. In some cases, we did not find appealing reasons to keep very strict coding
standard at the expense of simplicity. For example, in most cases, we import all
classes inside a package using the statement:

>>> from PackageName import *

instead of importing only certain classes as

>>> from PackageName import class1,class2

We did not enforce the latter case to keep the examples of this book short and con-
cise, so we could fit the code snippets into the pages of this book. Also, it is possible
that you may not like to type long lists of imported classes during a code prototyp-
ing (personally, I do not like this style), since this can be done later during code
deployment.
A professorial programmer might find some other odds, like why some object
containers are designed to store only double values (like the P1D class to be
discussed below), while it is more practical to store integer values when necessary.
Again, the motivation was not because of omissions. The reason was that the reader
may not want to dive into extra complexity of dealing with different types, since
integers are only a subset of float values. There are plenty of other classes which are
well suited for storing integer values (we will discuss them in this book).
The main motivation for the DMelt project was to develop an accessible and
friendly tool to be used in scientific search, with a syntax oriented toward scientists
rather than programmers. The design of this project was mainly motivated by
simplicity: there are many programming languages which are required to learn for
many years before starting to write useful scientific and engineering projects. The
approach discussed in this book is very different: generally, the reader does not need
to know any programming language to start writing analysis codes using DMelt
libraries. However, if it happens that the reader knows either Java or Python (or
both) already, he or she will find this book to be also interesting, since DMelt is not
just a simplified entry to the world of the Java and Python computer programming. It
6 1 Java Computational Platform

shows how to use programming for practical purposes such as numeric calculations,
statistics, and data analysis.
The reader may also notice that a little attention has been paid to how to write and
use Java or Jython classes. Of course, classes are necessary for any object-oriented
language. The reason for this is the following: for the majority of scientific data
analysis programs, the logic of scripting programs is linear, i.e., an analysis code
typically consists of a well-defined sequence of statements to be evaluated one by
one, from the top to the bottom of the code. It is very unlikely that data analysis logic
will contain highly parallel algorithmic branches as those for the usual graphical
user interface (GUI) development.1 Certainly, the classes are necessary when one
develops Java libraries to be used by a scripting language. But, in this book, we
mainly concentrate on the scripting examples based on the existing Java libraries of
DMelt, rather than discussing how to write classes for numerical computation to be
deployed as external libraries.

1.1.4 Errors

This book may contain typos, omissions, or even errors. DMelt can also contain
bugs. If you notice any errors or if you have suggestions regarding the book and
code examples, I would be happy to hear from you. You can send your comments
to:

dmelt@jworks.org

One can also post bug reports to the DMelt forum accessible from the main Web
page:

http://jwork.org/dmelt/

DMelt is not a software that stands still. Therefore, this book represents a snap-
shot of the time when the DMelt version 1.4 was in use, therefore, some examples
may fall out of date. Therefore, the reader is encouraged to look at the Web page
given above to find corrected examples.

1 We should probably say that this may not be totally true in future when multi-core machines will
be rather common and one will face with the question of how to parallelize analysis codes to gain
high performance. We briefly discuss this topic in this book.
Another random document with
no related content on Scribd:
found, no attempt being made to place it in a special receptacle. The
egg is placed on the ventral surface, well behind the feet, under a
mass of matter in the alimentary canal. Shortly after being hatched
the young destroyer penetrates with its head the skin of the victim,
and in this position commences to feed; it is necessary that it should
obtain its food without killing the Cetonia larva, for it cannot prosper
on decaying food, so that if the Cetonia larva die the Scolia larva
likewise perishes; the latter, accordingly, does not withdraw its head
from the interior of the victim, but remains always in the same
position, as it grows larger extending its head forwards into the front
part of the interior of its victim; the internal organs of the latter are
consumed in a systematic order so as to delay bringing about its
death till the last moment, and thus all the interior of the Cetonia
larva is appropriated till nothing remains but an empty skin. By a
series of experiments, Fabre showed how essential it is that this
apparently revolting operation should be carried on with all details
strictly en règle. If the head of the Scolia larva be taken out from the
victim and applied to another part of the body of the Cetonia, the
result is that it cannot eat; even if it be replaced in the original
situation, after being taken away, it frequently happens that the
Cetonia larva dies, its death involving also that of the destroyer. It is
necessary, too, that the victim should be paralysed, for if an intact
Cetonia larva be taken and bound down in such a position that it
cannot move, and if a small orifice in its skin be made in the proper
spot and a young Scolia larva be placed on it, the little parasite will
avail itself of the opportunity and commence to feed on the larva
provided for it, but the latter will speedily die, and the Scolia
necessarily perishes with it. Thus both the paralysis of the victim and
the special mode of eating are essential to the life of the Scolia. The
operation of stinging the larva so as to produce the necessary
paralysis, or rather insensibility, is a difficult one, and requires great
skill and patience. The Cetonia larva is of large size, and must be
pierced in one particular spot; in order to reach this the Scolia
mounts on its victim, and is frequently dislodged by its struggles;
sooner or later, however, the proper position is obtained by the wasp,
and the larva is then stung in the exact spot necessary to allow the
sting (and the poison introduced by it) to reach the most important of
the nervous ganglia that control the movements of the body, this spot
being, in the case of the Cetonia, the line of demarcation between
the pro- and meso-thorax, on the middle line of the ventral surface of
the body. The Scolia gives but one sting to the victim, and this it will
not administer until it can do so exactly in the proper place. This
practice of devouring the victim slowly, without killing it till all is
eaten, is very widely spread in the Hymenoptera, and it is
satisfactory to find that we may infer from Fabre's observations that it
is not so horrible as it would at first appear; for it is probable that the
stinging prevents decomposition of the victim, not by reason, as
some have supposed, of the poison injected by the wasp having an
antiseptic effect, but rather by means of destroying sensibility, so that
the creature does not die from the pain, as it is believed it did in
certain cases where Fabre induced the young Scolia larva to feed on
a victim that had not been stung. We may here remark that very little
exact information exists as to the operation of stinging. Fabre
attaches great importance to the sting being inflicted on a nerve-
ganglion. Whether a sting that did not reach this part might not have
a sufficient effect appears, however, doubtful.[46]

A remarkable form of Scoliides, with wings of smaller size than usual

and deeply divided, has been described by Saunders under the
name Pseudomeria graeca. Still more remarkable is Komarovia
victoriosa found in Central Asia; in this Insect the male retains the
appearance of a slender, pallid Scolia, but the female differs totally in
form, and has the peculiar wings so reduced in size as to be useless
for flight.

Sub-Fam. 4. Sapygides.—Closely allied to the Scoliides, but

possessing slender legs and antennae; also the first abdominal
segment is less disconnected from the second, so that the
outline is less interrupted; the eyes are deeply emarginate; the
hind body is not spinose at the apex.
 Fig. 41.—Sapyga 5-punctata ♀, Britain.

The economy of Sapyga, the only genus, has been the subject of
difference of opinion. The views of Latreille and others that these
species are parasitic upon bees is confirmed by the observations of
Fabre, from which it appears that S. 5-punctata lives in the burrows
of species of the bee-genus Osmia, consuming the store of
provisions, consisting of honey-paste, that the bee has laid up for its
young. According to the same distinguished observer, the Sapyga
larva exhibits hypermetamorphosis (i.e. two consecutive forms), and
in its young state destroys the egg of the bee; but his observations
on this point are incomplete and need repetition. We have two
species of Sapyga in Britain; they differ in colour, and the sexes of S.
5-punctata also differ in this respect; the abdomen, spotted with
white in both sexes is in the female variegate with red. Smith found
our British Sapyga 5-punctata carrying caterpillars.

Sub-Fam. 5. Rhopalosomides.—Antennae elongate,

spinigerous; ocelli very prominent; tarsi of peculiar structure,
their claws bifid.

Fig. 42—Rhopalosoma poeyi. A, female imago; B, front of head. Cuba.

(After Westwood.)
This sub-family has recently been proposed by Ashmead[47] for an
extremely rare American Insect that had previously been placed by
Cresson among parasitic Hymenoptera. Westwood classed
Rhopalosoma among Diploptera, saying of it "animal quoad
affinitates excrucians." We reproduce Westwood's figure, but not
being acquainted with the Insect we can express no opinion as to
whether it is allied to the Scoliidae or to the Sphegidae. The habits
are, we believe, quite unknown.

Fam. 2. Pompilidae.

Pronotum at the sides reaching the tegulae; hind body never

definitely pedicellate, though the first segment is sometimes
elongate and conical; hind legs long; eyes elliptic in form, not
emarginate.

The Pompilidae are perhaps the most extensive and important of the
groups of Fossores, and are distributed over all the lands of the
globe, with the exception of some islands and of the inclement arctic
regions. The sting of the Pompilidae, unlike that of most of the
Fossores, inflicts a burning and painful wound; the creatures
sometimes attain a length of two or three inches, and a sting from
one of these giants may have serious results. Although there is
considerable variety in the external form of the members of the
group, the characters given above will enable a Pompilid to be
recognised with approximate certainty. The elongation of the hind
legs includes all the parts, so that while the femur extends nearly as
far back as the extremity of the body—in dried examples at any rate
—the tibiae and the long tarsi extend far beyond it; thus these
Insects have great powers of running; they are indeed remarkable
for extreme activity and vivacity. They may frequently be seen
running rapidly on the surface of the ground, with quivering wings
and vibrating antennae, and are probably then employed in the
search for prey, or some other of the operations connected with
providing a store of food for their young. Spiders appear to be their
special, if not their only, prey. Several authors have recorded details
as to the various ways in which the prey is attacked. Fabre has
observed the habits of several species, and we select his account of
the modus operandi of species of the genera Pompilus and
Calicurgus, in their attacks on poisonous spiders that inhabit holes in
the ground or in walls. The wasp goes to the mouth of the spider's
burrow, and the latter then dashes to the entry, apparently enraged
at the audacity of its persecutor.

Fig. 43.—Calicurgus hyalinatus ♀. Britain.

The Calicurgus will not actually enter a burrow when there is a spider
in it, because if it did so the spider would speedily dispose of the
aggressor by the aid of its poisonous fangs. The Calicurgus,
therefore, has recourse to strategy with the object of getting the
spider out of its nest; the wasp seizes its redoubtable foe by one foot
and pulls; probably it fails to extract the spider, and in that case
rapidly passes to another burrow to repeat its tactics; sooner or later
a spider is in some moment of inattention or incapacity dragged from
its stronghold, and, being then comparatively helpless, feels itself at
a disadvantage and offers but a feeble resistance to the wasp, which
now pounces on its body and immediately inflicts a sting between
the fangs of the foe, and thus at once paralyses these dangerous
weapons; thereafter it stings the body of the spider near to the
junction of the abdomen and cephalothorax, and so produces
complete inactivity. Having secured its prey, the wasp then seeks a
suitable hole in which to deposit it; probably an empty burrow of a
spider is selected for the purpose, and it may be at a height of
several feet in a wall; the Hymenopteron, walking backwards, drags
its heavy prey up the wall to bring it to the den. When this is
accomplished an egg is deposited on the spider, and the wasp goes
in search of a fragment or two of mortar, with which the mouth of the
burrow is finally blocked. Fabre's accounts refer to the habits of
several species, and give a good insight into some points of the
instincts of both the spider and the wasp. It seems that a sense of
superiority is produced in one or other of the foes, according as it
feels itself in suitable conditions; so that though a spider out of its
burrow and on the ground is speedily vanquished by the Pompilid,
yet if the two be confined together in a vase, both are shy and
inclined to adopt defensive or even evasive tactics, the result
probably being that the wasp will be killed by the spider during the
night, that being the period in which the attacking powers of the
spider are more usually brought into play.

It seems to be the habit of some Pompilus to procure a victim before

they have secured a place for its reception; and Fabre took
advantage of this fact, and made very interesting observations on
some points of the instinct of these wasps. Having found a Pompilus
that, after having caught a spider and paralysed it, was engaged in
making a retreat for its reception, he abstracted the booty, which was
deposited at the top of a small tuft of vegetation near to where the
Pompilus was at work. In this case the burrow in course of
preparation was subterranean, and was formed by the Pompilus
itself, which therefore could not, while it was engaged underground,
see what took place near it. It is the habit of the wasp to leave its
work of excavation from time to time, and to visit the prey as if to
assure itself of the safety of this object, and to enjoy the satisfaction
of touching it with the mouth and palping it. Desirous of testing the
wasp's memory of locality, Fabre took the opportunity, while the
Insect was working at the formation of its burrow, of removing, as we
have said, the booty from the place where it had been deposited,
and putting it in another spot some half-yard off. In a short time the
Pompilus suspended work and went straight to the spot where it had
deposited its property, and finding this absent, entered on a series of
marches, counter-marches, and circles round the spot where it had
left the prey, as if quite sure that this was really the place where the
desired object ought to be. At last convinced that the paralysed prey
was no longer where it had been placed, the Pompilus made
investigations at a greater distance and soon discovered the spider.
Fabre recounts that its movements then appeared to indicate
astonishment at the change of position that it thus ascertained to
have occurred. The wasp, however, soon satisfied itself that this was
really the very object it was seeking, and seizing the spider by the
leg slightly altered its position by placing it on the summit of a small
tuft of vegetation; this latter proceeding being apparently always
carried out by this species of Pompilus. Then it returned to its
excavation, and Fabre again removed the spider to a third spot; the
wasp when it next rested from its work made its way immediately to
the second spot, where it had last left the spider, thus showing that it
possessed an accurate memory for locality; the wasp was very much
surprised at the absence of the valued prize and persisted in seeking
it in the immediate vicinity without once returning to the place where
it had been first located. Fabre repeated this manoeuvre five times,
and the Pompilus invariably returned at once to the spot where it had
last left its prey. The acute memory for localities displayed by this
Insect seems to be more or less general throughout the Aculeate
Hymenoptera, and is of very great importance to them. The power of
finding the object appears to depend on sight, for when Fabre, after
removing the spider to a fresh spot, made a slight depression in the
ground, placed the spider in it and covered it over with a leaf, the
wasp did not find it. At the same time, the Insect's sight must be a
very different sense from our own, for the wasp, when seeking its
lost booty, frequently passed within a couple of inches of it without
perceiving it, though it was not concealed.

Belt gives an example of the habits of the Mexican Pompilus

polistoides. He noticed it, when hunting for spiders, make a dart at a
web in the centre of which a spider was stationed; by this movement
the creature was frightened and fell to the ground, where it was
seized by the wasp and stung. The Pompilus then dragged its
prisoner up a tree and afterwards flew off with it, the burden being
probably too heavy for conveyance to the nest without the vantage of
an elevation to start from.
Several modifications adopted by Pompilidae in their mode of
stinging their spider-victims have been recorded by Ferton; these we
cannot allude to in detail, but will nevertheless mention that one
species stings the body of its spider-prey at random, and that in
other cases it would appear that the paralysis of the spider is
evanescent. In short, there are various degrees of perfection in the
details of the art of stinging.

The most remarkable of the forms of Pompilidae are the numerous

species of Pepsis, a genus peculiar to America, whence upwards of
200 species are already known.[48] Some of them attain a length of
two inches or more, and are able to conquer the largest spiders;
even the formidable Mygale avicularis succumbs to their agility and
skill. Some of these Pepsis have beautifully coloured wings;
according to Cameron, this may be due to scales. P. formosus, Say,
is called in Texas the tarantula-killer; according to Buckley, its mode
of attack on the huge spider is different from that made use of by its
European ally. When it discovers a tarantula it flies "in circles in the
air, around its victim. The spider, as if knowing its fate, stands up and
makes a show of fighting, but the resistance is very feeble and of no
avail. The spider's foe soon discovers a favourable moment and
darts upon the tarantula, whom it wounds with its sting, and again
commences flying in circles." The natural retreat of this huge spider,
Mygale hentzii, is in holes in the ground, and this account does not
inform us whether the spider allows itself to be overcome when in its
nest, or is only attacked when out of its retreat.

The genus Mygnimia includes a very large number of species, and

has a wider geographical distribution than Pepsis, being found in the
tropical regions of both the Old and New Worlds, some of them
rivalling in size and ferocity the larger specimens of the genus
Pepsis. In the Insects of this genus there is usually a more or less
distinct small space of more pallid colour on the middle of each front
wing. Parapompilus is a curious genus consisting of Insects of a
great variety of peculiar coloration, and having the wings short, so as
to be of little use for flight. P. gravesii is an inhabitant of Chili.
Agenia carbonaria and A. hyalipennis are small and feeble Insects
inhabiting the south of Europe. A. carbonaria extends to the south of
England. They construct, as nests for their offspring, small
earthenware vessels, differing in form according to the species,
those of A. hyalipennis being vase-like in shape, while those of A.
carbonaria are contracted near the mouth, something after the
fashion of a wide-mouthed bottle. The Insect is able by some means
—Fabre thinks by the use of saliva—to varnish the interior of the
vessel so that it will not absorb water; the outside of the cells is,
however, not so protected, and speedily crumbles away when
exposed to the action of water; hence the vessel is placed in a
protected situation, such as in a tree-stump, or a hole in a wall, or
even in an empty snail-shell under a heap of stones. The cells are
stored with spiders that have been paralysed by stinging and that
serve as food for the larva of the Agenia. The larva of A. carbonaria
has been described, and some particulars as to its habits have been
given by Verhoeff. It has been stated that this wasp does not
paralyse its prey by stinging, but substitutes a process of biting to
prevent the spider from hurting the larva that is to feed on it; and
Verhoeff's observations seem to show that the legs of the spider are
broken by some proceeding of the kind. The Agenia larva is of
peculiar shape, the head not being inflexed, while the pleurae of
each segment, from the second onwards, are prominent, so as to
give the outline of the body a scalloped appearance. This larva is
much infested by an Ichneumon that devours, it appears, not only
the larva itself, but also the spider that was destined to be food for
the larva. Verhoeff seems to have found some evidence that
Pompilus sericeus may also be a parasite on the Agenia.

The construction of earthenware cells, instead of the burrows usual

in Pompilidae, by the species of this genus is one of the cases
alluded to in our introductory remarks as to allied Fossores exhibiting
different habits. Mr. Pride has recently sent us from Brazil similar
earthen vessels constructed by some Pompilid.
The habits of Pompilids of the genus Ceropales are analogous to
those of the parasitic bees. Pérez has recently given us information
as to a very curious form of parasitism in this genus; he says that
when a Pompilus has obtained a spider as provision for its young, it
is pursued by a Ceropales, which lays an egg on the spider, thus as
it were substituting in advance its own young for that of the
Pompilus. Information as to the subsequent course of events in this
case is not at present forthcoming. In another case a Ceropales was
observed to oviposit on the spider, not while this is being carried in,
but subsequently by entering the nest for the purpose; a habit quite
similar to that of some parasitic bees. Ferton has recently made the
unexpected discovery that some Pompilus act as robbers; one
individual taking away by force the spider that another has captured
and is carrying off.

Lichtenstein described a Pompilid larva, that he afterwards

ascertained to be Calicurgus hyalinatus, as possessing the
extraordinary habit of feeding as an external parasite fixed to the
dorsal surface of a spider; thus repeating, it would appear, the habits
of some of the Ichnemonidae, though the perfect Insect (Fig. 143)
does not differ in structure from its congeners. Emery has given an
account of some Pompilids that do not bury their prey, but after
stinging it and depositing an egg, simply leave the spider on the
spot.

Buller has described the habits of a Pompilid in New Zealand; his

account is interesting because it shows a remarkable similarity in the
proceedings of this antipodean wasp to those of its congeners on our
own side of the world. The species is not scientifically named, but it
appears that it is known in New Zealand as "the Mason-bee." It
forms a nest of yellow clay consisting apparently of about eight cells,
each of which is filled with one or more spiders in a paralysed
condition. The figure given of the larva of this Insect by Buller shows
it to possess a peculiarly formed head.
It is pleasing to find that Pompilidae do not make use of cruel
methods when others will serve their purpose. We are informed that
a large Australian Pompilid—Priocnemis bicolor—may find a Cicada
sucking sap from a hole it has pierced in a tree. The Priocnemis has
not the art of making the puncture necessary to procure sap, so the
wasp seizes the Cicada, and shakes it till it leaves its hold and flies
away, when the Priocnemis takes its place and sips the sap. It is
added that the wasp never hurts the Cicada.

Fam. 3. Sphegidae.

Pronotum free from the tegulae; when the stigmatic lobes extend
as far back as the wing-insertion, they are placed below it and
separated by a space from it.

This large assemblage of Fossores is the one about which the

greatest difference of opinion prevails. It is based entirely on the
prothoracic characters mentioned above, and cannot be looked on
as natural. We shall, however, follow Kohl[49] in treating for the
present as only one family the divisions considered by many as
distinct families. They are ten in number.

Sub-Fam. 1. Sphegides.—Hind body with a slender pedicel of

variable length; two spurs on the middle tibia. The propodeum
usually horizontally elongate.[50]

This group includes a great number of species, about 200 of which

are referred to the genus Sphex.

The habits of one species of this genus have been fully described by
Fabre; he assigns to the species the name of S. flavipennis, but Kohl
considers that it is more probably S. maxillosus. This Insect forms its
nests, in the South of France, in the ground, excavating a main shaft
with which are connected cells intended for the reception of the
provisions for the young. The entrance to the burrow is formed by
piercing a hole in the side of a very slight elevation of the soil. Thus
the entrance to the construction consists of a horizontal gallery,
playing the part of a vestibule, and this is used by the Sphex as a
place of retreat and shelter for itself; at the end of the vestibule,
which may be two or three inches long, the excavation takes an
abrupt turn downwards, extending in this manner another two or
three inches, and terminating in an oval cell the larger diameter of
which is situate in a horizontal plane. When this first cell has been
completed, stored with food, and an egg laid in it, the entrance to it is
blocked up, and another similar cell is formed on one side; a third
and sometimes a fourth are afterwards made and provisioned, then
the Insect commences anew, and a fresh tunnel is formed; ten such
constructions being the number usually prepared by each wasp. The
Insect works with extreme energy, and as the period of its
constructive activity endures only about a month, it can give but two
or three days to the construction and provisioning of each of its ten
subterranean works. The provisions, according to Fabre, consist of a
large species of field-cricket, of which three or four individuals are
placed in each cell. Kohl states, however, that in Eastern Europe an
Insect that he considers to be the same species as Fabre's Sphex,
makes use of locusts as provisions, and he thinks that the habit may
vary according to the locality or to the species of Orthoptera that may
be available in the neighbourhood. However that may be, it is clear
from Fabre's account that this part of the Sphex's duties do not give
rise to much difficulty. The cricket, having been caught, is paralysed
so that it may not by its movements destroy the young larva for
whose benefit it is destined. The Sphex then carries it to the burrow
to store it in one of the cells; before entering the cell the Insect is in
the habit of depositing its prey on the ground, then of turning round,
entering the burrow backwards, seizing as it does so the cricket by
the antennae, and so dragging it into the cell, itself going backwards.
The habit of depositing its prey on the ground enabled Fabre to
observe the process of stinging; this he did by himself capturing a
cricket, and when the wasp had momentarily quitted its prey,
substituting the sound cricket for the paralysed one. The Sphex, on
finding this new and lively victim, proceeds at once to sting it, and
pounces on the cricket, which, after a brief struggle, is overcome by
the wasp; this holds it supine, and then administers three stings, one
in the neck, one in the joint between the pro- and meso-thorax, and a
third at the base of the abdomen, these three spots corresponding
with the situation of the three chief nervous centres governing the
movements of the body. The cricket is thus completely paralysed,
without, however, being killed. Fabre proved that an Insect so treated
would survive for several weeks, though deprived of all power of
movement. Three or four crickets are placed by the wasp in each
cell, 100 individuals or upwards being thus destroyed by a single
wasp. Although the sting has such an immediate and powerful effect
on the cricket, it occasions but a slight and evanescent pain to a
human being; the sting is not barbed, as it is in many bees and true
wasps, and appears to be rarely used by the Insect for any other
purpose than that of paralysing its victims. The egg is laid by the
Sphex on the ventral surface of the victim between the second and
third pairs of legs. In three or four days the young larva makes its
appearance in the form of a feeble little worm, as transparent as
crystal; this larva does not change its place, but there, where it was
hatched, pierces the skin of the cricket with its tiny head, and thus
begins the process of feeding; it does not leave the spot where it first
commenced to feed, but gradually enters by the orifice it has made,
into the interior of the cricket. This is completely emptied in the
course of six or seven days, nothing but its integument remaining;
the wasp-larva has by this time attained a length of about 12
millimetres, and makes its exit through the orifice it entered by,
changing its skin as it does so. Another cricket is then attacked and
rapidly consumed, the whole stock being devoured in ten or twelve
days from the commencement of the feeding operations; the
consumption of the later-eaten crickets is not performed in so
delicate a manner as is the eating of the first victim. When full-grown,
the process of forming a cocoon commences: this is a very elaborate
operation, for the encasement consists of three layers, in addition to
the rough silk that serves as a sort of scaffolding on the exterior: the
internal coat is polished and is of a dark colour, owing to its being
coloured with a matter from the alimentary canal: the other layers of
the cocoon are white or pale yellow. Fabre considers that the outer
layers of the cocoon are formed by matter from the silk-glands, while
the interior dark coat is furnished by the alimentary canal and applied
by the mouth of the larva: the object of this varnish is believed to be
the exclusion of moisture from the interior of the cocoon, the
subterranean tunnels being insufficient for keeping their contents dry
throughout the long months of winter. During the whole of the
process of devouring the four crickets, nothing is ejected from the
alimentary canal of the larva, but after the cocoon is formed the larva
ejects in it, once for all, the surplus contents of the intestine. Nine
months are passed by the Insect in the cocoon, the pupal state being
assumed only towards the close of this period. The pupa is at first
quite colourless, but gradually assumes the black and red colour
characteristic of the perfect wasp. Fabre exposed some specimens
of the pupa to the light in glass tubes, and found that they went
through the pupal metamorphosis in just the same manner as the
pupae that remained in the darkness natural to them during this
stage of their existence.

Sphex coeruleus is frequently stated to have the habit of provisioning

its nests with both Orthoptera and Spiders; but Kohl considers with
reason that this record is, as regards spiders, a mistake, arising
probably from a confusion with some other Insect of similar
appearance, such as Pelopaeus (Sceliphron) coeruleus. S.
coeruleus is no doubt the same as S. (Chlorion) lobatus, which
Rothney observed in East India, provisioning its nests with
Orthoptera. He discovered a nest in process of construction, and
during the absence of the mother-wasp abstracted from the burrow a
large field-cricket that she had placed in it; he then deposited the
Orthopteron near the cell; the parent Sphex on returning to work
entered the tunnel and found the provision placed therein had
disappeared; she came out in a state of excitement, looked for the
missing cricket, soon discovered it, submitted it to the process of
malaxation or kneading, and again placed it in the nest, after having
cleared it from some ants that had commenced to infest it. She then
disappeared, and Rothney repeated the experiment; in due course
the same series of operations was performed, and were repeated
many times, the Sphex evidently acting in each case as if either the
cricket had disappeared owing to its being incompletely stunned, or
to its having been stolen by ants. Finally, the observer placed the
cricket at a greater distance from the nest, when it recovered from
the ill-treatment it had received sufficiently to make its escape. The
points of interest in this account are the fact that the cricket was only
temporarily paralysed, and that the wasp was quite able to cope with
the two special difficulties that must frequently occur to the species
in its usual round of occupations.

The genus Ammophila is of wide distribution, and its species make

vertical tunnels in the ground. The habits of some of the species
found in France have been described by Fabre. The Insect does not
inhabit the burrow while it is in process of formation, but quits it; and
some of the species temporarily close the entry to the incomplete
nest with a stone. The tunnel is a simple shaft with a single cell at its
termination; this is stored with caterpillars, the different species of
Ammophila selecting different grubs for the purpose. A. hirsuta
hibernates in the perfect state, and carries on its work in the spring; it
chooses a single larva of considerable size belonging to one of the
nocturnal Lepidoptera, and this it paralyses by a series of about nine
stings, of which one is implanted in each segment from the first
thoracic ring backwards; it forms the burrow only after the food to be
placed therein has been obtained. The caterpillar used is
subterranean in habit, and the Ammophila detects the larva by some
sense, the nature of which appears at present quite uncertain. A.
holosericea chooses smaller larvae of the family Geometridae, and
uses only one or two stingings to paralyse each larva; several
caterpillars are used to provision a single cell, and they are often
selected of different colours.

Marchal has also published an important account of the proceedings

of A. affinis; he confirms Fabre's observations, and even adds to
their interest by suggesting that the Ammophila administers special
stings for the purpose of paralysing the mandibles of the caterpillar
and depriving it of any power of afterwards injuring the larva that will
feed on it. He thinks the mother-Ammophila herself profits by
appropriating an exudation from the victim.

Some species of Sphegides have the curious habit of choosing the

interiors of human habitations as the spots most suitable for the
formation of their own domestic establishments. Fabre has given a
charming account of the habits of Pelopaeus (Sceliphron) spirifex, a
species that inhabits the South of Europe, and that forms its nests in
the cottages of the peasants. The spot usually selected is a nook in
the broad, open fireplace, out of reach of the flames, though not of
the smoke; here the Pelopaeus forms a nest of earth, consisting of
ten to fifty cells, the material being mud or clay brought in little balls
by the aid of the Insect's mandibles; about twenty visits are required
in order to complete one cell, so that for the construction of a large
nest of fifty cells, about one thousand visits must be made by the
Insect. It flies in and out of the house apparently not at all
incommoded by the human habitants, or by the fact that the
peasant's potage may be simmering on the fire quite close to where
the fearless little creature is carrying on its architectural operations.
The cells are stored with spiders, of which the wasp has to bring a
plentiful supply, so that its operations extend over a considerable
period. The prey is captured by the Pelopaeus whilst on the wing,
and carried off at once, being probably stung by the wasp during the
process of transit; apparently it is killed by the operation, not merely
paralysed. Only small spiders are taken by this species, and the
larva of the Pelopaeus consumes them in a short time, one by one,
before the process of decomposition sets in; the egg, too, is laid on
the first spider introduced, and this is of course at the bottom of the
cell, so that the spiders are eaten by the wasp's larva in the order in
which they were brought to the cell. The cell is sealed up when full,
the number of spiders placed in it being on the average about eight.
The larva completes its task of consuming the store in about ten
days, and then forms a cocoon for its metamorphosis. Two or three
generations are produced in a single year, the autumnal one passing
eight or nine months in the clay cells, which are lodged in a nook of
the peasant's hearth, and exposed to the smoke of his fire during all
the months of winter. Pelopaeus (Sceliphron) is a genus including
many species;[51] several of them are known to be specially attached
to the habitations of human beings. Roth has given an account of the
habits of P. (Sceliphron) laetus in Australia; he says that in some
parts it is very difficult to keep these wasps out of the houses; the
nest is formed of mud, and constructed on the furniture or in any part
of a room that suits the fancy of the Insect. This it must be admitted
is, according to human ideas, liable to the charge of being very
capricious. Roth timed a wasp building its nest, and found that it
brought a fresh load of mud every two or three minutes. If the wasp
be allowed to complete the nest undisturbed, she does so by adding
to the exterior diagonal streaks of mud, so giving to the nest the look
of a small piece of the bark of a common acacia. The construction
consists of from ten to twenty cells, and when completed is
provisioned with spiders for the use of the young. This wasp is much
pestered by parasites, some of which prevent the development of
the larvae by consuming the spiders intended by the mother-wasp
for its young. A fly, of the Order Diptera, is said to follow the wasp
when carrying a spider, and to deposit also an egg on the food; as
the Dipterous larvae have more rapid powers of assimilation, the
Pelopaeus larvae are starved to death; and their mildewed remains
may be found in the cell, after their enemies have become fully
developed and have flown away. Another parasite is said to eat the
wasp-larva, and attains this end by introducing an egg through the
mud wall and the cocoon of the wasp—a habit that seems to indicate
a Leucospid parasite. Tachytes australis, a wasp of the sub-family
Larrides also dispossesses this Pelopaeus in a manner we shall
subsequently describe. This fragment of natural history from
Australia has a special interest, for we find repeated there similar
complex biological relations to those existing in the case of the
European congeners.

P. (Sceliphron) madraspatanus is common in the north-west

provinces of Hindostan, and is called the "mud-dauber" by the
European residents. According to Horne it constructs its cells in the
oddest places, but chiefly about the inhabited apartments in houses.
It is perfectly fearless when engaged in building: the cells are four to
six in number, and are usually provisioned with spiders to the
number of about twenty. On one occasion it was observed that green
caterpillars were stored instead of spiders. The species is said to be
protected by a peculiar odour as well as by its sting; it is also stated
that it disguises its edifice when completed by making it look like a
dab of mud, and on one occasion "rays of mud were observed round
the nest, even more exactly imitating a lump of mud thrown with
some force." P. (Sceliphron) bilineatus, formerly thought to be a
variety of P. madraspatanus, builds its nests in hedges and trees.

Sub-Fam. 2. Ampulicides.—Prothorax long and narrow,

forming a neck in front; clypeus beak-like; four submarginal cells,
the outer one being complete; metathorax elongate, the
posterior part of the metasternum deeply divided to allow a
perfect inflection of the abdomen.

Fig. 44—Ampulex compressa. Male. East India.

This is one of the smallest of the divisions of the Sphegidae, but has
a very wide distribution, being represented in both the Eastern and
Western Hemispheres. It is allied to the Sphegides, but differs by the
prolongation of the neck and of the head, and by the articulation
between the petiole and thorax being placed on the under surface of
the body; the wing-nervures are said to be of inferior importance
owing to their frequently differing in individuals of the same species.
These Insects appear to be rare in individuals, as well as few in
species, and but little has been recorded as to their habits; but it is
known that they live on cockroaches. Perkins has given a brief
sketch of the habits of Ampulex sibirica that is of great interest, but
requires confirmation. He says that this Insect, in West Africa, enters
apartments where cockroaches abound, and attacking one, that may
probably be four times its own size, succeeds, after a struggle, in
stinging it; the cockroach instantly becomes quiet and submissive,
and suffers itself to be led away and placed in confinement in some
spot such as a keyhole, and in one case was apparently prevented
from afterwards escaping, by the wasp carrying some heavy nails
into the keyhole. The larva of the Ampulex may be presumed to live
on the Blattid, as it is added that dead bodies of the cockroaches are
frequently found with the empty cocoon protruding from them. This
account, if correct, points to some features in the habits of this Insect
that are unique. A remark made by Rothney in reference to the
habits of A. (Rhinopsis) ruficornis seems to indicate some similar
instinct on the part of that species; he says, "I also saw two or three
of these wasps collar a peculiar cockroach by the antennae and lead
it off into a crack in the bark, but as the cockroach reappeared
smiling each time, I don't know what was up." The same observer
records that this species associates with Sima rufonigra, an ant it
greatly resembles in appearance, as well as with a spider that is also
of similar appearance (Fig. 72). Schurr has given a brief account of
the proceedings of Ampulex compressa, and his statements also
tend to confirm the correctness of Perkins' report. The habits of a
species of Ampulex were partially known to Réaumur, who described
them on the authority of M. Cossigni. The species is believed to be
A. compressa, which occurs not only in East India, but also in the
island of Bourbon, the locality where M. Cossigni made his
observation: his account is, like the others, a mere sketch of certain
points observed, the most important of which is that when Ampulex
cannot introduce the cockroach into a hole that it has selected as
suitable, it bites off some portions of the body in order to reduce the
poor Insect to the necessary extent.

From these fragmentary observations it would appear that the sting

of the Ampulex has not so powerful a paralysing effect as that of
most other Fossores; and that the Ampulex does not form any nest,
but takes advantage of suitable holes and crevices to store the victim
in; also that it displays considerable ingenuity in the selection of
materials with which to block up the cavity in which it has placed the
partially incapacitated creature.

The genus Dolichurus is by some entomologists considered the type

of a sub-family allied to the Ampulicides; it long consisted of a small
and rare European Insect, but some exotic species have recently
been added to it. It will probably prove not sufficiently distinct from
Ampulicides, although the pronotum is much shorter, but Handlirsch
has recently observed that the European species attacks Blattidae
as do the normal Ampulicides; and Ferton has recorded that D.
haemorrhous lives at the expense of Loboptera decipiens, the wasp
depositing its egg on the left intermediate femur of the prey. This is
placed in a solitary cell, and is entirely consumed by the larva, life
being preserved till within a few hours of the end of the repast, which
occupies altogether eight days.