MODAL WORLDS

Whitman Richards†

with an introduction by Jan Koenderink

D E C LOOTCRANS P RESS
Front cover: The famous “Blue Marble” (thanks to NASA) is a selfie by us earthlings. It
shows the earth as limited although it really has all our umwelts as microscopic parts. To
us the earth is the universe, too complicated to deal with except through the force of habit
and type. According to David Hume habit is the cement of the universe. He was right. In
this book Whitman Richards explores the ins and outs of it.

De Clootcrans Press
Utrecht The Netherlands
KoenderinkJan@gmail.com

Copyright c 2018 by Whitman Richards†

All rights reserved. Please do not redistribute this file in any form without my
express permission. Thank you!

pax / jan koenderink

First edition, 2018

10 9 8 7 6 5 4 3 2 1
MODAL WORLDS
Whitman Richards†
with an introduction by Jan Koenderink

D E C LOOTCRANS P RESS , T RAIECTUM MMXVIII

 Modal Worlds

When I received an email starting with “my life expectancy is down to . . . ”, it

was not unexpected. Whitman Richards was my best friend, and I was very aware
he had not long to go.
In the mail Whit asked me to take care of his book “Modal Worlds”, suggesting
I might finish it and promote myself to first author. A week after Whit died I
received a parcel with the manuscript, mailed to me by one of his daughters. It
put me in a quandary.
Of course, I wanted to see Whit’s book published, but I felt it should be his
book, every iota of it. The publisher had a review that suggested a swing to the
mainstream. Whit would never have done that, neither would I.
Eventually, after some hassles with the publisher, I decided to print Whit’s
book in my own press, with an introductory first chapter.
I did not change anything in his book. It came as a W ORD file with included
images. It is printed entirely as is.
Of course, you’re free to skip the introductory chapter. I enjoyed writing it,
it made me think of Whit and how we would have discussed it over drinks, me
gulping down Old Salt Ale, Whit sipping a gin-and-tonic double-shot.
Enjoy, and remember Whit, last of the independents.

Utrecht, June 21, 2018 — Jan Koenderink

Jan Koenderink
University of Leuven (KU Leuven)
Laboratory of Experimental Psychology
Tiensestraat 102 – bus 3711
3000 Leuven
Belgium
jan.koenderink@kuleuven.be

i
ii
W HITMAN R ICHARDS died September 2016 after a heroic fight with myelofi-
brosis. He was professor emeritus of MIT, where he worked for over sixty
years. Whitman was one of the first four PhD graduates of the Psychology Depart-
ment, started by Professor Hans–Lukas Teuber in 1960. Eventually Psychology
became the Department of Brain and Cognitive Sciences.
Whitman was professor of cognitive sciences and of media arts and sciences
and principal investigator in the Computer Science and Artificial Intelligence Lab-
oratory,
His importance for the field of V ISION S CIENCE can hardly be overestimated.
Whitman pushed David Marr at MIT, this in a sense started modern computational
social science. He was instrumental in pushing Bayesian statistical models of per-
ception and cognition. Whitman moved between classical psychophysics, com-
putational representations of perception and an amazing variety of formal models
of perception and cognition.
Whitman was a dedicated mentor and teacher. He told me his ideal was to
train a new generation of Leonardo’s. Many of his pupils had remarkable careers.
Whitman was always surprising and managed to boost creativity and unbridled
curiosity in others, although anyone had a hard time to keep up with his ideas.
I’ll never forget his wry humor and I sorely miss our talks during long walks.

A
B
Introduction to Modal Worlds

Jan Koenderink
A
W
HAT does it mean for the World to “be modal”? In a nutshell, it means
that it enables you to live your life, period. It is as important as that. A
non-modal World could not be known, in the sense of barring any form
of anticipation. Voluntary action and recognition would be logically impossible.
Listen to David Hume (1711–1776):1

Custom, then, is the great guide of human life. It is that principle alone
which renders our experience useful to us, and makes us expect, for the
future, a similar train of events with those which have appeared in the past.
Without the influence of custom, we should be entirely ignorant of every
matter of fact beyond what is immediately present to the memory and senses.
We should never know how to adjust means to ends, or to employ our natural
powers in the production of any effect. There would be an end at once of all
action, as well as of the chief part of speculation.

As Hume framed it, habit is “the cement of the Universe”. How right can you be?
In a sense, this fundamental insight is what Whit’s “Modal Worlds” develops.
Habit is based on “type”, as in Edmund Husserl’s (1859–1938) “typifying
apperception”. For Husserl’s Typus you may simply read “mode”. ‘Typifying ap-
perception” is modal awareness. The type is “sedimented experience”, it is not
conceptual, but closer to Hume’s idea of habit. It results from arbitrary associa-
tion, frequently multimodal.
Such habits and types often evoke awareness of a non-linguistic, perhaps
proto-conceptual nature. Immanuel Kant (1724–1804) already noticed this and
referred to it as “phantasmatic self–stimulation”.
We no doubt share this with many animals, or sentient beings in general. Etho-
logical research, as pioneered by Jakob von Uexküll (1864–1944), certainly sug-
gests as much. I’ll cite examples later.
Psychology has generally been hesitant to absorb such notions, although
William James (1842–1910) and the Gestalt psychologists were tuned in on the
same general wavelength. Especially Egon Brunswik (1903–1955) attempted to
quantify the modes of the Umwelt in a statistical sense.
Of the moderns one perhaps thinks of James Gibson (1904–1979), whose “af-
fordances” may be read as modes. More recently one thinks of Donald Hoffman’s
(born 1955; a student of Whitman Richards) “interface theory of perception”,
where the interface elements may be thought of as modes. Whitman Richards
presents a fresh opening though.
This is a very general and broad topic. It is impossible to discuss it as a neat
linear progression of concepts. Instead, I’ll consider modes in the physical world,
1
David Hume, An Enquiry concerning Human Understanding, 1748.

B
the realm of biological organisms, the phenomenology of awareness and finally,
the realm of human praxis and life.
In conclusion, I’ll attempt a synthetic view.

???

T HE cosmologist’s conviction that the universe started when emptiness ex-

ploded into everything (the “Big Bang”) is—on the face of it—rather similar
to Genesis 1:2 (King James version):

. . . And the earth was without form, and void; and darkness was upon the
face of the deep . . . .

In both cases the start is absolutely modeless, or formless (tohu wa–bohu). Modes
then evolve, or are created.
As we face it here and now, the cosmos is thoroughly modal on all scales. We
recognise only scales we can deal with, and consequently new ones are explored
regularly as science progresses.
We typically understand anything in terms of space and time. Modes in time
appear to us as cause, modes in space as repetition, or pattern. In both domains
the numbers are staggering.
In time we may (just an example) consider the number of collisions per sec-
ond of an air molecule at room temperature and normal atmospheric pressure, it is
about ten–to–the–tenth per second. Consider how often the Newtonian two parti-
cle collision mechanics (here the mode is a billiard balls engagement, the prover-
bial causal event) is tested in your very living room! The occasions to consider
the causal interaction of particles are beyond human imagination.
There may be 1080 electrons in the universe, yet Richard Feynman could say
that there is essentially only a single one: for they are all identical in the sense
of Wilhelm Gottfried Leibniz’s (1646–1716) Principle of the Identity of Indis-
cernibles. Quantum theory has put Leibniz’s famous Principle in the front row.
Wolfgang Ernst Pauli’s (1900–1958) notion of fermions and bosons has changed
our perspectives forever. According to the physicist, that we are supported by
the floor is due to the fermionic nature of electrons. All being identical fermions
prevents them from all being at the same place, which is why matter is extended.
I could go on quoting numbers. For instance, there are 1021 stars and
10 galaxies. Are these just “astronomical” numbers, fun perhaps, but irrelevant
11

to the terrestrial environment, or especially life?

No. There about three trillion trees on earth. There are about forty trillion
cells in your body and about as many bacteria. There are over ten million billion
mitochondria in your body. The haploid human genome occupies a total of just
over three billion DNA base pairs, that means six billion base pairs per diploid

C
cell. One could go on like that forever. Note that all your mitochondria look the
same, a major mode. Trilions of trees grow upright, another major mode. Modes
dominate structure on all spatial scales.
The world is endless repetition, both in space and in time. That is why science
is so successful. According to Edwin Thompson Jaynes (1922–1998) Bayesian
probability is the Logic of Science. No doubt, he was right about that. Formal
“logic” is a mere mathematical game that allows trivial deductions, it has no rela-
tion to science. Science starts with observation and intuitive induction, eventually
opening the road to rational inference. Such inferences are best bets, they in no
way command or force nature to act in a certain way.
Such “logic” strives on good priors and good odds for updating belief. Small
wonder this works as well as it does given the redundancies involved! It works so
well that we actually fool ourselves to have “discovered Laws of Nature”, where
all we noticed were natural modes.
Ludwig Wittgenstein (1889–1951) was surely right in holding that

The great delusion of modernity, is that the laws of nature explain the uni-
verse for us. The laws of nature describe the universe, they describe the
regularities. But they explain nothing.

Yet, such a dictum is a chutzpah for who does believe in God’s Eye, as probably
the majority of scientists today does. We have no possible “reason” to hold that
the electron should behave according to Paul Dirac’s (1902–1984) famous equa-
tion. That the sun will rise tomorrow is a (pretty good) bet, but we cannot claim
certainty. Shit happens, so much is for sure, but who knows when? Science has
no answer.
In science, there are quite reasonable ways to divide space–time into domains
that are mutually fairly independent. I mean, independent on a certain “funda-
mental” level. Within a domain laws are simple and neat, say simple systems of
partial differential equations. “General solutions” are rarely of interest. Interest-
ing solutions are those which satisfy certain “boundary conditions”. But these
latter come necessarily from outside the domain. That is where the trouble for
the fundamentalist starts. For, in practice, these “incidental” interactions between
domains largely decide on “what happens”.
Superordinate levels act as constraints on subordinate levels. Subordinate lev-
els supply the “raw material” for the superordinate levels. This also pesters the
fundamentalist, these kinds of interactions inject “noise” into their neat system,
deriving from other domains. It is a reason (boundary conditions being another
one) why weather prediction doesn’t work, although all the fundamental equations
(hydrodynamics, thermodynamics, . . . ) are in place and there is nothing wrong
with these.

D
This tends to be better understood by engineers than by physicists. The latter often
hold a variety of notions that might best be considered “romantic”. A minor phe-
nomenon like the presence of oceans on earth could never have been “predicted
from first principles”. It is one of the many charming romantic make–beliefs of
theoretical physicists though. They rarely notice that they predict in retrospect,
for that is really all they can do. But it surely defeats their romantic notion.
So much for physics, or the “exact sciences” in general. Now for something
more interesting.

???

B IOLOGY is next on my list. It is closer to the human condition, hence more

interesting than physics. At least, I would hold that it is more useful to
consider a human as a sentient being than as an incidental get-together of atoms
or molecules.
Of course, some people would object. For instance the behaviourists held that
human speech is nothing but the movements of air molecules. No doubt they were
right in some way, but who wants to know? I’ll take the position that humans are
sentient beings of some kind. I tend to think Whitman Richards and I generally
agreed on that.
As indicated above, biological numbers are “astronomical”. In 2010 there
were twenty billion chickens on the earth (two billion in Europe—where I live—
alone), whereas the total chicken egg output was about 1.1 1012 per annum. Just
imagine, over a trillion eggs, all looking the same! There is some mode for you.
“Who was first, the chicken or the egg?” is an age–old, famous philosophical
dillema. The common answer is provided by Genesis 1:25 (King James version):

And God made the beast of the earth after his kind, and cattle after their
kind, and every thing that creepeth upon the earth after his kind: . . . .

So the chicken (“cattle”) beat the egg to it.

But listen: somewhere around ten-thousand years ago—in a settlement in what
is now Thailand—the crossing of a wild Grey rooster with a tame Red hen pro-
duced an egg from which emerged the first modern chicken. Notice the clash with
the creation story. So, according to the latest science, the egg beat the chicken to
it!
Today the chick’s descendants are the most common bird on earth. In relation
to the evolutionary history the chicken–and–egg cycle runs blindingly fast. It
is a fast swirling eddy drifting along in the stream of evolution. We only see
the vortex, a circular process without beginning or end. In the eddy there is no
causation, unless you say that the chicken and the egg are “each other’s cause”.

E
This simple insight is a crucial one, because it solves a problem of teleology that
ended the careers of so many biologists in the mid twentieth century because they
were accused of being “vitalists”. Nowadays the theory of evolution is understood
in a more implicate sense and the (really unpleasant) witch hunt is over.
Perhaps unfortunately, such witch hunts were common during that era, being
triggered by convictions deriving from the philosophy of science. In psychology,
this led to the demise of Gestalt ideas and phenomenology and the growth of
behaviorism in the guise of “cognitive science”. In philosophy the “linguistic
turn” led to the hegemony of “analytic philosophy”. We still live in the aftermath
of these—in my view unfortunate—mid–twentieth–century events.
One of the victims of the vitalist witch hunt (with Konrad Lorenz (1903–1989)
acting as Inquisitor General) was Jakob von Uexküll. In retrospect it was an infor-
mal trial on which biology should look back in shame. At the time, the reliance
on a rather one-track version of evolution theory had all the characteristics of a
religious belief. This was perhaps understandable as biology had to fight in order
to be recognised as a true science. Yet, today the understanding of evolution is far
more intricate. It is not hard to render many “vitalist” notions salonfähig.
In retrospect, Jakob von Uexküll essentially constructed what is presently
known as ethology on his own steam.2 His insights on animal behaviour, the
Umwelt and the “functional tones” are extremely valuable, even today. He shows
experimentally how habits and type are formed from “sedimented experience”.
He illustrates how types are often odd, multimodal, apparently arbitrary combina-
tions of Merkmale (cues).
I would say that the biologist who deserves special credit for showing how to
get to grips with such problems is Rupert Riedl (1925–2005). Riedl worked out
the modal structure of the biological world in full detail from the submolecular to
the psychological and even sociological level. His books deserve close study as a
compliment to Whitman’s “Modal Worlds”.3
Riedl has a good understanding of the modal structure of the world, as well
as its importance to our thinking and behaviour. His examples are insightful and
interesting. A rabbit without a heart would make the headlines, so would a fish
with a hair. An archeologist/primatologist reconstructs a new human species from
a tooth, a mandible or a hipbone. These are all due to natural modes.

???

2
Perhaps interesting to know that Lorenz shared a Nobel Prize for the foundation of ethology.
3
Riedl, R. (1978) Order in Living Organisms: A Systems Analysis of Evolution. New York:
Wiley and Riedl, R. (1984) Biology of Knowledge: The Evolutionary Basis of Reason. Chichester:
John Wiley & Sons.

F
P SYCHOLOGY so far added apparently little to the Modal World ideas. An early
example is Egon Brunswik, who attempted to find the basis for simple Gestalt
Laws in the statistics of (retinal) images. At the time that was a really heroic (or,
perhaps, foolish) attempt, because essentially an impossible project. The idea was
certainly a good one though. Nowadays, thanks to the advent of the computer,
such researches have become possible. Indeed, today one witnesses increasingly
many attempts to collect such evidence.
In the mid-twentieth century and later James Gibson developed his notion of
“affordance”. An affordance is a mode that is recognised as such by an active
agent. However, Gibson placed the affordance “in the world”. Thus he considered
the throwability of a stone as a property of the stone, on a par with its size and
weight. Here he stood orthogonal to von Uxküll, who found the “throwability”
as a “seek image” or a “functional tone” in the mind of the agent. Notice that I
use “mind” in a very general sense, by no means implying “consciousness” or the
concept of a “self”. I would not hesitate to speak of the mind of any sentient being,
much in the vein of von Uexküll, who actually was close to being a behaviourist.
This brings up an interesting topic. If the “world is modal” then where are the
modes? If the modes are in the mind, then is the world in the mind? I would an-
swer yes, to the latter question4 , Gibson would surely say no, von Uexküll would
probably think of the modes as part of the Umwelt of the agent.
That latter notion is quite close to Donald Hoffman’s concept of perception as
a user interface. Here the modes would be internalised as interface elements. This
is quite close to von Uexküll and at a far remove from Gibson.
More recently psychology has developed notions of “mental models”, I think
especially of the work of Philip Johnson-Laird (1936) and George Lakoff (1941).
The mental models are modes by which the mind handles complexities of the
world. They are in many respects close to Husserl’s habit and type, Hoffman’s
user interfaces or von Uexküll’s structured Umwelts. They represent modal struc-
tures as filtered by the nature and life style of the agent, say its Nietzschian “Will
to Power”5 , a primordial “urge”, typical of life. I feel that such “urges” are like
proto–intentionality, as one finds them in unicellular organisms that one would be
hesitant to ascribe a “mind”. In humans they belong to pre–awareness, yet are
nevertheless prime movers.

???
4
But notice that I really think of body–mind–world as a T RINITY, thus the affirmative answer
should not be misconstrued.
5
Friedrich Nietzsche (1844–1900 ), Also sprach Zarathustra: Ein Buch für Alle und Keinen,
1883–1885 [Thus Spake Zarathustra, translated by Alexander Tille, Macmillan, New York and
London, 1896].

G
P HENOMENOLOGY is the link between the sciences and awareness. It is the
investigation of concrete actuality from the first person perspective. Concrete
actuality is a BRUTE FACT, thus its study should be truly considered physics.
However, concrete actuality from the first person perspective is the ultimate
subjective brute fact, which locates it outside the sciences proper, so no physics!
In my view a psychology that refrains from phenomenology should properly
be considered “dry physiology” (“dry” in the sense of drawing no blood, no tam-
pering with the body of the agent).
Mainstream psychology is of a different opinion though. Most psychologists
are ready to consider phenomenology as “unscientific” and therefore ignore it. Of
course, this is not a little hypocritical. Objectivity implies observations without
an observer, something that is even outside the reach of (proper) physics. In psy-
chology that would lead to the barest behaviorism. Yet psychologists and even
physiologists are only all too eager to describe their findings to the public in phe-
nomenological terms. Just think of brain scanning, pure physiology, which is
often suggested to reveal thoughts and feelings. Really foul play and on a grand
scale too! Yet it promotes huge research grants. The public craves for such cheats.
The scope of phenomenology is very wide. The “observations” may remain
individual, or may be communicated. Communication ranges from body language
and facial expression, over dance, music, painting, drawing . . . , to poetry and
prose. The topic may be in objective or subjective mode, being treated objectively
or subjectively, in any combination.
Here I am especially interested in analytical researches as of possible inter-
est to the scientist (“scientist” in the sense of investigator of brute facts). Ed-
mund Husserl is the prime example.
At this point Husserl’s notion of “typifying apperception” is very relevant.
The “types” are “sedimented experiences”, that are associations between sensory
structures (stimulations or changes of state at the sensitive body surfaces) that co-
incide with relevant events. Such types then may become “cues” (Von Uexküll’s
Merkmale) that trigger elements of awareness or even actions. A faint, unnoticed
lemon smell may cause a yellowish blotch to pop up as a “lemon” in your aware-
ness. You may even notice an acid taste in your mouth as a part of that awareness.
Neither awareness nor type recognises a division into Aristotle’s “five senses”,
smell, taste, touch, hearing and sight.
In the animal Kingdom one has even more striking examples, many recorded
in von Uexküll’s ethology. A feeling of warmth and a smell of butyric acid cause
a female tick to let go from her twig, in the (almost certain!) expectation of hairy
skin and a meal of mammalian blood on boring in. It is one of the few modes in
the tick’s Umwelt, but it is entirely crucial to its lifestyle. The ticks are blind and
deaf. They may wait for decades, hanging on a twig, waiting for a mammal to pass
by. In the meantime they neither eat nor drink, time and space are nonexistent for

H
them. A mammal is the mode: “a feeling of warmth and a smell of butyric acid”.
The mode allows a certain bet—at least, as certain as bets can be.
Husserl’s notions of habit and type essentially boil down to this. Concepts
from physics, physiology, biology, philosophy, psychology and phenomenology
converge here. The common denominator is the modal structure of the world.
That is why I think Whitman’s “Modal Worlds” is on a uniquely fundamental
issue.

???

P HILOSOPHY in the mode of the “natural philosophy” of the past, as opposed

to the rather sterile “analytic philosophy” of today, has added quite a few
concepts to the topic of modes.
David Hume was close to being a natural philosopher. Many of his ideas are
very close and very relevant to the topic of Modal worlds. Hume also understood
both biology and phenomenology in saying:

There is a secret tie or union among particular ideas, which causes the mind
to conjoin them more frequently, and makes the one, upon its appearance,
introduce the other.

So to Hume, the Principles of Association are basic. He identifies resemblance,

contiguity in time and place, and causation, although he grants that the deeper
reasons for these

are mostly unknown, and must be resolv’d into original qualities of human
nature, which I pretend not to explain.

I already mentioned earlier that Hume regards such Principles as

only ties of our thoughts, they are really to us THE CEMENT OF THE UNI -
VERSE , and all the operations of the mind must, in great measure, depend
on them. [my emphasis]

It is custom or habit, that are modes, that

determines the mind . . . to suppose the future conformable to the past,

as opposed to reason. Reason is trivial, like logical calculus, or deduction. But the
problem of life is induction and there custom and habit are the only tools except
for random choice. Of course, the latter may sometimes be the best course for
action. “Don’t just stand there, do something!” or just kicking around at random
are options that at least fit basic urges of life—fundamental prime movers. But
when in familiar territory custom and habit are by far the best bets. They are trivial

I
and “cookie cutter–style” (but crucial!) applications of the Logic of Science. This
is basic to the Modal Worlds notion.
Immanuel Kant had an important insight when he described phantasmatic
self–stimulation. What he noticed was that such “cookie cutter–style” (or per-
haps user–interface–style) actions are often acted out “behind the curtains” so to
speak, in the sense that they already happened on the moments you become aware
of them. They are apparently physiological mechanisms that somehow manage
to invade awareness. Of course, one has various well known instances of this
phenomenon, for instance, think of synestesias.
The phantasmatic self–stimulations set up the next stage of awareness on the
basis of actual, as yet uninterpreted sensory stimulation. You notice them in cases
they fail, as when you “see” things that aren’t there. But you never notice the—
immensely more numerous—cases where they cause you to see things that actu-
ally turn out to be there. Of course, you wouldn’t notice, you simply “see things”.
But without the (unnoticed) phantasmatic self–stimulation you would never see
anything to begin with! Seeing is not one of your actions, it happens to you. It is a
brute fact. Notice how this fits seamlessly in the “vision as user-interface notion”.

E NGINEERING , as in building bird’s nests, beaver dams, ant hills, mole tun-
nels, hermite crabs adapting empty shells for shelter and the like (just add
a few examples for yourself, there are countless ones!) changes the structure of
the Umwelt. Human cultures add to these examples big time. They change the
landscape both on large spatiotemporal scales (think of such practices as cutting
down forests, of cultivating crops and breeding and raising livestock) and on more
fine-grained ones (cities, roads, . . . ). They also change the available affordances
on the scale of the body (clothes, tools, vehicles, weapons, . . . ). Perhaps even
more importantly, they change the lifetime and spread of insights and know-hows
(language, books, educational systems, . . . ). This has a major, qualitative impact
on the course of the evolution of life on earth.
All this depends crucially on modal structures, indeed, would be impossible
without that. It is all habit and type. The modal structures themselves depend on
habit and type. The car engine is usually at the front of the driver, mainly because
horses used to draw (instead of push) carriages. The British drive on the left–
side of the road in order to leave the driver room for wielding his whip—perhaps
not so much in common use anymore. Military formal dress is imaged after the
defensive armour of the middle ages. One could go on for ever.
This makes that biological evolution has become a Lamarckian–style
biological–cultural evolution, with the implication of an enormous speed–up, a
greatly increased error rate and an even more uncertain prognosis, even on the
mid–term. “Modal Worlds” is also very much on this topic.

J
S O how to sum up such obviously important insights? What I will try to do here
is suggest some hints to a formalisation of Whit’s ideas. In his Modal Worlds
Whitman sketches a dim view on the possibilities of doing so and for good rea-
sons. After all, we are talking evolution here, which is the ultimate creative pro-
cess.
Creativity is the art of overcoming hurdles through novel means, or—more
correctly—the novel application of established means. “Novelty” is a difficult
concept. I would say the only true novelty is full unpredictability, thus random-
ness.6 Then creativity would be a kind of controlled, or canalised randomness.
The control is the result of two factors, one is a (possibly very vague) goal, the
other is the existence of constraints, or hurdles to overcome.
Here the problems starts, for if I were to say biological evolution has “goals”,
I would be branded a “vitalist”. I will conceive of goal as blind urge, similar to
Nietzsche’s Wille zur Macht (echoing Arthur Schopenhauer’s Wille). I’ll take it
for a fact that all life strives to increase its dominion, it seems simply a brute fact
to me. I’m not aware of any exceptions. Then biological evolution indeed fulfils
all the requirements for being considered a creative process.
There is hardly a doubt that biological evolution is a random process that ex-
ploits the modal structure of the natural world in any possible manner. One of
the numerous striking examples is the evolution of eyes. It happened many times
over, each independent of the others. It resulted in a great variety of eyes based on
different principles. Few of the known physical principles have not been exploited
in the evolution of eyes. No doubt our insights are still incomplete.
As one might expect, there are many similarities between technological evolu-
tion and biological evolution. Whitman’s Modal Worlds has much on that, Riedl’s
book on the evolution of rationality is also a great source.
What I will attempt here is to consider awareness as a creative, evolutionary
process on the short term. Indeed, the shortest term, because awareness is concrete
actuality, it only lasts for the present moment! In humans the psychogenesis of
awareness develops in about a tenth of a second. Although each moment exists
only for itself, moments evolve legato–style, so when one is done it dies, but then
the next one is already on the way, ready to take over.
I will have to explain why psychogenesis is best understood as an evolutionary
process and why the modal structure of the world is essential to it. The best source
to the understanding of psychogenesis in humans is (at least to my mind) Jason
Walter Brown (born 1938), a neurologist and neuropsychiatrist.7
Psychogenesis generates your concrete actuality, that is the present moment,
6
Notice that there cannot exist algorithms to compute randomness. Only Nature (quantum
events) generates truly random events—as far as we know. Thus we may generate randomness by
exploiting Nature, but not by algorithmic processes.
7
See Brown, J. W. (2014) Microgenetic Theory and Process Thought.

K
including the presence of the past and the presence of the future (Husserl’s reten-
tion and protention) on the basis of emotional state and urges, habits and phantas-
matic self–stimulation, imagery from dream-like to hallucinatory, all constrained
by the physical environment, including the states of the sensitive body surfaces.
The constraints are very important and most animals have specialised neural
nets to deal with them. In humans they are often called sensoria, brain areas
dedicated to sight, hearing, and other modalities. The sensoria are essentially
volatile buffers that are continuously overwritten with novel structure. They tend
to show up a neat organisation, for instance, optical structure is stored in terms
of various image elements as colours, gradients, movements, . . . . This allows
easy addressing of coherent entities, like “moving, red, circular patch on a gray
background”. There is a modal structure. I’ll refer to the visual sensoria (there
exists a variety) as the BLACKBOARD.
There is a complementary structure dedicated to motor actions. This allows
the triggering of complicated patterns of muscle contractions as coherent entities.
There is another modal structure. I refer to the motorium as the KEYBOARD.
There is no meaning in nature. That cannot be, for meaning exists only in re-
lation to agents. A tree has different meaning (or affordance) to a human, a dog or
a bird. Meaning is not found, computed or detected, it needs to be imposed. This
is Giovan Battista Vico’s (1668–1744) Verum Factum Principle.8 Von Uexküll’s
notions go a long way to capture this, but the first one to suggest a good heuristic9
for the genesis of awareness was Erwin Schrödinger (1887–1961).10 Psychogen-
esis probes physical constraints, if it encounters friction this elicits a spark of
awareness (my formulation).
I would liken the task of psychogenesis in the construction of concrete ac-
tuality as akin to the task set to a criminal investigator, elsewhere I called it the
“Sherlock Holmes Model” of awareness.
The investigator is faced with a possible criminal case and needs to come up
with a likely suspect. The case is closed if a suspect is legally found guilty beyond
reasonable doubt. The investigator has limited resources to solve the case within
some set time. Usually, the investigator starts from scratch, perhaps from the mere
report of a possible crime, as when a dead body is found (it could be a suicide, an
accident and so forth). Typically, there will be a lot of data, collected by local po-
8
Vico, Scienza Nuova (The New Science, 1725).
9
A heuristic is needed because there can be no science of the matter. A heuristic is a rule
that does not conflict with any scientific facts, past, present, or future. A good heuristic leads to
worthwhile developments. Most novel developments in science and mathematics derive from such
intuitive heuristics.
10
Erwin Schrödinger (1944), “What Is Life?: The Physical Aspect of the Living Cell”. Based
on lectures delivered under the auspices of the Dublin Institute for Advanced Studies at Trinity
College, Dublin, in February 1943.

L
licemen who initially investigated the scene of the crime. Think of plaster casts of
footprints, photographs of discarded cigarette butts, or blood stains, DNA–traces,
recordings or written reports of persons encountered in the neighbourhood and
so forth. I’lll refer to these as “THE FILE”. Most of the contents of the file will
be irrelevant to the case and had better be ignored. However, there is no a priori
rational way to make such a selection. The investigator may ask for additional
footwork, adding to the file. This is expensive, so there needs to be a reason for
it. Initially there are no such reasons. (“Round up the usual suspects!” does not
count as a reason.)
Notice that the file is analogous to the content of the blackboard. Most of
that structure has to be ignored by psychogenesis. How to achieve that? There
are evidently no rational, or computational methods. What this implies is that
“bottom up” methods that attempt to “compute” what is in front of the eyes on the
basis of “inverse optics algorithms” (methods of mainstream vision science) are
(even a priori) entirely useless.
Instead, the criminal investigator proceeds by conceiving of a likely “plot”. In
terms of the plot some structures in the file are rendered relevant, most are right
away discarded as not bearing on the case. The plot enables asking questions, the
answers can be sought for in the file, or collected by way of additional work by
lower order investigators, say “flatfoot men”.
The plot may not work out, but that is no big deal, for plots are cheap to come
by. The investigator will tune the plot, or even switch it for an alternative, at the
drop of a hat. The point is that the plot enables investigation instead of random
search, for instance “round up the usual suspects”, any fool could think of that.11
It is like kicking about at random, sometimes that is effective, albeit for no good
reason. Usually the Sherlock Holmes method beats that.
I’ll stick to the conventional “Sherlock Holmes” method, that is thinking of
plots and following up their leads.
“Plotting” is exactly what psychogenesis is doing. On the basis of some imag-
inary guess it selectively probes the blackboard. In this way it endows the mere
structure in the blackboard with possible meaning. For any meaning is in the ques-
tions, it is imposed. Questions have possible answers, the meaning of the answers
is in the questions. The “same answer” may well have different meaning to differ-
ent questioners. That is why physiological recordings of the blackboard (say area
V1 in primates) will reveal no secrets concerning visual awareness to the probing
electrodes. Electrodes and brain scans ask no questions.
Questions may lead to sharpened questions or turn out to be irrelevant or non-
11
There may also be political reasons to abstain from investigation. A common mode of achiev-
ing that is to arrest some stray tramp—preferably unknown in the neighbourhood—convict him
a.s.a.p. and be done with. In the old days a lynching would settle the matter fast and conclusively.

M
informative. Others may lead to unexpected findings that may be used to adjust or
change the plot.
In psychogenesis several (or many) investigators work in parallel. The one that
comes up with the best case wins and decides on the current concrete actuality.
There is no certainty, the actuality is merely “beyond reasonable doubt”, or “good
enough for judicial work”. The very next moment yields another shot at concrete
actuality.
The process of conceiving of plots and hunting for evidence, is an evolutionary
process in which many trials die and only the fittest survive. It yields the current
best bet of what is in front of the eyes as seen by the agent. It has been said (al-
though usually in a pejorative mode) that “perception is controlled hallucination”.
Indeed:

P SYCHOGENESIS, that is controlled hallucination is meaningful by

Vico’s Principle and generates awareness by Schrödinger’s heuris-
tic.12

How does the criminal investigator gets the “plots” in the first place? There is no
algorithm for that, no calculation or logical reasoning will do. The investigator
relies on his or hers “nose”. That is an understanding of the Modal Structure of
the World at the gut level. If there is any method, it must be like Hume’s habit.
That is basically what psychogenesis of awareness achieves.13 A thorough
understanding of the Modal Structure of the Umwelt is what drives it.
Here “understanding” is not rational understanding, it goes on “behind the cur-
tains” in the sense that all you come aware of is the awareness. The “gut level”
is Husserl’s “typifying apperception”, Kant’s “phantasmatic self-stimulation”,
von Uexküll’s “seek image” and “functional tone”. These are all implementations
of Hume’s habit.
Thus, in a nutshell, the “knowledge” is Husserl’s “sedimented experience”, for
which von Uexküll attempted neural explanations with his “new loop”, which has
immediate connections to what nowadays is referred to as“enaction” (Francisco
Varela (1946–2001), Alva Noe (born 1964), Kevin O’Regan (born 1948)).
It is largely implemented as habit and type, or user interface elements and
structures (Hoffman). Various of such generic structures are no doubt like the
“mental models” of Johnson–Laird and Lakoff.

12
Or, more properly said, controlled hallucination is awareness.
13
You may be disappointed that I don’t mention consciousness. It’s not that I’m afraid of the
C–word, but that I feel it is not appropriate in the present context. Consciousness implies a self.
In my view all sentient beings are aware, but only a minority of “social independents” can be
suspected of entertaining a “self”. Anyway, the notion is irrelevant to Modal Worlds.

N
So there is already much material that can be reaped in order to start building
formal models of M ODAL W ORLDS. A good project!

???

W HITMAN ’ S book is inspriring. It launched my thoughts on various trains of

thought. It is much like the kind of matter Whit would feed his students in
an attempt to set them off on new tracks. Remember that a (good) book is not a
final statement. Intellectual satisfaction is not to be found in any fixed goal, that
would be trivial. Research is a neverending journey. Well, till death us part.

Jan Koenderink
Utrecht, June 21, 2018

O
P
 Modal Worlds

Whitman Richards†
 Modal Worlds: the role of regularities
in Evolutionary Biology

Whitman Richards 16 May 16

Waban, Ma 02468 Hereford Rd.

1
 Contents
Forward………………

1. Introduction……………..6

Part I Regularities

2. City Designs…………..10

3. Niche & Dwellings……15

4. Psychometrics……… 26

5. Animals ……………….29

6. Fitness…………………37

Part II Artifacts

7. Artifacts………………41

8. Everyday Things……..52

9. Biotools………………56

Part III Representations

10. Tools for Thought…..66

11. Mens et Manus……..77

12. From Dawn to Destiny.…80

References…………….………84

Appendix
13. Animal Categorizations…88
14. Polyflowers ………….….91

2
 To Waltraud for her endearing support; and to
D. B. Smith, MD. for the extra years. My gratitude also to
Clarence M. Zener, my scientific advisor and friend.

3
 Acknowledgements.

Thanks to the many students who took my Cognitive Architectures class.

Special thanks to Ethan Loew and to Profs. Kobi Gal, Aaron Bobick, John
Rubin and three anonymous reviewers, all of whom provided useful
feedback.

Unless otherwise noted, colored figures are fractal images from the Web
(Mandelbrot, 1983).

4
 Forward

Modal Worlds began in 1986 with an MIT AI Memo entitled

“Classifying objects from visual information”. Written with Aaron Bobick,
the memo showed that acquiring new concepts was unlikely unless the
learning was based upon categorical world structures. These structures were
not random. An experimental test appeared later in “How to Play 20
questions with Nature and win”. In this game, one person picked a category,
such as oyster, muscle, elephant, grass, etc. and the other had the task of
determining what the examiner held in secret. Appropriate questions were:
Is it terrestrial? can it swim? does it have fur? is it green?, etc. Surprisingly,
often only 10 questions were needed to learn the hidden category. We tend
to categorize objects in the same way. The significance of this little game
was apparent to only a few (Koenderink, 2015). For example, if one feature
is a bony shell, and the size is less than your fist, then there is a high
probability that the object lives in salt water, etc. Thus, object features
tended to be correlated, with some categories being much more likely than
others. This clustering of object features we called “Natural Modes”. We
tend to categorize objects in the same way. Hence a few correct answers in a
row are sufficient to make an inference. The success thus becomes possible
because we share a modal view of the world about us.

Other examples of modes are cities, villages, dwellings, the smart phone,
the laptop – all of which have many correlated features or “regularities” in
their constructions. This book is a survey of Modal phenomena, limited
largely to the domain and scale of humans. It is understood that insects will
have their own regularities and hence their own set of modes, birds still
another, fish and shellfish, bacteria, etc. all have different modes. In each
domain, objects will have their own correlated regularities: observing one
infers several others. These become modes. Hence modal structures play an
important role in evolutionary advances. Of particular interest are the (non-
biological) tools, namely the artifacts and structures created by man. These
inventions lead to rapid evolutions, whereas Darwin’s notions for species
changes occur only infrequently unless assisted by humans by selective
breeding. Furthermore, the important role of a brain is discussed. Hence the
treatment suggests we should view Darwin’s hypotheses for natural selection
as only one aspect of evolution.

5
 I Introduction

The “Origin of Species” was one of the most influential works appearing
in the last two centuries. Darwin and Wallaces’ original proposal (1858) was
that evolution proceeded by adaptation to the environment. In Spencer’s
summary (1859), the fittest survived at the expense of those less fit. Over the
past century, additional ideas about evolution have been proposed, including
at least one that stresses new ideas in a culture that resemble evolutions,
such as Dawkin’s proposals (1976). Here, the proposal is that evolutions rest
on regularities, or more specifically modes, that guide or constrain future
changes. In other words, if our worlds are laced with regularities, then
evolution will be constrained by their forms.

Modal Worlds argues that for high order species such as homo sapiens,
new regularities are invented that are not just adaptations to the
environment, but nevertheless do increase fitness (McKibben, 1989). Roads
are created, mountains moved, plant life is modified, cities are built. Thus
the evolution of the species is not always due to Nature’s ways, but rather in
certain cases Nature itself is being molded to suit the species. As mentioned,
these changes occur by elaborating old regularities or by adding new ones to
create modes. The modes then become clusters of correlated regularities.
Almost all of these “modal” constructions are an outgrowth of earlier modes.
For example, new dwellings follow the designs of previous constructions,

Fig 1. A city with countless modes (left panel). Once the city is destroyed, only ruble is
left (right panel) with few regularities.

6
for they already have put into place some standard features that are easier to
build upon then starting from scratch. Simple examples are the width of
roads and pedestrian walkways, brick sizes, or the stud lengths and
separations used to create walls. To accomplish these changes, tools were
invented that were modified to become more powerful than the bio-tools of
our limbs, yet retained their earlier “good feel”. Thus clusters of old
regularities were modified to include later designs, keeping the advantages
of the older patterns. These regularities become an important property of our
daily lives.

We begin with an important example of a mode: namely the city

described in the first chapter. A city has a plethora of modes, more so than
any natural configuration. Fig. 1 makes this mode more explicit. In the left
panel of we have a city; in the right panel the same city appears as rubble –
as if hit by an earthquake. Regularities are scarce in the right panel.
Although the building blocks are all roughly same size (for convenient
handling by the builders) there is no obvious local relations among these
blocks. It is difficult to make a list of regularities in this pile, other than the
building blocks and their rough shape. In contrast, the city in the left panel
includes streets positioned on a rectangular grid; the buildings are aligned;
entrances are on facing streets. Windows, doors, etc. are all in reasonable
proportions and set at heights consistent with floor plans. There are
walkways and roads aligned in the same coordinate frame as the buildings.
Cars are positioned parallel to the midline of the streets. (Not shown is the
favored right-hand constraint for travel.) Observing any one of these
regularities implies with a high probability most of the others. This
implication is an important characteristic of a mode. Succinctly, a mode is a
set of correlated regularities. We will see that the approach extends well
beyond the physical sciences to social phenomenon, such as the design of
towns and governments, where modal relations are rapidly being created and
are changing the way we behave and how we regard others.

Underlying this book is the concern that there are views of our world that
are just as valid as those currently taught in school. Certainly scientific
methods have made enormous advances by probing deeply into particular
areas of inquiry, giving us powerful and predictive models of the world we
live in. Many of these models however are at space-time scales quite distant
from our own. In physics, for example, quantum theories of sub-atomic
structures at the finest scale are quite distinct from explorations at the
cosmic scale. Studies of genetics and protein folding and formation are at yet

7
another, intermediate scale. Although impressive and productive, to the
layman many of these advances seem largely independent, especially those
at the subatomic and cosmic inquiries (Hawking and Mlodinow, 2012 ).
Perhaps others, like myself, wonder how all these pieces of scientific
evidence at different scales of exploration fit together, especially regarding
the nature of living systems. A naïve view is that we will eventually build an
integrated framework of living systems from the parts, such as by showing
how atomic relates to molecular, then to biological and finally to creatures
living in a society. However, right now such an edifice seems almost
impossible to construct, because the theories at one level appear quite
independent of those at another. Furthermore, these models do not have the
same flavor as our own internal models that we use in our daily lives. When
the apple falls, there is no inkling of f=ma, but perhaps only a vague notion
of gravity. In contrast with the scientific formalisms, our internal models are
focused on the daily behaviors of objects, people, or other animals, not
precise mathematical relations. These internal models support our naïve
feeling that life at our scale (or any one scale) is a wholistic entity, not one
fractured into many parts. It is almost as if our scientific explorations have
shattered a panoramic window to our world, so we now see the fragments
very clearly, but have lost the real picture (Wilson, 1998.)

Modal Worlds is a new approach to Reality. The regularities are simple

and easy to understand. The key is that the observation of one property
allows the inference of other properties of the object under view. These are
the “modal models” we seek. The aim is to show that the world seen this
way is principled. The result is not one integrated edifice, but rather a web of
evolving correlations. These models have some aspects of scientific
formalisms, but emphasize particular regularities in our world that often
provide the underpinnings for evolution and analogies. They paint a different
picture of reality, one that is obvious yet largely ignored. As the role of
social structures increase in our anthropomorphic world, these kinds of
models become more important. They are likely to dominate how we view
the future - namely not as one broken into the many disciplines of science,
but rather a tangled web of Modal correlations. Evolution proceeds with
these at their core.

8
Part I Man-made Regularities

9
 2. City Designs

A city is an excellent example of a modal edifice. The key point is that

selections by humans, not natural selection, is the dominant force behind the
construction of cities. Thus cities offer a good beginning for understanding
modal regularities. Cities cannot afford haphazard designs or arrangements
of buildings, and these constraints lie at the heart of modes. They have an
enormous impact on our environment. If efficient flow through the
community is desired, then cities must accommodate streets, sidewalks,
NYC-GRID-1811.png (PNG Image, 750x2160 pixels) - Scaled ... http://upload

Here the East and Hudson rivers play

a major role in the road pattern, with
a North-South and East-West grid.
Only the tip of Manhattan violates
this design. The grid pattern allows
potential subdivisions of the city into
residential, business, and routes for
public transportation. Modeling
traffic flows becomes simpler than
when streets wind and intersect at
odd angles. Parking garages can be
optimally spaced, etc. Models for
optimizing this flow depend on traffic
volumes and speed, all of which are
dependent on road or highway size,
intersections, etc. The layout of a city
is thus not arbitrary. (At a larger
scale, we can see regularities in the
distances between cities and villages
that arose out of a need to bring
produce to markets, Hence many
villages are roughly one-half days’
travel apart ( at least before
superhighways).

Fig 2. Social conventions constrain city design

10
1 of 1
pedestrian crossings, street lamps, power lines, etc. These constraints are all
designed toward enabling the construction of an urban niche with the
greatest benefits at the lowest cost. Although the modes associated with
cities and dwellings are mostly physical entities, the social consequences
also become important factors. For example, the occupants need an efficient
flow through the community.

Whenever possible, modern cities are built with rectangular grid

patterns, or perhaps also a circumferential ring of roads. Clearly such
patterns facilitate traffic flow, and are easy to navigate, with city blocks
imposing strong constraints on building size, etc. These constraints lead to a
host of regularities and modal designs in modern cities, unlike the more
romantic arrangements of older cities like London (or Boston) where cow
paths dictated road positioning. However, even in these cases, roads do tend
to go to the edge of rivers to allow access to cargos carried by boats, with
roughly a perpendicular set of roads that are parallel to the river’s edge.
Manhattan is well laid out city (1807 map). The old (lower tip) and the new
(central park area) reflect when the parts of the island were occupied.

There are many regularities in a city that are not immediately recognized.
Codes are drawn up to specify minimum construction requirements for
buildings. There are codes for wiring and electrical outlets. Heights and
widths are not arbitrary. If a building is too tall and thin, the upper floors
will sway (counteracted in some cases with moveable ballasts). Water must
be provided – again according to some code. Waste must be discarded.
Similarly stairs, elevators and escalators are constrained. These codes and
constraints are largely hidden to us, but once mentioned are obvious.

One of the most significant modes of a city are the traffic patterns,
especially the roads and transportation for use in the morning and evening
commute. The traffic is constrained not just by the roads, however, but more
importantly by our dining hours. Breakfast typically precedes the commute
to work, and in the evening we have the reverse, where dinner occurs after
returning to one’s residence. These important constraints have led to the
congestion of our commutes, no matter what the availability of
transportation is. Similarly, one’s ability to navigate the city without a car is
constrained by pubic transportation – subways, light rails, buses etc. All
these means have schedules to be met that are designed to get travelers to
their destinations in reasonable time. These limitations are at the expense of
decreased fitness or benefit for the individual.

11
 A direct consequence of the modal character of cities is that laws and
conventions emerge to govern public traffic or pedestrian flow. In the USA,
traffic flow on streets follow the right side rule. Pedestrian flow is parallel to
the flow of cars, on walkways designed for 3 to 5 people abreast. At
intersections, there are conventions for turning across the opposite flow, or
for entering rotaries (round-abouts). Traffic signals are at roughly specified
heights, with signal colors almost universal (bottom: green-go, top: red-stop,
middle: yellow-caution) Street signs must be within 7 – 10 feet high for easy
reading by drivers, with type fonts such that they can be read at 100 feet or
so. This list of “norms” or conventions for city design (whether a rectangular
or polar layout) is extensive, with the regularities tightly correlated (Lynch,
1960; Alexander et al., 1977).

Families, tribes, communities, cities, states and nations all have some
kind of leadership. This is not uniquely human; we also see leadership and
dominance in the tribes and communities of the lower animals. Clearly a
system of governance must have been present in humans as early as 200K
years ago when communal groups were formed. The role of leadership is to
exercise management over the community. In lower animals, the most
primitive governance is when one individual carries out most of these tasks
through dominance. However, what distinguishes governance of the
complex societies from primitive precursors is the presence of a supporting
cast. Insect governance rests not with one queen, but utilizes the talents of
many lower cast insects to create a stable society. In humans we have
evolved in the Western, democratic world a tiered government structure: a
head of state, a team of advisors (or section heads), the members of
congress, political parties and the populace, as well as a judiciary. Families,
tribes and small communities can be seen as abridged versions of this
structure.

The need for a government is to maintain common lands and resources

and to coordinate or control actions in a society. This is true both for insect
societies as well as ours. However, with advanced symbolic tools, such as
exchange and money, taxes become the device used to provide funds for
these government services. These procedures were present in many cultures
over 10 K years ago, perhaps earlier in the Middle East. Their ubiquity and

12
regularities for exchange qualifies these organizations as modal. More on
this point later.

Unlike previous modal structures (and biological modes), modes of

governance need not be stable. The populace can rise up and contest its
government policies, striving for change, as has been happening currently in
the Middle East. The internet greatly favors such grass roots uprisings and
may well become the important medium for expressing dissatisfaction. Its
advantage is that thousands of individuals may be reached quickly, with
rapid exchanges. The fire can now be fueled quickly, more so than by going
through the slow process of physical meetings, or even relying on person to
person phone contacts. This is a game changer.

Although governance is modal, there are many forms of government,

with different kinds of leadership and means for exercising power and
control over the populace. There is a possibility, especially during unrest,
that one type of mode may be changed to another, such as a dictatorship to a
democracy, or vice versa. Obviously, in the initial stages of a revolution, the
mode is broken. The populace objects to the new regularities (or to their
absence), thus contesting the current modal structures. However, in all cases
known, new modes eventually reappear, again with the same abstract
structure but with a realignment of power. Governance is a clear example of
a mode that can undergo change and then reemerge (Jacobs, 1961).

Definitions

We now offer some tentative definitions:

(1) a regularity is a property, a relation, or a “law” often captured by a

simple arithmetic or geometric expression. The regularity may also be a
particular arrangement of the components of the object, namely a possible
configuration of its components.

(2) a modal object consists of several independent regularities, where one

observed regularity allows the observer to infer other regularities of that
object.

These definitions are not precise. Nevertheless, they convey the basic
ideas. The important point is that for physical structures, there are

13
constraints on the regularities, and the configurations are limited (modal).
Although most regularities obey some formal law, typically the regularity
will not meet the criteria for a precise scientific formulation. A geometric
pattern for example may be hard to describe quantitatively. However, we
could describe the regularity following Darcy Thompson (1919) and Huxley
(1932) where physical constraints limit the general shape of objects or
patterns. Unfortunately this may mean there would be no metric variable
describing a mode. For example, we know an egg when we see one, but the
ovoid may be very hard to describe – much more difficult than describing an
ellipsoid. This is quite unlike physics, where there is an assignment of
dimensions to all our variables. Thus, unlike classical physics, our models
will never be complete, in the sense of Einstein et. al (1935). Many are
topological arrangements. How to measure them is not always clear, yet later
we will see that they are often related to “degrees of freedom” DoF. The
more restrictive the DoF, the tighter is the mode.

In the following chapters, we will see that regularities may be structural

(in the form of arrangements such as symmetry), categorical, a type of
organization, or patterns. Smaller modes may also become embedded in
larger modes. Consider the city mode: it is occupied by a variety of species
such as rodents or dogs, etc., not just the human animal that built the city.
These animal modes are quite different from those of a city, but nevertheless
qualify as modes. The city, however, was a mode built by a collection of
submodes, such as the bricks or foundations around which the city was built.
We can imagine the city as a large encapsulated blue bubble, with its interior
occupied by much smaller red bubbles that characterize the human builders
and more specialized modes. This is an excellent example of how a species
builds one modal form to further its “fitness” to the environment. Not
surprisingly, the ensuing adaptation often becomes favored by another
species.

14
 3. Niches & Dwellings

All living creatures have a territory. Animals especially. Each type of

animal will create its own dwelling to increases its “fitness” to the
environment. These serve as a “castle” that serves as a retreat and center for
actions (von Uexkill, 1909, 1957, Lewontin, 1966, 1978; Wilson, 1971). The
dwelling could be a burrow, a den, a cave, nest, web, brush piles, holes in
trees or in some cases a shell (molluscs, turtles, etc.) For most chordates and
arthropods, the simplest niche provides protection during rest, a shelter, and a
nursery. But the surroundings are equally important. Typically, associated
with a niche or dwelling are not only the ground on which it is built, but also
includes the more distant environment, such as farmland, or for the city
dweller, the city itself – or at least a part of the city. (Recall if you dream of
your youth, often this dream includes not only your house, but also parts of
the town where your house is located.) Today, various aspects of a niche
have existed for thousand of years, with little change. None of this would be
possible without the invention of artifacts to shape the constructions.

Figure 3 Illustrates two kinds of niches for an insect. The giant mound built by the
leaf-cutter ants (right) and the much smaller niche of wasps (left). Note the symmetry in
these structures. (The man in the termite complex indicates the scale.)

Not all animals create such niches, however. Ungulates for example do not
build dwellings. Their lack of hands or feet with claws does not allow

15
them to mold the environment. Rather, an area is designated for the flock,
herd or troupe about which the animals can congregate. In contrast, most
animals with biological extensions to their limbs do indeed create niches.
These can include many rooms, some with gardens, others for food storage,
trash areas, defense mechanisms, sleeping areas, etc. as well as temperature
regulating mechanisms such as air conditioning (see Animal Architecture).
These, or course, include the human species but surprisingly are also created
by insects such as the leaf-cutter ants, who create very sophisticated earth
mounds that include many of the advantages we enjoy in our homes, such as
circulating air, nurseries, food warehouses, etc. We begin our treatment
with a focus on these ants, and their mound which can often become over six
feet high. This dwelling, although created by an insect, shows the modal
effects of social constraints on a very sophisticated creature.

Leaf-cutter ants

Although ants lack the thumb-hand addition to limbs, they still succeed in
building complex niches. This is accomplished by using more than one limb.
The mode arises from the use of two or more limbs that each have only a
three degrees-of-freedom (DoF) appendage, or by cooperative activities of
more than one creature. The ant niche may occupy more than 22 m^3 –a
huge niche as indicated in the depiction by a man to indicate the scale as
shown in Fig. 3 (from Odling-Smee). The figure illustrates the complex but
highly organized niche created by leaf-cutter ants (Hobldoler & Wilson,
1944.) These little creatures build gardens within their nests, growing fungi
on fresh vegetation they have cut and collected. The nests have hundreds of
chambers, with an organization that resembles to some degree a biological
form. The organizations and activities within this enormous ant city appear
to be designed by a grand, but invisible architect who pulls puppet strings to
coordinate the construction by thousands of ants. Of course, the organizing
capabilities come from constrained social interactions among the ants
themselves, with various individual goals, tools and activities coordinated by
poorly understood mechanisms (Werfel et al, 2014).

The Atta ant is but one of many examples of lower animals creating niches
that exhibit very regular designs. These are not haphazard structures, but
rather have many regularities. The most obvious is their overall symmetry.
With this and other regularities are coordinated behaviors that bring the
various constraints together. Bird nests, beaver constructions, spider webs
and earthworms are common examples. (See Tables 2.2 and 2.3 of Olding-

16
Smee.) However, it is not any particular design that interests us, but the
nature of the processes that underlie them. We argue that there is a strong
interaction among the creature’s physiology, the environment, and the
design of the niche. A spider’s 8 legs, with an extra claw on a leg that throws
silk, the role of a bird’s beak in weaving a nest of in manipulating sticks –
even in some cases the ability to tie knots are examples. Even the manner by
which an earthworm can ingest and excrete soil creating a host of tunnels –
illustrate the importance of the tools available to an animal to enable the
niche construction.

However, the roots go much deeper. Regularities arise from physical

constraints, such as the need for ventilation, passageways, etc. Hence,
although cognitive (and social) factors certainly play an important role in
advanced animate forms, the root of the modal design lies in the coincidence
of regularities in the process of construction.

It is quite surprising that two quite unrelated species build complex

niches: namely human primates and advanced insects. To gain perspective,
consider the abridged phylogenetic tree shown in Fig. 4.
Atta Ants Apes
Humans
Molluscs
wasps Ungulates

Carnivores

spiders
rodents

Birds

Fig. 4 Abridged phylogenetic tree. Size of ellipses indicate the complexity of the niches.

17
If we use a filled circle to indicate crudely the relative size and complexity
of niches in the Animal Kingdom, that of the ants and humans completely
overshadows the niches of other animals. Note that even subhuman primates
(apes, monkeys..) exhibit little or no complexity on niche construction, often
opting just to stake out a territory. Similarly, for the arthropods at the top of
the evolutionary tree, we see a complex nest for ants (and other such
insects), but typically only simple forms for the other animals on the lower
portions of this branch. Hence the surprising finding here is that complex
niches appear only in advanced animal forms for two unrelated species. This
observation raises the question why not earlier - why not a more gradual
evolution of forms ? Two ideas come to mind: (1) these two species are
unique in social complexity and (2) they have highly developed tools that
favor manipulations of aspects of their environments.

Humans

Humans are distinguished from other species by having invented tools

that give them tremendous power to alter their environments. Although the
Atta ant has not invented new implements, their biological tool capabilities
are impressive: their limbs are well coordinated, the insects can operate in
pairs, they cut leaves to shape, they have glue, they can stitch leaves to
desired shapes. Like the Atta ant, the human also has become a
sophisticated food preparer (hence a kitchen), can grow food--an agriculture;
created sleeping quarters, and now has sophisticated heating and cooling
systems. These developments began with a home that was merely a cave,
then matured to a lean-to (or teepee), to a cabin, to a house, then multi home
complexes .

time before 2000 AD dwelling

100,000 cave
50,000 lean-to (wood)
20,000 cabin (one room) stone or wood
10,000 multi-rooms
4000 complex multi-homes

This evolution could occur only with the advent of stone cutters, axes,
fasteners (cord, notches in wood, nails, etc.) now including modern methods
for mass production of brick, timber, boards, sheet rock, etc. used in present

18
day dwellings. We divide the analysis of these (Western) dwelling & niches
into four parts: (1) the logic of the architecture, (2) construction materials.
(3) furnishings and (4) social considerations.

Fig 5. Reconstruction of a Stone Age dwelling in Finland. The base is about 30 ft.

As social groups were formed, there was an advantage in creating

independent dwellings in a small community. The stone or wood one-room
cabin would be an example. But antedating this were versions of the lean-to.
Typically these dwellings measured 20 x 10 feet or so, and covered 200 –
400 sq. ft. Because these were not portable, fire was key to a successful
abode. A logic of architectural design, which today includes dining, nursery,
play, and a community of dwellings greatly improved security, and allowed
the individual residents with different talents to specialize as hunters,
agriculture, food preparation, etc. Later, dwellings were partitioned into
areas with different functions, such as cooking, general living, sleeping, etc.
These different activities drove the design. Almost every home has room
arrangements serving these functions, with related functions in adjacent
rooms (i.e. kitchens are adjacent to dining, pantries adjacent to kitchens,
bathrooms near bedrooms, etc.) When one enters a home for the first time,
these highly modal arrangements are anticipated. For example, a kitchen is
almost never approached immediately from the front door. Instead, it
typically is situated near the back door.

Construction modes

19
 Although early stone or wood dwellings had modal features, today’s
constructions are still further constrained by the availability of materials

20
 Wood-framed_house.jpg (JPEG Image, 1083x546 pixels) - Scal... http://upload.wikimedia.org/wikipedia/commons/5/50/Wood-fr...

Fig 6. Modern spacing of the vertical studs. This regularity dates back tens of thousand of
years, to lean-to constructions in the Stone Age.

with standardized dimensions, that reduce cost through mass production and
volume. Studs or support beams are 4x4, 4x6, 6x6 inch etc. Plywood sheets
for siding are 4 x 8 ft, consistent with the stud separation of 16 inches (note:
3 x 16 = 48). Such types of standardizations go back thousands of years
where the lean-to supports were equally spaced. Interior designs are also
standardized. In Western cultures, rooms have 8 ft ceilings. Stairs elevate by
8 inches with 9 inch treads to accommodate easy ascents by the human
body. The doors to rooms are roughly 3 feet x 7 feet for easy passage; door
knobs are at waist height, with circular knobs easy to grip; windows are
about 30" x 48 ", situated so one can look out when sitting or when standing
(hence the middle of a window is about 4 ft. from floor.) Unlike a burrow,
the human dwelling is highly modal, with almost countless numbers of
regularities meshed to create convenient and pleasant sub-niches. Although
there are differences among cultures, each still has a powerful pattern of
1 of 1 11/21/11 2:41 PM
regularities satisfying abstract similarities. In addition, all are subject to
allometric constraints on the mechanical strengths of a building, which
follows principles analogous to tree height vs trunk diameter, but now based
on reinforced metal factors that allow larger constructions (Burton, 1998).

Furnishings

Interiors are designed for the dweller’s comfort. Chairs are needed for
sitting. The standardized seat height is about 17 inch (tailored to be one-half

21
adult leg length), thus favoring sitting down and getting up. There is a back
for additional body support. There are four legs. Tables are 28” for
convenient use of arms, with clearance for legs when sitting. A chair, and its
cousin, the table, are highly modal in our Western world. We also have
counters for preparing food, stoves for cooking, and sinks in the kitchen for
washing dishes, all roughly standardized to a three ft. height. In Fig 7, just
seeing what is on a table or counter top identifies the room type. Furnishings
and knick-knacks in rooms are so highly modal and expected that their
presence seems like a trivial observation. In another culture, there will

Fig 7. Typical kitchen and living room

be differences, but the point is the same – a room and its furnishings all “go
together” . Function dominates these choices. Individual tastes tweak these
designs, but do not reduce their roles.

Furnishings constrain room size. A bedroom with even one bed must be
at least 10 x 15 ft to accommodate the human body. Also there will be a
chest of drawers and likely a chair. Similarly minimum dimensions can be
placed on the dining area, living rooms, bathrooms and kitchen. The
minimum footprint for a one bedroom, one story house will be about 1500
sqr ft.

22
 Social Modes
j0409578.jpg (JPEG Image, 1280x1024 pixels) - Scaled (67%) http://4.bp.blogspot.com/-9ThGjMWskA0/Tll41NyGYUI/A...

Fig. 8 Conventions for dining.

Within the niche itself, the most important social mode is the family
dinner. Figure 8 shows two examples, each from a different culture.
1 of 1 11/17/11 1:38 PM

Although there are some differences, each still expresses a "ritualistic”

arrangement: a group sitting around a table that displays dishes for that day,
with serving or eating utensils (silverware or chopsticks) near the table’s
edge, etc. The face to face arrangement is one of the most important and
universal social modes (Gal et al. 2008). With this arrangement, everyone
can directly address and observe the other's expressions and reactions to
discussions. We have a modal event, with a collection of regularities that
includes the seating arrangement, the placement of eating utensils & dishes,
etc. with different specifics for each culture, but fitting a common abstract
framework. Note that a group telling stories around a campfire is an
analogous, but somewhat weaker mode, its principal strength resting to a
large part on the ritual nature of the meeting. We can key into these rituals
just by observing one of the minor regularities, such as a piece of silverware
on the dinner table, or a fire in the middle of the group of campers. In each
case, there are certain etiquettes to be observed, such as when to start eating,
the order of the dishes to be consumed, or the acceptable portions for each
dish. Such etiquette norms help strengthen the correlated regularities.

The family setting is just one of many ritualistic social modes. Another
example of a powerful mode would be a religious service, such as Sunday at
church. Although the organizational arrangement is different, with the
parishioners seated in rows facing forward toward the minister or priest, who

23
is typically attired in a distinctive wardrobe. This is reinforced by the
commanding vaulted architecture of the church, windows with striking
symbols or artistry, all contributing to a powerful emotional setting. These
are important modal details, just as is the sequence of prayers, songs and
communion. Even the “Sunday” dress code reinforces this mode. Again, any
one regularity in the service or environment serves to index into the
collection of regularities that contribute to a modal event.

In addition to activities in dwellings or churches, there are many

examples having an abstract modal character with distinguishing regularities
forming the mode. Children going to school: typically walking in small
groups at certain times of the day; a dress code, etc. This parallels the adults
who head to the city to work, or farmers rising early to tend to crops and
livestock. A quick glance usually reveals the culture even though the spatial
temporal modal patterns are similar.

Such social regularities are also observed among lower animals. Already
the niche of Atta ant was seen to be a highly organized structure designed to
facilitate efficient interactions among ant types. Presumably within the niche
there are rules or norms for traversing tunnels (Is there an “up” tunnel and a
separate “down” tunnel so creatures can flow fluidly without colliding?)
Food pads are likely near breeding and nurturing areas. Likewise, for
honeybees, an important characteristic of the hive is one “meeting hall” in
which a returning forager be can communicate the location of a honey
source to other members of the hive who observe the forager’s dance. This
in itself is a remarkable sub-mode which combines direction of a dance to
indicate food direction with respect to the hive’s entrance, distance by wing
rate, etc. But a key part of the communication is that all bees are facing the
center actor, forming an encompassing circle, much like the ritual of
campers sitting around a campfire. We note in passing that here we have two
completely different species, humans and insects, who use similar “circular”
arrangements as a key part of the mode. Hence the spatial organizations are
not arbitrary or haphazard.

Although there is indeed a collection of correlated regularities in the above

examples, there is still no standard used to define a “mode”. The hurdle of
defining a metric for modes still remains. As mentioned earlier, the difficulty
to formalizing modes in the social domain is a general lack of parametric
measurements. Instead, most of our examples are lists of correlated events.

24
However, we already have noted at least one modal event, common to both
insects and humans, that can be understood formally. This is the circular
topological arrangement between a leader or speaker (if present) and other
members of a group seen at campfires, or the honey bee dance. The ring
with a central focus is a distinctive topological arrangement. A second form,
seen in chapel services, are rows of parishioners facing the minister, or
Congress listening to a Presidential address, or to students listening to a
lecture. Again, this arrangement is special. These are the two most obvious
topologies for one speaker addressing a group. Never-the-less, although a
formal specification is lacking, the notion of modes should now be clear.

25
 4. Psychometrics

Modes are defined by their regularities. But the regularities we are

invoking, namely patterns, categorizations, ribbons of an unspecified nature,
etc., do not have obvious scientific metrics. Thus it is not clear how one can
build quantitative models for modes. For example, what makes the mode for
a city so much greater than that for a watch ? Figure 9 shows plots for

200 2M Washington, D.C.

Wrist!
160 Watch 1600
Pocket!
Modes
Modes

80 Watch 1200

40 800

20 400

20 19 18 17 16 20 19 18 17 16
Time, 100 years Time, 100 years

200 200
Tools!
160 Fireplace 160 for!
Modes
Modes

Thought
80 80

40 40

20 20

20 15 10 0.5 0 -0.5 20 15 10 0.5 0 -0.5

Time, 100 years Time, 100 years

Fig, 9. Modal trends. (Ordinates are arbitrary.)

Several modes. These are illustrative only; the ordinate scale is arbitrary,
because we do not have a definitive “measure” for the regularities.
Nevertheless, the trends are clear. For example, in the upper right we show
mode estimates for Washington, DC. Excepting the British fire in 1812
which razed several government buildings, the trend is upward as the
population grows. But the real impetus was the introduction of the
electricity, the telephone, the TV, cable and finally the digital explosion over
the last two decades. In contrast, if we look at the fireplace as a mode, we
see a rather flat curve, followed by a drop when central heating took over.

26
Many of the regularities were discarded as the fireplace became ornamental.
Likewise, the pocket watch (upper right) was replaced by the wrist watch
with many more options as well as convenience. Later a treatment for “Tools
for Thought “ will explain the shape of the curve in the lower right.

Our first task is to find a proper ordinate scale. Inspired by Darwin’s

observations of “fitness”, Galton (1859) proposed scales for measuring
human traits. Slightly later, Fechner (1860) founded a psychophysics with
proposals along similar lines. (See also Stevens, 1946.) These scales rested
upon one’s intuitions, but nevertheless could be used to assess the
magnitudes and properties of certain psychological traits, such as
intelligence, pain, or other emotions, etc. The key idea is to consider fitness.
If fitness increases then presumably the creator is more talented. Here, we
prefer fitness to be measured in energy expenditure. However, another
measure might be the number of people who benefit.

Regarding the latter, mode measures in principle, should be independent

of the population size, but dependent only on the number of regularities of
the object. However, there is a correlation. First, if the mode is discontinued
(as a pocket watch), the number of people using this device will drop to
zero. Likewise, as technology brings more options to society, the regularities
will increase, and this growth is often correlated with population size (as in
the growth of a city vs. the cave man.) Hence the number of instances of a
mode is relevant, but not deterministic.

The second method for measuring modes is the savings in cost that results.
If we use dollars or Euros to evaluate the cost (and hence the presumed
benefit), first we need a normal reference point independent of the
fluctuations or escalations in monetary value. This is a difficult task.
Perhaps more devastating is that modes occur not only for humans and other
animals, but also for devices at other scales. On a molecular scale there will
be many modes (or internal regularities), but again the costs for each
regularity are more or less obscure. Perhaps the most convincing argument
however is that it will be very difficult to measure the cost of increased
fitness for non-human forms such as insects, etc. that do not have a monetary
system.

Finally, one of the useful properties of a mode measure is that it can be

negative. When one’s property is taken by eminent domain, or when there is
a theft, or stock market crash, fitness can decrease. Indeed, for some people,

27
taxes are seen as invasive, and governance will be seen as “unfit”. We need
to be able to place crude measures on these events. The most obvious is a
decrease in the number of regularities (as when a city is destroyed.)

28
 5. Animals

Animals have distinctive modes. Their regularities are quite different

from that of a city. Most obvious are the bilateral symmetries. Here,
Darwin’s hypothesis is more fruitful because one can argue these forms are
biological advances that increase fitness to their environments. Animals
have two (or more) legs and arms; they have two eyes, two ears, etc. Their
modal designs give them the ability to locomote, and to find better food

Fig. 10. Three animal types covering more than two orders of magnitude. The human
form, depicted by LeCorbusier’s LeModulator, is roughly the geometric mean of this
range.

sources, and to create unusual dwellings or niches. Some of these modal

regularities are illustrated in Fig. 10. One of the most popular analyses is by
Courvoisier’s “LeModulator”, shown in the middle panel. The symmetry is
obvious. But each limb also has its own symmetry in the upper and lower
segments. The legs and arms are positioned with respect to the body to
provide good reach as well as the ability to provide workmanship.
Courvoisier reveals his impressions by a set of ellipses that underlie the
form, shown at the right edge of the panel. This picture shows some major
aspects of modes. What is unusual is that for advanced designs, many of
these body parts can be configured in several ways, giving the appendage
many degrees of freedom (DoF). For example, although a leg is principally
for walking, there is also dance, ballet, track, etc. In contrast, consider the
cow, domesticated for better milk and meat. It has limited movements –

29
principally grazing. In contrast, the human (or most primates) and termites
can coordinate their body parts to accomplish more difficult tasks. This
ability to create multiple configurations is a capability that permits greater
flexibility during alteration of the environment.

Definition: a bio-mode is a body part that for advanced species may have
many possible configurations or degrees of freedom.

One of the symmetric features of animal shapes are its limbs that allow an
animal to be erect. These must be sufficiently strong to support the animal
body. As shown by McMahon (1973) and Alexander (1983), leg length and
diameter are related to body size. Larger animals (especially bipeds) have

100
Baboons

Vervet monkey
Macaque

Squirrel monkey
10
Tamarin
Chest size

1.0

0.1 1.0 10 100

Body wgt

Fig 11 Children have “thin” legs compared to adults (from McMahon)

body size that is highly correlated with leg size and diameter. Consequently,
children have thin legs relative to their bodies, whereas adults have larger
legs. Galileo was the first to point out that this rule follows from the nature
of the thigh bone. Larger animals have a larger body mass and consequently
need stronger bones for support. Correlated with this property will be larger
limbs. As body mass increases as the cube of the diameter D^3, there will

30
 be an increasing larger diameter of the cylindrical limb, with will increase as
d^2. This is the science of allometry.
Allometric regularities: Shape and Support

Diameter, D

Length, L

Fig 12 Galileo was among the first to note that the thigh bone of an elephant was much
thicker in relation to its length than was a chicken bone.
Body wgt

In all these cases, the support is against a gravitational force. Hence for
any erect structure, such as trees or buildings, the general rule will apply,
and object shape will be constrained.
height

A second property of erect structures will be self-generated oscillations

that can be controlled under gravitational
D = L 0.5
forces. A tree and other plants
have structures that are pinned at the bottom, hence they oscillate like an
inverted
sequoia pendulum. Animals exhibit similar pendulum-like oscillations with
diameter 24
the period set by leg length. The tree and the animal are inverted versions of
the other. Because the body size determines the leg length, generally the
larger animals will have longer stride periods (see Fig. 13). We consider
first these modes, then move on to explore other complexities that a
locomoting creature may have, including derivatives of their natural
symmetries.

Limb designs

All limbs have at least two segments and three joints. This property
maximizes the number of positions that can be reached about the body,
allowing the limb to fold back on itself (i.e. for removing parasites, foreign
objects, grooming, etc.) This design also can serve to coordinate the use of
pairs of paws or hands. Hence the two segments have equal length. (Imagine
the handicaps if your forearm had half the length or less of your upper arm.)
For locomotion, there will be constraints on optimal stride and stability.

31
Again, the two equal segment limb is a good choice. If stride is too large and
one segment of the limb is too short, the large angle between the limb and
the terrain will require large muscular forces to raise the body and move it
forward. On the other hand if the stride is very short, then many movements
are required for high speed—consuming extra energy and taxing the control
system. The optimal angle will be the one favoring a pendular movement of
the limb with a period consistent with a walking stride – ie about 60 degrees.
Equal arm length and equal leg length further reduces the effort required to
move the limbs in concert.

A second consideration is the overall length of the limb segments. Imagine

you are a four-legged animal lying on the ground, and your knee joints are
rigid. In effect your legs would be stick-like stilts. Getting your body upright
then becomes a challenge if the legs are too long. Do you stretch your legs to
the side and raise one part of your body, then repeat? Without the leverage
afforded by a two-segment limb with a knee joint, this task is very difficult.
With the lower extremity folded back on the upper segment, the torso can be
raised with the least effort. Almost all two and four-legged animals follow a
two equal segment rule with three joints (hip, knee, ankle.) We will see that
two segment limbs tend to be correlated with strong functional body parts,
whereas for a single body part, positioning becomes much more critical.

For insects and spiders, limbs have 6 or 8 segments. Clearly these

additional segments impose a new hurdle requiring a more complex control
of segments. What is the advantage? Obviously with more limbs, body
support becomes easier. Also stability is improved. However, the bio-modal
advantage is that any limb can be positioned in many more ways than a two
segment limb. This allows two or more limbs to coordinate actions such a
when a spider builds a web, or when termites tend fungal gardens. Hence
there is a weak mode: insect limbs are correlated with the body structure as
well as a more simplified grasping tool namely the mandible claw
(essentially a form of pliers). Also the use changes dramatically compared
with a two segment limb. The net result is more coordinated activity among
the limbs. The spider is an excellent example, weaving a complex web that
is more complex than most nests built by birds.

32
 Stability

The first extensive scientific treatments related to stability were by

Thompson (1919) and Huxley(1932), who showed how constraints on form,
such as the minimization of energy (or effort), or ideal packing in a closed
volume, etc. could explain many natural shapes and their growth patterns. A
similar generalization applies to the motions of legs and limbs. To a first
approximation, motions such as walking can be considered almost effortless,
as compared with running or hopping where the body is significantly
elevated (Tedrake et al. 2014). Again, a pendulum model can be invoked,
with the arm of the pendulum equal to limb length, L. (The knee joint is
ignored here.) Fig 13 shows how the stride rate depends on animal size.
way Legs and Stride rate

Fig 13. Relation between animal size (e.g. leg length} and stride frequency.
Wings and Beats
The stride regularity is an allometric relation between body and leg size.
But the relation that applies here is not support, which is determined by leg
diameter, but rather leg length. For this relation to be valid, leg length must
be a scalable function of body size, with larger animals having longer legs.
Two factors may be relevant: first, limbs may need to reach other parts of
the body, such as for “scratching” and grooming, A second factor is related
to step size and speed. Longer legs will have greater strides, and hence lead
to faster locomotion. However, the stride cannot exceed the body size (for
quadrupeds) without risking entanglement with other limbs. Also long
strides in relation to body size will lead to buckling because the body/limb
support angle becomes very large.

25
33
 Wings and Fins

Although our focus is on terrestrial animals, the generalizations of modes

can be seen with an excursion into non-terrestrial animals. The dynamics are
different, but observers are not surprised to observe that wing beats for large
birds are much slower than beats for small birds or insects. A recent
analysis is given by Landry et. al. (2015).
1000
100

Insects
Beat rate, cps

Hummingbirds
10

Large birds
1

1 10 100 1000
Wing length, cm
Fig 14. Crude relation between frequency of wing beats versus wing size

The wave motions of swimming fish and aquatic mammals have

analogous properties. Large fish (or dolphins, etc.) have slow undulations of
their bodies, but the minnow’s undulations are very fast (see Fig 14.) Again
the dynamics is quite different from bird wings, or legs of animals, or plant
motions in the wind. However, observers accept the basic notion of
oscillation frequency decreasing with body size, thus making reliable
inferences about the size and behaviors of other objects, even if outside their
natural habitats. In other words, terrestrial modes are applied to non-
terrestrial animals.

34
 crude !
16
beat frequency, c/sec

8
T M
4 A
S
2 C D
W
1
2 4 8 16 32 64 128 256! 500
!
body length, cm

Fig. 15 Approximate log-log relation between tail beat and body length (T=tadpole,
M=mackerel, A= anchovie S= sockeye, C=Chinook, D = dolphin, W= Beluga whale

Sound and Size

Animals have vocal communications. This ability leads us closer to the

multi-modal property of modes. Colloquilly, elephants or lions “roar”, but
mice “squeak”. These differences are due to the mouse having a short vocal
tract, whereas the larger animals have longer vocal tracts. Fig. 16 shows

Mouse
piccolo
Cat
Pitch, Hz

Dog flute
clarinet
Human
oboe
Cow
Lion tuba

Size, cm

Fig. 16. Animal sounds increase in pitch with decrease in size, just as wind instruments
like the difference between a piccolo and oboe.

35
this relation for animal sounds. These sounds can be modeled by a vocal
tract cylinder, where frequency f is proportional to the speed of sound c
divided by vocal tract length L: f = c/4L. Because L is related to body size,
we have a modal regularity.

A Modal Picture

Although we have stressed the differences in terrestrial forms, it is

important to keep in mind that they are simply parts of an animal that
coordinate many of their actions to solve a task. This need for coordination
will lead to a major point. This is seen clearly when the task is eating. The
mouth size accommodates whatever the hand holds. The arm changes
position to favor the hand. The symmetries in body design favor arms and
legs of the same length, etc. (see fig 17.) These relations are at a grand scale
– namely that of the body size. They will change (at least for eating)
depending whether we use our fingers, or silverware, chopsticks, etc. Thus
there is another modal regularity in play, one that is obvious but is not easily
seen, as for the submodes in house or city design. The task, along with body
symmetries, constrains the “solution”. The complex submodal nature of the
body parts will become clearer in a following chapter.

POSITION
Stride Frequency

Leg Length

GRIP SOUND
Vocal Pitch
Paw, Hand Size

Body Size

Leg Length Leg Length Leg Length

Mouth Size

EAT

Leg Length

Fig. 17. Abstraction of how body parts correlate to carry out a task.

36
 6. Fitness

“Survival of the fittest” is a term coined by Malthus (1798) and

Spenser (1860) to characterize evolutionary theories such as described by
Lamark (1809), Darwin (1859) and Wallace (1860). Thus, evolution became
viewed as an improvement of the species that allowed them to better adapt to
the their environment. Although genetic mutation was one important factor,
the species could also change by selection of a (small) segment of the
population that performs better in the current environment (such as in
basketball, tall players are favored.) However, we argue here that fitness will
improve most dramatically as the species invents tools that extend its
capabilities to manipulate and improve its surroundings or way of life. These
tools include bio-tools such as the hand, as well as non-biological artifacts
such as the hammer, screwdriver, excavator or other devices that increase
the power of the species. Controlled domestication is yet another example.

Most advances with tool design are human-based. However, a few non-
primates have also invented artifacts that increase their capabilities. These
include the nest building of birds (weaving), the raven or crow who
occasionally use sticks for probing, and especially primates who again used
sticks or stones for various tasks. This latter group could take advantage of
their hands with an opposing thumb which gave them 5 or 6 DOF.
However, the principal beneficiaries of non-biological tools are the human
species. The advances took place about 100,000 years ago, roughly as
language was being developed. Indeed, as we begin to see the power of
modal regularities in our lives, one might wonder what the language
connection might be, and whether Modal ideas deserve a place near
language in the importance of evolutionary advantages. If a region of our
brains is devoted to language (Pinker, 1994), why should not our thumbs
also become a prime candidate for advancements?

To summarize, we will consider fitness to improve as the species invents

tools that extend its capabilities. Specifically, our definition will be as
follows:

Definition: Fitness is the energy gain (or loss) that results when an animal
can accomplish a task with less (more) effort than previously used.

37
Basically it is the energy gain (or loss) found by carrying out a task with a
new method, either through a new biological additions (such as a hand) or
by inventing a physical artifact. Non-biological artifacts would include the
hammer, screwdriver, excavator, or tokens for money. To model “fitness”
has been difficult. Especially compelling however is a representation
depicting energy bond interactions between two main variables (Paynter,
1961.) The first is an “effort” that acts on a body, and the second is a “flow”
resulting from that effort. The product yields the power or energy flow rate.
In the case of the sway of a tree, the “effort” would be exerted by the wind,
and the flow would be the bending of the tree. Note that the “flow” is a
counter-force or resistance R to the effort E. Hence if the flow is nil, so will
be the exchange of energy or power flow. These relations can be captured as
follows:

m(t)
{E}   {R}
ω(t)

where m(t) is a variable which is dependent on a flow velocity ω (t). Hence

ω is an input to the energy source E and m(t) is an effect of the output of
the energy source.

The simple example of the interaction between these two systems, E and
R, would be a mechanical spring with a frictional force in series which acts
like an impedance (resistance to flow.) A mass and a frictional element
sharing a common velocity V1 would express the admittance of a force
providing the flow. More formally,
•
x1 = −(k / b1 )x1 + V1
F1 = kx1

where x1 is the spring displacement, V1 the input velocity F1 the output force
and (k,b) are constants. Similarly, for the impedance,
•
x2 = −(b1 / m)x2 + (1 / m)F2
•
V2 = k x 2

38
If the coupling imposes a common velocity (flow) on the two systems, then
to satisfy conservation of energy, the forces (effort) must be equal and
opposite:
V1 = V2
F1 = −F2

Solving the equations for this system yields the following condition for
stability (Hogan, 1988):

(k / b )(b / m)+ (k / m) > 0

1 2

Of special interest is that if each system is stable in isolation (which they are
in the allometric examples for small forces), then they are stable when
coupled. In such systems, the energy will always be decreasing unless
continued to be supplied by the admittance. This is indeed the case for
animal motions, and relates to the metabolic energy consumed (Kleiber’s
rule, 1932 ). Our intuitive generalizations across the four biological forms is
thus plausible.

39
Part II Artifacts

40
 7. Tools & Artifacts

Although our appendages provide powerful biological tools,

additions to these body parts have given us considerable added power.
We argue that these tool developments are the major impetus to
evolution. Through these tools, fitness is also increased, principally by
allowing us to make significant changes to our environments. These
advances led not only to wood and metal-working tools, but also to earth
movers such as excavators or bulldozers, as well as additional other
improvements in construction techniques. These developments began
with simple implements such as sticks and stones.

For some species these earliest tools continue to be used today. One might
see a crow grasp a stick with its beak and use the stick to probe a hole or
crevice. Chimps use the same technique to gather termites from a nest. Bears
may use this method to test a beehive for honey. Dogs play with sticks. Even
human children seem attracted to sticks, swishing them like broad swords to
cut ferns, tall grass, or thin branches of trees. Stones are used by
chimpanzees and apes to crunch or crack open nuts. We may surmise that
unfinished sticks and stones are among the most primitive tools.

Fig. 18 An Oldowan early stone tool design

from Western Sahara (Wiki/StoneAge.)

Among humans, we know that stones were fashioned with points

and cutting edges as early as 2.5 million years ago. The first of these
implements were river pebbles, with flakes hammered off one side to
create a sharp surface on the other side, leaving a smooth rounded
surface for convenient gripping (Fig 18.) This tool has some
resemblance to the molar tooth of an ungulate, which is suited for

41
chewing grass, plants, etc. Hence these stones are regarded as
“choppers”. Later craftsmen created more complex shapes, with one
sharp edge for cutting or butchering, or a pointed edge for penetrating
animal hide (See Fig 19.) In turn these could be used to create hunting
implements or to shape wood, and to help create garments for cold
climates from animal hides. Later periods (200,000 years ago) reveal a
preference for a small set of shapes, specifically triangular-like stones
with sharp points, or orange-sliced shapes with knife-like edges. Many
of these implements were still used even 50K years ago. These shape
regularities are precursors to modes. Further support for modes is the
appearance of several of these kinds of tools, such as in a cave where
butchering or other tasks might occur. In today’s world, if we see a
hammer, then we infer that nails and boards to be fastened are handy;
for a screw driver we anticipate screws.

Other early tools include the spear: a sturdy stick with a sharpened
pointed tip. (Easily fashioned from “chopper-like” stones.) In some cultures,
the effectiveness of the point was enhanced by charring with fire (e.g.
possibly Peking man.) Another tool is the wooden club, fashioned from a
knotted root. Serrated flint knives, with teeth to facilitate sawing, help
harvest appropriate specimen roots or tree knurls. Although evidence is
weak (wood deteriorates and such tools are hard to document), some
relevant artifacts have been found in deep excavations in China, with the
implements dated to about 700K years ago.

The important step from a simple, isolated implement to a modal

artifact is when correlations among tools appear together. One of the
first was when a sharpened stone or bone was attached to a shaft to
allow greater force to be applied for penetrating animal hide. This
advance may not have occurred until 200K to 300K years ago and
implied the invention of some type of fastener, such as twine or leather
cord which was wrapped around a honed stick to hold an embedded
stone-knife (or bone) in place. We now have a hunting spear, greatly
improved over a pointed stick. It is more lethal, with greater mass and
penetrating power.

42
 Fig. 19 Early modal tools

To further increase the effectiveness of a spear, it can be launched by a

second stick with a cup at one end. The cup fits onto the end of the spear,
with the two parts held in place with one hand, which holds the free end of
the launcher. This is the atlatls. Travel distances can now be tripled. This
device is still used by primitive cultures today. Note one advantage of this
design is that there is no need for the two parts to be fastened together –
once launched, the spear travels on its own.

In parallel with the development of the hand launched spear, we can

anticipate the invention of the bow and arrow, created by many distinct and
different cultures – each arriving at this new artifact as a natural extension of
earlier tools. With twine fasteners and knots, a flexible shaft could be bent to
form a bow. The arrow is just a miniature spear. Its novelty is a design for
flight which placed severe constraints on its length, weight, size, etc.
Similarly, the size of a bow and its maximum tension is constrained by the
archer’s body size and arm strength. A bow and arrow is an important early
mode, dating back about 60,000 years, with the basic design unchanged even
today.

43
 The age at which these important modal advances first occurred is not
clear. We estimate these inventions were created at least 75,000 years ago,
and perhaps as early as 150K years ago, to be reinvented many times in
other cultures. To attach two tools (arrow head and stone handle) to one
another was an enormous conceptual advance, increasing the ability to
manipulate the environment. Not only was hunting more effective, but
agriculture was favored with grub-type hoes for clearing and turning soil
(Savanna in Africa.)

As mentioned, the fastener of the stone (or bone) to a shaft using twine
occurred as early as 75,000 years ago – roughly the same time as glue.
Strong support comes from the analysis of match and matting in Subichu
Caves during this period. The construction of such matting required some
elementary knowledge of knots and braids, and perhaps might have been
triggered in part by early humans noting the woven nests of birds.

The role of fasteners in the development of stone age tools has been
underestimated. To fasten the two parts firmly to one another, the cord
fastener would need to be knotted in some manner, or some type of weaving
of the cord around the parts would be necessary. Knots are conceptually
difficult to tie (recall the difficulty a 4 year old has learning to tie ones
shoes.) Weaving is simpler, but the fastener would be much weaker. One
might also need some “glue” to hold strapping in place. Early humans
invented a glue made up of plant gum, red ochre and partly beeswax or fat,
thus duplicating glues similar to those used by insects. Before attaching a
crafted stone to a stick to create a spear, the glue mixture might have
undergone a controlled heat treatment. The combination of a crafted shaft,
the pointed stone (or bone), glue and cord is an important step forward
toward the development of modal tools.

If one examines the assembly of these parts, there will be a sequence of

moves needed to position the parts and to wrap and tie the cord. Although
Apes have the opposing thumb, they are not noted for conceiving the
sequence of operations needed to accomplish this kind of task. As noted by
Heidigger (1951), craftsmanship requires a mental counterpart to make the
appropriate design. Hence it should be no surprise that the initial advances
with fasteners could have taken place before the appearance of modern
language. Note that the conceptual advance underlying both language and
knot-tying is an ordered sequence of finger movements or of concepts.

44
Today, we regard the prefrontal cortex as the locus of sequencing control.
Hence the role of this area of the brain may be critical, controlling fine
muscular movement of the fingers, as well as the fine movement of the
articulation of sounds. In chapter 10, we explore this relationship between
hand and mind in more detail.

“Good Feel”: early mechanics

The invention of fasteners led to a clear modal advance in tool

implements, with changes in designs the result of tools with compound parts.
Not only was the stone sharpened (or flattened) for the task, but so must the
handle be properly shaped. The length, diameter, and handle grip (ovoid, not
round) were all needed to give the orientation of spear the head. Together,
these modal advances meshed to form and effective tool giving the human
more power over his environment. These developments occurred
independently, throughout most of the inhabited world. Many of these early
tools still survive today in some cultures. Obviously the developments
required an optimal balance between the weight of the stone, its placement,
and the shaft diameter and length, and the length of the arm, hand, etc.
Properly designed, they gave the weapon a “good feel”, namely that the tool
is simply an extension of our own body parts.

An experiment.. “Good-feel”
In class, I give a blind-folded student a “hammer”, positioning the hammer
as it would be in use. The student can wave the hammer around, but can not
change his hand position. Without hesitation or sight, the student confidently
states “it’s a hammer”. Note that there are actually an infinity of arrangements
of handle length, placement of the metal piece, and shaft design that would
give the same inertial moments and leverage. But the student “feels” that the
mass is at the end of the handle, and properly infers the moments of inertia.
(He senses that the flattened face is aligned with the ovoid shape of the
handle.) The artifact is so highly modal that the inference “hammer” is made
immediately.

An important factor in the design of these early tools was an intuitive

physics based on “good feel”. The design of a spear or a hammer should be
aimed at giving the user the greatest possible power, subject to the ability of
his arm to wield and control the tool. The length of the spear, the weight of
the stone head, the weight of the shaft, how the shaft is oriented to control
inertial moments all come into play. The result is a highly modal artifact.

45
 The concept of “good feel” applies to the wide range of useful tools.
Underlying this mode is a strand of intuitive physics. We do not consciously
calculate the moments of inertia, center of mass, etc. Rather, our insights
have been derived from the use of our own body parts (i.e. limbs, hands,
fingers…), which formed the basis for an anthropological model capable of
being used by analogy.

Modern tool developments (7000 K years ago) are largely based on an

extension of intuitive physics, formalized by the Greeks and recorded with
written documents. Hence we regard the “pencil and paper” – or their early
versions - as very significant tools, helping to foster more sophisticated and
powerful designs.

Tool Date Observed (ref: 2K AD) Location

Unshaped shaped stone 2500 K

Flaked stone 1500 K Ethiopia
Advanced flaking 500 K

Spear (Honed stick) 400K

Fire (embers, lightening) Peking man

Language 100K ?

Spear and stone head 150K

Hatchet, axe 150K

Atlatls 100K

Fastener (weaving) 100 K Sibidu cave

(Glue)
Needle (bone) 65K

Bow & arrow 65K Cro-Magnon

Wheel 6000 Mesopotamia, Central Europe

Lever 7000 Greece

Bronze age tools 6000 Eurasia

46
 Wedge

Diamond drill 5000 Egyptians

Deep Drill 2000 Chinese

Screw

Pulley, Crane 3500 Egypt

Gear Train

Writing on paper

Mechanical Advantage

The stone age tools served to give the user increased physical
power, or more technically, increase mechanical advantage over
biological body parts. Their evolution has a significance and impact in
the physical world that parallels and equals the significance of language
in the cognitive world. Today, we attribute mechanical advantage to
several types of tools: the lever, the wheel, the wedge, the inclined
plane and the screw. These inventions were of human origin and do not
appear in other species.

The lever: If a stiff branch is used to move a fallen tree, or to pry a partly
buried stone, then a greater force can be applied if the end of the branch is
leveraged against a pivot, such as another rock. Such leverage is necessary
to prepare soil for agriculture. The mechanical advantage is proportional to
the relative lengths of the branch on each side of the pivot point. Once
illustrated, kids easily understand the concept. Two such sticks joined near
the ends of each are extensions of this concept, creating implements such as
the class “pliers” or wrenches.

The wedge: If two sides of a stone form a triangle to form a wedge, then a
crack in a log can be pulled apart by hammering the stone into the crack
eventually splitting the log. The mechanical advantage is due to the lateral
force applied by the inclined sides of the wedge. The inclined plane is
related, using the opposite relation between two forces: the weight of the
object and the force needed to move the object up the incline.

47
The screw: Wedges and inclined planes gain their advantage from forces
applied at small angles. The screw is a modern (3k years ago) version of this
principle. Each ridge of the screw acts like a wedge, with the ridges crafted
about a shaft. When the shaft is turned, the wedge applies a force against the
grooves in which the ridge sits. Hence if two pieces of plates, each with
grooves are joined by a screw with wedge-ridges, the turning moves the
plates together, such as occurs in the modern vice. At the top of the shaft,
enormous torque (and hence leverage) can be created if the shaft has a lever-
like handle facilitating the turning of the screw.

The drill: Related is the drill. In the simplest version, the end of the shaft
has a cutting tool, such as a sharpened stone, or fabricated metal piece.
Rotation can be achieved using a bow with its string wrapped around the
shaft. However, with shafts that have screws, more power can be elicited by
when turning the drill, especially when the turning is augmented by a lever
arm on the end of the shaft. The Chinese successfully drilled 2000 ft in 5BC
using this technique.

The wheel: There are two main motions for mechanical advantage: a
translation and a rotation. Precursors to the wheel used logs as rollers for
translating heavy objects. To advance this invention, someone conceived of
the idea of a disc mounted on a shaft, with the shaft piercing the center of the
disc. Two discs at the ends of the shaft then provided support for a platform,
perhaps something held in place by a triangular set of rods, with one end
used to move the arrangement – like a two-wheeled wheelbarrow. This step
did not occur until about 10k years ago. An alternate path could also have
come from the idea of the wheel as several levers to aide in creating more
power, such as for a deep drill. It is interesting that these advances took
place independently in several cultures, but not all. For example, in the
American continent, natives created triangular platforms for transporting
equipment and supplies, but were not tuned to the idea of a wheel.

Later, we see the creation of a pulley, where a wheel is used to guide a

strong rope to improve mechanical advantage in lifting or moving heavy
objects. A crane also follows.

The wheel and its derivatives were a significant, creative leap in tool
design. Not only did it give more mechanical advantage, but it led the way
to the development of carts for transporting goods, chariots for greater

48
fighting ability, and their use for water power, etc. The wheel has to be
ranked as one of man’s greatest inventions.

Compound modes

Many tools use several of the elementary mechanical machines.

This is especially clear when large equipment is needed for
construction, such as excavations and demolitions. The back hoe has an
arm with a claw, the arm serving as a powerful lever. Assisting the
power are hydraulic devices, where small increments in distance lead to
powerful forces. Engines have plates held in place by bolts— a screw—
as well as moving shafts that convey torques. Even the simple bicycle
has wheels and gear trains, pedal levers, etc. Not all advanced tools are
hand held—which greatly augments the population of tools types.
However, the underlying principles remain the same, as well as the
need for these compound machines to have a ”good feel” allowing the
user to model and control their behavior (Conant & Ashby, 1970.)

Modal Composites

As mentioned, often two independent modal objects can be merged to

create another, distinctive mode. This is different from simply adding
together many different regularities. A clear example is the addition of a
flattened stone to the end of a stick to create a “hammer”, or the cart which
normally carries goods, now adapted to transport people. However, rather
than an open wagon, an obvious advance is to create an enclosure for
protection against wind, rain, snow, etc. In effect a mini dwelling or room is
added to the cart. We now have two modal advances coupled to create
another mode. Later, once engines were invented yet another mode was
added that led to automotive vehicles. This merging of modal artifacts is
very common. Indeed, the evolution is similar to that seen in other domains,
for example in the evolution of languages. Although there may be a wide
variety in the languages, the modal roots are very similar.

49
 Towards a “Theory” of Modal Evolution

We now have the background needed to see the relation between

our early attempts to explain modal phenomena and the ideas that follow.
Let us depict the evolutionary process as a downward growing tree, as in
Fig. 11. Then we will now have strong analogies between early tools (and
Niches or dwellings), modal clusters, which have just been mentioned, and
finally language. The figure thus summarizes what has been covered, and
how advances may be related.

Towards a “Theory” of Modal Evolution

Modal Evolution Modal Clusters Language

Allometric laws Generative models Grammars

(e.g. polyworld) 45

Fig. 20. Similarities between Modal models, clusters, and grammars.

50
51
 8. Everyday Things

Everyday things include the many artifacts that are scattered about us.
Many of these artifacts come and go. Modes may wither away, like the
fireplace, which is now largely ornamental, being replaced by central
heating. Or lighting, where gas or candles were replaced by light bulbs with
the advent of electricity. Other everyday things include chairs, tables,
refrigerators, books, scissors, typewriters, etc. – artifacts we encounter daily
but may not always be optimally designed (Norman, 2013.) From the
perspective of Modal Worlds, often the artifact may be just a convenience,
but does not necessarily improve fitness. Consider the light switch.
Typically for wall mounted switches, when the lever is pushed up, the
electricity is “on”. The opposite when it is in the down position. A proper
design follows these rules. Although there is at least one regularity, the light
switch is not a mode. There are too many versions, such as those on a lamp.
It is also fairly recent, as electricity has become commonplace only two
centuries or so ago. Furthermore, we do not know how long switches will be
used. If switches are replaced by verbal commands, new modal aspects may
emerge.

It is important to realize that cultures may have artifacts that are unique to
that culture. These may or not be modes, or may be part of another mode.
For example, the Habib worn by south Asian or middle Eastern women are
common for that culture. The Habib is part of the mode for that culture. It
immediately triggers a notification of the culture. It is symbolic and
expresses part of a religious faith. In our country, the Amish exhibit a mode.
Not only is their dress unique, but also special is their avoidance of
automobiles in favor of a horse and buggy. But typically one’s dress will not
be a mode. Whether a Western woman wears a pink or blue-striped shirt is
of little consequence for us. Manhattan is peppered with such fashions; but
these are not modes. Other qualifications may be shoes (or sandals), perhaps
a belt, for these increase fitness and have been around for centuries. There
are many modes we have ignored, especially in other cultures. Not
mentioned is that umbrellas are common in Japan, even though it may not
rain, Sandals are common in India and other South Asian cultures; Eskimos
can not survive in Winter with out snowshoes. These are all modal artifacts
in that culture.

52
 In contrast, many everyday things are modal and occur in almost all
cultures. An example is the wrist watch. This device stands alone in its
uniqueness and origin. The modern wrist watch not only tells time, but also
the date, the day of the week, perhaps that year. The second hand may be
used as a stop watch. Perhaps it also has an alarm, or a GPS readout. There
are many features associated with wrist watches, separate from the tendency
to wear the device on the left wrist. Modern watches now do not need to be
wound, but can operate almost indefinitely using a small solar panel.

Although watches have been around for centuries. Modern technology has
now afforded miniaturization that has replaced the earlier pocket watch and
other varieties we now would consider cumbersome. An important modal
regularity is the standardization of the numbers, with 12 being at the top of
the clock, and the other numbers marked clockwise around the dial.
Typically we have a 12 hr interval, requiring the “AM” and “PM” notations.
Time is important to us (Cowan, 1958).

Our time keepers are still improving. Smart phones have clocks as well
as many other functions. In the future, we might expect that modal
developments include some of these functions, assuming they can be
incorporated without the confusion and clutter currently associated with the
navigation panel on many automobile dashboards.

Time is one of the six fundamental properties of physics, namely mass,

length, time, temperature, charge and photons. These variables underlie
almost all models for events. The measurement of distance with a ruler or
tape measure may qualify as modal, and perhaps also temperature
(thermostats control our heating). The specification of length is interesting,
for there are two systems: linear and metric. In the US, the linear system has
its roots in anthropomorphic designs, such as the foot, or digit. The unit 12
for a ruler has the advantage of being divisible into 4 parts (2, 3, 4, 6)
whereas in the majority of the world we have a base 10 system that is metric,
with only two integer dividers (2, 5). Although measurement of length goes
back over thousands of years, we still have not made a universal choice. This
leads to a breakdown in the dimensions of manufactured parts, especially in
the US where linear is common, but with some imported parts based on a
metric system.

An obvious everyday thing is the computer keyboard, which is now almost

universal. Its origin begins with the typewriter in 1752. Not only is the size

53
fairly standard to accommodate our fingers, but so is its “QWERTY”
organization of the keyboard introduced in 1854. A similar mode is that used
on wall phones, with three columns of digits and a few extra characters at
the bottom. This design now appears not only on phones, but also on ATM
machines, where their inclusion has led to a rapid acceptance of off hours
bank withdrawals.

Within our dwellings, the table and chair can be with certainty be
considered modal, as well as the doorknobs or handles (almost every door
has these, which are located at elbow height for easy grasping) or keys.
Other objects are more recent conveniences and do not necessarily have the
requisite many regularities all coming together. For tables and chairs,
however, the dimensions are set by body size: the seat is supported by legs
roughly of the same length as ours; the height of the back is about two feet,
comparable to what is needed for support if one leans backwards. Similarly,
the table is constrained by our arm length and chair size. Likewise, similar
arguments apply to door handles.

Other everyday things include scissors, money, the newspaper (perhaps on

its way out), the stove, the toaster, and many wall hangings, pictures or
knick-knacks. Most of these are not modes: they are conveniences but do not
have a host of regularities that define them. Some exceptions include money
– all countries have currency – and scissors, which may be a surprise.
Consider two knives, such as the modes described earlier. To create scissors,
one needs a rivet that hinges the two blades, leaving also two handles. We
have then the combination of two modes to create a new mode, useful in the
kitchen, in opening packages, or simply cutting a sheet of cloth or paper.
This is an excellent example of two modes being merged to create a new
mode.

Keys have been around for centuries, largely to lock boxes as well as
houses. Only recently has the mechanical key been replaced by a card, or a
pad with numbers to be chosen. The fashioning of a mechanical key has
many regularities: its grooves, the size and length, the handle, as well as the
lock into which it is inserted. A key card also has many regularities, but
these are not as obvious.

One modal feature often overlooked is the book. Book size is very
regular – roughly 6 by 8 inches in the linear scheme. Even before the
printing press, books were so constrained for easy handling and reading.

54
Book size also interacts with type size, which is designed to cover as much
of the pages as legible. Books also are typically only 1-1/2 inches thick,
being less than 300 pages. Longer books are difficult to assimilate in only a
few sessions. Perhaps also we tend to limit the amount of material presented
in one setting.

Modes therefore appear in many of our everyday artifacts, but not all.
When we enter a room, there may be a million dollar painting on the wall.
And in addition there may be some knick knacks of little monetary value,
but which have meaning to us. A price can not be set on such artifacts; it
either increases our fitness (and hence acts like a constraint), or it is simply a
choice we have made among many.

One may ask what is the difference between everyday things that are
modal, and those that are not? In some cases, the distinction is slim. But a
general rule, aside from the merging of regularities required for modes, is
that modes act like constraints, whereas the items not modal are typically
conveniences.

55
 9. Bio-Tools

During tens of thousands of years, biological forms have developed

appendages that function as tools. These “tools” can manipulate the animal’s
environment. In some sense, they are the complement to modal artifacts, in
that they allow many configurations, not just a restricted number as do
modal artifacts. In some stage in their development there is a confluence of
regularities that increased their power. A hand is such a tool.

Hands

With the exception of a brain, a primate hand is the most powerful bio-
tool. Each has almost countless numbers of possible configurations. A single
finger can probe or grease a hole from almost any angle. Two fingers, one
being a thumb, can provide the grasp of some objects to be picked up. If
three fingers are used, the grasp is stable with friction. The grasp can occur
with many configurations of the hand and arm. However, from a physical
systems view, such a mechanical object will have at least six degrees of
freedom (DoF), namely the three orientation directions and the inertial
moments about these directions (Salisbury, 1988). However, our definition
of a regularity goes deeper, and allows for several ways (i.e. configurations)
to satisfy the degrees of freedom required by physics. Thus, with frictional
grasp, the primate hand masters the challenge of recovering these moments
including the center of gravity of the mass. The choice of configuration is
one that has the greatest “fitness” in that environment.

One obvious advantage of a hand is that it can grasp edible objects in

many ways. But nevertheless hand size is still constrained by mouth size.
Similarly, the hand cannot be excessively large in relation to the limb to
which it is attached. The latter also is related to body size. Furthermore, limb
designs will favor an ability to position a hand for flexibility of a grip. Hence
we see a correlation between mouths, hands and other grippers, limb design
and body shape, as shown earlier in Fig 20. Like modes, observing one
regularity we infer another. In this case, however, the regularity indexes to
the entire animal design.

Hence for an animal, bio-tools as a group can be considered a collection of

regularities at comparable scale that are highly correlated with one another.
For legged animals, we see the correlations between pitch of voice, stride
frequency, and leg diameter. We can also add the noise of footfalls:

56
elephants make loud low frequency sounds, small creatures make higher
frequency sounds of low power. These correlated laws describe regularities
over a range of spatial scales, lying in a five or more multidimensional
space. Although our measuring instruments are different, they all reach a
common conclusion, effecting a dramatic dimensionality reduction that
isolates the main variables, allowing also predictive inferences across a
range of scales. We have discovered a powerful index into the hidden
processes and mechanisms that nature devised to generate these biological
forms.
If a body part has multiple uses, the underlying functional variable
becomes clouded. Hence the hidden modal strand becomes more difficult to
identify. Nevertheless, given the body parts reviewed, we can extract a
glimmer or modest insight into modal variable with strands that span a range
of species as shown in Fig. 22.

Tasks
The evolution of some of the very first biological tools can be traced to four
main tasks. These are:

1. Limb design (positional flexibility)

2. Grasping
3. Chewing
4. Digging and burrowing

Table 1 provides a limited first glimpse at the scope of these tools and tasks
for insects, avian and terrestrial animals. The columns indicate the tool or
biological advance; the rows show the relative success of the for a task

Glue mandible beak Claw Teeth hoof Finger Thumb

Dig X X X x X
Burrow X X
Crush/crack X X Xx x
Grasp Xx Xx xx Xx Xxx X
Weave X X Xxx
Pickup X Xx X X Xx Xxx Xx
Fasten Xx X X X
Knot Xxx Xx

57
 (number of xx’s.) Note that these tools are species specific; mandibles are
associated with insects, beaks with birds, etc. Given these adaptations, how
can we find submodal strands? The approach will be to focus on three types
of animals: insects, avians, and terrestrial legged animals, attempting to
identify possible strands. We then show that all can be seen as the result of
functional pressure, at least for the advanced species. The implication is that
animals with the most effective body parts exhibit the most modal forms.

Insect Mandibles

One of the simplest body tools is the mandible of an insect such as a

termite, its mandible being a two-part gripper that resembles a pair of pliers
with curved tongs. For attachments to the head of an insect, the two parts
can overlap, like a claw. The latter form is similar when attached to a limb.
Unlike bird bills however, that close vertically like a mammalian jaw,
mandible design closes the two components horizontally. Furthermore, most
mandibles are rough on the inside, with crude teeth-like grippers, favoring
shredding, grinding and grasping. There is also a scissors or shearing action
due to the two parts of the mandible crossing one another. This action and
design is a very consistent regularity across insect species.

A similar tool also appears as a claw at the end of the insect (termite)
limb. Hence there is a strong modal correlation in the design of the claw and
the head mandible. However, the simplicity of this plier-like tool is a
disadvantage in manipulating objects: the degrees of freedom (DoF) that the
tool can stabilize is very limited, something on the order of 3 or 4,
depending on the roughness of the inside of the mandible and the softness of
what is gripped. To completely control an object, we must immobilize 6
DoF mentioned earlier. One scheme would be to glue the grip to the object,
or have a deep probe like a talon or two. Another would be to bring several
claw-like grippers together about the object. The latter requires a lot of
flexibility of limb position. For an insect with 6 segment limbs, this is the
simplest option, bringing two or more limbs and their extensions into
proximity without difficulty. This union between the body part and a large
number of limb segments is a further demonstration of modal correlates.
Advanced insects (termites, spiders) use multiple limbs and their grippers in
coordination for some tasks, thus reducing the remaining degrees of freedom
of movement of the object being manipulated.

58
 Bird Bills and Feet

Unlike the uniformity of insect mandibles, the bird’s beak takes on a

variety of forms that are similar but modified for specific tasks. (See Fig 21).
Except for adaptations by the raven and crow, most bills are designed
primarily to favor one function, such a probing, chiseling, cracking seeds,
etc. Obviously long, pointed bills will be best for probing, but seed cracking
requires a more blunt bill to gain maximum force. Similarly, chiseling into a
tree trunk, needs a combination of a blunt bill for strength, but some length
to create a deep hole. Clearly the task influences the designs. This variety in
bill design masks the underlying modal structure. Nevertheless, there are
some mild shared regularities.

Consider two variables: (i) the aspect ratio of a bill, namely its width at
where the bill joins the head versus its length; and (ii) the curvature of the
beak, as measured by the change in tangent angle at the head to that near the

Fig 21. The bill and feet as tools for several types of bird

59
tip of the bill. (Many use the underside of the upper bill to avoid the pointed
extremity.) In contrast, hummingbirds have a small aspect ratio,
with a bill of little curvature, allowing for deep probing. Raptors, however,
need high curvature to tear meat, etc., and have short, thick bills for strength.
Fig 22 also shows a variety of bird feet, ordered by the curvature of talons.
Clearly, like beaks, there is a functional pressure on these designs: curved
talons of raptors favor the ripping of flesh, but also they improve grip. The
arrangement of talons and their curvature immobilize their prey. At the
opposite extreme are straight talons with poor gripping power. But like little
swords, they could serve as a weapon for protection (especially for birds that
don’t fly.) Another, quite different adaptation is the water fowl’s webbed
foot that favors swimming, yet which fails as a gripper.

60 Rp

Tr
40
Talon curvature

Ph

20 Wp

Du
0
0 20 40 60 80
Beak Curvature

Fig 22 Evidence for a submode strand between some birds’s feet ( talons) and its beak
design. Rp (raptor), Tr (thrush), Ph (pheasant). Wp (woodpecker), Du (duck)

For some birds, there is a correlation between the bird’s beak and its feet,
suggesting an underling common functional submode strand between two
body tools. On one hand consider the raptor with a high curvature beak and
its feet with curved talons; or, on the other hand, the flat low curvature of
duck’s bill and similarly the low curvature of its feet. Fig 22 plots the
curvature of bills and feet, showing a high correlation for some bird species.

60
 A variable underlying this correlation could be functional use. For example,
if predatory skill is the underlying variable, then the raptor with curved
talons and bill is very adept, whereas the waterfowl with flat bill and webbed
feet will fail as a predator. This suggests a possible correlation between the
predatory nature of the bird and the curvature of both beaks and feet talons.
Of interest then is the biological process that designs both these physically
different body parts. Why should beak design and feet design be related?
Is a one-gene alteration sufficient for both parts? Another possibility is that
together they supported increased fitness, as proposed by Darwin and
Wallace.

Paws and Claws

For terrestrial animals we see a variety of designs, from paws with no

claws to hands with strong fingers that can exercise control over an object.
These body tools are associated largely with two segment limbs. Hence,
unlike insects, positioning about the body space is limited. Furthermore,
paws have soft pads that cushion the limb during locomotion. Their success
as grippers comes from their claws, or by the use of two paws acting in
concert. The latter is not that successful because two segment limbs do not
offer much flexibility for paw orientation. Claws, however, can be
formidable. They play the key role for bears, raccoons etc. and squirrels and
other rodents, who use claws for climbing, as well as improving the grip on
objects. In this respect, paws and bird feet share a common feature. To
improve grip, we need finger like segments and talons as in bird feet, or
flexible fingers of a hand. Placing one finger opposing the others was a big
evolutionary step for a gripper, The thumb was the highlight, which greatly
increased grip stability, as well as improving tactile assessments of object
shape. Three segment fingers had allowed a hand to grip curved objects with
many points of contact, especially if the hand had four fingers. But this
configuration alone does not furnish backside contact for large objects.
Some birds solved this problem by placing one finger opposite the others.
But it is the thumb that provided the best solution, greatly increasing grip
stability (as well as tactile assessment of object shape).

62
 Teeth and Paws

For animals with soft paws and only small claws, the mouth becomes the
principle gripper. Now limb flexibility is almost lost because there are few
DoF of movement for the head. Hence the grip has to be unusually secure
and powerful. This is a characteristic of carnivores and rodents, with a jaw in
which the teeth are embedded. The combination of a muscular jaw with
teeth provides one of nature’s most useful and powerful tools. The jaw
muscles can provide up an enormous force especially at the opening of the
mouth where the incisors are located. (These incisor teeth are typically used
for cutting and shredding). Further back, the canines are the next significant
type of tooth, followed by a set of molars. The canines can tear into meat,
the molars can crack nuts, or in the case of ungulates, grind down grasses or
other edibles. Perhaps what is most remarkable is that these three types of
dental forms have persisted for tens of thousands of years, suggesting a near
optimal solution in the context of feeding.

Together with the jaw, teeth can also be used for grasping objects. Indeed
those species without hands or deep claws (such as dogs) use this tool quite
effectively. For other animals, such as mother cats, the teeth and jaw enable
infant young to be carried by grasping a skin fold behind the neck. These
adaptations, however, might be more properly viewed as an extended use of
this tool, not its primary purpose. Once again, we note the mouth as a
gripper is very powerful, and an adaptation that balances for the lack of
ability to orient this tool.

Multi-tool use

One important issue is whether the optimal tool is designed to support

many tasks, or just one. A Swiss army knife has many uses, such as for
cutting, nail clipping, or turning screws. With the constraint that these
implements fit together into a compact space, each individual tool is
sufficient but not optimal. We see the same conflict in some body parts used
for more than one task. A bird’s beak is best designed for gripping food,
probing for insects or grasping a twig or grass blade when nest building.
However, although grasp is suitable for weaving a nest, it is far from
optimal. Similarly, the “feet” of the limb of a canine have the primary
purpose of cushioning when traversing uneven ground, but also can be used
for scraping and digging (excluding the use of claws.) In some cases, when a
tool is designed for more than one task, the multi-task modifications do not

63
interfere with one another. For the canine, claws at the extremity of the paws
do not interfere with cushioning, for example. Another is the palm of the
human hand that can be used to cup water or to collect cracked nuts, etc.
The mouth also can chew, grind food, crack nuts or seeds, and is often used
to grasp objects. Birds, use their beaks to dig and probe. Both may be
classified as multi-use tools. For most animals, however, the extensions
limbs (claws, paws, hands) are preferred for digging, burrowing, probing or
grasping, whereas the mouths serve primarily for chewing.

If a body part has multiple uses, the underlying functional variable

becomes clouded. Hence the hidden modal strand becomes more difficult to
identify. Nevertheless, given the body parts reviewed, we can extract a
glimmer or modest insight into modal variable with strands that span a range
of species.

Functional strands: two solutions

Consider the gripping and manipulation of an object. To position the

gripper, the limb must have at least six degrees of freedom of movement
plus a couple more DoF for redundancy to allow for different configurations
if there are obstructions or other constraints on positions. This redundancy is
especially important when two or more limbs come into play to orient
grippers about an object. Similarly, the gripping segments are typically not
straight line segments, but have joints that allow the “finger” to be wrapped
around the object. This increases stability of a grip and also provides some
redundancy for positioning. We thus have two factors that govern an
animal’s ability to control and object: the flexibility and DoF of the limb,
and the DoF and flexibility offered by the gripper. With redundancy, the
DoF of these body parts can easily reach 8 (the necessary 6 DoF plus 2 DoF
for redundant positioning. Of course, these are minimum requirements –
degrees of configuration needed to judge modes permits more flexibility.

If we now include body parts (or bio-tools) in our modal world, we have
described three distinct modes: cities, animals, and body types. Whether the
regularities are global or local makes an enormous difference in the nature of
the mode. There are many others, but the reader should now see that digging
deeper into each category, or moving laterally at the same scale to another
domain will reveal yet more modes. The world is thus a collection of modes,
with each modal cut quite distinctive.

64
Part III Representations

65
 10. Tools for Thought

The brain or its antecedents was one of the original biotools. Although
entirely internal, this organ was the result of the need for a creature to decide
how to assemble modes. One of the earliest we encounter would be the
mounting of a stone to a stick to form a hammer. How should these parts be
attached? For this artifact, there is no known antecedent to be copied. The
brain undoubtedly was introduced to this problem, perhaps noting that lower
animals formed fasteners by winding vines or twine around the two pieces.
However, if the stone is merely attached to the side of the stick, the device
will be fragile as compared with a stick attached on both sides of the stone.
Hence a forked stick would be more appropriate (assuming the fastener
problem is solved.) Another solution would be to chisel a hole in the stone,
and drive the stick through this hole. Both solutions are seen at the early
stages of hammer design. Hence we conclude the brain must be involved in
artifact design in the early stages of development.

Thus, the brain had a role in new designs, as well as their construction,
and implementation. However, being internal, its role was underestimated.
But to perform simple tasks, like pick up or step forward, a creature needed
to coordinate the various movements of each part. Each of these might have
several degrees of freedom. These aspects underlie the innate control of bio-
tools, such as when we carry out a simple operation such as “pick-up”. Not
only is the brain the primary organ for carrying out such procedures, it also
regulates most of our body functions. More important however is its role as
the decision-maker who choses and controls other bio-tools or invents new
ones. It organizes our limbs to produce effective action sequences; it can
easily switch from one representation to another, such as from pictures to
words as in a map; it knows when to regard a sine wave as a picture, and
when to transform the representation into a power series such as the Fourier
transform, etc. To enjoy these different representations, the brain easily
switches from one to the other, as in a geographical form showing spatial
distances between points, or as a topology like a subway map that
emphasizes routes to locations.

Charts

Underlying any procedure is a representation. A chart is one of the

simplest. For example, Mr. Beck’s depiction of the London subway map is
one of the earliest clear examples (Garland.1994). Beck’s insight was to

66
create a “vivid representation” of the domain. The frame set the horizontal
and vertical dimensions in the map. (See Boston map in Fig 24.) Colors are
also used to identify the routes. In this case the labels are oriented at an
angle to the routes. The simplest and typically most important routes lie
parallel to this frame, with North at the top. Extra routes then appear as 45
deg diagonals, avoiding wherever possible the odd diagonals. All routes are
labeled with colors. The red line takes you directly from Harvard square to
down town. We do not worry whether or not the route is straight as the crow
would fly. Stops are labeled with landmarks, such as Park Street or Harvard
Sq. This is a “vivid” representation, because the answer is almost
immediately obvious (namely the route to be taken to get from A to B.) An
earlier example of vividness was the map of Manhattan. Here switching
from street number to written addresses is easy. This is a typical
regularization approach. The reader can look at the map and easily find the

Fig. 24. The subway map for Boston, showing regularization, favoring
the presentation of lines parallel to the coordinate frame.

solution she seeks. Thus regularization is a term we use to present data in a

frame or coordinate system that is easy to understand (Brachman et al.
2006). In some cases, we change the representational form to facilitate the
solution. In addition to moving from pictorial to linguistics forms, in
mathematics, we have symbols that can be represented in many ways (i.e.
the cosine, or Fourier or Laplacian transforms). Again a switch allows the

67
reader to look at a plot and easily find the solution she seeks. Only a brain
can do this. This is but one example of a brain’s ability to choose a
representation convenient for the domain.

A second example are charts that show a trend such as the stock market
or other economic issues. In Fig, 25 an entirely different trend is shown –
that of the emergence of political parties. Note that the plot is aligned with
the coordinate frame of the picture. Although hard to read, it is clear that
language alone would be a poor choice to describe this trend. However
language is still needed to provide the details. Both language and pictures
together can often provide a rich representation of the trends.

Fig. 25 Chart showing political parties in the US from 1780 to 1940.

Yet another form for a chart is the picture shown in Fig. 26. This
representation depicts a solution to the Pythagorean theorem. Again, the
regularization immediately gives the reader insight into the proof. Note that
a + b lie on the same line and the four instances form a square. With a little
simple algebra we now see the relationship between c^2 and (a + b)^2. The
regularity plays a key role in the proof.

68
 a b
b
c
a
c c
a
c
b
b a
2 2
. ( a + b) = c + 4a b /2
Fig. 26. Both pictures and algebra some together for a proof of the Pythagorean theorem.
Note that each a +b lies on the same line, and are the regularity that enables a simple.
obvious proof.

Bartering
One of the earliest transformations of representation was the switch from
verbal to pictorial forms or vice versa. This ability would occur when
symbols were introduced for barter. On another front, special needs must
be satisfied. For example, with a brain able to relate symbols and shapes, it
made sense to design special pebbles with shapes and symbols representing
goods for bartering such as livestock, grain, and probably other implements
of general uses such as tools or pottery (See Fig. 27). This was one of the
first steps toward systems that represented an object - a leap of faith if you
will. To be acceptable, these tokens would have to be approved by the
society using them. Pressures to move in this direction would come from
economics – the need to make simple pebbles more informative about
wealth than shape alone. These simple tools appeared as early as 5 – 10K
years ago.

Fig. 27 Some early tokens used for bartering

69
 Early tokens are obvious precursors to the use of symbols for exchange,
and their sophistication paralleled the evolution of language and writing. An
important step was to invent a numerical system to represent multiple tokens
of the same kinds, such as by marking the outside of clay containers that
housed the tokens. Record keeping became easier and more accurate.

By 3000 BC (5K years ago) various cultures used special tokens for
money. In Mesopotamia, the shekel was the ancient unit, referenced to a
specific weight of barley, which was also used to value silver, copper, etc.
Metals became favored because of durability. Gold usage can be traced back
to 4 BC, when bars of set weight were used for exchange by the Egyptians.
Coins were introduced as early a 799 BC in Lydia, and this kind of exchange
spread rapidly. To bring these changes about, the society must rely on its
Governance. The role of leadership is to exercise management over the
community. In lower animals, the most primitive governance is when one
individual carries out most of these tasks through dominance. However,
what distinguishes governance of the complex societies from primitive
precursors is the presence of a supporting cast. Insect governance rests not
with one queen, but utilizes the talents of many lower cast insects to create a
stable society. In humans we have evolved in the Western, democratic world
with a tiered government structure: a head of state, a team of advisors (or
section heads), the members of congress, political parties and the populace,
as well as a judiciary. Families, tribes and small communities can be seen as
abridged versions of this structure. The need for a government is to maintain
common lands and resources and to coordinate or control actions in a
society. This is true both for insect societies as well as ours. However, with
advanced symbolic tools, such as exchange and money, taxes become the
device used to provide funds for these government services. These
procedures were present in many cultures very early - at least since 10 K
years ago. Note that the steps between different denominations are set to
optimize their usage. We do not have dollar bills of denomination $1.50 as
this too close to the dollar. These steps are modal, set by custom for
convenience.

Unlike previous modal structures, modes of governance need not be

stable (Jacobs, 1961). The populace can rise up and contest its government
policies, striving for change, as has been happening currently in the Arab
Spring. The internet greatly favors such grass roots uprisings and may well
become the important medium for expressing dissatisfaction. Its advantage
is that thousands of individuals may be reached quickly, with rapid

70
exchanges. The fire can be fueled quickly, more so than going through the
slow process of physical meetings, or even relying on person to person
phone contacts. This is a game changer.

For most of our waking hours, we access alternatives (largely actions),

choosing among many possibilities. There is almost always a variety of
choices, not just one. Imagine one of the simplest bio-tools – the hand. To
coordinate both hands the relevant organ must have paralleled the
development of the tool. Hence the aspects of a brain of interest to us must
have begun its evolutionary development along with the development of the
tool. For homosapiens, we can see that this occurrence began tens of
centuries ago. Similarly for the role of other parts, such as the legs which
eventually enable to creature to walk upright, etc. (Recall how difficult it is
for an infant to go from a four-limb crawl to standing and to move upright.)
At this early stage of evolution, we might already have a rudimentary
language, and the ability to draw shapes. (e.g. Lascaux cave drawings).

In parallel with the development of the brain as a coordination device, we

also begin to see that inventions began to come into play. Rather than simply
coordinate the biotools, the brain could imagine the design or a new use of
an artifact for some particular task of interest. Now the brain becomes the
driving force, not just a part player in the use of biotools. The role of the
brain has begun to shift so that it becomes the main player, not the biotool.
This is the beginning of a new evolution, one that will dominate all further
homo sapien advances.

Preceding the invention of tokens for exchange we have other tasks that
are better solved by using multiple modes. One of these is a map for
navigation. It is common for a map to use both pictures (spatial information)
and language to indicated landmarks. Animals lay trails. Ants use
pheromones, apes may break twigs to indicate routes, the terrain itself
controls routes of passage. But these techniques did not show others the
preferred rote. Surprisingly, the honey bee solved this problem by indicating
a direction during a dance in the hive, and well as a distance by the dance
length. So these simple creatures had the ability to combine information
about distance and direction, and communicated this information by the
pattern of the dance. This advance came later for other creatures who
created the first multimodal maps as we know today. As stressed, a key

71
component was the representational frame used to assimilate the
information, and the action sequence of the dance to guide followers.

Numerical Calculations

With the advent of bartering, and the recognition of symbolic artifacts

such as stones or beads to assist counting, there was a need for an arithmetic.
The brain made some of the key choices. When counting, why not group by
11 or 17, and not ten? The primary numbers do not divide, and hence are not
as practical as using the fingers. Grouping by hands leads to the implicit
notion of a decimal system. (Note that a twelve number basis would be more
flexible, but there is no obvious duo-decimal anthropological device.) The
roots of mathematical thought – simple counting - hence lie in a simple tool,
the hand. Only when computers were being invented, did one see the
advantages of binary or octal bases for these machine calculations.

In 2000 – 3000 BC several cultures introduced the abacus. This device was
first introduced in Mesopotamia. By 500 BC several cultures had versions of
the abacus. However, these early forms were merely counting boards. The
user would perform calculations in his head, and then use the abacus to keep
track of the answers (eg sums). Not until the 19 century did more
th

sophisticated abaci appear for multiplication and division, and still

Fig 28. The 20 century Russian abacus is very similar to the Chinese and Roman abaci
th

dated about 2000 BC..

72
Later, a scheme for finding square and cube roots was devised. One of the
most advanced was a Russian design shown in Fig. 28. Although not
common in most Western cultures, the abacus has continued to aid number
crunching even today. This is a unique achievement, and, unlike the slide
rule, survived the advent of mechanical and electronic calculators. Tools for
numerical calculations are one category of mode. The interface for modern
calculators and typewriters is another. Although only a century or two old,
together with the modern computer, the interface has become a standard.
Perhaps later, a speech interface will replace this. Nevertheless, the trend is
apparent. This is an advance in artifacts due to brain design, in a way no
different in principle from the century old artifacts of hand tools that enable
more power over the environment. For proper use, both require a set of
action sequences, which in the case of calculators and computers is often set
up “on the fly”.

Procedures

A procedure is a sequence of steps needed to complete a task. Hence

these are a generalization of a set of fixed action patterns. When stone age
individuals attached a sharpened stone to a stick, there was a sequence of
steps, such a sharpening the stone, crafting the wooded handle, and fastening
the stone to the handle. If the fastener was a cord, then the end had to be
knotted or secured in some manner. Although all steps in the sequence are
not unique, one clear set of instructions must have been passed on verbally
or by example. Similarly, when preparing an animal for a meal. Cooking
recipes are among the earliest written records, dating from about 3000 BC.
Procedures are an inherent modal part of our lives, saving many trial and
error manipulations to solve a task.

Today, recipes and computer programs are probably the most common
use of procedures. Their detailed structure need not be the same. For
example, many computer programs use a divide and conquer strategy,
assembling many small procedures into a larger program. One novel aspect
of this approach is recursion, where the same procedure is applied for n
iterations to the outcome of the previous iteration. Recursion is a rather
recent development, and not naturally a human mode of thought. Although
common in many economic approaches, some training is typically needed to
invoke recursive solutions (just as is needed for logics and the calculus.)
This is in the same category as a change in representation, such as the many

73
ways to specify a sine function or use of the Fourier transform to solve a
particular part of a wave equation.

Summary
To briefly summarize some of our ideas, we can place them as advances
in a table, including several not mentioned: Most require a representation,
and
several involve procedures.

Time before 2000 AD Core Ideas Artifacts

2,000,000 ?? hand

100,000 Language stories

50,000 Numbers pebbles

Symbols (writing) stick, stone, sand, charcoal

10000 Drawings paper, pencil, pen

Value tokens shaped stones (1-2 cm)

8000 Toys stones, stick dolls

5000 Geometry (sine, cos) straight edge, ruler, compass

3000 Logic, counting abacus

1000 Maps stars, compass

500 Books/library printing press

250 Nomogram, charts chart with scales

200 Machine multiplication Babbage calculator

100 Differential calculus harmonic analyzer

50 Complex calculations computer, tablet

20 Knowledge retrieval search engines

10 Virtual links smart phones, etc.

74
 We have a rather long preliminary list of tools for thought. Language,
sketching, numerical calculations are three of the most obvious modes
underlying these tools because of the duration they have been in use. These
devices are modal constructs, in same sense as our cities. They involve the
use of procedures – namely various action sequences needed to carry them
out. These then are obvious collections of regularities, or modal tasks. There
are also simulations resulting from play, as well as written records and
books. One key point is that many of these tools used more than one tool
simultaneously, such as language and pictures. These present no problems
for the coordination of action sequences, which like perceptual abilities are
often multimodal.

An essential point is that ways of thinking are tools that differ in an

important way from just one action sequence effected by a biological mode
rooted in an animal’s design. Thinking is a cognitive ability, tied to the
design of our brains – i.e. our perceptual, sensory and motor control systems.
If our brain design changes, then so will the representation under use and
also our modes. This limitation leaves room for creative thought “outside
the channels” of cognitive modes.

In future years the brain may be augmented by a prosthetic or mechanical

advantage as is the case for the hand and other bio-tools. Already many
devices can perform better than the human brain for complicated math
problems or search, for example. But these require interfaces to the brain
and as such are not an inherently independent part of the organism itself.
The first advances of this kind will occur when there is a genetic change in
the design of our brain. An earlier example could be the language module
proposed by Chomsky (1955) and discussed by Pinker (1994). Note that
this change occurred many centuries ago. Hence genetic evolution is
typically slow moving. But, in contrast, for advanced species like homo
sapiens, the evolution can be relatively rapid. Thus, contrary to Darwin’s
original ideas, evolution proceeds in part by homo sapiens changing his
environment to increase fitness, not vice versa where Nature is imposing
forces that alter species designs.

75
 11. Mens et Manus

The “Origin of Species” was one of the most influential works appearing
in the last two centuries. Darwin & Wallace’s proposal was that evolution
proceeded by the species adapting to changes in the environment. In
Spenser’s (1859) summary, he said that “The fittest species adapts most
rapidly” and thus advances evolution.

Modal Worlds disagrees and argues the opposite: the most advanced
species modify their environments to increase fitness. Homo sapiens is our
prime example. Man modifies the environment at will. The evolution is a
result of these changes, and the artifacts built to enable them. Over hundreds
of centuries, the brain has slowly become the dominant factor in evolution,
with its force still growing stronger. The result is a landscape suited to Man,
with new structures capitalizing on past regularities and designs in existing
structures.

76
 For most of our waking hours, we access alternatives (largely actions),
choosing among many possibilities. There is almost always a variety of
choices, not just one. Imagine one of the simplest bio-tools – the hand. To
coordinate both hands the relevant organ must have paralleled the
development of the tool. Hence the aspects of a brain of interest to us must
have begun its evolutionary development along with the development of the
tool. For homosapiens, we can see that this occurrence began tens of
centuries ago. Similarly for the role of other parts, such as the legs which
eventually enable to creature to walk upright, etc. (Recall how difficult it is
for an infant to go from a four-limb crawl to standing and to move upright.)
At this early stage of evolution, we might already have a rudimentary
language, and the ability to draw shapes. (e.g. Lascaux cave drawings).

The brain or its antecedents was one of the original biotools. Although
entirely internal, this organ was the result of the need for a creature to
coordinate its body parts. Thus, the brain was involved not only in the
creation of a new non-biological artifact, but also had a role in its design, its
construction, and its implementation (Heidigger, 1971). However, being
internal, its role was probably underestimated. But to perform simple tasks,
like pick up or step forward, a creature needed to coordinate the various
movements of each part. Each of these might have several degrees of
freedom. These aspects underlie the innate control of bio-tools, such as when
we carry out a simple operation such as “pick-up”. Not only is the brain the
primary organ for carrying out such procedures, it also regulates most of our
body functions. More important however is its role as the decision-maker
who choses and controls other bio-tools or invents new ones. It organizes
our limbs to produce effective action sequences; it can easily switch from
one representation to another, such as from pictures to words as in a map; it
knows when to regard a sine wave as a picture, and when to transform the
representation into a power series as the Fourier transform, etc. To enjoy
these different representations, the brain easily switches from one to the
other, as in a geographical form showing spatial distances between points, or
as a topology like a subway map which emphasizes routes to locations.

The brain’s role occurs almost daily when we choose a sequence of

actions for our limbs. “Pick up” for example, depends on whether the object

77
is an axe or spoon or sledge hammer. These different representations for
actions have one thing in common: their representations are “vivid”. In
particular, when the choice is made, the result of the sequencing for the
example will be obvious. The final action sequence will be an ordering of
those body tools that are very simple, chosen among the countless degrees of
freedom available. To pick up and wield the dinner spoon the sequence will
be a series of finely tuned finger movements, whereas for the sledge hammer
requiring a large force, the entire arm must be invoked. Just to highlight the
complexity of the final procedure, this objective requires (1) first the turning
of the head and then the eye to focus on the object, (2) a step toward the
object, (3) the reaching of the arm, (4) the positioning of the hand, (5)
grasping, (6) then pick up – at least six steps in a sequence governed by our
brain. These movements are not arbitrary, but tend to use fixed action
patterns found useful in the past (Tinbergen, 1951)). In other words, there is
a set of coordinated regularities. “Pick-up” is a mode of behavior using legs,
body, arms hands, fingers and only one of the many we use. It is a sequence
that we have learned involving perception and action. (Stratton, 1897;
Kohler, 1951; Held & Hein, 1963). “Pick-up” is but one of many activities
carried out by the brain, choosing and coordinating a set of regularities
offered by possible action sequences. In addition to coordinating the
movements of bio-tools, the brain also easily learned to coordinate different
modalities, especially speech or language and pictures, It would also decide
the representational form best suited for the task. There is almost always a
variety of choices, not just one. The choices optimize the possibilities, and
make extensive use of regularities we have found useful in the past. Contrary
to popular opinion, the final selection is not necessarily a winner take all –
the loudest shout - but rather a cooperative decision to give the most their
best (Richards, 2015). These decisions are multi-modal, typically using
grammars, numbers, and/or drawings. An important observation is that we
often switch from one representational form to another, such as from
language-like to pictorial forms…a trick that is very hard for machines. To
do this we typically use related regularities, as is shown in Fig. 26. A brain
thus assimilates its multimodal interfaces and creates a modal regularity by
itself (von Uexkill, 1926; Hoffman, 2010). A brain makes internal modes
possible.

In parallel with the development of the brain as a coordination device, we

also begin to see that inventions began to come into play. Rather than simply
coordinate the biotools, the brain could imagine the design or a new use of
an artifact for some particular task of interest. Now the brain becomes the

78
driving force, not just a part player in the use of biotools. The role of the
brain has begun to shift so that it becomes the main player, not the biotool.
This is the beginning of a new evolution, one that will dominate all further
homosapien advances.

12. From Dawn to Destiny

Our world is graced with regularities. Modal Worlds lists over one
hundred of these. Only when several of these regularities come together do
we achieve the status of a “Mode”. A strand of wire or a stone or brick is not
a mode. However, when placed in the context of a house, these become part
of a set of correlated regularities. Hence a mode is a collection of
regularities. One unsolved task is to design a theory that can tell us whether
a modal collection of regularities is stable, or easily broken by some
perturbation. Such an analysis would be a semantics for pattern matching
stimuli (Kahl et al, 2006). (See also Wolfram, Chpt. 8 , 2002). Although
some of these regularities can be modeled, most can not, at least with current
mathematical methods. There is no metric such as used by scientists. Hence
a mode, being a collection of regularities can not be modeled in the physical
sense. In other words, we can not identify all the variables to be modeled, let
alone specify how they will be measured. We may have more than enough
variables, or too few, but never that particular set required to complete our
model. Modal Worlds is thus not a science, even though its structure will be
influenced by Science. Nevertheless, once a mode is identified, it will seem
obvious to us that this is a good description of the world at the scale of the
mode. We do not have an algorithm. Hence we can not make advances as we
did in the 70’s with an algorithm that implemented a theory (Marr, 1982).
For example, although we can build one city by copying another, there is no
algorithm that tells us what features of a city are modal and which are not.
How do we know that the wiring, plumbing, heating, etc. are some of the
modes? Unless the regularities are special, such as non-accidental
alignments (Lowe, 1985), we need the information from many cities to
identify their modes. Hence a mode is like an object – we know one when
we see it, but have no definition, only a general description. This
observation leads us to conclude that Modal descriptions of the world about
us lie in a domain that is semi-independent of Science. Hence we have two
domains of study: Modes and Scientific inquiry.

79
 Underlying our dilemma of describing the character of our world is that
regularities are endemic and ubiquitous. They are so common that they are
almost invisible – a bit like our inability to see dark matter, even though it
occupies more space than visible objects (Randall, 2015). We accept them
but do not make explicit their presence. Yet once described, most will
consider them an important part of our world.

Language is clearly a mode of behavior. In America we speak one

language (English), whereas in Japan we have another language serving a
similar purpose. Neither convention will be easily overturned. However,
these regularities can not be modeled; perhaps the best attempt was by
Chomsky, who realized that there may be a “universal grammar” that
underlies all languages. But this grammar is an obscure regularity, whereas
the languages are not. To complicate this regularity, we note that verbs can
come after the subject, or before, or perhaps be embedded elsewhere in our
utterances. In a similar vein, the Brain coordinates aspects of out biotools,
selecting an appropriate series of actions from many degrees of freedom of
movements. Modeling the selection is difficult. Yet we tend to use the same
(or similar) set for different tasks. This is a regularity chosen by our Brain,
probably one that is very efficient.

It is discouraging that Modal Worlds are difficult to model. Or at least

that our scientific methods do not apply. However, if we step back and ask
what a physical model is good for, then there is a partial answer. The
physical model serves as a framework for something we build. Each element
of the model has a counterpart in the real world, and ideally there are no
extra variables (Einstein et al., 1935). Hence what is built according the
model should work and gain leverage in advancing our activities. This
approach is especially important for a prostheses: if an artificial heart can
pump blood in the same way as our own biological devices, this artifact can
be substituted for the original device. We can not do this for Modal
constraints. This also means that predictions are very difficult. Who knows
what the next convention or procedure will be that increases our fitness ?
Yes, we do have a measure for improvements (increased fitness), but we can
not easily estimate how the changes will affect our actions.

The situation is like describing the property of an object. We can

measure the current going through a wire, or the temperature of a device.
Each requires different units of measurement. However, in general, we are

80
unable to relate the temperature of one object to the charge of another,
although we can describe each of these properties in detail. We have results,
but no common denominator. It is like placing the kitchen at the rear of a
house (away from the front entrance), and trying to relate this to the design
of a hammer. Each has quite different constraints and properties. Hence, at
the moment at least, we should accept the curves in Fig. 9 as good
descriptors of their modal characters, and not try to relate these to each other
or to analogous curves for insect behaviors or other modal events.

The aim of scientific exploration is to offer an explanation for an event, or

a collection of events. The motion of the planets, or of the electrons in the
Bohr atom provided adequate models of two important scientific
observations in the early 20 century. But these models or theories became
th

overturned as more data were gathered. This is the history of science: our
models are continually being replaced by “more accurate” models that can
predict more events. A recent example is the impact of quantum
considerations at both the microscopic and cosmic scales (Hawking &
Mlodenov, 2012). This process will continue: our models and theories will
continue to change as we move forward (Quine, 1969). Our predictions
about events in their context will improve. Thus, when we build a device
according to the “latest theory”, that prediction will be better. Not so for a
mode.

Science is therefore an evolving enterprise. There is a glimpse of the

future. Theories improve or are revised and in concert so do the machines or
queries we create. In contrast, Modal Worlds change at a much slower pace.
Typically they are not available to “hands on” manipulation. However, they
are there, often the product of the latest science and engineering insights. We
can rely on them. But the picture they paint of the world is one with many
strands of regularities or procedures, with these regularities linked together
in an obscure manner. It is like we are viewing a loose ball of colored twine,
with each strand another regularity. In some sense this picture is more
digestible that the one offered by Science. Moving from molecular biology
to molecules to chemistry to physics, etc. is a recipe, but not one with a
coherence or sense of purpose that somehow emerges from modal insights.

Theories can be overturned with one blow, in a very short time interval.
Modes in contrast may last for centuries, often slowly evolving into a
sharper mode such as done with the keyboard, printers, books, and

81
especially dwellings, some of which with designs that lasted for millennia.
This can not be said of astronomy, where a flat earth was replaced by a
spheroid, or the trajectory of the sun about earth was reconsidered. Hence
Modes are relatively stable, unlike scientific discoveries, being free to
solidify more or to fade away. Growth is often by the merger of two modes,
such as the automobile or RV. Hence many future modes will retain some
of the properties of modes today, with today’s modes becoming part of
future trends. But we have little idea what these new modes will look like.
Our guess is that most will likely be associated with increasing brain power.
Here, science and engineering again may play an important role. Under these
constraints, evolution will proceed gracefully.

82
 References

Alexander, C. S. Ishikawa, M. Silverman, M. Jacobson, I. Fiksdahl-King, S,

Angel (1977) A Pattern Language Oxford Univ, Press.
Alexander, R. McN (1983) Animal Mechanics, Blackwell, Oxford
Bateson, G. (1979) Mind and Nature: A Necessary Unity,
Bobick, A. & W. Richards (1986) Classifying objects from visual
information. MIT AI Memo 879
Brachman, R., H. Levesque, R. Reiter (1992) Principles of Knowledge
Representation and Reasoning. MIT Press, 1992
Burton, R. F. (1998) Biology by Numbers, Cambridge Univ. Press.
Carrel, A. (1935) Man, The Unknown. New York and London: Harper
Conant, R. C. & W. R. Ashby (1970) Every good regulator must be a model
for the system. Int. Jrl. System Sci.
Cowan, H. J.(1958) Time and its measurement, World, Cleveland
Darwin, C. R.. (1859) Origin of species by means of Natural Selection,
Murray, London 2, 89-97.
Deng W., F. H. Gage (2015) The effect of immature adult-born dentate
granular cell of hyponeophagial behavior is related to their roles in
learning and memory. Front Syst Neurosci. 9, 34. Doi
10.3389/fnsys.2015.00034. eCollection 2015. PubMed PMID: 25798094.
Einstein, A. , B. Podolsky, & N. Rosen (1935) Can a quantum mechanical
model be complete ? Phys. Rev.
Fechner, G. (1860) Elemente der Psychophysik (1860). 2 volumes.
Breitkopf und Härte
Feldman, J. (2014) Bayesian models of Perceptual Organization
Gal, Y, R. Kasturirangen, A. Pfeffer, W. Richards (2008) How tacit
knowledge guides action. Workshop on Naturally Inspired Intelligence,
AAAI 2008 Fall Symposium, Arlington, VA.
Galton, F. (1869) Hereditary Genius
Garland, K. (2994) Mr Beck's Underground Map. Harrow Weald,
Middlesex: Capital Transport, 1994.
Gilbert, S. (1997) Mapping mental spaces. PhD dissertation Dept. Brain &
Cognitive Sci. http://nimrod.mit.edu/depts/document_services/docs.html
Gould, S. J. & N. Eldrege (1977) Punctuated equilibria: the tempo and
mode of evolution reconsidered, Paleobiology 3, 115-191.
Gunkel, P. M. (2000) Ideonomy. Works by Gunkel presented on web site
Hawking, S. & L. Mlodinow (2012) The Grand Design, Bantam Books
Haskall, J. M. Ritchie & H. Off (2002) Fractal geometry predicts varying
body size scaling relationships for mammal and bird home ranges, Letter

83
 to Nature
Heidigger, M. (1951) What calls for thinking? (Was heisst Denken?) Harper
& Row 1958 (English)
Held, R. & J. Bauer (1975) Comparison of visually guided reaching
behavior in normal and deprived infant monkeys: Animal Beh. Proc. Jrl.
Exp. Psychol. 1, 298-308
Held, R. & A. Hein ((1963) Movement-produced stimulation in the
development of visually guided behavior. Jrl. of Comp. and Physiol.
Psychology 56(5): 872-876.
Hoffman, D. D. (2009) The interface Theory of Perception. In S. Dickenson,
M. Tarr, A. Leonardis, & B. Schiele (Eds) Object Categorization
Computer and human vision perspectives (pp. 148-165) Cambridge Univ.
Press, NY
Hölldobler, B. & E.O. Wilson (1990) The Ants, Harvard Univ.Press
Hogan, N. (1988) Physical systems theory In Natural Computation, W.
Richards (ed) MIT Press
Huxley, J. S. (1932) Problems of Relative Growth, Metheun, Lond.
Jacobs, J. (1961) Death & Life of American cities. Random House
Kahl, W, J. Carette, J. Xiaoheng (2006) Bimonadic semantics for basic
pattern matching caculi, MPC'06 Proceedings of the 8th international
conference on Mathematics of Program Construction
Kemp, C. , A. Perfors, & J. Tenenbaum (2006) Learning overhypotheses
with hierarchical Baysian models. Developmental Science, 10, 307-321
Kleiber, M. (1932) Body size and metabolic rate. Physiol. Rev. 27, 511-541
Krantz, D.; D. Luce, P. Suppes, A. Tversky (1971) Foundations of
Measurement; Academic Press
Kohler, I. (1964) The formation and transformation of the perceptual world.
Psychol. Issues 3, 1-173.
Koenderink, Jan ( 2015) Ontology of the Mirror World Gestalt Theory, 119-
140.
Landry, B., R. Deits, P. R. Florence, and Russ Tedrake.(2015) Aggressive
quadrotor flight through cluttered environments using mixed integer
programming. In Under review.
Lamarck, J. B. (1809) Philosophie, Zoologique, Paris. (see Packard)
Lewontin, R. C. (1966) Is nature probable or capricious? Biosci. Univ.
Calif. Press; (1978) Adaptation, Sci. Amer. 239, 212-228
LeCorbusier (1948) The Modulator (see Packard)
Lowe, D. G. (1985), Perceptual Organization and Visual Recognition,
Kluwer Academic Publishers, Boston, Mass.
Lynch, K. (1977) The Image of the City, MIT Press, Cambridge, MA.

84
Malthus, T. (1798) Principle of Population
Mandelbrot, B. (1983) Fractral geometry of nature. Freeman, San Francisco
Marr, D. C. (1974) Simple memory: Theory of Archicortex, Phil. Trans.
Royal Soc. Lond. 262, 23-81
McKibben, W. (1989) The End of Nature, Anchor Press
McMahon, T. A. (1973) Size and Shape in Biology, Science, 179, 1201-
1204.
McMahon, T. A. & R. E. Kronauer (1976) Tree structures: deducing the
principles of mechanical design, Jr. Theor. Biology, 59, 443-466.
Norman, D. (1993) Things that make us smart. Addison Wesley Press.
Norman, D. (2013 ) Design of Everyday Things, Basic Books
Odling-Smee, F. J., K. N. Leland, and M. Feldman (1996) Amer. Naturalist,
147, 641-648
Packard, A. (1901) Lamark:founder of evolution, Longman, Green, NY
Paynter, H. M. (1961) The gestation and birth of bond graphs (c. 2000)
Pallasmaa, Juhan ( 2002) Animal builders: ecological functionalism of the
animal World. In Animal Architecture; Museum of Finnish architecture
Pedley, T. J. (1977) Scale effects in Animal locomotion (Ed.) Academic
Press, Lond. 201
Pinker, S. (1994) The Language Instinct, Morrow.
Quine, W. V. O. (1969) Epistemology Naturalized.
Randall, L. (2015) Dark matter and the dinosaurs, Harper Collins
Reiss, M.J. (1989) The Allometry of Growth and Reproduction Cambridge
Univ. Press.,Cambridge
Richards, w. (1988) Natural Computation (Ed), MIT Press
Richards, W. (2015) Anigrafs: experiments in cooperative cognitive
architectures. MIT Press.
Richards, W. & A. Bobick (1986) How to play 20 questions with Nature
and Win. MIT Artificial Intellignece (CSAIL) Pub. No. 186
Richards, W. & J. J. Koenderink ( 1995). Trajectory Mapping: a new
nonmetric scaling technique. Pereption, 24, 1315 – 1331.
Salisbury, K. N. (1988) Articulated hands. In Natural Computation, W.
Richards (Ed.)
Salakhutinov, R., J. Tenenbaum and A. Torralba (2012) One-shot learning
with a hierarchical nonparametric Bayesian model
Shepard, R. (1962) The analysis of proximities: Multidimensional scaling
with an unknown distance function. Psychometrika, 27, 125 – 140.
Spenser, H. (1860) System of synthetic philosophy, Lond.
Stevens, S. S. (1946). On the theory of scales of measurement. Science 103
(2684): 677–680.

85
Stiny, G. (1980) Introduction to shape and shape grammars. Environment
and Planning B: Planning and Design 7(3), 343-351.
Stratton, G. (1897) Vision without inversion of the retinal image.
Psychological Review, 4, 341-360; 463-481.
Tedrake, R., S. Kuindersma, R. Deits, and K. Miura. (2015) A closed-form
solution for real-time ZMP gait generation and feedback stabilization. In
Proceedings of the International Conference on Humanoid Robotics,
Seoul, Korea.
Tinbergen, N. (1951) A theory of instinct
Thompson, D’Arcy ( 1919) On Growth and Form. Cambridge Univ. Press
Valiant, L. G. (2009). Evolvability. J. ACM 56, 1, Article 3 (Jan 2009), 21
pages. DOI=10.1145/1462153.1462156 http://doi.acm.org/10.1145/
1462153.1462156
Valiant, L. G. (2013) Probably Approximately Correct, Basic Books.
Von Uexkill, J. (1909) Umwelt und Innenwelt der Tiere, Springer-Verlag,
Berlin
Von Uexkill, J. (1926) Theoretical Biology. Harcourt, Brace, NY
Wallace, A. R. (1860) Considerations in the theory of natural selection.
Marchant & Co.
Werfel, J. , K. Peterson, and R. Nagpal (2014)Designing collective behavior
in a termite-inspired robot construction team. Science 343(6172): 754-
758
Wilson, E. O. (1998) Consilience: the unity of knowledge, Knopf,NY.
Wilson, E. O. (1971) Insect Societies, Harvard Univ. Press., Cambridge
Wolfram, S, (2002) A new kind of science, pp383, Wolfram Media,
Champaign, Il.

86
87
 Appendix

Although man-made regularities are the most obvious, Nature also

exhibits distinctive types of structures. They can be used for object
classification such as the textures of animals as contrasted to their form, or
the partitioning of natural objects, such as flowers.

13. Categorization of Animals

Different animals have many different features. The same holds for
flowers, trees and other plants. It is these regularities that allow us to
distinguish species types in finer detail. A dog has teeth, paws, claws, and
certain facial features. These differ from those of a pig, for example. The pig
has no body hair; its tail is distinctive, so is its snout, and the sound that it
makes. In each case we have a set of correlated features that we use to
categorize these animals. These correlated regularities constitute a mode that
for us “defines” the pig, or dog, or elephant, etc. Hence animals (and plants)
can be classified by the regularities of their submodes, as well of course by
the features that define their animal class, such as flight ability (wings),
swimming (fins), legs (terrestrial), etc. If we note some correlated features of
a set of objects, we tend to classify them together, just as we do here for
animals.

Imagine we see a group of animals that look different. We then

differentiate among them, placing each into an appropriate category. To
accomplish this we analyze a set of features, and make attributions to the
animals. Our list may look something like this:

body_hair tail nose snout pointed_ears flat_ears four_legs claws bark

grunt panting wag_tail meow sniff griwl paw independent growl….

With such a list we can now check off those features that apply to the animal
(which we observe) and leave blank those that do not. Our list can easily
grow to 100 features just for animals. So it is clear that we may have up to
2^100 categories. But with such a list we will NOT necessarily have a mode.
Rather what we need to do is to show that a half-dozen or so of the features
are correlated and do not appear in any of the other animals. For example, if

88
we hear an “oink”, should we infer that the animal is a pig? Rather, a better
strategy is to note that this is one of at least four or more features that are
always correlated. Then our search is truncated and we have evidence for an
animal type. Obviously, observers may differ as to their choices. However if
their regularities are modal, then the same type of animal will be inferred
even though their feature lists may differ.

As an example, consider we know in advance there are three types of

animals: dogs, cats, and pigs in a large group. We see one type that has no
fur. This implies a “pig”, but we need to have a couple of other regularities
correlated with this feature. For example, we may note that these furless
animals “grunt”, they have a snout, and only a little curled tail. (We could
pick other features, but only a few will be needed here.) Now it is easy to tell
a pig from a dog or cat, but more difficult to distinguish between cats and
dogs. Hence we have to continue to regard our list of features to make this
distinction, at the same time making sure that the regularity we pick is not
exhibited by pigs. Appropriate choices could be the sounds the animals
make (growl, meow, bark), and their facial features. Dogs have long jaws
ending at a distinctive nose, whereas cats and pigs do not; they have tails
which they wag, which cats do not, they prefer groups and are playful; they
do eat scraps of food, but these are different for cats and especially for pigs.
In such a manner we will quickly find features that exhibit regularities for
the animal type.

However, once we have obtained satisfactory classifications, our

“solution” will not be unique. Given that we may want four regularities in
order to make a classification, clearly there are many sets of “four”. And if
only three regularities are sought, there will be still more “solutions”.
Observers will thus most likely classify animal types in different ways, but
still hit upon the same modes.

Similarly, although a large city is clearly modal, being subjected to many

constraints and codes, cities may also be differentiated. This differentiation
can occur simply based on the design of the city, without resorting to
possible difference in dress codes. For example, Venice is noted for its
canals, whereas London with its old style buildings and roads with left
driving, is quite different. Similarly Kyoto has its own distinctive character,
as does New York city. They are all identified by their submodal characters.
The rectangular street pattern of New York is not seen in London, nor in
Venice. We have a different set of features associated with each. This

89
assortment is still modal, however, but perhaps not quite as obvious as the
features that distinguish New York from London, or Amsterdam from
Venice. Thus our world may be further compartmentalized by considering
the features of a mode. Here, perhaps animal submodes are the clearest.

The importance of categories was noted by Marr (1974), in his paper on

“Theory of Archicortex”. However the details were not flushed out until
Bobick and Richards (1986) explored an artificial domain of flowers, which
they called “polyflowers”. Chapt. 14 shows the strategy in detail. All
flowers in this domain can be created by a simple logo program. The
possible list of features were those chosen by classroom participants, who
were asked to partition nine of these polyflowers into categories. So the
flowers are analogous to the modes of Animals (or city), and the submode
will be the categorizations or partitioning of the set. Now, however, the
observers do not know how many categories of flowers are shown. For the
nine polyflowers exhibited, there can be a many as 21000 partitions. We find
only three, with the requirement that the choices be modal (ie a correlation
among types of regularities). So the analysis is not of the features directly,
but rather of the regularities of the features, and whether another feature also
exhibits the same regularity. This strategy allows us to search for only a
small number of features, rather than explore over one million.

The categorization of polyflowers is interesting because it relates to one

kind of learning. Imagine I show you a line segment oriented on the page at
62 deg. If you are than asked to make another copy, you will likely duplicate
the segment (some, however, simply draw another line, perhaps at a
different orientation.) If we now show two examples, the “modal” aspects
click in and the subject will draw a third example identical to the first two.
The task is similar to twenty questions, in that the subject infers the intent.
One quickly requires a modal mind-set. A version of this task has been
studied by Valiant (2013) in a PAC learning framework (probably
approximately correct.) Every instance of a repeated token further
strengthens the conclusion that the target is a line segment of a certain
length, oriented at 62 deg. Children often use this technique to extrapolate
their knowledge. Although not rote learning as taught in schools, this type of
learning is very fast (“one-shot”) and useful (Feldman, 2014; Kemp et al.
2014). But it can apply only in a world that is modal.

We have described three distinct modes: cities, animals, and animal

types. We may think of these as modes, submodes, and sub-submodes –

90
namely three levels of analysis. There are many others, but the reader should
now see that digging deeper into each category will reveal yet other modes,
as well as moving laterally at the same scale to another domain. The world is
thus a collection of modes, with each modal cut quite distinctive.

To glimpse other domains, recall the insects, or sea creatures, or airborn

animals. We have a host of other submodes here, many at a scale quite
different from ours. For clarity, however, we have focused on modes at a
human scale, or those created by us at a larger scale, such as cities. But if
we consider the total scope of science, then we realize that there are modes
at galactic scales, as well as those of a quantum nature, or of dark matter.
These, however, are not as accessible to us, and hence we do not
immediately see their impact on a daily basis.

14. Polyflowers
(adapted from Bobick & Richards)

In Chapter 13, a set of animal modes was further subdivided into

categories using sets of correlated features typical of one type of animal,
such as dog, pig, elephant, etc. but not others. The scheme showed that
modes could occur at many scales as one searched for more refinements.
Here we illustrate this construction using the domain of artificial flowers.
The construction has three variables of a Logo program. These are: (1)
segment length, (2) turning angle at one end to forma petal; and (3) the
number of turn or cycles around the circular center. The latter is set by the
division of the petal angle into 2pi (this excludes irrational divisors of 360
deg.)
Hence there are really only two variables, since we require that the
segments return to the starting location. The constructions will show how a
simple generative function can lead to many “features” (i.e. observation
measurements) that are highly correlated. The approach is reminiscent

91
 37

Fig 29. Construction of polyflower blossoms.

of Wolfram’s (2002) point that simple processes can lead to complex,

correlated properties. Fig. 29 illustrates a simple logo program that captures
the essence of a flower blossom. The petals are outlines, then transformed

Fig. 30. A set of polyflowers. The groupings tend to be ACE, FJH, and BDG

92
into a symmetric pattern (lower right.) The actual objects used in the
study are shown in Fig. 30. The groupings are those seen most frequently by
the subjects.

The subjects also reported what features they used in their classification.
These shouldFeatures subjects used for classification (partial list)
be regarded as observation “measurements”. When we look at
a dog, the shape of the head might be used for classification, or the hair
large
Features: star braid poly rough pointy intersect smooth >90 turns In-Cir concave
Polygon
Star
Braid ?

Fig 31. Assignment of “features” to the three categories.

length and color. Size may make a difference, but almost everyone ignored
this variable.
41

Because the logo has only two parameters, many of the features in the
tables must be correlated: given one feature, the likelihood of some others is
Questions:
high. For example,Whytheresoare no lines intersecting for the polygons. There is
few categories ?
also only one cycle of line segments around the clock (not in table 31),
unlike the other twoWhy is everyone
groups. so consistent lines”
Hence “intersecting ? (or not) can be used
to make a cut in the classes. Also, we know that intersecting lines must be
…..given 21K possible ways of choosing partitions
correlated with polyflower edges that need to complete more than one
(modulo) 360 deg. cycle before returning to the start point. When the line
Answer:there
segments intersect, Generating Functionthat
will be “stars” (two
havekeypointed
parameters)
petals (turn angle
< 90 deg.) Whereas polygons and braid
….. hence many highlyhave turnfeatures
correlated angles(~modal
greaterrelations)
than 90 deg.

! = turn angle
!
N = number of turns around clock

#vertices = f (2N"/!)
42

Fig 32. Logo program used to generate polyflowers.

93
This is another possible classification rule, which also is correlated with
three other different features as shown in table 31. For example, this rule
affects the size of the inner inscribed circle (eg small vs large) which some
observers used as a classification feature. Also related is the textured quality
of the outer boundary of the shape.

A peek at the logo generating algorithm makes these correlations clearer

(Fig 29). As mentioned, there are only three parameters: line segment length
which sets the size (not labeled), the number of turns about the clock, and
the turn angle before drawing another line segment of the same length. Note
that the length of the line segment was ignored by almost all observers. In
other words, the observers inferred from the variety of sizes and shapes, that
size was an irrelevant parameter (See Feldman, 2014 for explanation.)
Hence there were only two relevant parameters, N and a.

We can cast the possibilities in a table of number of vertices (columns) vs

number of cycles (rows) (See Fig. 33). For example, to return to the start
position after one cycle, the top row (red) specifies the interior angle
required (pi-alpha). This lists all the polygons. For return after two clock
cycles, the second row shows the possible polyflowers, with the row one
duplicated crossed out.

Looking at subsequent rows, we see interior angles greater than 90 deg,

and some less. However, note that along the bottom diagonal, all interior
angles are less than 45 deg the small angles. Obviously these

Essence of Generating Process

(abbreviated table of interior angles)

Fig 33. Table showing turn angles for all possible closed polyflowers.

94
43
polyflowers will be very sharp, pointed stars. (The sampling used for
classification was one diagonal element above the minimum.) Intermediate,
we can find a crude diagonal set of entries from the table with interior angles
>90 deg, as shown in green. These are the “braids”. Hence the generative
algorithm and selection process is very simple, and ”orderly.” The surprising
result is that the observers “recognized” these hidden regularities, even
though their choice of features differed, inferring the commonality between
members of teach of the three classes. Obviously, the fact that more than
one feature will” support” a given class category adds evidence to the
inference process. The correlations among different measurements that arise
from an underlying generative process lies at the heart of Modal Worlds.
Furthermore, this modal inference implies that the observation of one
regularity along a measurement dimension will suggest another regularity
seen along another dimension.

95
 end of

Modal Worlds

Whitman Richards†
a
D E C LOOTCRANS P RESS

b
OTHER E B OOKS FROM T HE C LOOTCRANS P RESS :

1. Awareness (2012)
2. MultipleWorlds (2012)
3. ChronoGeometry (2012)
4. Graph Spaces (2012)
5. Pictorial Shape (2012)
6. Shadows of Shape (2012)
7. Through the Looking Glass: on Viewing Aids (2012)
8. Painting to Marble (2012)
9. Experimental Phenomenology: Art & Science (2012)
10. The Spirit of the New Style (2013)
11. World, Environment, Umwelt and Innerworld (2013).
12. Part & Whole (2013)
13. Skin Deep Only (2015)
14. Shape Tutorial (2016)
15. Points (2017)
16. The Way of the Eye (2018)

(Available for download here.)

A BOUT T HE C LOOTCRANS P RESS

The Clootcrans Press is a selfpublishing initiative of Jan Koenderink. Notice

that the publisher takes no responsibility for the contents, except that he gave it
an honest try—as he always does. Since his books are free you should have no
reason to complain.

T HE “C LOOTCRANS ” appears on the front page of Simon Stevin’s (Brugge,

1548–1620, Den Haag) De Beghinselen der Weeghconst, published 1586 at
Christoffel Plantijn’s Press at Leyden in one volume with De Weeghdaet, De
Beghinselen des Waterwichts, and a Anhang. In 1605 there appeared a supple-
ment Byvough der Weeghconst in the Wisconstige Gedachtenissen. The text reads
“Wonder en is gheen wonder”. The figure gives an intuitive “eye measure” proof
of the parallelogram of forces.

c
 The key argument is

de cloten sullen uyt haer selven een eeuwich roersel maken, t’welck
valsch is.

Simon Stevin was a Dutch genius, not only a mathematician, but also an en-
gineer with remarkable horse sense. I consider his “clootcrans bewijs” one of the
jewels of sixteenth century science. It is “natural philosophy” at its best.

T RAIECTUM is the city of Utrecht in the Netherlands where I live. Traiectum

was a castrum on the frontier of the Roman Empire in Germania Inferior.
The fortress was built of wood, it measured a little over a hundred by a hundred
meters and held five hundred troops, a cohors quingenaria — the II Hispanorum
peditata, a standing non-citizen infantry corps of the Imperial Roman army.
The region was designated frontier by the Emperor Claudius. It must have
been much like what you see in the familiar fifties western movies. The castellum
was destroyed by the Franks before 270, although there remained Roman presence
up to the fourth century.
The fort’s exact location is the current Utrecht city centre, where I live close.
My Alma mater takes her name Rheno–Traiectina or Universitas Ultraiectina
from this.

d
e
f