Annali di Matematica (2016) 195:935–956

DOI 10.1007/s10231-015-0498-7

Hamiltonian elliptic systems in R2 with subcritical

and critical exponential growth

Manassés de Souza1 · João Marcos do Ó1

Received: 24 November 2014 / Accepted: 16 April 2015 / Published online: 28 April 2015
© Fondazione Annali di Matematica Pura ed Applicata and Springer-Verlag Berlin Heidelberg 2015

Abstract Our aim in this paper was to establish by a variational method the existence of
nontrivial solutions for strongly coupled Hamiltonian systems of the form

−u + V (x)u = g(x, v), v > 0 in R2 ,
−v + V (x)v = f (x, u), u > 0 in R2 ,

when the potential V is neither bounded away from zero, nor bounded from above. The
nonlinear terms g(x, s) and f (x, s) are superlinear at infinity and have exponential subcritical
or critical growth of the Trudinger–Moser type. Typical features of this class of problems
are the lack of compactness because of the unboundedness of the domain and the critical
growth. Moreover, the Lagrangian functional associated with this class of systems is strongly
indefinite, that is, it has a saddle-point geometry where both positive and negative subspaces
of the quadratic form are infinite-dimensional. To overcome these difficulties, we use an
inequality of Trudinger–Moser type combined with Galerkin methods and a linking theorem.

Keywords Variational methods · Elliptic systems · Trudinger–Moser inequality ·

critical exponents

Mathematics Subject Classification Primary 35J50 · 35J55; Secondary 35Q55

1 Introduction

Our aim in this paper was to establish the existence of nontrivial solutions for the following
class of elliptic system

Research was partially supported by the National Institute of Science and Technology of Mathematics
INCT-Mat, CAPES and CNPq/Brazil.

B Manassés de Souza
manassesxavier@hotmail.com
1 Departamento de Matemática, Universidade Federal da Paraíba, João Pessoa, PB 58051-900, Brazil

123
936 M. de Souza, J. M. do Ó


−u + V (x)u = g(x, v), v>0 in R2 ,
(S)
−v + V (x)v = f (x, u), u>0 in R2 .

Here u, v : R2 → R are unknowns function, while V : R2 → R is given potential, which

may change sign and vanish. The nonlinear terms g, f : R2 × R → R are given measur-
able functions satisfying suitable assumptions to apply Trudinger–Moser type inequality
in combination linking theorems. Solutions of (S) can be used to represent a standing
wave to the time-dependent nonlinear Schrödinger equation. They also arise when look-
ing for stationary solutions to other classes of physical problems, such as cross-diffusion
and Klein–Gordon equations (for a discussion, see for example [2,21,24] and references
therein).
Here we impose certain conditions on the potential and the nonlinearities, which allow
us to study this class of elliptic systems by using a variational formulation, in the sense that
weak solutions of (S) are critical points of the associated energy functional

I (u, v) = u, v − J (u, v), (1.1)

where 
u, v = (∇u∇v + V (x)uv) dx (1.2)
R2

and 
J (u, v) = (F(x, u) + G(x, v)) dx (1.3)
R2

and the functions F and G are the primitives of f and g. The first difficulty that we face is
that the quadratic part of I , ·, · is strongly indefinite in the sense that the origin (0, 0) is a
saddle point for ·, · having infinite Morse index. We also stress that in this work, we are
interested in the limiting case of Sobolev embedding, which for this case corresponds when
nonlinearities g(x, s) and f (x, s) are in the critical growth range with maximal growth in
s, which allows us to treat problem (S) variationally in a suitable function space by using
the functional I . Moreover, in this work, we are interested in establishing the existence of
solutions of (S) when g(x, s), f (x, s) do not have to be bounded in variable x, but must have
the growth in x controlled by V (x) at infinity, and the potential V : R2 → R may change
sign. Because of this and the fact that the equations in (S) are defined in the whole R2 , we
face two more difficulties, which are the lack of compactness for the associated Palais–Smale
sequences and the second term of I ; functional J is not weakly continuous. Hence, with this
geometry and the lack of compactness, it is not clear for us how to apply the generalized
linking theorem and its variants to obtain the existence of nontrivial critical points of I . To
deal with these difficulties, we use a Galerking method. It is worthwhile mentioning that
such approximation-type argument to study a class of problems was first used by Rabinowitz
[18,19] in the famous articles.
In order to apply variational methods, we consider a suitable subspace of the Sobolev
space H 1 (R2 ), namely
  
HV1 = u ∈ H 1 (R2 ) : V (x)u 2 dx < ∞ .
R2

Under the conditions on the potential:

(V1 ) There exist B > 0 such that V (x) 2≥ −B, for2all

 x ∈R ;
2

(V2 ) λ1 := inf {u∈H 1 : u 2 =1} R2 |∇u| + V (x)u dx > 0;

V

123
Hamiltonian elliptic systems in R2 with subcritical… 937

we can see that HV1 is a Hilbert space when endowed with the scalar product

u, v = (∇u∇v + V (x)uv) dx.
R2

Moreover, HV1 is continuously embedded into H 1 (R2 ) (see [23, Lemma 2.1] for details).
We use the following notation: if  ⊂ R2 is open and s ≥ 2, we set
  
 |∇u| + V (x)u
2 2 dx
νs () = inf  2/s ,
 |u| dx
u∈H01 ()\{0} s

and we put νs (∅) = ∞. In order to obtain a compactness result, we shall consider the
following assumptions:
∞ (R2 ), with A(x) ≥ 1, and constants μ > 1,
(V3 ) There exist a function A(x) ∈ L loc
c0 , R0 > 0 such that
  1/μ
A(x) ≤ c0 1 + V + (x) ,

for all |x| ≥ R0 , where V + (x) = max x∈R2 {0, V (x)};

(V4 ) lim R→∞ νs (R2 \B R ) = ∞ for some s ∈ [2, +∞) or for any r > 0 and any sequence
(xk ) ⊂ Rn , which goes to infinity, for some s ∈ [2, +∞), limk→∞ νs (B(xk , r )) = ∞.

It is also well known that assumptions (V3 ) − (V4 ) imply that embeddings of HV1 into L s (R2 )
are compact for all s ∈ [2, +∞) (see [23, Proposition 3.1]).
We now state our main assumptions on g(x, s) and f (x, s) to apply minimax procedure.

(H1 ) g, f : R2 × R → R are continuous and g(x, s) = f (x, s) = 0 for all x ∈ R2 and

s ≤ 0;
(H2 ) g(x, s)/A(x) = o(|s|) and f (x, s)/A(x) = o(|s|) at the origin and uniformly on
x ∈ R2 ;
(H3 ) g(x, s) and f (x, s) are locally bounded in s, that is, for any bounded interval J ⊂ R,
there exists C > 0 such that

|g(x, s)| ≤ C and | f (x, s)| ≤ C for all (x, s) ∈ R2 × J.

(H4 ) there is a number θ > 2 such that

⎧  s
⎪
⎪
⎨ 0 < θ F(x, s) := θ f (x, t) dt ≤ s f (x, s)
0 s for all (x, s) ∈ R2 × (0, +∞).
⎪
⎪
⎩ 0 < θ G(x, s) := θ g(x, t) dt ≤ sg(x, s)
0

(H5 ) there exist constants M > 0 and s1 > 0 such that


0 < F(x, s) ≤ M f (x, s)
for all (x, s) ∈ R2 × [s1 , +∞).
0 < G(x, s) ≤ Mg(x, s)

We are interested in the case that the nonlinearities g(x, s) and f (x, s) are of maximal
growth, which allows us to treat problem (S) variationally in a suitable function space. Pre-
cisely, we shall treat the subcritical and critical growth in the Trudinger–Moser sense, which

123
938 M. de Souza, J. M. do Ó

we define as follows: We say g(x, s) ( f (x, s) resp.) has subcritical growth if for all α > 0
(β > 0 resp.), uniformly in x, we have
⎧
⎪ g(x, s)
⎪
⎨ s→+∞
lim =0
A(x)eαs
2

⎪
⎪ f (x, s)
⎩ lim =0
s→+∞ A(x)eβs 2

and has critical growth if there exists critical exponent α0 > 0 (β0 > 0 resp.) such that,
uniformly in x,
⎧ 
⎪
⎪ g(x, s) 0, for all α > α0 ;
⎪
⎪ lim =
⎨ s→+∞ A(x)eαs 2 +∞, for all α < α0 ;

⎪
⎪ f (x, s) 0, for all β > β0 ;
⎪
⎪ lim =
⎩ βs
s→+∞ A(x)e 2
+∞, for all β < β0 .

This notion of criticality is motivated by Trudinger–Moser inequality (see [17,25] and pioneer
works of Adimurthi [1] and de Figueiredo et al. [6]).
We denote the product space E = HV1 × HV1 endowed with the inner product

(u, v), (ϕ, ψ) E = [∇u∇ϕ + V (x)uϕ + ∇v∇ψ + V (x)vψ]dx (1.4)
R2
1/2
for all (u, v), (ϕ, ψ) ∈ E, to which corresponds the norm (u, v) E = (u, v), (u, v) E .
Here, we search weak solutions of (S), that is, functions (u, v) ∈ E such that
  
[∇u∇ψ + V (x)uψ + ∇v∇ϕ + V (x)vϕ]dx − f (x, u)ϕ dx − g(x, v)ψ dx = 0
R2 R2 R2

for all (ψ, ϕ) ∈ E. Notice that weak solution of (S) turns out to be critical points of the
energy functional
  
I (u, v) = [∇u∇v + V (x)uv]dx − F(x, u)dx − G(x, v)dx.
R2 R2 R2

For more details, see Sect. 3.

1.1 Statement of main results

Next, we state our existence results.

Theorem 1.1 If g(x, s) has exponential subcritical growth, f (x, s) has exponential subcrit-
ical growth or critical growth, (V1 ) − (V4 ) and (H1 ) − (H4 ) are satisfied. Then (S) possesses
a nontrivial weak solution (u, v) ∈ E.

Theorem 1.2 If g(x, s) and f (x, s) have exponential critical growth, (V1 )−(V4 ) and (H1 )−
(H5 ) are satisfied. In addition, we assume that there exist p1 , p2 > 2 and λ > 0 such that

g(x, s) ≥ λs p1 −1 , f (x, s) ≥ λs p2 −1 for all (x, s) ∈ R2 × [0, +∞). (1.5)

Then, there exists λ0 > 0 such that (S) possesses a nontrivial weak solution (u, v) ∈ E for
all λ > λ0 .

123
Hamiltonian elliptic systems in R2 with subcritical… 939

Remark 1.3 1. Our results improve the main results in [4,7,9,12,27] because our hypothe-
ses allow the case of different critical nonlinearities, that is, g(x, s) ∼ eα0 s and
2

β 2
f (x, s) ∼ e 0 at infinity, where α0 and β0 might be different.
s

2. Our results also improve the main result in [9] in the sense that p1 and p2 in (1.5) may
be different.
3. We point out that there are some results considering V (x) a continuous function, which
is bounded below by a positive constant (see [3,9,10,12–14,27]). Motivated by Sirakov
[23], we extend this discussion to the case in which V (x) can change its sign.
4. Notice that, if g(x, s) has exponential critical growth, it would be enough to assume,
uniformly in x, that
g(x, s)
lim inf ≥ μ p1 > 0, (1.6)
s→0+ s p1 −1
instead of (1.5), because (H1 ) − (H4 ) and (1.6) imply (1.5). However, we prefer to
consider (1.5) to make ease the presentation of this paper and avoid certain technicalities.
5. We point out that the hypotheses (H1 ) − (H5 ) have been used in many papers, see for
instance [3,9,10].
6. We would like to mention that a suitable rescaling transforms the system in a new system
with nonlinearities with the same critical exponent; indeed, consider the model case

−u + V (x)u = v(eα0 v − 1),
2

−v + V (x)v = u(eβ0 u − 1),

2


and set u = αβ00 u. Then one gets
⎧ 
⎨ −u + V (x)u = β0 v(eα0 v 2 − 1),
 α0
⎩ −v + V (x)v = α0 u(eα0 u 2 − 1).
β0

Motivated by this fact and to improve the presentation of this paper, in the proof of our
results, we consider α0 = β0 .
7. As it is well known in dimension N > 2, the nonlinearities are required to have polyno-
mial growth at infinity, so that one can define associated functionals in Sobolev spaces.
Coming to dimension N = 2, much faster growth is allowed for the nonlinearity. In fact,
the Trudinger–Moser estimates in N = 2 replace the Sobolev embedding theorem used
in N > 2.
We point out that in dimension N > 2, Hamiltonian elliptic systems in bounded domains were
studied in [5,11,15] and reference therein. The problem in the whole space was considered
in [8,22,26].

The sequel of the paper is organized as follows: The next section contains some technical
results, which are crucial tools to prove our main theorems. Section 3 is devoted to the
variational framework and the geometry of the linking theorem. The proof that a Palais Smale
sequence is bounded is presented in Sect. 4. Furthermore, we prove some convergence results,
which will be used later to prove that a (P S) sequence converges weakly, up to a subsequence,
to a weak solution of (S). The finite-dimensional approximation for system (S) (a Galerkin
approximation procedure) is presented in Sect. 5. Section 6 contains the proof of existence
of solutions in the subcritical case for (S), Theorem 1.1. Estimates for the minimax level
are obtained, in the critical case, and is presented in Sect. 7. Finally, by using the estimate

123
940 M. de Souza, J. M. do Ó

obtained in the previous section and some convergence of the approximation is deduced in
Sect. 8 the existence of solution for (S) in the critical case, Theorem 1.2.

2 Some preliminary results

In this section, we state some technical results, which will be used in the proof of our main
results. First, we need of the following result which has been proved in [3]:
 
Lemma 2.1 If α > 0 and u ∈ H 1 R2 , then

(eαu − 1) dx < ∞.
2
(2.1)
R2

Moreover, if 0 < α < 4π, ∇u 2 ≤ 1 and u 2 ≤ T , then there is a positive constant

C = C(α, T ) such that 
(eαu − 1) dx ≤ C(α, T ).
2
(2.2)
R2

Let us consider the weighted Lebesgue space L sA(x) (R2 ) with the usual norm
 1/s
u L s (R2 ) = A(x)|u|s dx .
R2

In the next result, we collect some facts which have been proved in
[23, Lemma 2.1, Lemma 3.1 and Proposition 3.1].

Lemma 2.2 Suppose (V1 ) − (V4 ) hold. Then

(1) HV1 is continuously embedding into H 1 (R2 );
(2) HV1 is continuously embedding into L s (R2 ) for all s ≥ 2;
(3) HV1 is continuously embedding into L sA(x) (R2 ) for all s ≥ 2;
(4) HV1 is compactly embedded into L s (R2 ) for all s ≥ 2;
(5) HV1 is compactly embedded into L sA(x) (R2 ) for all s ≥ 2.

The Lemma 2.1 in [23] provides an inequality, which will be used throughout the paper:
More precisely, there exists ζ > 0 such that

u 2 ≥ ζ |∇u|2 dx, (2.3)
R2

for all u ∈ HV1 , where u = u, u1/2 .

Now, we prove a Trudinger–Moser inequality. This result will be essential for proof of
our results.

Lemma 2.3 Suppose that (V1 ) − (V3 ) are satisfied. Then for all u ∈ HV1 , q ≥ 2 and α > 0,

A(x)|u|q (eαu − 1) dx < ∞.
2
(2.4)
R2

Moreover, if αμ u 2 /(μ − 1) < 4πζ , there exists C > 0 such that


A(x)|u|q (eαu − 1) dx ≤ C u q .
2
(2.5)
R2

123
Hamiltonian elliptic systems in R2 with subcritical… 941

Proof First, we write

 
A(x)|u|q (eαu − 1) dx = A(x)|u|q (eαu − 1) dx
2 2
L(u) =
R2 |x|≤R0

A(x)|u|q (eαu − 1) dx.
2
+
|x|>R0

By (V3 ), we obtain
 
αu 2
[1+(V + (x))1/μ ]|u|q (eαu −1) dx
2
L(u) ≤ A L ∞ (B R0 ) |u| (e − 1) dx +c0
q
|x|≤R0 |x|>R0
 
V (x)1/μ |u|q (eαu − 1) dx + C |u|q (eαu − 1) dx.
2 2
≤ c0
[V >0] R2
(2.6)
Now our goal is to estimate each of these integrals. Consider s, s  > 1 such that 1/s+1/s  = 1
μ
and 1 < s  < μ−1 , then by Hölder inequality and Lemma 2.2-(2), we get
  1/s   1/s 
  2
|u|q (eαu −1) dx ≤ u qs ((eαu − 1))s dx (eαs u − 1) dx
2 q 2
≤ C u q .
R2 R2 R2
(2.7)
Hence, by (2.1), for u ∈ HV1 , we have

|u|q (eαu − 1) dx < ∞.
2
(2.8)
R2

Now, if αμ u 2 /(μ − 1) < 4πζ , we have αs  ∇u 22 < 4π, it follows by (2.2) that there
exists C > 0 such that

 2
(eαs u − 1) dx ≤ C.
R2

Thus, by (2.7), we obtain


|u|q (eαu − 1) dx ≤ C u q .
2
(2.9)
R2
μ
Now, let τ > 1 be sufficiently large and γ > μ−1 such that 1/μ + 1/τ + 1/γ = 1. By
Hölder inequality, we have
  1/μ  1/τ
2
V (x)1/μ |u|q (eαu − 1) ≤ |u|(q− μ )τ
2
V (x)|u|2
[V >0] [V >0] [V >0]
 1/γ
(eαu − 1)γ
2
.
[V >0]

Consequently, from the conditions (V1 ) − (V2 ), it holds

   1/μ
q− μ2
V (x)1/μ |u|q (eαu − 1) ≤ C u 2 + B
2
u2 u
[V >0] [V ≤0] (q− μ2 )τ
 1/γ
(eαγ u − 1) dx
2
.
R2

123
942 M. de Souza, J. M. do Ó

Hence,
    1/γ
B 1/μ 2 q− 2
V (x)1/μ |u|q (eαu − 1) ≤ C 1+ (eαγ u − 1) dx
2 2
u μ u μ2 .
[V >0] λ1 (q−μ )τ R2
(2.10)
Since (q − μ2 )τ ≥ 2 for τ sufficiently large, by Lemma 2.2-(2), we obtain by (2.1)

V (x)1/μ |u|q (eαu − 1) dx < ∞.
2
(2.11)
[V >0]

Hence, from (2.6), (2.8) and (2.11), we conclude that


A(x)|u|q (eαu − 1) dx < ∞.
2

R2
This completes the first part of the Lemma.
μ
Now, if αμ u 2 /(μ − 1) < 4πζ , we can choose γ sufficiently close to μ−1 such that
αγ ∇u 22 < 4π. Thus, by (2.2), we get

(eαγ u − 1) dx ≤ C.
2

[V >0]

Then, using Lemma 2.2-(2) and (2.10), there exists C > 0 such that

V (x)1/μ |u|q (eαu − 1) dx ≤ C u q .
2
(2.12)
[V >0]

Combining (2.6), (2.9) and (2.12), we conclude the second part of the Lemma. 


3 The variational framework

By the hypothesis (H1 ) − (H3 ), if f (x, s) and g(x, s) have subcritical growth, then for each
β > 0, α > 0, ε > 0 and q > 2, there exists a constant C1 = C1 (ε, q) > 0 such that for all
(x, s) ∈ R2 × [0, +∞),

F(x, s) ≤ 2ε A(x)s 2 + C1 A(x)s q (eβs − 1),
2

(3.1)
G(x, s) ≤ 2ε A(x)s 2 + C1 A(x)s q (eαs − 1).
2

Similarly, if f (x, s) and g(x, s) have critical growth with critical exponent α0 > 0, then for
each β > α0 and α > α0 , there exists a constant C1 = C1 (ε, q) > 0 such that (3.1) holds.
Thus, by (2.4) and using (3.1), we have F (x, u), G (x, v) ∈ L 1 (R2 ) for all (u, v) ∈ E.
Therefore, the functional energy I : E → R given by
  
I (u, v) = [∇u∇v + V (x)uv]dx − F(x, u)dx − G(x, v)dx
R2 R2 R2

is well defined. Moreover, using standard arguments, we can see that I ∈ C 1 (E, R) with

I  (u, v)(ϕ, ψ) = [∇u∇ψ + V (x)uψ + ∇v∇ϕ + V (x)vϕ] dx
R2
 
− f (x, u)ϕ dx − g(x, v)ψ dx, (3.2)
R2 R2

for all (ϕ, ψ) ∈ E.

123
Hamiltonian elliptic systems in R2 with subcritical… 943

We use the following notation:

E + = {(u, u) ∈ E} and E − = {(v, −v) ∈ E}.
It is easy to check that E = E + ⊕ E − .
The next lemmas are essential to establish the geometry of the Linking Theorem of the
associated functional.
Lemma 3.1 Suppose that (V1 ) − (V3 ) and (H1 ) − (H3 ) hold. Then there exist ρ, σ > 0
such that I (z) ≥ σ , for all z ∈ S := ∂ Bρ ∩ E + .
 
Proof If u ≤ δ, with δ > 0 such that δ 2 < min 4π ζαμ (μ−1) 4π ζ (μ−1)
, βμ . From (2.5) and
(3.1), we have

I (u, u) ≥ u 2 − ε A(x)|u|2 dx − C1 u q .
R2
By Lemma 2.2-(3), there exists C > 0 such that
I (u, u) ≥ (1 − Cε) u 2 − C1 u q .
Therefore, for ε > 0 sufficiently small, we can find ρ, σ > 0 such that I (u, u) ≥ σ > 0
whenever u = ρ. This completes the proof of the Lemma. 

Let y ∈ HV1 \ {0} be a fixed nonnegative function and
Q y = {r (y, y) + w : w ∈ E − , w E ≤ R0 and 0 ≤ r ≤ R1 }.
The constants R0 and R1 will be chosen in the next lemma.
Lemma 3.2 Suppose that (V1 ) − (V3 ) and (H1 ) − (H4 ) hold. Then there exist positive
constants R0 , R1 , which depend on y, such that I (z) ≤ 0 for all z ∈ ∂ Q y .
Proof Notice that the boundary ∂ Q y of the set Q y in the space R(y, y) ⊕ E − is composed
of three parts. On these parts, the functional I is estimated as follows:
(i) If z ∈ ∂ Q y ∩ E − we have I (z) ≤ 0 because, for all z = (u, −u) ∈ E − ,

I (z) = − u 2 − [F(x, u) + G(x, −u)] dx ≤ 0.
R2
(ii) If z = R1 (y, y) + (u, −u) ∈ ∂ Q y with (u, −u) E ≤ R0 ,

I (z) = R12 y 2 − u 2 − [F(x, R1 y + u) + G(x, R1 y − u)] dx. (3.3)
R2
It follows from assumption (H4 ) that for all δ > 0, there exists c1 = c1 (δ) > 0 such that
F(x, s), G(x, s) ≥ c1 s θ − δs 2 , for all (x, s) ∈ R2 × [0, +∞).
Thus, we obtain by (3.3) that there exists C1 = C1 (y) > 0 such that
I (z) ≤ R12 y 2 + C1 (R12 − R1θ ).
Finally, we take R1 = R1 (y) sufficiently large such that I (z) ≤ 0.
(iii) If z = r (y, y) + (u, −u) ∈ ∂ Q y with (u, −u) E = R0 and 0 ≤ r ≤ R1 ,

I (z) = r y − u −
2 2 2
[F(x, r y + u) + G(x, r y − u)] dx
R2
1
≤ R12 y 2 − R02 .
2

123
944 M. de Souza, J. M. do Ó

√
Thus, I (z) ≤ 0 if 2R1 y ≤ R0 . This completes the proof of the Lemma. 


So, the geometry of the linking theorem holds.

4 On Palais–Smale sequences

It is well known that the absence of the (PS) compactness condition creates difficulties in
the study of this class of elliptic problems involving critical growth and unbounded domains.
In this section, we study the compactness of Palais–Smale of I . First of all, we need the
following lemma:

Lemma 4.1 The following inequality holds

2
√
3
st ≤ t 2 (et − 1) + s(log s)1/2 , for all t ≥ 0 and s ≥ e1/ 4
.

2
Proof For each s > 0, consider supt≥0 {st − t 2 (et − 1)}. Let ts denote the (unique) point
where the supremum is attained. Then,
2 2
s = 2ts [(ets − 1) + ts2 ets ].

Consider now two cases:

Case 1 ts ≥ √ 3 ts2
3 ; then s ≥ 2ts e , which implies (log s)
1 1/2 ≥ t . Thus,
s
2

2
sup{st − t 2 (et − 1)} ≤ sts ≤ s(log s)1/2 .
t≥0

√
3 √
3
Case 2 0 ≤ ts ≤ √3
1
and s ≥ e1/ 4 ; then sts ≤ √1
3 s and √1
3 s ≤ s(log s)1/2 iff s ≥ e1/ 4.
2 2 2
This completes the proof of the Lemma. 


Proposition 4.2 Suppose that (V1 ) − (V3 ) and (H1 ) − (H4 ) hold. Let (u m , vm ) ∈ E such
that

(I1 ) I (u m , vm ) = c + δm , where δm → 0 as m → +∞;

(I2 ) | I  (u m , vm )(ϕ, ψ) |≤ εm (ϕ, ψ) E , for ϕ, ψ ∈ E, where εm → 0 as m → +∞.
Then

u m ≤ C, vm ≤ C,
 
f (x, u m )u m dx ≤ C, g(x, vm )vm dx ≤ C,
R  R
2 2

F(x, u m ) dx ≤ C, G(x, vm ) dx ≤ C.
R2 R2

Proof Taking (ϕ, ψ) = (u m , vm ) in (I2 ), we have

   
 
2u m , vm  E − f (x, u m )u m dx − g(x, vm )vm dx  ≤ εm (u m , vm ) E , (4.1)

R2 R2

123
Hamiltonian elliptic systems in R2 with subcritical… 945

which together with (I1 ) and (H4 ) imply


[ f (x, u m )u m + g(x, vm )vm ] dx
R2

≤2 [F(x, u m ) + G(x, vm )] dx + 2c + 2δm + εm (u m , vm ) E
R
2

2
≤ [ f (x, u m )u m + g(x, vm )vm ] dx + 2c + 2δm + εm (u m , vm ) E .
θ R2
Thus,

θ
[ f (x, u m )u m + g(x, vm )vm ] dx ≤ (2c + 2δm + εm (u m , vm ) E ) . (4.2)
R2 θ −2
Next taking (ϕ, ψ) = (vm , 0) and (ϕ, ψ) = (0, u m ) in (I2 ), we obtain

vm 2 − εm vm ≤ f (x, u m )vm dx
R
2

u m 2 − εm u m ≤ g(x, vm )u m dx.
R2
Setting Um = u m / u m and Vm = vm / vm , we get

vm ≤ f (x, u m )Vm dx + εm (4.3)
R
2

u m ≤ g(x, vm )Um dx + εm . (4.4)

R2
First let us estimate (4.3). Notice that

f (x, u m )Vm dx ≤ Am + Bm ,
R2
where

Am = f (x, u m )Vm dx,
W
Bm = f (x, u m )Vm dx,
H √ 
f (x, u m ) 1/ 3 4
W= x ∈R : 2
≤e ,
C A(x)
 √ 
f (x, u m ) 3
H = x ∈ R2 : ≥ e1/ 4 ,
C A(x)
with C > 0 to be chosen later. Using Young’s inequality, we obtain that
   
1 f (x, u m ) 2
Am ≤ A(x) + Vm dx
2
2 W A(x)
 
1 f (x, u m ) 1
= f (x, u m ) dx + A(x)Vm2 dx. (4.5)
2 W A(x) 2 W
Since Vm = 1, by Lemma 2.2-(3), we reach

A(x)Vm2 dx ≤ C. (4.6)
W

123
946 M. de Souza, J. M. do Ó

By (H1 ) − (H2 ), there exist constants c > 0 and s0 > 0 such that f (x, s) ≤ cs A(x) for
s ∈ [0, s0 ] and x ∈ R2 , and hence,
 √ 
f (x, s) f (x, s) 3
≤ cs, for x ∈ R2 : s ∈ [0, s0 ] and ≤ e1/ 4 ,
A(x) C A(x)
while for s > s0
√
3  
f (x, s) e1/ 4 f (x, s) √
3
≤ s, for x ∈ R2 : s ∈ (s0 , +∞) and ≤ e1/ 4
.
A(x) s0 C A(x)
Consequently,  
f (x, u m )
f (x, u m ) dx ≤ C f (x, u m )u m dx. (4.7)
W A(x) W
From (4.5), (4.6) and (4.7), we conclude

Am ≤ C + f (x, u m )u m dx. (4.8)
R2
By the hypothesis (H1 ), if f (x, s) has subcritical growth, for each β > 0, there exists a
constant C > 0 such that for all (x, s) ∈ R2 × [0, +∞),
f (x, s)
≤ Ceβs .
2
(4.9)
A(x)
Similarly, if f (x, s) has critical growth with critical exponent α0 > 0, for each β > α0 , there
exists C > 0 such that (4.9) holds. Note also that for δ > 0

C f (x, u m )
Bm = A(x) δVm dx.
δ H C A(x)
Thus, by Lemma 4.1, we obtain
   
1 f (x, u m )
A(x)Vm2 (eδ Vm − 1) dx +
2 2
Bm ≤ δC f (x, u m ) log1/2 dx. (4.10)
R2 δ H C A(x)
Choosing δ > 0 such that δ 2 < 4πζ , we have by (2.5) that

A(x)Vm2 (eδ Vm − 1) dx ≤ C
2 2
(4.11)
R2
and by (4.9),
   
f (x, u m ) 1/2
f (x, u m ) log dx ≤ C f (x, u m )u m dx. (4.12)
H C A(x) R2

Combining (4.10), (4.11) and (4.12), we reach


Bm ≤ C + f (x, u m )u m dx. (4.13)
R2
Hence, from (4.8) and (4.13), we get
 
f (x, u m )Vm d x ≤ C1 + C2 f (x, u m )u m dx. (4.14)
R2 R2
This estimate together with (4.3) implies

vm ≤ C1 + C2 f (x, u m )u m dx. (4.15)
R2

123
Hamiltonian elliptic systems in R2 with subcritical… 947

Repeating the same argument above, we get


u m ≤ C1 + C2 g(x, vm )vm dx. (4.16)
R2

Joining the estimates (4.15) and (4.16) and using (4.2), we obtain
(u m , vm ) E ≤ C(C1 + δm + εm (u m , vm ) E + εm ),
which implies that (u m , vm ) E ≤ C. From this estimate and inequality (4.2), we obtain the
other estimates in the statement of the proposition. 


To show that the weak limit of a Palais–Smale sequence in E is a weak solution of (S),
we will use the following convergence result due to de Figueiredo et al. [6, Lemma 2.1].

Lemma 4.3 Let  ⊂ R2 be a bounded domain and f : R2 × R → R a continuous function.

Then for any sequence (u m ) in L 1 () such that u m → u in L 1 (),

f (x, u m ) ∈ L () and
1
| f (x, u m )u m | dx ≤ C,


up to a subsequence, we have
f (x, u m ) → f (x, u) in L 1 ().

Similar to Lam and Lu in [16], we have the following:

Lemma 4.4 Suppose that (V1 ) − (V4 ) and (H1 ) − (H5 ) hold. If (u m , vm ) ⊂ E is a sequence
such that I (u m , vm ) → c, I  (u m , vm ) → 0 and (u 0 , v0 ) is its weak limit, then
   
F(x, u m ) dx → F(x, u 0 ) dx and G(x, vm ) dx → G(x, v0 ) dx.
R2 R2 R2 R2

Proof By using Proposition 4.2 and Lemma 4.3, we have that for all R > 0
 
f (x, u m ) dx → f (x, u 0 ) dx.
BR BR

Thus, there exists p(x) ∈ L 1 (B R) such that

f (x, u m (x)) ≤ p(x) almost everywhere in B R .
Let B = {x ∈ B R : u m (x) ∈ [0, s1 ] for all n ∈ N}. Then, using (H1 ), (H3 ) and (H5 ), we
conclude that
F(x, u m (x)) ≤ sup F(x, u m (x)) + M f (x, u m (x))
B

almost everywhere in B R . By Generalized Lebesgue dominated convergence theorem, we

obtain
 
F(x, u m ) dx → F(x, u 0 ) dx.
BR BR

Thus, to prove that

 
F(x, u m ) dx → F(x, u 0 ) dx
R2 R2

123
948 M. de Souza, J. M. do Ó

it is sufficient to show that given δ > 0, there exists R > 0 such that
 
F(x, u m ) dx < δ and F(x, u 0 ) dx < δ.
R2 \B R R2 \B R

To prove this, we first recall the following facts: There exists C > 0 such that for (x, s) ∈
R2 × [0, +∞):
F(x, s) ≤ C A(x)s 2 + C f (x, s),
F(x, s) ≤ C A(x)s 2 + C A(x)s(eβs − 1),
2

  (4.17)
f (x, u m )u m dx ≤ C and F(x, u m ) dx ≤ C.
R2 R2
Note that
  
F(x, u m ) dx ≤ C A(x)|u m |2 dx +C f (x, u m ) dx
{x∈R2\B R : |u m |>K } R2\B R {x∈R2\B R : |u m |>K }
 
C C2
≤ A(x)|u m |3 dx + f (x, u m )u m dx
K R2 \B R K R2

C1 C2
≤ u m 3 + f (x, u m )u m dx.
K K R2
Thus, since u m is bounded, by using (4.17), we can choose K large enough such that

F(x, u m ) dx < 2δ/3.
{x∈R2 \B R : |u m |>K }
Next, we have
 
F(x, u m ) dx ≤ C(β, K ) A(x)|u m |2 dx
{x∈R2 \B R : |u m |≤K } {x∈R2 \B R : |u m |≤K }

≤ 2C(β, K ) A(x)|u m − u 0 |2 dx
{x∈R2 \B R : |u m |≤K }

+ 2C(β, K ) A(x)|u 0 |2 dx.
{x∈R2 \B R : |u m |≤K }
q
Now, using the compactness of embedding HV1 → L A(x) (R2 ) for q ≥ 2 and noticing that
(u m , vm )  (u 0 , v0 ) weakly in E, again we can choose R such that

F(x, u m ) dx < δ/3.
{x∈R2 \B R : |u m |≤K }
Combining all the above estimates, we have the fact that
 
F(x, u m ) dx → F(x, u 0 ) dx.
R2 R2
Similarly, we can prove for G. This completes the proof of the Lemma. 


5 Finite-dimensional problem

Since the functional I is strongly indefinite and defined in an infinite-dimensional space, no

suitable linking theorem is available. We therefore approximate problem (S) with a sequence
of finite-dimensional problems (a Galerkin approximation procedure).

123
Hamiltonian elliptic systems in R2 with subcritical… 949

Associated with the eigenvalues 0 < λ1 < λ2 ≤ λ3 ≤ · · · ≤ λ j → +∞ of (− +

V (x), HV1 ), there exists an orthonormal basis {ϕ1 , ϕ2 , . . .} of corresponding eigenfunctions
in HV1 . We set,

E n+ = span{(ϕi , ϕi ) : i = 1, . . . , n},
E n− = span{(ϕi , −ϕi ) : i = 1, . . . , n},
E n = E n+ ⊕ E n− .

Let y ∈ HV1 be a fixed nonnegative function and

Q n,y = {r (y, y) + w : w ∈ E n− , w E ≤ R0 , and 0 ≤ r ≤ R1 },

where R0 and R1 are given in Lemma 3.2. We recall that these constants depend on y only.
We use the following notation:
+
Hn,y = R(y, y) ⊕ E n , Hn,y = R(y, y) ⊕ E n+ , Hn,y
−
= R(y, y) ⊕ E n− .

Furthermore, define the class of mappings

n,y = {h ∈ C(Q n,y , Hn,y ) : h(z) = z on ∂ Q n,y }

and set

cn,y = inf max I (h(z)).

h∈n,y z∈Q n,y

Using an intersection theorem (see Proposition 5.9 in [20]), we have

h(Q n,y ) ∩ (∂ Bρ ∩ E n+ )  = ∅, for all h ∈ n,y ,

which in combination with Lemma 3.1 implies that cn,y ≥ σ > 0. On the other hand,
an upper bound for the mini–max level cn,y can be obtained as follows. Since the identity
mapping I d : Q n,y → Hn,y belongs to n,y , we have for z = r (y, y) + (u, −u) ∈ Q n,y
that

I (z) = r y − u −
2 2 2
[F(x, r y + u) + G(x, r y − u)] dx ≤ R12 y 2 .
R2

Therefore, we have 0 < σ ≤ cn,y ≤ R12 y 2 . We remark that the upper bound does not
depend of n, but it depends on y.
Let us denote by In,y the functional I restricted to the finite-dimensional subspace Hn,y .
So, in view of Lemmas 3.1 and 3.2, we see that the geometry of the linking theorem holds
for the functional In,y . Therefore, applying the linking theorem for In,y (see Theorem 5.3
in [20]), we obtain a (PS)-sequence, which is bounded in view of Proposition 4.2. Finally,
using the fact that Hn,y is a finite-dimensional space, we get the main result of this section.

Proposition 5.1 For each n ∈ N and for each y ∈ HV1 a fixed nonnegative function, the
functional In,y has a critical point at level cn,y . More precisely, there is a z n,y ∈ Hn,y such
that

In,y (z n,y ) = cn,y ∈ [σ, R12 y 2 ] and (In,y ) (z n,y ) = 0.

Furthermore, z n,y E ≤ C where C does not depend on n.

123
950 M. de Souza, J. M. do Ó

6 Proof of Theorem 1.1

In this section, we assume that g has subcritical growth and f has subcritical or critical
growth.
Let y ∈ HV1 be a nonnegative function fixed. Applying Proposition 5.1, we have a sequence
z n,y = (u n,y , vn,y ) ∈ Hn,y such that (u n,y , vn,y ) E ≤ C and

In,y (u n,y , vn,y ) = cn,y ∈ [σ, R12 y 2 ],

(In,y ) (u n,y , vn,y ) = 0, (6.1)
(u n,y , vn,y )  (u 0 , v0 ) weakly in E.

Using Proposition 4.2, we reach

 
f (x, u n,y )u n,y dx ≤ C and g(x, vn,y )vn,y dx ≤ C, (6.2)
R R2
2

F(x, u n,y ) dx ≤ C and G(x, vn,y ) dx ≤ C. (6.3)
R2 R2

Taking as test function (0, ψ) and (ϕ, 0) in (6.1), where ψ and ϕ are arbitrary functions in
C0∞ (R2 ), we get
 
(∇u n,y ∇ψ + V (x)u n,y ψ)dx = g(x, vn,y )ψ dx, (6.4)
R  R
2 2

(∇vn,y ∇ϕ + V (x)vn,y ϕ)dx = f (x, u n,y )ϕ dx. (6.5)

R2 R2

Next, using Lemma 4.3 and (6.2), we have

   
f (x, u n,y )ϕ dx → f (x, u 0 )ϕ dx and g(x, vn,y )ψ dx → g(x, v0 )ψ dx.
R2 R2 R2 R2

Thus, taking the limit in (6.4) and (6.5) and using the fact that C0∞ (R2 ) is dense in HV1 , it
follows that
 
(∇u 0 ∇ψ + V (x)u 0 ψ)dx = g(x, v0 )ψ dx, for all ψ ∈ HV1 , (6.6)
R2 R2
 
(∇v0 ∇ϕ + V (x)v0 ϕ)dx = f (x, u 0 )ϕ dx, for all ϕ ∈ HV1 . (6.7)
R2 R2

Finally, it only remains to prove that u 0 and v0 are nontrivial. Assume by contradiction that
u 0 ≡ 0. This implies that v0 ≡ 0. Since g has subcritical growth, we see that for all α > 0,
there exist C1 , C2 > 0 such that

g(s) ≤ C1 A(x)s + C2 A(x)s 2 (eαs − 1), for all s ∈ [0, +∞).

2

Combining Hölder inequality and Lemma 2.3, and choosing α > 0 and q > 1 sufficiently
close to 1 such that αq vn,y 2 < 4πζ , 1/q + 1/s = 1 and s > 2, we obtain

g(x, vn,y )u n,y dx
R2
≤ C1 vn,y L 2 u n,y L 2
A(x) (R
2)
A(x) (R
2)

123
Hamiltonian elliptic systems in R2 with subcritical… 951

 1/q
A(x)|vn,y |2q (eαq vn,y
2 (v
n,y / vn,y )
2
+C2 − 1) dx u n,y L s
A(x) (R
2)
R2
≤ C1 vn,y L 2 u n,y L 2 + C u n,y L s .
A(x) (R
2)
A(x) (R
2)
A(x) (R
2)

Thus,
 
(|∇u n,y |2 + V (x)|u n,y |2 )dx = g(x, vn,y )u n,y dx → 0,
R2 R2

p
since vn,y → 0 and u n,y → 0 in L A(x) (R2 ) for all p ≥ 2. Consequently, u n,y → 0 in HV1 .
This implies that 
(∇u n,y ∇vn,y + V (x)u n,y vn,y )dx → 0. (6.8)
R2

Then, by (In,y ) (u n,y , vn,y ) = 0 and (6.8), we obtain that

 
f (x, u n,y )u n,y dx → 0 and g(x, vn,y )vn,y dx → 0.
R2 R2

Using these limits and (H4 ), it follows that

 
F(x, u n,y ) dx → 0 and G(x, vn,y ) dx → 0. (6.9)
R2 R2

Finally, using (6.8) and (6.9), we see cn,y = 0, which is a contradiction. Consequently, we
have a nontrivial critical point of I and thereby conclude the proof of the Theorem 1.1.

7 On the minimax level

In this section, we assume that f and g have critical growth with critical exponent α0 . Let
us to denote C p1 by
 1/2
(|∇u|2 + V (x)u 2 ) dx
R2
C p1 := inf  1/ p1 .
u∈HV1 \{0}
|u| p1 dx
R2

Lemma 7.1 Suppose that (1.5) is satisfied. Then, there exists a nonnegative function u ∈ HV1
such that
4π ζ (μ − 1)
sup I (η) < ,
η∈R+ (u,u)⊕E − α0 μ

where ζ is given in (2.3).

Proof It is clear that C p1 > 0 and since the embedding HV1 → L p1 (R2 ) is compact, there
exists a nonnegative function u p1 ∈ HV1 such that

C p1 = u p1 and u p1 L p1 (R2 ) = 1. (7.1)

123
952 M. de Souza, J. M. do Ó

Thus, for each η = t (u p1 , u p1 ) + (v, −v) with t ≥ 0 and v ∈ HV1 , we have

 
I (η) = t 2 C 2p1 − v 2 − F(x, tu p1 + v) dx − G(x, tu p1 − v) dx
R2
 R
2

≤ t 2 C 2p1 − τ F(x, tu p1 + v) dx − τ G(x, tu p1 − v) dx,
C1 (v) C2 (v)

where C1 (v) = {x ∈ R2 : v(x) ≥ 0} and C2 (v) = {x ∈ R2 : v(x) < 0}. Consequently, by

(1.5),
 
λ λ
I (η) ≤ t 2 C 2p1 − (tu p1 + v) p2 dx − (tu p1 − v) p1 dx
p2 C1 (v) p1 C2 (v)
 
λ λ
≤ t 2 C 2p1 − (tu p1 ) p2 dx − (tu p1 ) p1 dx. (7.2)
p2 C1 (v) p1 C2 (v)

We shall divide the proof into three cases.

Case 1 p2 = p1
From (7.1) and (7.2), we get

λ
I (η) ≤ t 2 C 2p1 − t p1 |u p1 | p1 dx
p1 R 2
 
λ
≤ max t 2 C 2p1 − t p1
t≥0 p1
  2/( p1 −2) 2 p1 /( p1 −2)
p1 − 2 2 C p1
= .
p1 λ2/( p1 −2)
Taking λ0 > 0 such that
  2 p /( p1 −2)
p1 − 2 22/( p1 −2) C p1 1 4π ζ (μ − 1)
2/( p −2)
< ,
p1 λ0 1 α0 μ

we obtain the desired result, for λ > λ0 .

Case 2 p2  = p1
We assume p2 > p1 . From (7.2), we get
 
λ λ
I (η) ≤ t C p1 −
2 2
(tu p1 ) dx −
p2
(tu p1 ) p1 dx
p2 C1 (v) p1 C2 (v)
  
λ
≤ t C p1 −
2 2
min{t , t }
p1 p2
|u p1 | dx +
p2
|u p1 | dx .
p1
(7.3)
p2 C1 (v) C2 (v)

Notice that, by Hölder inequality, we have

   p2   p2
p1 1 p1
|u p1 | p2 dx ≥ |u p1 | p1 dx p2 − p1 ≥ |u p1 | p1 dx .
C1 (v) C1 (v) |C1 (v)| p1 C1 (v)
(7.4)
From (7.3) and (7.4), we get
  p2  
λ p1
I (η) ≤ t 2 C 2p1 − min{t p1 , t p2 } |u p1 | p1
dx + |u p1 | p1
dx .
p2 C1 (v) C2 (v)

123
Hamiltonian elliptic systems in R2 with subcritical… 953


C2 (v) |u p1 | dx ≤ 1, by (7.1), we obtain
Since p1

  p2   p2 
λ p1 p1
I (η) ≤ t 2 C 2p1 − min{t , t }
p1 p2
|u p1 | dx
p1
+ |u p1 | dx
p1
p2 C1 (v) C2 (v)
p1 − p2
λ2 p1
≤ t 2 C 2p1 − min{t p1 , t p2 },
p2
where we have used that (a + b)q ≤ 2q−1 (a q + bq ), for all q > 1 and a, b ≥ 0.
p2 − p1
If λ > 2 p1 C 2p1 p2 , we find
⎡ ⎤
p1 − p2   2 p /( p2 −2)
λ2 p1 p2 − 2 22 p2 / p1 ( p2 −2) C p1 2
I (η) ≤ max ⎣t 2 C 2p1 − t p2 ⎦
= .
t∈[0,1] p2 p2 λ2/( p2 −2)

Taking λ0 large enough such that

  2 p /( p −2)
p2 − p1 p2 − 2 22 p2 / p1 ( p2 −2) C p1 2 2 4π ζ (μ − 1)
λ0 > 2 p 1 2
C p1 p2 and 2/( p2 −2)
< ,
p2 λ α0 μ
0
we obtain the desired result, for λ > λ0 . This completes the proof of the Lemma. 


8 Critical case: Proof of Theorem 1.2

In this section, we assume that f and g have critical growth with critical exponent α0 .
By using Lemma 7.1, there is δ > 0 such that
4π ζ (μ − 1)
cn := cn,u ≤ − δ,
α0 μ
where cn,u is defined in Proposition 5.1.
Next, applying Propositions 4.2 and 5.1, we have a sequence z n := z n,u = (u n , vn ) ∈ Hn,u
such that (u n , vn ) E ≤ C and
 
4π ζ (μ − 1)
In,u (u n , vn ) = cn ∈ σ, −δ ,
α0 μ
 (8.1)
(In,u ) (u n , vn ) = 0,
(u n , vn )  (u 0 , v0 ) weakly in E.
By using Proposition 4.2, we conclude that
 
f (x, u n )u n dx ≤ C and g(x, vn )vn dx ≤ C. (8.2)
R2 R2

So, arguing as in subcritical case, from (8.1), we obtain

 
(∇u 0 ∇ψ + V (x)u 0 ψ) dx = g(x, v0 )ψ dx, for all ψ ∈ HV1 , (8.3)
R2 R2
 
(∇v0 ∇ϕ + V (x)v0 ϕ)dx = f (x, u 0 )ϕ dx, for all ϕ ∈ HV1 . (8.4)
R2 R2
Finally, it only remains to prove that u 0 and v0 are nontrivial.

123
954 M. de Souza, J. M. do Ó

Assume by contradiction that u 0 ≡ 0. This implies that v0 ≡ 0. Now, if u n → 0, then

we get directly (8.10) below, and then a contradiction. Thus, we assume that u n ≥ b > 0
for all n ∈ N and consider 
u n 2 = g(x, vn )u n dx. (8.5)
R2

Note that considering u n = ( 4π ζα(μ−1)

− δ)1/2 uu nn , by using Lemma 4.1 with s =
0μ
√
g(x, vn )/C A(x) and t = α0 u n and arguing as in Proposition 4.2, we get
 1/2 
4π ζ (μ − 1)
−δ u n = g(x, vn )u n dx
α0 μ R2

C g(x, vn ) √ (8.6)
= √ A(x) α0 u n dx
α0 R2 C A(x)
≤ J1,n + J2,n + J3,n + J4,n ,
where

√
A(x)u 2n (eα0 u n − 1)dx,
2
J1,n = α0 C √
{x∈R2 : g(x,vn (x)) ≥e1/
34
}
C A(x)
   
1 g(x, vn ) 1/2
J2,n = √ √ g(x, vn ) log dx,
α0 {x∈R2 : g(x,vn (x)) ≥e1/ 3 4 } C A(x)

C A(x)
1
J3,n = √ A(x)u 2n dx,
2 {x∈R2 : g(x,vn (x)) ≤e1/ 3 4 }
C A(x)

1 g(x, vn )
J4,n = √ g(x, vn ) dx.
2 {x∈R2 : g(x,vn (x)) ≤e1/ 3 4 } A(x)
C A(x)

Let us estimate these integrals. Since u n 2 = 4π ζα(μ−1)

0μ
− δ, arguing as in Lemma 2.3
and using Lemma 2.2, from (2.6), (2.7) and (2.10), we obtain that J1,n → 0, and by Lemma
2.2-(5), we get J3,n → 0. We observe that by (H1 )-(H3 ),
 √ 
g(x, vn ) g(x, vn (x)) 3
g(x, vn ) ≤ C A(x)|vn |2 in x ∈ R2 : ≤ e1/ 4
A(x) C A(x)
and thus, by Lemma 2.2-(5), we obtain

g(x, vn )
J4,n = √ g(x, vn ) dx → 0.
{x∈R2 : g(x,vn (x)) ≤e1/ 4 }
3
A(x)
C A(x)

Now, using (4.9), we can estimate J2,n by


1
J2,n ≤ √ g(x, vn )(α0 + ε)1/2 vn dx
α0 R2
  
ε 1/2
≤ 1+ g(x, vn )vn dx,
α0 R2

and hence, by (8.6), we get

 1/2   
4π ζ (μ − 1) ε 1/2
−δ u n ≤ on (1) + 1 + g(x, vn )vn dx. (8.7)
α0 μ α0 R2

123
Hamiltonian elliptic systems in R2 with subcritical… 955

Repeating the same argument with


vn 2 = f (x, u n )vn dx,
R2

we see that
 1/2   
4π ζ (μ − 1) ε 1/2
−δ vn ≤ on (1) + 1 + f (x, u n )u n dx. (8.8)
α0 μ α0 R2

By Lemma 4.4, we have

 
F(x, u n ) dx → 0 and G(x, vn ) dx → 0. (8.9)
R2 R2

Thus, we conclude from Lemma 7.1 that

4π ζ (μ − 1)
|u n , vn  E | ≤ on (1) + − δ,
α0 μ

which together with (8.1) implies that

 
4π ζ (μ − 1)
f (x, u n )u n dx + g(x, vn )vn dx ≤ on (1) + − δ.
R 2 R 2 α0 μ

So, from (8.7) and (8.8), we obtain

   1/2
ε 1/2 4π ζ (μ − 1)
u n + vn ≤ on (1) + 1 + −δ
γ0 α0 μ
 1/2
4π ζ (μ − 1)
≤ −δ ,
α0 μ

for ε > 0 sufficiently small and n sufficiently large. It follows that there is a subsequence of
(u n ) or (vn ) (without loss of generality, we can assume that the sequence is (vn )) such that
 1/2
4π ζ (μ − 1) δ
vn ≤ − .
α0 μ 2

Consequently, arguing as in Lemma 2.3 and using Lemma 2.2, from (2.6), (2.7) and (2.10),
we obtain

g(x, vn )vn dx → 0.
R2

Hence,
u n , vn  E → 0, (8.10)

which together with (8.9) implies cn = 0, which is a contradiction. This completes the proof
of Theorem 1.2.

Acknowledgments We would like to thank the referee for the careful review and the valuable comments,
which provided insights that helped to improve the paper.

123
956 M. de Souza, J. M. do Ó

References
1. Adimurthi.: Existence of positive solutions of the semilinear Dirichlet problem with critical growth for
the n-Laplacian. Ann. Scuola Norm. Sup. Pisa Cl. Sci 17, 393–413 (1990)
2. Berestycki, H., Lions, P.-L.: Nonlinear scalar field equations, I. Existence of a ground state. Arch. Ratl.
Mech. Anal. 82, 313–345 (1983)
3. Cao, D.M.: Nontrivial solution of semilinear elliptic equation with critical exponent in R2 . Commun.
Part. Differ. Equ. 17, 407–435 (1992)
4. de Figueiredo, D.G., do Ó, J.M., Ruf, B.: Critical and subcritical elliptic systems in dimension two. Indiana
Univ. Math. J. 53, 1037–1054 (2004)
5. de Figueiredo, D.G., Felmer, P.L.: On superquadratic elliptic systems. Trans. Am. Math. Soc. 343, 97–116
(1994)
6. de Figueiredo, D.G., Miyagaki, O.H., Ruf, B.: Elliptic equations in R2 with nonlinearities in the critical
growth range. Calc. Var. Part. Differ. Equ. 3, 139–153 (1995)
7. de Figueiredo, D.G., do Ó, J.M., Ruf, B.: Elliptic equations and systems with critical Trudinger–Moser
nonlinearities. Discrete Contin. Dyn. Syst 30, 455–476 (2011)
8. de Figueiredo, D.G., Yang, J.: Decay, symmetry and existence of solutions to semilinear elliptic systems.
Nonlinear Anal. TMA 33, 211–234 (1998)
9. de Souza, M.: On a singular Hamiltonian elliptic systems involving critical growth in dimension two.
Commun. Pure Appl. Anal. 11, 1859–1874 (2012)
10. de Souza, M., do Ó, J.M.: On a class of singular Trudinger–Moser type inequalities and its applications.
Math. Nachr. 284, 1754–1776 (2011)
11. Ding, Y., Li, S.: Existence of entire solutions for some elliptic systems. Bull. Aust. Math. Soc. 50, 501–519
(1994)
12. do Ó, J.M., Maia, L.A., Silva, E.A.B.: Standing wave solutions for system of Schrodinger equations in
R2 involving critical growth (to appear)
13. do Ó, J.M., Medeiros, E., Severo, U.B.: A nonhomogeneous elliptic problem involving critical growth in
dimension two. J. Math. Anal. Appl. 345, 286–304 (2008)
14. Giacomoni, J., Sreenadh, K.: A multiplicity result to a nonhomogeneous elliptic equation in whole space
R2 . Adv. Math. Sci. Appl. 15, 467–488 (2005)
15. Hulshot, J., Mitidieri, E., Van der Vorst, R.: Strongly indefinite systems with critical Sobolev exponents.
Trans. Am. Math. Soc. 350, 2349–2365 (1998)
16. Lam, N., Lu, G.: Existence and multiplicity of solutions to equations of N-Laplacian type with critical
exponential growth in R N . J. Funct. Anal. 262, 1132–1165 (2012)
17. Moser, J.: A sharp form of an inequality by N. Trudinger. Indiana Univ. Math. J. 20, 1077–1092 (1970/71)
18. Rabinowitz, P.H.: Some global results for nonlinear eigenvalue problems. J. Funct. Anal. 7, 487–513
(1971)
19. Rabinowitz, P.H.: Periodic solutions of a Hamiltonian system on a prescribed energy surface. J. Differ.
Equ. 33, 336–352 (1979)
20. Rabinowitz, P.H.: Minimax methods in critical point theory with applications to differential equations.
In: CBMS Regional Conf. Ser. in Math., vol. 65, AMS, Providence, RI (1986)
21. Schrödinger, E.: Collected Papers on Wave Mechanics, 3rd edn. AMS Chelsea Publishing, New York
(2003)
22. Sirakov, B.: On the existence of solutions of Hamiltonian elliptic systems in R N . Adv. Differ. Equ. 5,
1445–1464 (2000)
23. Sirakov, B.: Existence and multiplicity of solutions of semi-linear elliptic equations in R N . Calc. Var.
Part. Differ. Equ. 11, 119–142 (2000)
24. Strauss, W.A.: Existence of solitary waves in higher dimensions. Commun. Math. Phys. 55, 149–162
(1977)
25. Trudinger, N.S.: On the embedding into Orlicz spaces and some applications. J. Math. Mech. 17, 473–484
(1967)
26. Yang, J.: Nontrivial solutions of semilinear elliptic systems in R N . Electron. J. Differ. Equ. Conf. 06,
343–357 (2001)
27. Zhang, G., Liu, S.: Existence result for a class of elliptic systems with indefinite weights in R2 . Bound.
Value Probl. 2008, 1–10, Art ID 217636. doi:10.1155/2008/217636

123