Problem 3. Let a, b be positive integers. a, b and ab are not perfect squares.

Prove
that at most one of following equations
ax2 – by2 = 1 và ax2 – by2 = –1.
has solutions in positive integers.

Solution.
Firstly we prove following lemma

Lemma. Let be given the equation Ax2 – By2 = 1 (1)

where A and AB are not perfect squares. Let (a, b) are minimal positive roots of
associated Pell equation x2 – ABy2 = 1 (2)
Assume that (1) has positive integral solutions and (x 0; y0) be its minimal positive
roots. Then (x0; y0) are the only roots of system of equations
a = Ax2 + By2
b = 2xy
Proof. Let (x0; y0) be minimal roots of (1). Let u = Ax 02 + By02, v = 2x0y0 then we
have
u2 – ABv2 = (Ax02 + By02)2 – AB(2x0y0)2 = (Ax02 - By02)2 = 1.
This shows that (u; v) are roots of (2). But (a; b) are minimal roots of this equation
so u  a, v  b.
We will prove that u = a, v = b.
Indeed, assume that u > a ; v > b. We have

In other hand

Put s = ax0 – Bby0, t = ay0 – Abx0 then the last inequalities can be rewritten as

Now, we have (As2 – Bt2) = A(ax0 – Bby0)2 – B(ay0 – Abx0)2

= (a2 – ABb2)(Ax02 – By02) = 1.1 = 1.
We see that s > 0 since
s > 0 ax0 > Bby0 a2x02 > B2b2y02  a2x02 > Bb2(Ax02-1)
 (a2-ABb2)x02 > - Bb2  x02 > - Bb2.
The last inequality holds, so s > 0.
We also have that t  0 since t = 0  ay0 = Abx0  a2y02 = A2b2x02 (ABb2+1)y02
= Ab2(By02+1)  y02 = Ab2. But the last one is impossible because A is not perfect
square
.
If t > 0 then (s; t) are positive roots of (1), due the minimality of (x 0 ; y0) we have
(1) s  x0 ; t  y0, thus
, that contradicts (3).
Similarly, if t < 0 then (s ; -t) are positive roots of (1) and we come to a inequality
that contradics (4).
Hence u = a, v = b or in other words, (x0 ; y0) are roots of abovementioned system.

Let return to the problem. Assume that bot equations

ax2 – by2 = 1 (5)
and
bx2 – ay2 = 1 (6)
have solutions
Let (m ; n) be minimal roots of x2 – aby2 = 1, (x1; y1) – minimal roots of (5) and
(x2, y2) – minimal roots of (6). Apply the lemma, we have
m = ax12 + by12
n = 2x1y1
and
m = bx22 + ay22
n = 2x2y2
Since ax12 = by12 + 1 and ay22 = bx22 – 1 it follows that
ax12 + by12 = bx22 + ay22
2by12 + 1 = 2bx22 – 1
 b(x22-y12) = 1.
But it is impossible since b > 1.