Normalized solutions for a class of Sobolev critical

Schrödinger systems*
Houwang Li1 , Tianhao Liu2 , Wenming Zou3
1. Beijing Institute of Mathematical Sciences and Applications, Beijing, 100084, China.
arXiv:2410.15750v1 [math.AP] 21 Oct 2024

2. Institute of Applied Physics and Computational Mathematics, Beijing, 100088, China.

3. Department of Mathematical Sciences, Tsinghua University, Beijing, 100084, China.

Abstract

This paper focuses on the existence and multiplicity of normalized solutions for the
following coupled Schrödinger system with Sobolev critical coupling term:
 αν α−2 β
p−2
 − ∆u + λ1 u = ω1 |u| u + 2∗ |u|
 |v| u, in RN ,



 βν
− ∆v + λ2 v = ω2 |v|p−2 v + ∗ |u|α |v|β−2 v, in RN ,

 Z Z 2



 2
u dx = a , 2
v dx = b2 ,
2
RN RN

where N ≥ 3, a, b > 0, ω1 , ω2 , ν ∈ R \ {0}, and the exponents p, α, β satisfy

α > 1, β > 1, α + β = 2∗ , 2 < p ≤ 2∗ = 2N / (N − 2) .

The parameters λ1 , λ2 ∈ R will arise as Lagrange multipliers that are not prior given.
This paper mainly presents several existence and multiplicity results under explicit
conditions on a, b for the focusing case ω1 , ω2 > 0 and attractive case ν > 0:
(1) When 2 < p < 2 + 4/N , we prove that there exist two solutions: one is a local
minimizer, which serves as a normalized ground state, and the other is of mountain-
pass type, which is a normalized excited state.
(2) When 2 + 4/N ≤ p < 2∗ , we prove that there exists a mountain-pass type
solution, which serves as a normalized ground state.
(3) When p = 2∗ , the existence and classification of normalized ground states are
provided for and N ≥ 5, alongside a non-existence result for N = 3, 4. These results
reflect the properties of the Aubin-Talenti bubble, which attains the best Sobolev em-
bedding constant.
Furthermore, we present a non-existence result for the defocusing case ω1 , ω2 < 0.
This paper, together with the paper [T. Bartsch, H. W. Li and W. M. Zou. Calc. Var.
Partial Differential Equations 62 (2023) ], provides a more comprehensive under-
standing of normalized solutions for Sobolev critical systems. We believe our meth-
ods can also address the open problem of the multiplicity of normalized solutions for
Schrödinger systems with Sobolev critical growth, with potential for future develop-
ment and broader applicability.

Key words: Sobolev critical system; Normalized solution; Existence and multiplicity.
2020 Mathematics Subject Classification: 35A01, 35J10, 35J50.

* E-mails: lhwmath@bimsa.cn (H. W. Li), liuth19@mails.tsinghua.edu.cn (T. H. Liu), zou-wm@mail.tsinghua.edu.cn (W. M. Zou)

1
1 Introduction and statement of results
The following Schrödinger system describes the dynamics of coupled nonlinear waves

 ∂ αν


 − i Φ1 = ∆Φ1 + ω1 |Φ1 |p−2 Φ1 + |Φ1 |α−2 |Φ2 |β Φ1 ,

 ∂t α+β
∂ βν (1.1)

 − i Φ2 = ∆Φ2 + ω2 |Φ2 |p−2 Φ2 + |Φ1 |α |Φ2 |β−2 Φ2 ,

 ∂t α + β


Φj = Φj (x, t) ∈ C, (x, t) ∈ RN × R, j = 1, 2,
where i is the imaginary unit, and the parameters satisfy
N ≥ 3, ω1 , ω2 , ν ∈ R \ {0} and α, β > 1, 2 < p, α + β ≤ 2∗ := 2N / (N − 2) .
System (1.1) arises in various physical models and has been extensively studied in recent years. For instance,
it appears in Hartree-Fock theory for a double condensate, specifically describing a binary mixture of Bose-
Einstein condensates in two different hyperfine states. In particular, when p = 4, α = β = 2, system (1.1)
reduces to the well-known Gross-Pitaevskii system, which is widely applied in nonlinear optics. Moreover, cer-
tain models for ultracold quantum gases involve different exponents; see [2, 3, 23, 29, 45]. For more background
on system (1.1), we refer to [1, 19, 20] and the references therein.
From a physical perspective, the solutions Φ1 and Φ2 represent condensate amplitudes corresponding to
different condensates, while the parameters ω1 and ω2 correspond to self-interactions within each component.
These self-interactions are called focusing when the sign is positive, and defocusing when negative. The cou-
pling constants αν/(α + β) and βν/(α + β) define the strength and type of interaction between components Φ1
and Φ2 . The sign of ν whether the interaction between the two states is attractive or repulsive: it is attractive
when ν > 0, and repulsive when ν < 0, indicating strong competition between the two states.
An important feature of system (1.1) is that any solution satisfies the conservation of mass, which plays a
crucial role in the dynamics and stability of the system. More precisely, the following L2 -norms
2 2
kΦ1 (·, t)kL2x (RN ) and kΦ2 (·, t)kL2x (RN )
are independent of time t ∈ R. These norms have important physical significance: in Bose–Einstein conden-
sates, the L2 -norms represent the number of particles of each component; in nonlinear optics framework, they
correspond to the power supply; and in Hartree-Fock theory, they represent the mass of condensate.
Among all the solutions, the study of solitary waves is particularly important for system (1.1), as the ground
state plays a crucial role in the dynamics of the solutions; for example, the determination of the threshold of the
scattering solutions, see [21, 22, 33, 34, 41, 42], and the universal profile of collapsing solutions for the critical
nonlinear Schrödinger equations, see [32, 47, 50] and the references therein. The ansatz Φ1 (x, t) = eiλ1 t u(x)
and Φ2 (x, t) = eiλ2 t v(x) for solitary wave solutions of system (1.1) leads to the following steady-state coupled
nonlinear Schrödinger system
 αν

 − ∆u + λ1 u = ω1 |u|p−2 u + |u|α−2 |v|β u, in RN ,

 α + β

βν (1.2)
 − ∆v + λ2 v = ω2 |v|p−2 v + |u|α |v|β−2 v, in RN ,

 α + β


u(x), v(x) → 0 as |x| → ∞,
For the study of system (1.2), there are two distinct options concerning the frequency parameter λ1 and λ2 ,
leading to two different research directions. One direction is to fix the parameters λ1 , λ2 > 0. The existence
and multiplicity of solutions to (1.2) have been extensively investigated over the past two decades. Numerous
relevant studies can be found in the literature; we refer to [1,5,6,13,14,19,20,44,52] and the references therein.
In this paper, we focus on a different direction, specifically investigating the solutions to system (1.2) that
have a prescribed L2 -norm, which are commonly referred to as normalized solutions. In this direction, the
parameters λ1 , λ2 ∈ R cannot be prescribed but appear as multipliers with respect to the constraint L2 -torus
n o
T (a, b) := (u, v) ∈ H : kuk22 = a2 , kvk22 = b2 .

where H := H 1 (RN ) × H 1 (RN ) and kukp to denote the Lp -norm of u.

2
 From now on, we focus on the following normalized problem
 αν


 − ∆u + λ1 u = ω1 |u|p−2 u + |u|α−2 |v|β u, in RN ,

 α +β

 βν
− ∆v + λ2 v = ω2 |v|p−2 v + |u|α |v|β−2 v, in RN , (1.3)

 α + β

 Z Z

 2 2
 u dx = a , v 2 dx = b2 ,
RN RN

From a variational point of view, normalized solutions to problem (1.3) can be obtained as critical points
of the energy functional I(u, v) : H 1 (RN ) × H 1 (RN ) → R,
Z Z Z
1
1 ν
I(u, v) := |∇u|2 + |∇v|2 dx − (|u|p + |v|p ) dx − |u|α |v|β dx
2 RN p RN α + β RN
on the constraint T (a, b) with parameters λ1 , λ2 ∈ R appearing as Lagrange multipliers. Since α, β > 1,
it is standard to conclude that I(u, v) is of class C 1 . Mathematicians and physicists typically focus on the
solution with the least energy, as this often corresponds to good” behavior in physical terms. In this paper, we
are particularly interested in the normalized ground state that minimizes energy. Moreover, the existence of a
normalized excited state is also significant in physics.
Definition 1.1. A solution (u0 , v0 ) is called a normalized ground state to system (1.3) if it satisfies
I|′T (a,b) (u0 , v0 ) = 0, and I(u0 , v0 ) = inf{I(u, v) : I|′T (a,b) (u, v) = 0, (u, v) ∈ T (a, b)}.
A solution (u0 , v0 ) is called a normalized excited state to system (1.3) if it satisfies
I|′T (a,b) (u0 , v0 ) = 0, and I(u0 , v0 ) > inf{I(u, v) : I|′T (a,b) (u, v) = 0, (u, v) ∈ T (a, b)}.

The normalized ground state energy mg (a, b) can be defined by

mg (a, b) := inf I(u, v).
(u,v)∈T (a,b)

Moreover, the existence of normalized ground state solution depends on whether mg (a, b) can be attained.
From a mathematical perspective, new difficulties arise in the search for normalized solutions, making this
problem particularly challenging. For instance, the appearance of the constraint T (a, b) makes lots of classical
variation methods can not be applicable directly since λ1 , λ2 are not given prior, and the existence of bounded
Palais–Smale sequences requires new arguments (the classical method used to establish the boundedness of any
Palais–Smale sequence for unconstrained Sobolev-subcritical problems fails to apply in this case). Moreover,
in the mass supercritical case where p > 2 + N4 or α + β > 2 + N4 , there are bounded Palais-Smale sequences
that do not have a convergent subsequence and converge weakly to 0, since the L2 -torus T (a, b) is not a weak
compact submanifold in H (even in the radial case).
System (1.3) can be viewed naturally as a counterpart to the following scalar equation, that is
− ∆u + λu = |u|p−2 u in RN , kuk22 = a2 . (1.4)
which has received a lot of attention in recent years. Over the past decades, many articles have investigated
the existence of normalization solutions for equation (1.4) and systems (1.2), proposing various methods to
overcome the above difficulties. We refer the reader to [37–40, 54, 55, 57] for scalar equations, [7–11, 25, 28, 43]
for system of two equations, and [46, 51] for system of k equations. However, these papers on system mainly
focus on the Sobolev subcritical case, and there are few results on the Sobolev critical system.
Recently, Bartsch, Li and Zou [12] investigate the existence of normalized solutions to the system (1.3)
with a Sobolev critical nonlinearity p = 2∗ and a subcritical coupling term 2 < α + β < 2∗ when N = 3, 4.
They proposed the issue of multiplicity as an open problem in [12, Remark 1.3]. To the best of our knowledge,
there are no paper consider existence, multiplicity of normalized solutions to system (1.3) under the following
assumptions
N ≥ 3, a, b > 0, ω1 , ω2 , ν ∈ R \ {0} and 2 < p ≤ 2∗ , α + β = 2∗ , α, β > 1. (1.5)
The research we present here is a contribution to improve the picture of the Sobolev critical situation.

3
 First, for the defocusing case ω1 < 0 and ω2 < 0, we observe that the following non-existence results.
Theorem 1.1. Let N ≥ 3, ω1 < 0 and ω2 < 0. Assuming that the exponents satisfy 2 < p < 2∗ and α+β = 2∗
with (
α > 1, β > 1, if ν > 0,
(1.6)
α ≥ 2, β ≥ 2, if ν < 0,
then we have the following
(1) if N = 3, 4, then the system (1.3) has no positive normalized solution (u, v) ∈ H 1 (RN ) × H 1 (RN ).
(2) if N ≥ 5, then the system (1.3) has no positive normalized solution (u, v) ∈ H 1 (RN ) × H 1 (RN ) satisfying
the additional assumption that u, v ∈ Lq (RN ) for some q ∈ (0, N/(N − 2)].

Therefore, by Theorem 1.1, we will now focus on the focusing case ω1 > 0 and ω2 > 0. Without loss of
generality, we always assume ω1 = ω2 = 1 for simplicity. To present our results regarding the Sobolev critical
system (1.3), we first need to introduce some established results in [54] for the scalar equation (1.4), which play
a crucial role in our argument. In variational approach, normalized solutions of (1.4) are obtained as critical
points of the associated energy functional
Z Z
1 1
E(u) := |∇u|2 dx − |u|p dx, for u ∈ H 1 (RN ),
2 RN p RN

on the constraint S(a) = u ∈ H 1 (RN ) : kuk22 = a2 . Moreover, the least energy is defined by
n o
e(a) := inf E(u) = inf E(u) : u ∈ H 1 (RN ), kuk22 = a2 and k∇uk22 = γp kukpp , (1.7)
u∈S(a)

where the parameter γp is denoted as

N (p − 2)
γp :=. (1.8)
2p
We then have the following results, which are established in [54].

Theorem A. Let N ≥ 3, p ∈ (2, 2∗ ) \ 2 + N4 . Then up to a translation, scalar equation (1.4) has a unique
positive normalized solution up with Lagrange multiplier λ > 0, and
(1) if p < 2 + N4 , then
e(a) = inf 2
E(u) = E(up ) < 0;
kuk2 =a2
4
(2) if p > 2 + N, then
N N
e(a) = inf
2
max E(e 2 t u(et ·)) = max E(e 2 t up (et ·)) = E(up ) > 0;
kuk2 =a2 t∈R t∈R

Moreover, in both cases, the least energy e(a) is strictly decreasing with respect to a > 0.

From a variational point of view, besides the Sobolev critical exponent 2∗ = 2N/(N − 2) for N ≥ 3 and
∗
2 = +∞ for N = 1, 2, a new L2 -critical exponent p̄ = 2 + 4/N arises that plays a pivotal role in the study
of normalized solutions to (1.3). This threshold significantly impacts the structure of functional I(u, v) (or
E(u), respectively) on the constraint T (a, b) (or S(a), respectively), and, consequently, influences the choice
of approaches when searching for constrained critical points. We divide our results into three situation: (1)
L2 -subcritical case 2 < p < 2 + 4/N ; (2) L2 -critical and L2 -supercritical (but Sobolev subcritical) case
2 + 4/N ≤ p < 2∗ ; (3) Sobolev critical case p = 2∗ .

4
1.1 L2 -subcritical 2 < p < 2 + N

Now we consider the case ν > 0 of attractive interaction. It is straightforward to verify that the functional
I(u, v) becomes unbounded from below on T (a, b) when ν > 0. To establish the existence of normalized
solutions to (1.12) in such cases, we draw inspiration from [12, 40, 55]. Specifically, solutions of (1.12) satisfy
the Pohozaev identity
Z Z Z


P (u, v) := |∇u|2 + |∇v|2 dx − γp (|u|p + |v|p ) dx − ν |u|α |v|β dx = 0. (1.9)
RN RN RN

4
Setting the Pohozaev manifold
n o
P(a, b) := (u, v) ∈ T (a, b) : P (u, v) = 0 .

Then P(a, b) 6= ∅. The Pohozaev identity implies that any solution of (1.3) belongs to P(a, b), so that if (u, v)
is a minimizer of the following minimization problem

m(a, b) := inf I(u, v).

(u,v)∈P(a,b)

then (u, v) is a normalized ground state solution of (1.3), that is m(a, b) = mg (a, b). By introducing a L2 -
norm invariant transformation and a suitable fiber map (detailed in Section 1.4), we decompose the Pohozaev
manifold into three submanifolds:

P(a, b) = P + (a, b) ∪ P 0 (a, b) ∪ P − (a, b).

If 2 < p < 2 + 4/N and ν > 0, the fiber map exhibits both convex and concave geometries under certain
conditions, such as when the total mass a2 + b2 is small. Such specific geometric structure plays a crucial role
in searching for the multiplicity of normalized solutions.
We now state that the multiplicity results of normalized solutions to system (1.12) for the L2 -subcritical
case. These solutions can be characterized as the local minimizer (normalized ground state) and the mountain
pass solution (normalized excited state) of functional I on the constraint P(a, b) respectively. More precisely,
we have the following results.
Theorem 1.2. Let N ≥ 3, ν > 0 and the exponents satisfy α > 1, β > 1, α + β = 2∗ and
(
2 + N4 , when N = 3, 4,
2<p< 2
2 + N −2 , when N ≥ 5.

There exists a constant C0 > 0, such that if ν satisfies

2−pγp  −1
0<ν 2∗ −2 < C0 ap(1−γp ) + bp(1−γp ) , (1.10)

then we have
(1) I|T (a,b) has a critical point of local minimum type (u+ , v+ ) at negative level

m(a, b) = inf I(u, v) < 0,

P + (a,b)

which is also a normalized ground state of system (1.3).

(2) I|T (a,b) has a second critical point of mountain pass type (u− , v− ) at positive level

l(a, b) = inf I(u, v) > 0,

P − (a,b)

which is also a normalized excited state of system (1.3)

(3) Both components of (u+ , v+ ) and (u− , v− ) are positive functions, are radially symmetric, and solve
system (1.3) for suitable λ1,+ , λ2,+ > 0 and λ1,− , λ2,− > 0.
Remark 1.1. Actually, we can give the explicit expression of C0 ,
2∗ (2−pγp )
  ∗
2−pγp
2(2∗ −2)
∗
1 p(2 + 2 − pγp ) (2 − 2)(2 − pγp ) 2∗ −2 Sα,β
C0 = min , ,
γp 22∗ C(N, p)(2∗ − pγp )
2∗ −pγp
2∗ −2

where Sα,β is defined by (1.19), and C(N, p) is defined by (1.20).

To our best knowledge, Theorem 1.2 appears to be the first multiplicity result of normalized solutions to
Sobolev critical system (1.3). In this part, we outline the ideas for establishing the key multiplicity of solutions.

5
 On the one hand, in order to obtain the local minimum type solution (u+ , v+ ) of the functional I(u, v), we
first compare the ground state m(a, b) with the local minimum energy (see Lemma 2.2), namely:

m(a, b) = inf I(u, v).

AR0 (a,b)

We then establish an upper estimate of the normalized ground state level m(a, b) < min {e(a), e(b)} < 0, which
allows us to recover the compactness of Palais-Smale sequence at level m(a, b) and obtain a local minimum
type normalized ground state.
On the other hand, since the functional I(u, v) is unbounded from below on T (a, b), the structure of the
functional suggests that there may exist another normalized solution at the mountain pass level. To obtain the
second solution, we need to construct a mountain pass level σ1 (see (2.12)), and find a Palais-Smale sequence at
such level by applying the Ghoussoub min-max principle described in [24, Section 5]. Then using the Pohozaev
identity, we can show that such Palais-Smale sequence is bounded in H. However, the weak limit of Palais-
Smale sequence in H may potentially possess vanishing components. To overcome this difficulty, a precise
threshold for the mountain pass level is required to ensure that the weak limit for the Palais-Smale sequence
has two nontrivial components. More precisely, the mountain pass level σ1 must be smaller than m(a, b) +
ν −(N −2)/2 N/2
N Sα,β , where Sα,β is defined by (1.19). In this paper, the energy estimate can be established by
observing the following result (see Lemma 2.3),

σ1 = inf I(u, v) := l(a, b).

P + (a,b)

The proof of energy estimate (2.16) depends on the choice of suitable test function; see Lemma 2.8 and 2.9. Our
approach is inspired by the pioneering work [40, 57], which focuses on the multiplicity of normalized solutions
to the following scalar equation with combined nonlinearity
∗
−2
− ∆u + λu = µ|u|q−2 u + |u|2 u in RN , kuk22 = a2 .

However, due to the presence of coupling terms, the geometry of the Pohozaev manifold for system (1.3) is
more complicated than that for (1.4), which means that the methods used in [40, Propositions 1.15 and 1.16]
and [57, Lemma 3.1] cannot be applied directly here. Thus, we require more careful calculations in this paper.

For the L2 -subcritical case p < 2 + N4 , we find that the Sobolev critical system (1.3) under assumption
(1.5) is more interesting than other cases. This is reflected not only in the multiplicity result for the attractive
case ν > 0 in Theorem 1.2, which reveals a complex structure of the corresponding functional I(u, v), but
also in the following existence result for the repulsive case ν < 0. This contrasts with the non-existence result
for ν < 0 presented in [12] for the system (1.3) with a Sobolev critical nonlinearity p = 2∗ and a subcritical
coupling term 2 < α + β < 2∗ .
Our next existence result can be stated as follows.
Theorem 1.3. Let N = 3 and ν < 0. Assuming that the exponents satisfy
4
2<p<2+ , and p ≤ α, β < 2∗ , α + β = 2∗ ,
N
we get that I|Tr (a,b) has a critical point (ū, v̄) at negative level

mr (a, b) := inf I(u, v) < 0,

(u,v)∈Pr (a,b)

where Tr (a, b) = T (a, b) ∩ Hrad and Pr (a, b) := P(a, b) ∩ Hrad (Hrad is the subspace of radial symmetric
functions in H). Moreover, (ū, v̄) is a normalized solution with least energy among all radially symmetric
normalized solution of (1.3) with Lagrange multipliers λ̄1 , λ̄2 > 0.

4
1.2 L2 -critical and L2 -supercritical 2 + N
≤ p < 2∗
4
Now we focus on the case 2 + N ≤ p < 2∗ and ν > 0. Our results are as follows.

6
Theorem 1.4. Let N = 3, 4, ν > 0, and the exponents satisfy
4
α > 1, β > 1, α + β = 2∗ , 2 + ≤ p < 2∗ .
N
We further assume that
2 4
a4/N + b4/N < (1 + )C(N, p), when p = 2 + .
N N
where C(N, p) is the best constant in Gagliardo-Nirenberg inequality (1.20). If one of the following conditions
holds:
(C1) there exists a0 > 0 such that 2 + N4 < p < 2∗ and a, b ≤ a0 ;
(C2) there exists ν1 = ν1 (a, b, α, β) > 0 such that ν > ν1 ;
(C3) a ≤ b and α < 2;
(C4) b ≤ a and β < 2,
then I|T (a,b) has a critical point of mountain pass type (û− , v̂− ) at positive level m(a, b) > 0, which is also a
normalized ground state of (1.3) with λ̂1 , λ̂2 > 0.
4
Remark 1.2. Actually, when 2 + N < p < 2∗ , for fixed ν > 0, the constant a0 is defined by
  p(1−γ
1
 −(N −2)/2
 2−pγ
2
p p)
1 2pγp ν N/2
a0 = a0 (ν) :=  S  . (1.11)
γp C(n, p) N (pγp − 2) α,β

By the properties of e(a) in Theorem A, a0 is actually choosed to satisfy

ν −(N −2)/2 N/2

e(a0 ) = Sα,β .
N
Therefore, under condition (C1), we obtain e(a), e(b) ≥ e(a0 ). Furthermore, by Lemma 2.13, it can be
observed that under conditions (C1)-(C4), there holds m(a, b) < min {e(a), e(b)}.
Remark 1.3. Since T (a, b) is not a weak compact submanifold in H, a crucial step in recovering the the
compactness of Palais-Smale is to prove that the Lagrange multipliers λ1 > 0, λ2 > 0. As shown in Proposition
2.1, this step can be achieved through a contradiction argument and Liouville-type theorems (see [35, Lemma
A.2] and [49, Theorem 8.4]), which are applicable only for N = 3, 4 when 2 + N4 ≤ p < 2∗ . This is the only
reason we consider the cases N = 3, 4 in Theorem 1.4. Finally, we would like to emphasize that the key energy
estimates in Lemmas 2.12 and 2.13 remain valid for N ≥ 5.

1.3 Sobolev critical p = 2∗

Our previous results primarily address the case 2 < p < 2∗ and α + β = 2∗ . We recall that the existence of
normalized ground state solution to (1.3) with the case p = 2∗ and 2 < α+β < 2∗ has already been investigated
in [12]. Moreover, the case where 2 < p < 2∗ and 2 < α + β < 2∗ has been thoroughly studied in [43]. A
natural question is what happens in the fully Sobolev critical case p = α + β = 2∗ . Does a normalized solution
still exist in this particular case? We will provide an answer to this problem in this subsection.
Before proceeding, we introduce some results concerning the following coupled Sobolev critical system
 αν

 − ∆u = |u|2 −2 u + ∗ |u|α−2 |v|β u, in RN ,
∗


 2

2∗ −2 βν α β−2 (1.12)
 − ∆v = |v| v + ∗ |u| |v| v, in RN ,

 2

u, v ∈ D1,2 (RN ), N ≥ 3.

Such system has been widely studied in these years, we refer to [15–17, 26, 31] and references therein. Define
the Sobolev space D := D1,2 (RN ) × D1,2 (RN ) and the Nehari manifold
 Z 
2∗ 2∗
N := (u, v) ∈ D \ {(0, 0)} : k∇uk22 + k∇vk22 = kuk2∗ + kvk2∗ + ν |u|α |v|β dx .
RN

7
It is easy to see that any nontrivial solution of (1.12) belongs to the Nehari manifold N . A solution (u0 , v0 ) is
called ground state solution if it has least energy among all nontrivial solutions, that is,
I(u0 , v0 ) = inf I(u, v) =: C . (1.13)
(u,v)∈N

Moreover, in a standard way, there holds

2∗
1 k∇uk22 + k∇vk22 2∗ −2
C= inf   2∗2−2 . (1.14)
(u,v)∈D\{(0,0)} N 2∗ 2∗ R
α
kuk2∗ + kvk2∗ + ν RN |u| |v| β

The existence and classification results for ground state solution were shown in [31]. Following the notations
in [31], we define the polynomial F : R2 → R by
∗ ∗
F (x1 , x2 ) = |x1 |2 + |x2 |2 + ν|x1 |α |x2 |β ,
and denote by X the set of solutions to the maximization problem
F (x̃1 , x̃2 ) = Fmax := max F (x1 , x2 ), with x̃21 + x̃22 = 1.
x21 +x22 =1

Then it follows from [31] that the system (1.12) has a ground state solution of the form
   
− N 4−2
eε,y , Veε,y := x̃1 Fmax − N −2
U Uε,y , x̃2 Fmax4 Uε,y , where y ∈ RN , (x̃1 , x̃2 ) ∈ X , ε > 0, (1.15)

and Uε,y is the Aubin-Talenti bubble, see (1.18) ahead. Therefore it follows from (1.14) that

C = I(U eε,y , Veε,y ) = 1 Fmax

− N 2−2 N
S2. (1.16)
N
Our first existence results for N ≥ 5 are as follows.
Theorem 1.5. Let N ≥ 5, ν > 0 and the exponents satisfy α > 1, β > 1, α + β = p = 2∗ . We further assume
that x̃1 6= 0, x̃2 6= 0 and the mass a, b satisfy that a|x̃2 | = b|x̃1 |, then
1 − N 2−2 N
m(a, b) = inf I(u, v) =Fmax S 2 = C.
(u,v)∈P − (a,b) N
 
Moreover, system (1.3) has a normalized ground state solution, given by U eε0 ,y , Veε0 ,y defined in (1.15) for
y ∈ RN and the e e
 unique choice of ε0 > 0 such that kUε0 ,y k2 = a and kVε0 ,y k2 = b. Furthermore, the function
e e
Uε0 ,y , Vε0 ,y solves (1.3) with λ1 = λ2 = 0.
Remark 1.4. The assumptions x̃1 6= 0, x̃2 6= 0 guarantee that the system (1.12) possesses a ground state
solution with two nontrivial components. Particularly, for a special case 2α = 2β = 2∗ , Chen and Zou [18]
proved that there exists a constant ν̃ > 0 such that the following nonlinear problem

 ν 2∗ 2∗
k 2 −2 + ∗ k 2 −2 l 2 = 1,
∗

2
 ν 2∗ 2∗ −2
+ l2 −2 = 1.
∗
 k l 2 2
2 ∗

has a positive solution (k0 , l0 ) with k0 > 0, l0 > 0 if ν > ν̃. In this case, the ground state solution of (1.12) is
unique of the explicit expression (k0 Uε,y , l0 Uε,y ). Therefore, as an application of Theorem 1.5, if we assume
that al0 = bk0 then system (1.3) has a normalized ground state solution (k0 Uε,y , l0 Uε,y ) ∈ T (a, b) for y ∈ RN
and suitable value of ε > 0.

For the special case N = 3, 4, we have the following non-existence result.

Theorem 1.6. Let N = 3, 4, ν > 0 and the exponents satisfy α > 1, β > 1, α + β = p = 2∗ , then system (1.3)
has no positive normalized solution.
Remark 1.5. Theorem 1.5 together with Theorem 1.6 provide an answer to the question whether a normalized
solution to system (1.3) still exist in the Sobolev critical case p = α+β = 2∗ . The distinction between N = 3, 4
and N ≥ 5 is significant, as it reflects the fact that the Aubin-Talenti bubble Uǫ,y belongs to L2 (RN ) if and only
if N ≥ 5, and this distinction plays a crucial role in the analysis.

8
1.4 Notations
Throughout this paper, we always use the notations kukp to denote the Lp -norm of u, and we simply denote
H := H 1 (RN ) × H 1 (RN ). Let Hrad be the subspace of radial symmetric functions in H. We use A ∼ B
to represent C1 B ≤ A ≤ C2 B for some positive constants C1 , C2 > 0. We use “→” and “⇀” to denote the
strong convergence and weak convergence in corresponding space respectively. The capital letter C will appear
as a constant which may vary from line to line.
Sobolev inequality. Recall the best Sobolev embedding constant

k∇uk22
S := inf 2 , (1.17)
u∈D 1,2 (RN )\{0} kuk2∗

R
1
where D1,2 (RN ) is the completion of C0∞ (RN ) with respect to the norm kuk := RN |∇u|2 dx 2 . It is well
known that S is achieved by u if only if
( p )
N (N − 2)ε
N2−2 N
u ∈ Uε,y (x) : Uε,y (x) = 2 , ε > 0, y ∈ R . (1.18)
ε + |x − y|2

Moreover, ∗ N
k∇Uε,y k22 = kUε,y k22∗ = S 2 .
To simplify the notations, we denote Uε (x) := Uε,0 (x). Define
R 2 2


RN |∇u| + |∇v| dx
Sα,β := inf R
2/2∗ , (1.19)
u,v∈D 1,2 (RN )\{0} α β
|u| |v| dx
RN

then from [4] we know that

 β/2∗  α/2∗ !
α β
Sα,β = + S,
β α
where S is defined by (1.17).
Gagliardo-Nirenberg inequality. Recall the Gagliardo-Nirenberg inequality
1−γp γ
kukp ≤ C(N, p)kuk2 k∇uk2p , ∀u ∈ H 1 (RN ), (1.20)

where γp is defined by (1.8), and C(N, p) is the sharp constant satisfying

1−γp γ
kuk2 k∇uk2p γp 1 γp 1− 2
C(N, p)−1 = inf = (pγp ) 2 (1 − pγp ) p − 2 kZk2 p . (1.21)
u∈H 1 (RN )\{0} kukp

Here Z is the unique solution of



 − ∆Z + Z = |Z|p−2 Z in RN ,

Z > 0 and Z(x) → 0 as |x| → ∞,


Z(0) = max Z(x).
x∈RN

For more details, we refer to [58]. Moreover,the function Z κ,ρ (x) := κZ(ρx) satisfies

−∆Z κ,ρ + ρ2 Z = κp−2 ρ2 |Z κ,ρ |p−2 Z κ,ρ in RN ,

and (1.21) is achieved by u if and only if


u ∈ Z κ,ρ (· + y) : κ > 0, ρ > 0, y ∈ RN .
N
An importmant transformation. We introduce a L2 -norm invariant transformation t⋆u(x) := e 2 t u(et x)
and
(u, v) ∈ T (a, b) 7→ t ⋆ (u, v) := (t ⋆ u, t ⋆ v) ∈ T (a, b).

9
We define the following fiber map
Z Z
1
1
Φ(u,v) (t) := I(t ⋆ (u, v)) = e2t |∇u|2 + |∇v|2 dx − epγp t (|u|p + |v|p ) dx
2 R N p R N
Z (1.22)
ν 2∗ t α β
− ∗e |u| |v| dx.
2 RN

By a direct computation, we observe that Φ′(u,v) (t) = P (t⋆(u, v)), where P (u, v) is defined by (1.9). Therefore,
it holds n o
P(a, b) = (u, v) ∈ T (a, b) : Φ′(u,v) (0) = 0 .

In this direction, we decompose P(a, b) into three disjoint submanifolds P(a, b) = P + (a, b) ∪ P 0 (a, b) ∪
P − (a, b), which are given by
n o
P + (a, b) := (u, v) ∈ P(a, b) : Φ′′(u,v) (0) > 0 ,
n o
P 0 (a, b) := (u, v) ∈ P(a, b) : Φ′′(u,v) (0) = 0 ,
n o
P − (a, b) := (u, v) ∈ P(a, b) : Φ′′(u,v) (0) < 0 .

1.5 Structure of the paper

In the remaining sections of this paper, we provide proofs for our main results. In Section 2, we address
the attractive case, where ν > 0, and present the proofs for Theorem 1.2 in Subsection 2.1 and for Theorem
1.4 in Subsection 2.2. Moving on to Section 3, we consider the repulsive case ν < 0, and provide the proof for
Theorem 1.3. In Section 4, we establish the non-existence result for the defocusing case, as stated in Theorem
1.1. Lastly, in Section 5, we explore the Sobolev critical case with p = α + β = 2∗ and present the proofs for
both Theorem 1.5 and Theorem 1.6

2 Existence for the attractive case ν > 0

In this section, we study the existence of normalized solution of (1.3) for the attractive case ν > 0.
Throughout this section, we always work under the assumptions (1.5). To obtain the compactness of Palais-
Smale sequence, we need the following additional assumptions
2
2 < p < 2∗ when N = 3, 4; 2<p<2+ when N ≥ 5, (2.1)
N −2
2 4
a4/N + b4/N < (1 + )C(N, p) when p = 2 + . (2.2)
N N
Now we give the following compactness result for ν > 0.
Proposition 2.1. Assume that (2.1), (2.2) hold and
m(a, b) ≤ m(a1 , b1 ) for any 0 < a1 ≤ a, 0 < b1 ≤ b.
Let {(un , vn )} ⊂ T (a, b) be a sequence consisting of radial symmetric functions such that
I ′ (un , vn ) + λ1,n un + λ2,n vn → 0 for some λ1,n , λ2,n ∈ R, (2.3)
I(un , vn ) → c, P (un , vn ) → 0,
u− −
n , vn → 0, a.e. in RN , (2.4)
with
c 6= 0, c 6= e(a), c 6= e(b), (2.5)
and
ν −(N −2)/2 N/2
c< Sα,β + min { 0, e(a), e(b), m(a, b) } . (2.6)
N
1
Then there exists u, v ∈ Hrad (RN ), u, v > 0 and λ1 , λ2 > 0 such that up to a subsequence (un , vn ) → (u, v)
in H (R ) × H (R ) and (λ1,n , λ2,n ) → (λ1 , λ2 ) in R2 .
1 N 1 N

10
Proof. The proof is divided into three steps.
Step 1.) We show that {(un , vn )} is bounded in H 1 (RN ) × H 1 (RN ).
If 2 < p < 2 + N4 , there is pγp < 2. Then using P (un , vn ) → 0, one can see from the Gagliardo-
Nirenberg inequality that for n large enough,
1
c + 1 ≥ I(un , vn ) − P (un , vn )
2∗
1
2∗ − p

= k∇un k22 + k∇vn k22 − ∗ kun kpp + kvn kpp
N 2 p
1

pγp /2
≥ k∇un k22 + k∇vn k22 − C k∇un k22 + k∇vn k22 ,
N
4
for some C > 0, which implies that {(un , vn )} is bounded. Now if 2 + N ≤ p < 2∗ , then pγp ≥ 2 and for n
large enough
1
c + 1 ≥ I(un , vn ) − P (un , vn )
2 Z
pγp − 2 p p

ν
= kun kp + kvn kp + |un |α |vn |β dx.
2p N RN
Combining (2.2) with the fact that P (un , vn ) = o(1), we conclude that {(un , vn )} is bounded. Moreover, from
1 1
λ1,n = − I ′ (un , vn )[(un , 0)] + o(1) and λ2,n = − I ′ (un , vn )[(0, vn )] + o(1),
a b
we know that λ1,n , λ2,n are also bounded. So there exists u, v ∈ H 1 (RN ), λ1 , λ2 ∈ R such that up to a
subsequence
(un , vn ) ⇀ (u, v) in H 1 (RN ) × H 1 (RN ),
(un , vn ) → (u, v) in Lq (RN ) × Lq (RN ), for 2 < q < 2∗ ,
(un , vn ) → (u, v) a.e. in RN ,
(λ1,n , λ2,n ) → (λ1 , λ2 ) in R2 .
Then (2.3) and (2.4) give that (
I ′ (u, v) + λ1 u + λ2 v = 0,
(2.7)
u ≥ 0, v ≥ 0,
and hence P (u, v) = 0.
Step 2.) We show that the weak limit u 6= 0 and v 6= 0.
Without loss of generality, we assume that u = 0 by contradiction. There are two cases. If v = 0, then
from P (un , vn ) = o(1) we obtain that
Z
−2∗ /2
2∗ /2
k∇un k22 + k∇vn k22 = ν |un |α |vn |β dx + o(1) ≤ νSα,β k∇un k22 + k∇vn k22 + o(1).
RN

−2∗ /2 −2∗ /2
Assuming k∇un k22 + k∇vn k22 → l ≥ 0, we immediately conclude l ≤ νSα,β l , which gives that l = 0
N/2
or l ≥ ν −(N −2)/2 Sα,β . As a consequence c = 0 or

l ν −(N −2)/2 N/2

c = lim I(un , vn ) = ≥ Sα,β ,
n→∞ N N
and both of them are contradictions. Now if v 6= 0, by maximum principle we know that
(
− ∆v + λ2 v = v p−1 , in RN ,
v > 0.

Using (2.1), [35, Lemma A.2] and [49, Theorem 8.4], we obtain λ2 > 0. Let v̄n = vn − v. Then

k∇vn k22 = k∇v̄n k22 + k∇vk22 + o(1),

11
 kvn kpp = kvkpp + o(1),
Z Z
|un |α |vn |β dx = |un |α |v̄n |β dx + o(1).
RN RN
It follows that
Z
o(1) = P (un , vn ) = k∇un k22 + k∇v̄n k22 − ν |un |α |v̄n |β dx + P (u, v) + o(1)
RN
Z
2 2
= k∇un k2 + k∇v̄n k2 − ν |un |α |v̄n |β dx + o(1).
RN

N/2
Similar as before, there holds k∇un k22 + k∇v̄n k22 → 0 or lim inf n→∞ k∇un k22 + k∇v̄n k22 ≥ ν −(N −2)/2 Sα,β .
If k∇un k22 + k∇v̄n k22 → 0, i.e., un , v̄n → 0 in D1,2 (RN ), then

k∇v̄n k22 + λ2 kv̄n k22

= (I ′ (un , vn ) + λ1,n un + λ2,n vn ) [(0, v̄n )] − (I ′ (u, v) + λ1 u + λ2 v) [(0, v̄n )] + o(1)
= o(1).

That is vn → v in H 1 (RN ). As a result,

1

c = lim I(un , vn ) = lim k∇un k22 + k∇v̄n k22 + E(v) = e(b),
n→∞ n→∞ N

N/2
which is a contradiction. On the other hand, if lim inf n→∞ k∇un k22 + k∇v̄n k22 ≥ ν −(N −2)/2 Sα,β , we have
again
1
ν −(N −2)/2 N/2
c ≥ lim k∇un k22 + k∇v̄n k22 + e(kvk2 ) ≥ Sα,β + e(b),
n→∞ N N
a contradiction.
Step 3.) We show the strong convergence.
Let (ūn , v̄n ) = (un − u, vn − v). Then
Z
o(1) = P (un , vn ) = k∇ūn k22 + k∇v̄n k22 − ν |ūn |α |v̄n |β dx + o(1).
RN

Similar to before, there are two cases

N/2
whether k∇ūn k22 + k∇v̄n k22 → 0 or lim inf k∇ūn k22 + k∇v̄n k22 ≥ ν −(N −2)/2 Sα,β .
n→∞

If the second case occur, then

1
ν −(N −2)/2 N/2

c ≥ lim k∇ūn k22 + k∇v̄n k22 + m(kuk2 , kvk2 ) ≥ Sα,β + m(a, b),
n→∞ N N
which is a contradiction. So k∇ūn k22 + k∇v̄n k22 → 0, i.e., un , v̄n → 0 in D1,2 (RN ). Moreover by maximum
principle, (u, v) is a positive solution of (2.7), and from [35, Lemma A.2] and [49, Theorem 8.4], we obtain
immediately λ1 , λ2 > 0. Noting that

k∇ūn k22 + λ1 kūn k22 + k∇v̄n k22 + λ2 kv̄n k22

= (I ′ (un , vn ) + λ1,n un + λ2,n vn ) [(ūn , v̄n )] − (I ′ (u, v) + λ1 u + λ2 v) [(ūn , v̄n )] + o(1)
= o(1),

we obtain (un , vn ) → (u, v) in H 1 (RN ) × H 1 (RN ). We complete the proof.

12
 4
2.1 The case 2 < p < 2 + N
Recall the fiber map defined by (1.22)
Z Z Z
1
1 ν ∗
Φ(u,v) (t) := e2t |∇u|2 + |∇v|2 dx − epγp t (|u|p + |v|p ) dx − ∗ e2 t |u|α |v|β dx.
2 R N p R N 2 R N

We define
1 2 ∗
h(ρ) := ρ − Aρpγp − Bρ2
2
with  
1 1 −2∗ /2
A := C(N, p) ap(1−γp ) + bp(1−γp ) , B := ∗
νSα,β .
p 2
Then for any u, v ∈ H 1 (RN ),

1/2
I(u, v) ≥ h( k∇uk22 + k∇vk22 ).
Futher, there is
h′ (p) = ρpγp −1 (g(ρ) − pγp A)
with g(ρ) := ρ2−pγp − 2∗ Bρ2 −pγp
∗
. Let
 1/(2∗ −2)
2 − pγp
ρ∗ := . (2.8)
2∗ (2∗ − pγp )B

It is easy to check that g(ρ) is strictly increasing in (0, ρ∗ ) and is strictly decreasing in (ρ∗ , +∞). By direct
computations, assumption (1.10) gives that g(ρ∗ ) > pγp A and h(ρ∗ ) > 0, which means that h(ρ) has only two
critical points 0 < ρ1 < ρ∗ < ρ2 with

h(ρ1 ) = min h(ρ) < 0,

0<ρ<ρ∗

h(ρ2 ) = max h(ρ) > 0.

ρ>0

Moreover, there exist R0 and R1 such that h(R0 ) = h(R1 ) = 0 and h(ρ) > 0 iff ρ ∈ (R0 , R1 ).
Using (1.10) again, we can also prove in a standard way that P 0 (a, b) = ∅, and P(a, b) is a smooth
manifold of codimension 2 in T (a, b), see [54, Lemma 5.2] (or [8]) for more details. This fact can in turn be
used in the following lemma.
Lemma 2.1. For every (u, v) ∈ T (a, b), Φ(u,v) (t) has exactly two critical points t+ (u, v) < t− (u, v) and two
zeros c(u, v) < d(u, v) with t+ (u, v) < c(u, v) < t− (u, v) < d(u, v). Moreover,
(a) t ⋆ (u, v) ∈ P + (a, b) if and only if t = t+ (u, v); t ⋆ (u, v) ∈ P − (a, b) if and only if t = t− (u, v),

1/2
(b) k∇t ⋆ uk22 + k∇t ⋆ vk22 ≤ R0 for every t ≤ c(u, v) and
n
1/2 o
I(t+ (u, v) ⋆ (u, v)) = min I(t ⋆ (u, v)) : t ∈ R and k∇t ⋆ uk22 + k∇t ⋆ vk22 ≤ R0 < 0,

(c) Φ(u,v) (t) is strictly dereasing and concave on (t− (u, v), +∞) and

Φ(u,v) (t− (u, v)) = max Φ(u,v) (t),

t∈R

(d) the maps (u, v) 7→ t+ (u, v) and (u, v) 7→ t− (u, v) are of class C 1 .
Proof. The proof is completely analogue to the one in [54, Lemma 5.3], so we omit the details.
For any R > 0, let
n
1/2 o
AR (a, b) := (u, v) ∈ T (a, b) : k∇uk22 + k∇vk22 <R .

Lemma 2.2. The following statements hold.

(1) m(a, b) = inf AR0 (a,b) I(u, v) < 0,
(2) m(a, b) ≤ m(a1 , b1 ) for any 0 < a1 ≤ a, 0 < b1 ≤ b.

13
Proof. (1) From Lemma 2.1, we have P + (a, b) ⊂ AR0 and

m(a, b) = inf I(u, v) = inf I(u, v) < 0.

P(a,b) P + (a,b)

Obviously m(a, b) ≥ inf AR0 (a,b) I(u, v). On the other hand, for any (u, v) ∈ AR0 (a, b), there is

m(a, b) ≤ I(t+ (u, v) ⋆ (u, v)) ≤ I(u, v).

It follows that m(a, b) = inf AR0 (a,b) I(u, v).

(2) Recall the definition of ρ∗ in (2.8), we see that

m(a, b) = inf I(u, v). (2.9)

Aρ∗ (a,b)

We need only to show that for any arbitrary ε > 0, one has

m(a, b) ≤ m(a1 , b1 ) + ε.

Let (u, v) ∈ Aρ∗ (a1 , b1 ) be such that I(u, v) ≤ m(a1 , b1 ) + 2ε and φ ∈ C0∞ (RN ) be a cut-off function
(
0, if |x| ≥ 2,
0 ≤ φ ≤ 1 and φ(x) =
1, if |x| ≤ 1.

For any δ > 0, we define uδ (x) = u(x)φ(δx) and vδ (x) = v(x)φ(δx). Then (uδ , vδ ) → (u, v) in H as δ → 0.
As a consequence, we can fix a δ > 0 small enough such that
ε
1/2
I(uδ , vδ ) ≤ I(u, v) + and k∇uδ k22 + k∇vδ k22 < ρ∗ − η, (2.10)
4
for a small η > 0. Now we take ϕ ∈ C0∞ (RN ) such that supp(ϕ) ⊂ B(0, 1 + 4δ ) \ B(0, 4δ ), where B(0, r) is
the ball with a radius of r and a center at the origin. Set
p p
a2 − kuδ k22 b2 − kuδ k22
wa = ϕ and wb = ϕ.
kϕk2 kϕk2
Then for any λ < 0, noting that

(supp(uδ ) ∪ supp(vδ )) ∩ (supp(λ ⋆ wa ) ∪ supp(λ ⋆ wb )) = ∅,

1/2
one has (uδ +λ⋆wa , vδ +λ⋆wb ) ∈ T (a, b). Since I(λ⋆(wa , wb )) → 0 and k∇λ ⋆ wa k22 + k∇λ ⋆ wb k22 →
0 as λ → −∞, we have
ε
1/2 η
I(λ ⋆ (wa , wb )) ≤ and k∇λ ⋆ wa k22 + k∇λ ⋆ wb k22 ≤ (2.11)
4 2
for λ < 0 sufficiently close to negative infinity. It follows that

1/2
k∇(uδ + λ ⋆ wa )k22 + k∇(vδ + λ ⋆ wb )k22 < ρ∗ .

Now using (2.10), (2.11), we obtain

m(a, b) ≤ I(uδ + λ ⋆ wa , vδ + λ ⋆ wb ) = I(uδ , vδ ) + I(λ ⋆ wa , λ ⋆ wb ) ≤ m(a1 , b1 ) + ε,

which finish the proof.

Proof of the Theorem 1.2 (1) and (3) . First we construct a Palais-Smale sequence for IT (a,b) at level m(a, b).
By the symmetric decreasing rearrangement, one can easily check that m(a, b) = inf AR0 (a,b)∩Hrad I(u, v).
Choosing a minimizing sequence (ũn , ṽn ) ∈ AR0 (a, b) ∩ Hrad , we assume (ũn , ṽn ) are nonnegative by re-
placing (ũn , ṽn ) with (|ũn |, |ṽn |). Furthermore, using the fact that I(t+ (ũn , ṽn ) ⋆ (ũn , ṽn )) ≤ I(ũn , ṽn ), we
can assume that (ũn , ṽn ) ∈ P + (a, b) ∩ Hrad for n ≥ 1. Therefore, by Ekeland’s varational principle, there

14
is a radial symmetric Palais-Smale sequence {(un , vn )} for I|T (a,b)∩Hrad (hence a Palais-Smale sequence for
I|T (a,b) ) with the property ||(un , vn ) − (ũn , ṽn )||H → 0 as n → ∞, which implies that

P (un , vn ) = P (ũn , ṽn ) + o(1) → 0 and u− − N

n , vn → 0 a.e. in R .

Now we want to apply Proposition 2.1 with c = m(a, b), it remains to check conditions (2.5), (2.6). For
that purpose, since e(a), e(b) < 0, we only need to prove that m(a, b) < min {e(a), e(b)}. Without loss of
generality, we only prove m(a, b) < e(a). Let u0 be the unique solution in Theorem A. From h(R0 ) = 0, we
pγ 2−pγ
have 21 R02 > p1 C(N, p)ap(1−γp ) R0 p , that is R0 p > p2 C(N, p)ap(1−γp ) . It follows that
pγp 2−pγp pγ
k∇u0 k22 = γp ku0 kpp ≤ γp C(N, p)ap(1−γp ) k∇u0 k2 < R0 k∇u0 k2 p ,

which gives k∇u0 k2 < R0 . Let v0 = ab u0 . Then (u0 , −s ⋆ v0 ) ∈ AR0 (a, b) for s > 0 large enough. Therefore
for large s > 0,

e−2s e−pγp s
m(a, b) ≤ I(u0 , s ⋆ v0 ) ≤ E(u0 ) + k∇v0 k22 − kv0 kpp < e(a).
2 p
1
According to Proposition 2.1, there exists u, v ∈ Hrad (RN ), u, v > 0 and λ1 , λ2 > 0 such that up to a
subsequence (un , vn ) → (u, v) in H (R ) × H (R ) and (λ1,n , λ2,n ) → (λ1 , λ2 ) in R2 . And hence (u, v) is
1 N 1 N

a normalized ground state solution of (1.3).

Now we search for the second normalized solution of (1.3) at the mountain pass level of the Energy func-
tional. Define the energy level
l(a, b) := −inf I(u, v).
P (a,b)

According to Proposition 2.1, the first step of finding a second critical point for I|T (a,b) is to obtain a
Palais-Smale sequence {(un , vn )} ⊂ T (a, b) ∩ Hrad for I|T (a,b) at level l(a, b), with P (un , vn ) → 0 and
u− − N
n , vn → 0 a.e. in R as n → ∞ .

For that purpose, we will follow the standard way as in [38]. Set Tr (a, b) = T (a, b) ∩ Hrad . Define
n o
Γ1 := γ ∈ C([0, 1], R × Tr (a, b)) : γ(0) ∈ (0, P + (a, b)), γ(1) ∈ (0, I 2m(a,b) ) ,

where we denote the sublevel set by I c := {(u, v) ∈ H : I(u, v) ≤ c}.

Lemma 2.3. Let
σ1 := inf max J(γ(t)) (2.12)
γ∈Γ1 t∈[0,1]

with J(s, (u, v)) := I(s ⋆ (u, v)). Then l(a, b) = σ1 .

Proof. Indeed, for any (u, v) ∈ P − (a, b), let γ(t) := (0, ((1 − t)t+ (u, v) + tt0 ) ⋆ (u, v)) with t0 > 0 large
enough. So γ ∈ Γ1 , and hence

σ1 ≤ max J(γ(t)) ≤ max Φ(u,v) (t) = I(u, v),

t∈[0,1] t∈R

which gives σ1 ≤ l(a, b). On the other hand, for any γ = (γ1 , γ2 ) ∈ Γ1 , we can assume that γ1 = 0. Since
γ2 (0) ∈ P + (a, b) we have t− (γ2 (0)) > t+ (γ2 (0)) = 0; since I(γ2 (1)) ≤ 2m(a, b) we have t− (γ2 (1)) < 0 (in
fact, if t− (γ2 (1)) ≥ 0, then m(a, b) ≤ I(γ2 (1)) ≤ 2m(a, b), which is impossible). It follows that there exists
τγ ∈ (0, 1) such that t− (γ2 (τγ )) = 0, that is γ2 (τγ ) ∈ P − (a, b), and hence

l(a, b) ≤ I(γ2 (τγ )) ≤ max J(γ(t)),

t∈[0,1]

which gives l(a, b) ≤ σ1 . Then we obtain that l(a, b) = σ1 . We conclude also that

γ2 ([0, 1]) ∩ P − (a, b) 6= ∅, ∀γ ∈ Γ1 . (2.13)

15
 Now we can construct a Palais-Smale sequence at level l(a, b).
Lemma 2.4. There exists a Palais-Smale sequence (un , vn ) ⊂ T (a, b) ∩ Hrad for I|T (a,b) at level l(a, b), with
P (un , vn ) → 0 and u− − N
n , vn → 0 a.e. in R as n → ∞.

Proof. Following the strategies in [24, Section 5], it is easy to check that F = {A = γ([0, 1]) : γ ∈ Γ1 } is
a homotopy stable family of compact subsets of X = R × Tr (a, b) with boundary B = (0, P + (a, b)) ∪
(0, I 2m(a,b) ). Set F = (0, P − (a, b)), then (2.13) gives that

A ∩ F \ B 6= ∅, ∀A ∈ F .

Moreover, we have
sup I(B) ≤ 0 < σ1 ≤ inf I(F ).
So the assumptions (F ′ 1) and (F ′ 2) in [24, Theorem 5.2] are satisfied. Therefore, taking a minimizing sequence
{γn = (0, βn )} ⊂ Γ1 with βn ≥ 0 a.e. in RN , there exists a Palais-Smale sequence {(sn , wn , zn )} ⊂ R ×
Tr (a, b) for J|R×Tr (a,b) at level σ1 , that is

∂s J(sn , wn , zn ) → 0 and ∂(u,v) J(sn , wn , zn ) → 0 as n → ∞, (2.14)

with the additional property

|sn | + distH ((wn , zn ), βn ([0, 1])) → 0 as n → ∞. (2.15)

Let (un , vn ) = sn ⋆ (wn , zn ). The first condition in (2.14) implies P (un , vn ) → 0, while the second condition
gives

kdI|T (a,b) (un , vn )k


= sup |dI(un , vn )[(φ, ψ)]| : (φ, ψ) ∈ T(un ,vn ) T (a, b), k(φ, ψ)kH ≤ 1

= sup |dI(sn ⋆ (wn , zn ))[sn ⋆ (−sn ) ⋆ (φ, ψ)]| : (φ, ψ) ∈ T(un ,vn ) T (a, b), k(φ, ψ)kH ≤ 1

= sup |∂(u,v) J(sn , wn , zn )[(−sn ) ⋆ (φ, ψ)]| : (φ, ψ) ∈ T(un ,vn ) T (a, b), k(φ, ψ)kH ≤ 1

≤ k∂(u,v) J(sn , wn , zn )k sup k(−sn ) ⋆ (φ, ψ)]kH : (φ, ψ) ∈ T(un ,vn ) T (a, b), k(φ, ψ)kH ≤ 1
≤ Ck∂(u,v) J(sn , wn , zn )k → 0 as n → ∞.

Finally, (2.15) implies that u− − N

n , vn → 0 a.e. in R .

According to Proposition 2.1, the subsequent step in finding a second critical point of I|T (a,b) involves
establishing the inequality:
ν −(N −2)/2 N/2
0 < l(a, b) < m(a, b) + Sα,β . (2.16)
N
It is noteworthy that for any u ∈ P − (a, b), we can deduce from Lemma 2.1 that 0 is the unique maximum point
of Φ(u,v) . Consequently, we obtain

I(u, v) = max I(t ⋆ (u, v))

t∈R

1/2
≥ max h( k∇(t ⋆ u)k22 + k∇(t ⋆ v)k2 ) = max h(t) > h(ρ∗ ) > 0,
t∈R t∈R

which means l(a, b) > 0. However, the proof of the second inequality is notably more challenging. We begin
by establishing several properties of l(a, b).
Lemma 2.5. The following statements hold.
(1) For any 0 ≤ a1 ≤ a and 0 ≤ b1 ≤ b, there is l(a, b) ≤ l(a1 , b1 ).
(2) Set n o
Γ(a, b) := γ(t) ∈ C([0, 1], T̄ (c, d)) : γ(0) ∈ V̄ (a, b), γ(1) ∈ I 2m(a,b) . (2.17)
with [ [
T̄ (a, b) := T (c, d), V̄ (a, b) := V (c, d)
c∈[a/2,a],d∈[b/2,b] c∈[a/2,a],d∈[b/2,b]

16
and n o
V (c, d) := (u, v) ∈ T (c, d) : I|′T (c,d) (u, v) = 0, I(u, v) = m(c, d) .

Then l(a, b) ≤ inf γ∈Γ(a,b) maxt∈[0,1] I(γ(t)).

Proof. (1) The proof is inspired by [39, Lemma 3.2], here we give the details for the completeness. It is
sufficient to prove that for any arbitrary ε > 0, one has

l(a, b) ≤ l(a1 , b1 ) + ε.

By the definition of l(a1 , b1 ), there exists (u, v) ∈ P − (a, b) such that

ε
I(u, v) ≤ l(a1 , b1 ) + . (2.18)
2
Let ϕ ∈ C0∞ (RN ) be radial and such that

0 ≤ ϕ ≤ 1; ϕ(x) = 1, when |x| ≤ 1; ϕ(x) = 0 when |x| ≥ 2.

For any small δ > 0, we define uδ (x) := u(x)ϕ(δx), vδ (x) := v(x)ϕ(δx). Since (uδ , vδ ) → (u, v) in H as
δ → 0+ , by Lemma 2.1, one has limδ→0+ t− (uδ , vδ ) = t− (u, v) and hence

t− (uδ , vδ ) ⋆ (uδ , vδ ) → t− (u, v) ⋆ (u, v) in H as δ → 0+ .

As a consequence, we can fix a δ > 0 small enough such that

ε
I(t− (uδ , vδ ) ⋆ (uδ , vδ )) ≤ I(u, v) + . (2.19)
4
Now take ψ ∈ C0∞ (RN ) such that supp(ψ) ⊂ B(0, 4δ )c and set
p p
a2 − kuδ k22 b2 − kvδ k22
ψa = ψ and ψb = ψ.
kψk2 kψk2

Then for any τ ≤ 0, one has

(supp(uδ ) ∪ supp(uδ )) ∩ (supp(τ ⋆ ψa ) ∪ supp(τ ⋆ ψb )) = ∅,

and hence
(ũτ , ṽτ ) := (uδ + τ ⋆ ψa , vδ + τ ⋆ ψb ) ∈ T (a, b).
Let tτ := t− (ũτ , ṽτ ) which is given in Lemma 2.1. From P (tτ ⋆ (ũτ , ṽτ )) = 0, i.e.,
Z
e(2−pγp )tτ (kũτ k2 + kṽτ k22 ) = kũτ kpp + kṽτ kpp + νe(2 −pγp )tτ
∗
|ũτ |α |ṽτ |β dx,
RN

we see that lim supτ →−∞ tτ < +∞, since (ũτ , ṽτ ) → (uδ , vδ ) 6= (0, 0) as τ → −∞. Then one has that
tτ + τ → −∞ as τ → −∞ and that for τ < −1 small enough
ε
I((tτ + τ ) ⋆ (ψa , ψb )) < . (2.20)
4
Now using (2.18), (2.19) and (2.20), we obtain

l(a, b) ≤ I(tτ ⋆ (ũτ , ṽτ ))

= I(tτ ⋆ (uδ , vδ )) + I((tτ + τ ) ⋆ (ψa , ψb ))
ε
≤ I(tν (uδ , vδ ) ⋆ (uδ , vδ )) +
4
ε
≤ I(u, v) + ≤ l(a1 , b1 ) + ε,
4
which completes the proof.

17
(2) Take a γ(t) ∈ Γ(a, b). Since γ(0) ∈ V (c, d) for some c ∈ [a/2, a] and d ∈ [b/2, b] we have t− (γ(0)) >
t+ (γ(0)) = 0; since I(γ(1)) < 2m(a, b) we have t− (γ(1)) < 0. So there exists a tγ ∈ (0, 1) such that
t− (γ(tγ )) = 0, i.e., γ(tγ ) ∈ P − (c, d) for some c ∈ [a/2, a] and d ∈ [b/2, b]. Then
max I(γ(t)) ≥ I(γ(tγ )) ≥ l(c, d) ≥ l(a, b),
t∈[0,1]

which completes the proof.

Let
ηε (x) = φUε , (2.21)
where Uε is defined by (1.18) and φ ∈ C0∞ (RN ) is a radial cut-off function
0 ≤ φ ≤ 1; φ(x) = 1 when |x| ≤ 1; φ(x) = 0 when |x| ≥ 2.
Then we have the following estimations, and the proofs can be found in [40, 55].
Lemma 2.6. As ε → 0+ , we have
k∇ηε k22 = S N/2 + O(εN −2 ), kηε k22∗
∗
= S N/2 + O(εN ),

2

O(ε ) if N ≥ 5,
2
kηε k2 = O(ε2 | log ε|) if N = 4,


O(ε) if N = 3,

N −(N −2)p/2

O(ε ) if N ≥ 4 and p ∈ (2, 2∗ ) or if N = 3 and p ∈ (3, 6),
p p/2
kηε kp = O(ε ) if N = 3 and p ∈ (2, 3),


O(ε3/2 | log ε|) if N = 3 and p = 3.

Observe that for p > 2 and α, β > 1,

(1 + t)p ≥ 1 + tp + pt, ∀t > 0,
α β β
(1 + t1 ) (1 + t2 ) ≥ 1 + tα
1 t2 , ∀t1 , t2 > 0. (2.22)
Then for any u, v > 0, t > 0, there holds
p p Z √ p 
√ √
I(u + t αηε , v + t βηε ) ≤ I(u, v) + I(t αηε , t βηε ) + t α∇u + β∇v · ∇ηε dx (2.23)
RN
Lemma 2.7. For small ε > 0,
√ √
(1) there exists a t∗ > 0 independent with ε such that maxt≥t∗ I(t αηε , t βηε ) + t ≤ 2m(a, b);
√ √ −(N −2)/2 N/2
(2) maxt>0 I(t αηε , t βηε ) < ν N Sα,β + O(εN −2 ) − Ckηε kpp
Proof. (1) Observe that
√ p α+β 2 ναα/2 β β/2 2∗ ∗
I(t αηε , t βηε ) ≤ t k∇ηε k22 − ∗
t kηε k22∗ . (2.24)
2 2
∗
We have that k∇ηε k22 → S N/2 √ and kη
2
√ε k2∗ → S
N/2
, see Lemma 2.6. Thus there exists a t∗ > 0 independent
with ε such that maxt≥t∗ I(t αηε , t βηε ) + t ≤ 2m(a, b).
√ √ −(N −2)/2 N/2
(2) In view of (2.24), if t → 0, we have I(t αηε , t βηε ) < 12 ν N Sα,β . Moreover, for a small t0 > 0,
√ p α+β 2 ναα/2 β β/2 2∗ ∗
max I(t αηε , t βηε ) ≤ max t k∇ηε k22 − ∗
t kηε k22∗ − Ckηε kpp
t≥t0 t>0 2 2
" #N/2
1 (α + β)k∇ηε k22
=
∗
∗ 2/2
− Ckηε kpp
N α/2 β/2
να β kηε k2∗ 2

−(N −2)/2
 β/2∗  α/2∗ !
ν α β
N/2
= + S + O(εN −2 ) − Ckηε kpp
N β α
ν −(N −2)/2 N/2
= Sα,β + O(εN −2 ) − Ckηε kpp .
N

18
Thus we conclude that
√ p ν −(N −2)/2 N/2
max I(t αηε , t βηε ) < Sα,β + O(εN −2 ) − Ckηε kpp .
t>0 N

Now let
a2ε := a2 − 2t2∗ kηε k22 , b2ε := b2 − 2t2∗ kηε k22 .
Clearly aε → a, bε → b as ε → 0. Take (uε , vε ) ∈ V (aε , bε ) the positive solution obtained in Theorem 1.2 (1).
Since uε , vε are radially symmetric, there holds
sup {|uε (x)|, |vε (x)|} ≤ Cε |x|−(N −2)/2 ,
x∈RN

with Cε > 0. Then using a similar approach as [40, Lemma 5.5 and 5.6], we can find a sequence yε ∈ RN such
that √
aε ≤ kuε (· − yε ) + t αηε k2 ≤ a, ∀0 ≤ t ≤ t∗ ,
p
bε ≤ kvε (· − yε ) + t βηε k2 ≤ b, ∀0 ≤ t ≤ t∗ ,
and Z √ 
p
α∇uε (· − yε ) + β∇vε (· − yε ) · ∇ηε dx ≤ kηε k22 . (2.25)
RN
Then we can prove the following energy estimate.
Lemma 2.8. If N ≥ 4, we have
ν −(N −2)/2 N/2
l(a, b) < m(a, b) + Sα,β .
N
√ √
Proof. Let γε (t) := (uε (· − yε ) + tt∗ αηε , vε (· − yε ) + tt∗ βηε ). Clearly γε ∈ Γ(a, b) for ε > 0 small, see
(2.17) and Lemma 2.7. In view of (2.23), (2.25), Lemma 2.5 and Lemma 2.7, we obtain that
√ p
l(a, b) ≤ max I(γε (t)) ≤ I(uε , vε ) + max I(t αηε , t βηε ) + t∗ kηε k22
t∈[0,1] t>0
−(N −2)/2 (2.26)
ν N/2
≤ m(aε , bε ) + Sα,β + O(εN −2 ) − Ckηε kpp + Ckηε k22
N
Now we give an upper bound estimate of m(aε , bε ). Let (u0 , v0 ) ∈ V (a, b) be the positive solution obtained in
Theorem 1.2 (1). Let wε = aaε u0 and zε = bbε v0 . Thus kwε k2 = aε , kzε k2 = bε and

1/2
1/2
k∇wε k22 + k∇zε k22 ≤ k∇u0 k22 + k∇v0 k22 < ρ∗ ,
which means (wε , zε ) ∈ Aρ∗ (aε , bε ). From (2.9), it follows that
m(aε , bε ) ≤ I(wε , zε ) = m(a, b) + I(wε , zε ) − I(u0 , v0 )
  p 
1  a p 
ε 1 bε
≤ m(a, b) + 1− ku0 kpp + 1− kv0 kpp
p a p b
!
ν  a α  b β Z
ε ε β (2.27)
+ ∗ 1− uα
0 v0 dx
2 a b RN

≤ m(a, b) + C1 (a2 − a2ε ) + C2 (b2 − b2ε )

≤ m(a, b) + Ckηε k22 .
Combining (2.26) and (2.27), we deduce from Lemma 2.6 that
ν −(N −2)/2 N/2
l(a, b) ≤ m(a, b) + Sα,β + O(εN −2 ) − Ckηε kpp + Ckηε k22
N
ν −(N −2)/2 N/2
= m(a, b) + Sα,β + O(ε2 ) − O(εN −(N −2)p/2 )
N
ν −(N −2)/2 N/2
< m(a, b) + Sα,β ,
N
where we use the fact N ≥ 4.

19
Lemma 2.9. If N = 3, we have

ν −(N −2)/2 N/2

l(a, b) < m(a, b) + Sα,β .
N
Proof. By Theorem 1.2 (1), we obtain a normalized ground state solution (u+ , v+ ) ∈ P + (a, b) of system (1.3)
with λ1,+ , λ2,+ > 0, which is positive and radially symmetric. Recall the definition of ηε (see (2.21)), we take
√ p
wε,t = u+ + t αηε , vε,t = v+ + t βηε for t > 0.

Now we define
1 1
Wε,t (x) = τ 2 wε,t (τ x), Vε,t (x) = ξ 2 vε,t (ξx). (2.28)
Since N = 3, there holds
k∇Wε,t k22 = k∇wε,t k22 , k∇Vε,t k22 = k∇vε,t k22 ,
kWε,t k22 = τ −2 kwε,t k22 , kVε,t k22 = ξ −2 kvε,t k22 , (2.29)
pγp −p pγp −p
kWε,t kpp =τ kwε,t kpp , kVε,t kpp =ξ kvε,t kpp .

By choosing
kwε,t k2 kvε,t k2
τ= ≥ 1, ξ = ≥ 1,
a b
we obtain that (Wε,t , Vε,t ) ∈ T (a, b). Then by Lemma 2.1, there exists a unique kε,t > 0 such that

 3 3

W ε,t , V ε,t := kε,t
2
Wε,t (kε,t x), kε,t
2
Vε,t (kε,t x) ∈ P − (a, b),

which implies that

Z
2

pγ
2∗
kε,t k∇Wε,t k22 + k∇Vε,t k22 = γp kε,tp kWε,t kpp + kVε,t kpp + νkε,t |Wε,t |α |Vε,t |β dx. (2.30)
R3

Since (u+ , v+ ) ∈ P + (a, b) and it follows from Lemma 2.1 that kε,0 > 1. On the other hand, since 2 < p <
2 + N4 and 0 < pγp < 2, one can deduce from (2.29), (2.30) that kε,t → 0 as t → ∞. By using Lemma 2.1
again, kε,t is continuous for t, which implies that there exists tε > 0 such that kε,tε = 1 for ε small enough.
Hence,
l(a, b) = inf I(u, v) ≤ I(W ε,tε , V ε,tε ) = I(Wε,tε , Vε,tε ) ≤ sup I(Wε,t , Vε,t ). (2.31)
(u,v)∈P − (a,b) t>0

Note that (u+ , v+ ) is the normalized ground state solution of system (1.3) and ηε is a positive function, one can
deduce from (2.29) and Lemma 2.6 that there exists t0 > 0 such that

ν −(N −2)/2 N/2 (2.32)

I(Wε,t , Vε,t ) ≤ m(a, b) + Sα,β − δ̃
N
for t < t−1
0 and t > t0 with δ̃ > 0.
On the other hand, for t0−1 ≤ t ≤ t0 , by (2.22), (2.28) and (2.29) we have
1
1 1
I(Wε,t , Vε,t ) = k∇wε,t k22 + k∇vε,t k22 − τ pγp −p kwε,t kpp − ξ pγp −p kvε,t kpp
2 p p
Z
ν α β
− ∗τ ξ
2 2 |wε,t (τ x)|α |vε,t (ξx)|β dx
2 R3
p p
α+β 2 ναα/2 β β/2 2∗ ∗ α2 + β 2 p
≤ I(u+ , v+ ) + t k∇ηε k22 − ∗
t kηε k22∗ − t kηε kpp
2 2 p (2.33)
Z p
√ 1 − τ pγp −p 1 − ξ pγp −p
+t ( α∇u+ + β∇v+ ) · ∇ηε + kwε,t kpp + kvε,t kpp
R3 p p
Z Z 
ν α β α β
α β
+ ∗ |wε,t (x)| |vε,t (x)| dx − τ ξ
2 2 |wε,t (τ x)| |vε,t (ξx)| dx
2 R3 R3
ν
=: I1 + I2 + ∗ I3 .
2

20
For t−1
0 ≤ t ≤ t0 , since I(u+ , v+ ) = m(a, b), we deduce from Lemma 2.6 and (1.19) that
 
α+β 2 2 ναα/2 β β/2 2∗ 2∗
I1 ≤ I(u+ , v+ ) + sup t k∇ηε k2 − t kηε k2∗ − Ckηε kpp
t>0 2 2∗
(2.34)
ν −(N −2)/2 N/2
≤ m(a, b) + Sα,β .
N
Note that (u+ , v+ ) is a radially symmetric solution of (1.3) and decays to zero as r → ∞, one has
Z Z 1ε   12 Z
5 1 2 1 1
u+ ηε dx ∼ ε 2
2
r dr ∼ ε , and
2 v+ ηε dx ∼ ε 2 .
R 3 1 1 + r R 3

Then we have √ Z Z
2 2 αt t2 α 1
τ =1+ u+ ηε dx + 2 |ηε |2 dx = 1 + O(ε 2 ),
a2 3 a 3
√ ZR 2 Z
R
(2.35)
2 2 βt t β 2 1
ξ =1+ 2 v+ ηε dx + 2 |ηε | dx = 1 + O(ε ).2
b R3 b R3
Since (u+ , v+ ) is a solution of system (1.3) with λ1,+ , λ2,+ > 0, it follows from a similar argument as used
in [40, Lemma 5.5] that
Z Z Z Z
p−2 αν
∇u+ · ∇ηε dx = −λ1,+ u+ ηε dx + |u+ | u+ ηε dx + ∗ |u+ |α−2 |v+ |β u+ ηε dx
R3 R3 R3 2 R3
Z Z
2
≤ −λ1,+ u+ ηε dx + Ckηε k2 = −λ1,+ u+ ηε dx + O(ε).
R3 R3

Similarly, we have Z Z
∇v+ · ∇ηε dx ≤ −λ2,+ v+ ηε dx + O(ε).
R3 R3
Therefore, combining these with (2.35) we have
Z p
√ 1 − τ pγp −p 1 − ξ pγp −p
I2 = t ( α∇u+ + β∇v+ ) · ∇ηε dx + kwε,t kpp + kvε,t kpp
R3 p p
√  Z Z 
t α
≤ 2 −λ1,+ a2 u+ ηε dx + (1 − γp )kwε,t kpp u+ ηε dx
a R3 R3
√  Z Z 
t β
+ 2 −λ2,+ b2 v+ ηε dx + (1 − γp )kvε,t kpp v+ ηε dx + O(ε).
b R3 R3


Note that the fact λ1,+ a2 + λ2,+ b2 = (1 − γp ) ku+ kpp + kv+ kpp which comes from the Pohozaev identity
P (u+ , v+ ) = 0, then it is easy to verify that
Z p
√ 1 − τ pγp −p 1 − ξ pγp −p
I2 = t ( α∇u+ + β∇v+ ) · ∇ηε dx + kwε,t kpp + kvε,t kpp = O(ε). (2.36)
R3 p p
Now we estimate
Z Z
α β
I3 = |wε,t (x)|α |vε,t (x)|β dx − τ 2 ξ 2 |wε,t (τ x)|α |vε,t (ξx)|β dx.
R3 R3

Note that τ, ξ ≥ 1,
Z
α β
I3 = (1 − τ 2 ξ 2 ) |wε,t (x)|α |vε,t (x)|β dx
R3
Z Z 
α β
α β α β
+τ 2ξ2 |wε,t (x)| |vε,t (x)| dx − |wε,t (τ x)| |vε,t (x)| dx
3 R3
ZR Z 
α β
α β α β
+τ ξ2 2 |wε,t (τ x)| |vε,t (x)| dx − |wε,t (τ x)| |vε,t (ξx)| dx
R3 R3
Z
α β
≤ τ 2 ξ 2 (1 − τ ) α|wε,t (x)|α−1 |vε,t (x)|β (x · ∇wε,t (x)) dx + o(|1 − τ |)
R3
Z
α β
+ τ 2 ξ 2 (1 − ξ) β|wε,t (τ x)|α |vε,t (x)|β−1 (x · ∇vε,t (x)) dx + o(|1 − ξ|).
R3

21
For small ε > 0 and t−1
0 < t < t0 , we may assume that τ, ξ ∈ [1, 2]. Moreover, one can easily check that
Z Z Z Z
∗ ∗ ∗ ∗
|ηε |2 dx = |η1 |2 dx, |x · ∇ηε |2 dx = |x · ∇η1 |2 dx,
R3 R3 R3 R3

Hence, with the help of Hölder inequality, it is easy to see that

Z Z
α|wε,t (x)|α−1 |vε,t (x)|β (x · ∇wε,t (x)) dx and β|wε,t (τ x)|α |vε,t (x)|β−1 (x · ∇vε,t (x)) dx
R3 R3

are bounded independent of ε. By (2.35), there exists a constant K > 0 independent of ε such that
Z Z 
1 1 1
I3 ≤ −K u+ ηε dx + v+ ηε dx + o(ε 2 ) ≤ −Kε 2 + o(ε 2 ). (2.37)
R3 R3

Therefore, by (2.33),(2.34),(2.36),(2.37), we have

ν −(N −2)/2 N/2 1 1

I(Wε,t , Vε,t ) ≤ m(a, b) + Sα,β − Kε 2 + o(ε 2 ) + O(ε)
N
ν −(N −2)/2 N/2
< m(a, b) + Sα,β
N
for ε small enough and t−1
0 < t < t0 . Then it follows from (2.32) that

ν −(N −2)/2 N/2

sup I(Wε,t , Vε,t ) < m(a, b) + Sα,β .
t>0 N

Finally, the conclusion follows directly from (2.31). This completes the proof.

Proof of Theorem 1.2 (2) and (3). With the help of Lemma 2.4, Lemma 2.5, Lemma 2.8 and Lemma 2.9, the
conclusion is just a combinition of Proposition 2.1 and above lemmas.

4
2.2 The case 2 + N
≤ p < 2∗
4
In this subsection, we consider the case 2 + N ≤ p < 2∗ and assume that (2.2) hold. Following the
strategies in [54], we have the following lemma.
Lemma 2.10. For every (u, v) ∈ T (a, b), Φ(u,v) (t) has exactly one critical point t− (u, v). Moreover,
(a) P(a, b) = P − (a, b);
(b) t ⋆ (u, v) ∈ P(a, b) if and only if t = t− (u, v),
(c) Φ(u,v) (t) is strictly dereasing and concave on (t− (u, v), +∞) and

Φ(u,v) (t− (u, v)) = max Φ(u,v) (t) > 0,

t∈R

(d) the map (u, v) 7→ t− (u, v) is of class C 1 .

Moreover, in a standard way, we can prove
Lemma 2.11. There exists a Palais-Smale sequence (un , vn ) ⊂ T (a, b) ∩ Hrad for I|T (a,b) at level m(a, b),
with P (un , vn ) → 0 and u− − N
n , vn → 0 a.e. in R as n → ∞.

Proof. Noting that m(a, b) = inf γ∈Γ2 maxt∈[0,1] J(γ(t)) with J(s, (u, v)) := I(s ⋆ (u, v)) and

Γ2 := γ ∈ C([0, 1], R × Tr (a, b)) : γ(0) ∈ (0, Ak (a, b)), γ(1) ∈ (0, I 0 ) ,

for a small k > 0. Then such a required Palais-Smale sequence can be found as in Lemma 2.4. For more details,
we refer to [8].
Now in view of Proposition 2.1, to obtain the compactness of the Palais-Smale sequence, one need some
fine estimations about m(a, b), which will be done in the remainder of this section.

22
Lemma 2.12. Let N = 3, 4. Then
(1) for any 0 ≤ a1 ≤ a and 0 < b1 ≤ b, there is m(a, b) ≤ m(a1 , b1 );
−(N −2)/2 N/2
(2) there holds 0 < m(a, b) < ν N Sα,β .
Proof of Lemma 2.12. (1) The proof is just the same as that of Lemma 2.5 (1).
(2) It is natural that m(a, b) > 0. Indeed, for any (u, v) ∈ P(a, b), there is
Z
k∇uk22 + k∇vk22 = γp (kukpp + kvkpp ) + ν |u|α |v|β dx
RN
pγp 2∗
≤ C1 (k∇uk22 + k∇vk22 ) 2 + C2 (k∇uk22 + k∇vk22 ) 2

which implies inf P(a,b) k∇uk22 + k∇vk22 > 0. So we have

1
m(a, b) = inf I(u, v) = inf I(u, v) − P (u, v)
P(a,b) P(a,b) pγp
Z
pγp − 2 2∗ − pγp
= (k∇uk22 + k∇vk22 ) + ∗ ν |u|α |v|β dx
2pγp 2 pγp R N

≥ C inf k∇uk22 + k∇vk22 > 0.

P(a,b)

√ √
Now let ηε be defined by (2.21). Take uε = kηεαc k2 ηε and vε = kη
βc
ηε for a small constant c > 0. Let
√ √ε k2
tε := t− (uε , vε ) be given by Lemma 2.10. So tε ⋆ (uε , vε ) ∈ P( αc, βc) and hence for a proper c > 0 we
obtain
√ p
m(a, b) ≤ m( αc, βc) ≤ I(tε ⋆ (uε , vε ))
Z
1
ν ∗ 1 pγp tε

= e2tε k∇uε k22 + k∇vε k22 − ∗ e2 tε uα β
ε vε dx − e kuε kpp + kvε kpp
2 2 RN p
 α/2 β/2 ∗

α+β 2 να β
s2 kηε k22∗ − Ckηε kpp kηε k−p
∗
≤ max s k∇ηε k22 − 2 e
pγp tε
s>0 2 2∗ (2.38)
" #N/2
2
1 (α + β)k∇ηε k2
=
∗
∗ 2/2
− Ckηε kpp kηε k−p
2 e
pγp tε
N ναα/2 β β/2 kηε k2∗ 2
−(N −2)/2
ν N/2
= Sα,β + O(εN −2 ) − Ckηε kpp kηε k2−p epγp tε .
N
4
We claim that etε ≥ Ckηε k2 for some constant C > 0 as ε → 0. Now we check it. If p = 2 + N, by
definition P (tε ⋆ (uε , vε )) = 0 and hence

∗
−2)tε k∇uε k22 + k∇vε k22 kuε kpp + kvε kpp
e(2 = R β
− γp R β
ν RN u αε vε dx ν RN u α ε vε dx
 
2C(N, p) 4/N k∇uε k22 + k∇vε k22
≥ 1− (a + b4/N ) R β (2.39)
p ν N uα ε vε dx R
∗
−2
k∇ηε k22 kηε k22
≥C ,
kηε k22∗
∗

∗
where we used (2.2). By Lemma 2.6, we have k∇ηε k22 ∼ 1 and kηε k22∗ ∼ 1, which in turn together with (2.39)
gives that etε ≥ Ckηε k2 . If p > 2 + N4 , by definition P (tε ⋆ (uε , vε )) = 0 and hence
Z
(2∗ −2)tε
e ν uα β 2
ε vε dx ≤ k∇uε k2 + k∇vε k2 ,
2
RN

whence it follow that !1/(2∗ −2)

k∇uε k22 + k∇vε k22
etε ≤ R β
. (2.40)
ν RN u α ε vε dx

23
By (2.40) and using the fact that pγp > 2,

∗
−2)tε k∇uε k22 + k∇vε k22 kuε kpp + kvε kpp (pγp −2)tε
e(2 = R β
− γp R β
e
ν RN u α ε vε dx ν RN u α ε vε dx
! pγ p −2

k∇uε k22 + k∇vε k22 kuε kpp + kvε kpp k∇uε k22 + k∇vε k22
2∗ −2

≥ R − γ p R R (2.41)
β β β
ν RN u α ε vε dx ν RN u α ε vε dx ν RN u α ε vε dx
!
−2 kηε kpp
∗
k∇ηε k22 kηε k22 −2
2∗ −pγp
2∗ −2
2∗
pγp −2
2∗ −2
= C1 − C2 k∇ηε k2 kηε k2∗ .
kηε k22∗ p(1−γp )
∗
kηε k2
∗
Using Lemma 2.6, we obtain k∇ηε k22 ∼ 1, kηε k22∗ ∼ 1 and that
( 6−p
kηε kpp O(ε 4 ) if N = 3,
p(1−γp )
= p(1−γp ) (2.42)
kηε k2 O(| log ε|− 2 ) if N = 4.

Going back to (2.41), it results that etε ≥ Ckηε k2 .

Substituting etε ≥ Ckηε k2 into (2.38), we obtain

ν −(N −2)/2 N/2 kηε kpp

m(a, b) ≤ Sα,β + O(εN −2 ) − C p(1−γp )
,
N kηε k2

ν −(N −2)/2 N/2

and hence using (2.42) we infer that m(a, b) < N Sα,β for any ε > 0 small, which is the desired result.

By Lemma 2.12, we observe that the energy level m(a, b) satisfies (2.6) because e(a) > 0, e(b) > 0.
However, we still need to show that m(a, b) 6= e(a) and m(a, b) 6= e(b). Actually, there holds
Lemma 2.13. For any a, b > 0, the following energy estimate holds

m(a, b) < min {e(a), e(b)} ,

under one of the following conditions:

(1) a ≤ a0 and b ≤ a0 , when 2 + N4 < p < 2∗ , where a0 is defined in (1.11);
(2) ν > ν1 for some ν1 = ν1 (a, b, α, β) > 0;
(3) α < 2 and a ≤ b;
(4) β < 2 and b ≤ a.
Proof. The proofs of (2)-(4) in Lemma 2.13 is very similar as that of [43, Lemma 4.6], so we omit the details
here. Now we give a simple proof for (1). Recalling the definition of a0 (see (1.11)), by the properties of e(a)
in Theorem A, a0 is actually choosed to satisfy

ν −(N −2)/2 N/2

e(a0 ) = Sα,β .
N
Note that e(a) is strictly decreasing with respect to a > 0, we can see from Lemma 2.12 that

m(a, b) < min {e(a), e(b)} ,

since a ≤ a0 and b ≤ a0 . This completes the proof.

Remark 2.1. We point out that the conclusions of Lemma 2.12 and 2.13 still hold true when N ≥ 5.
Proof of Theorem 1.4. With the help of Lemma 2.11, 2.12 and 2.13, the conclusion is just a combinition of
Proposition 2.1 and above lemmas.

24
3 Existence for the repulsive case ν < 0
In this section, we study the existence of normalized solution of (1.3) for the repulsive case ν < 0.
Throughout this section, we always work under the following assumptions
4
N = 3, 2 < p < 2 + , and p ≤ α, β ≤ 2∗ , α + β = 2∗ . (3.1)
N
Now we give the following compactness result for ν < 0.
Proposition 3.1. Assume that (3.1) holds and

mr (a, b) ≤ mr (a1 , b1 ) for any 0 < a1 ≤ a, 0 < b1 ≤ b.

Let {(un , vn )} ⊂ Tr (a, b) be a sequence consisting of radial symmetric functions such that

I ′ (un , vn ) + λ1,n un + λ2,n vn → 0 for some λ1,n , λ2,n ∈ R, (3.2)

I(un , vn ) → c, P (un , vn ) → 0,
u− − N
n , vn → 0, a.e. in R , (3.3)
with
c < min {e(a), e(b)} and c 6= 0.
1
Then there exists u, v ∈ Hrad (RN ) with u, v > 0 and λ1 , λ2 > 0 such that up to a subsequence (un , vn ) →
(u, v) in H (R ) × H (R ) and (λ1,n , λ2,n ) → (λ1 , λ2 ) in R2 .
1 N 1 N

Proof. We divide the proof into three steps.

Step 1.) We prove that {(un , vn )} is bounded in H 1 (RN ) × H 1 (RN ).
By P (un , vn ) → 0, one can obtain that for n large enough,
1
c + 1 ≥ I(un , vn ) − P (un , vn )
2∗
1
2∗ − p

= k∇un k22 + k∇vn k22 − ∗ kun kpp + kvn kpp
N 2 p
1

pγp /2
≥ k∇un k2 + k∇vn k2 − C k∇un k22 + k∇vn k22
2 2
,
N
for some C > 0, which implies that {(un , vn )} is bounded because pγp < 2.
Moreover, from
1 1
λ1,n = − I ′ (un , vn )[(un , 0)] + o(1) and λ2,n = − I ′ (un , vn )[(0, vn )] + o(1),
a b
we know that λ1,n , λ2,n are also bounded. So there exists u, v ∈ H 1 (RN ), λ1 , λ2 ∈ R such that up to a
subsequence
(un , vn ) ⇀ (u, v) in H 1 (RN ) × H 1 (RN ),
(un , vn ) → (u, v) in Lq (RN ) × Lq (RN ), for 2 < q < 2∗ ,
(un , vn ) → (u, v) a.e. in RN ,
(λ1,n , λ2,n ) → (λ1 , λ2 ) in R2 .
Then (3.2) and (3.3) give that (
I ′ (u, v) + λ1 u + λ2 v = 0,
u ≥ 0, v ≥ 0,
and hence P (u, v) = 0.
Step 2.) We prove that u 6= 0, v 6= 0 and λ1 , λ2 > 0.

25
 Without loss of generality, we assume that u = 0 by contradiction, thenR two cases will occur. For the case
v = 0, then from P (un , vn ) = o(1) we obtain that k∇un k22 + k∇vn k22 = ν RN |un |α |vn |β + dx + o(1). Note
that ν ≤ 0, we have k∇un k22 + k∇vn k22 → 0, that is, un → 0 and vn → 0 strongly in D1,2 (RN ). Then
1

c = lim I(un , vn ) = lim k∇un k22 + k∇vn k22 = 0,
n→∞ n→∞ N

which contradicts to the fact c 6= 0. For the case v 6= 0, then by maximum principle we know that v is a solution
of (
− ∆v + λ2 v = v p−1 , in RN ,
v > 0.
and by using [35, Lemma A.2] and [49, Theorem 8.4], we obtain λ2 > 0. Let v̄n = vn − v. Similar to
Proposition 2.1, we obtain that
Z
k∇un k22 + k∇v̄n k22 = ν |un |α |v̄n |β dx + o(1),
RN

therefore k∇un k22

+ k∇v̄n k22 → 0 because ν < 0, which implies that un → 0 and vn → v strongly in
D1,2 (RN ). Then
k∇v̄n k22 + λ2 kv̄n k22
= (I ′ (un , vn ) + λ1,n un + λ2,n vn ) [(0, v̄n )] − (I ′ (u, v) + λ1 u + λ2 v) [(0, v̄n )] + o(1)
= o(1).
That is vn → v in H 1 (RN ). Therefore,
1

c = lim I(un , vn ) = lim k∇un k22 + k∇vn k22 + E(v) = e(b),
n→∞ n→∞ N
which is impossible because c < min {e(a), e(b)}. Then we have u 6= 0 and v 6= 0. Hence by the maximum
principle, u > 0, v > 0.
Since ν < 0, in order to show that λ1 , λ2 > 0, we will borrow some ideas from [35, Lemma A.2]. By
contradiction, we may assume that λ1 ≤ 0. It follows from [43, Corollary B.1] that (u, v) is a smooth solution,
and belongs to L∞ (RN )×L∞ (RN ), thus |∆u|, |∆v| ∈ L∞ (RN ). A standard gradient estimates for the Poisson
equation (see [27]) shows that |∇u|, |∇v| ∈ L∞ (RN ). Then from u, v ∈ L2 (RN ), we get u(x), v(x) → 0, as
|x| → ∞. Recalling the assumption (3.1), we have
αν
−∆u = |λ1 |u + (1 + ∗ |u|α−p |v|β )|u|p−2 u ≥ 0
2
for |x| > R0 with R0 > 0 large enough, so u is superharmonic at infinity. From the Hadamard three spheres
theorem [48, Chapter 2], we can see that the function m(r) := min|x|=r u(x) satisfies
m(r1 )(r2−N − r22−N ) + m(r2 )(r12−N − r2−N )
m(r) ≥ , for R0 ≤ r1 < r < r2 .
r12−N − r22−N
Since u(x) → 0 as |x| → ∞, we can see that m(r2 ) → 0 as r2 → ∞ and rN −2 m(r) ≥ R0N −2 m(R0 ) for all
r ≥ R0 . Note that N = 3, we have
Z +∞ Z +∞
kuk22 ≥ C |m(r)|2 rN −1 dr ≥ C r3−N dr = +∞
R0 R0

for some C > 0, which is impossible because kuk2 = a. Therefore, λ1 > 0. Similarly, we can prove that
λ2 > 0.
Step 3.) We prove the strong convergence.
Let (ūn , v̄n ) = (un − u, vn − v). Similar to before, we can show that un → u and vn → v strongly in
D1,2 (RN ). Note that λ1 , λ2 > 0 and
k∇ūn k22 + λ1 kūn k22 + k∇v̄n k22 + λ2 kv̄n k22
= (I ′ (un , vn ) + λ1,n un + λ2,n vn ) [(ūn , v̄n )] − (I ′ (u, v) + λ1 u + λ2 v) [(ūn , v̄n )] + o(1)
= o(1),
we obtain (un , vn ) → (u, v) strongly in H 1 (RN ) × H 1 (RN ). This completes the proof.

26
 For any (u, v) ∈ Tr (a, b), we present an important result about Φ(u,v) : R → R
Z Z Z
1
1 ν ∗
Φ(u,v) (t) := e2t |∇u|2 + |∇v|2 dx − epγp t (|u|p + |v|p ) dx − ∗ e2 t |u|α |v|β dx.
2 RN p RN 2 RN

Lemma 3.1. For all ν ≤ 0 and (u, v) ∈ Tr (a, b), there exists a unique t(u, v) ∈ R such that t(u, v) ⋆ (u, v) ∈
Pr (a, b); t(u, v) is also the unique critical point of Φ(u,v) (t) and a strict minimum point at a negative level.
Moreover, we have
(a) Pr (a, b) = Pr+ (a, b); t ⋆ (u, v) ∈ Pr (a, b) if and only if t = t(u, v);
(b) Φ(u,v) (t) is strictly decreasing on (−∞, t(u, v)), and strictly increasing and convex on (t(u, v), +∞), and

Φ(u,v) (t(u, v)) = min Φ(u,v) (t);

t∈R

(c) The map (u, v) 7→ t(u, v) is of class C 1 .

Proof. Observe that
Z Z Z 


Φ′(u,v) (t) = e2t |∇u|2 + |∇v|2 dx − γp e(pγp −2)t −2)t
∗
(|u|p + |v|p ) dx + |ν|e(2 |u|α |v|β dx
RN RN RN
2t
=: e G(u,v) (t),
one can easily see that G(u,v) (t) is strictly increasing because pγp < 2 < 2∗ . A direct computation shows that
G(u,v) (−∞) = −∞ and G(u,v) (+∞) = +∞, then G(u,v) has a unique zero t(u, v), which is also a unique
zero of Φ′(u,v) . And we can see that t ⋆ (u, v) ∈ Pr (a, b) if and only if t = t(u, v). Moreover, Φ′(u,v) (t) < 0 for
t < t(u, v) and Φ′(u,v) (t) > 0 for t > t(u, v), which implies that t(u, v) is a strict minimum point of Φ(u,v) at a
negative level. For any (u, v) ∈ Pr (a, b) \ Pr+ (a, b), there hold
Z Z Z


|∇u|2 + |∇v|2 dx − γp (|u|p + |v|p ) dx + |ν| |u|α |v|β dx = 0,
RN RN RN

and Z Z Z
2 2

p p ∗
2 |∇u| + |∇v| dx − γp pγp (|u| + |v| ) dx + 2 |ν| |u|α |v|β dx ≤ 0.
RN RN RN
Therefore, we have
Z Z
0 ≥ (2 − pγp )γp (|u|p + |v|p ) dx + (2∗ − 2)|ν| |u|α |v|β dx,
RN RN

which is impossible because pγp < 2 < 2∗ . Consequently Pr (a, b) = Pr+ (a, b). Finally, the implicit function
theorem implies that the map (u, v) 7→ t(u, v) is of class C 1 .
4
Lemma 3.2. For all ν ≤ 0 and 2 < p < 2 + N, we have the following.
(a) For 0 < a1 ≤ a and 0 < b1 ≤ b, then

mr (a, b) ≤ mr (a1 , b1 ) + mr (a − a1 , b − b1 ) ≤ m(a1 , b1 );

(b) For all a, b > 0, we have

mr (a, b) ≤ e(a) + e(b) < min {e(a), e(b)} ,

where e(a), e(b) are given by (1.7).

Proof. (a) By the density of C0∞ (RN ) into H 1 (RN ), for arbitrary ε > 0, we can choose radial symmetric
functions (u1 , v1 ), (u2 , v2 ) ∈ C0∞ (RN )×C0∞ (RN ) with ku1 k2 = a1 , kv1 k2 = b1 and ku2 k2 = a−a1 , kv2 k2 =
b − b1 such that
ε
I(u1 , v1 ) ≤ mr (a1 , b1 ) + ,
2
ε
I(u2 , v2 ) ≤ mr (a − a1 , b − b1 ) + .
2
Observe that I(u, v) is invariant by translation, then we may assume that supp(u1 )∩supp(u2 ) = ∅, supp(u1 )∩
supp(v2 ) = ∅, supp(v1 ) ∩ supp(u2 ) = ∅ and supp(v1 ) ∩ supp(v2 ) = ∅, which implies that ku1 + u2 k2 = a

27
and kv1 + v2 k2 = b. By Lemma 3.1, there exists a unique strict minimum point t = t(u1 + u2 , v1 + v2 ) ∈ R
such that t ⋆ (u1 + u2 , v1 + v2 ) ∈ Pr (a, b) and therefore

m(a, b) ≤ I(t ⋆ (u1 + u2 , v1 + v2 )) ≤ I(u1 + u2 , v1 + v2 )

= I(u1 , v1 ) + I(u2 , v2 ) ≤ mr (a1 , b1 ) + mr (a − a1 , b − b1 ) + ε

Then by the arbitrariness of ε, we obtain that mr (a, b) ≤ mr (a1 , b1 ) + mr (a − a1 , b − b1 ). Finally, since

2 < p < 2 + N4 , by Lemma 3.1 we know that mr (a − a1 , b − b1 ) ≤ 0 for 0 < a1 ≤ a and 0 < b1 ≤ b, and then
mr (a, b) ≤ mr (a1 , b1 ).
(b) For 2 < p < 2 + N4 , it follows from [54] that e(a) and e(b) are attained, then for arbitrary ε > 0 we
can take two radial symmetric functions u0 , v0 ∈ C0∞ (RN ) with ku0 k2 = a and kv0 k2 = b such that

E(u0 ) ≤ e(a) + ε, E(v0 ) ≤ e(b) + ε.

We can further assume that the supports of u0 and v0 are disjoint, then I(u0 , v0 ) = E(u0 ) + E(v0 ). Since
(u0 , v0 ) ∈ Tr (a, b), by Lemma 3.1, there exists t(u0 , v0 ) ∈ R such that t(u0 , v0 ) ⋆ (u0 , v0 ) ∈ Pr (a, b) and
Φ(u0 ,v0 ) (t(u0 , v0 )) = mint∈R Φ(u0 ,v0 ) (t). Hence, we have

m(a, b) = inf I(u, v) ≤ I(t(u0 , v0 ) ⋆ (u0 , v0 ))

(u,v)∈P(a,b)

= Φ(u0 ,v0 ) (t(u0 , v0 )) ≤ Φ(u0 ,v0 ) (0)

= I(u0 , v0 ) = E(u0 ) + E(v0 )
≤ e(a) + e(b) + 2ε.

Then we obtain mr (a, b) ≤ e(a) + e(b) from the arbitrariness of ε. Finally, since 2 < p < 2 + N4 , it follows
from [54] that e(a) < 0 and e(b) < 0, and then mr (a, b) < min {e(a), e(b)}. This completes the proof.
Proof of Theorem 1.3. Since 2 < p < 2 + N4 and ν < 0, one can easily check that −∞ < mr (a, b) < 0,
then we take a radial minimizing sequence {(ûn , v̂n )} ∈ Pr (a, b) such that I(ûn , v̂n ) = m(a, b) + o(1).
Then passing to (|ûn |, |v̂n |), we may assume that (ûn , v̂n ) is nonnegative. By using the Ekeland variational
principle (see [53]), we can take a radially symmetric Palais-Smale sequence (un , vn ) for I|Tr (a,b) such that
k(un , vn ) − (ûn , v̂n )kHrad → 0 as n → ∞. Then

P (un , vn ) = P (ûn , v̂n ) + o(1), as n → ∞.

By Lagrange multiple rules, there exist λ1,n , λ2,n ∈ R such that I ′ (un , vn ) + λ1,n un + λ2,n vn → 0 as n → ∞.
Then combining Lemma 3.2 and Proposition 3.1 for c = mr (a, b), we obtain that (un , vn ) → (u, v) in Hrad
and (λ1,n , λ2,n ) → (λ1 , λ2 ) in R2 . By the strong convergence, (un , vn ) ∈ Pr (a, b) is a normalized solution of
(1.3).

4 Nonexistence for the defocusing case ω1 < 0 and ω2 < 0

In this section, we will prove the Nonexistence results of normalized solution Theorem 1.1.
Proof of Theorem 1.1. Assume by contradiction that the system (1.3) has a positive normalized solution (u, v) ∈
H 1 (RN ) × H 1 (RN ) with kuk22 = a2 , kvk22 = b2 . Then from the Pohozaev identity, we obtain that


λ1 a2 + λ2 b2 = (1 − γp ) ω1 kukpp + ω2 kvkpp < 0,

since ω1 , ω2 < 0, γp < 1. Then one of λ1 and λ2 is strictly less than zero. Without loss of generality, we may
assume that λ1 < 0. By a similar argument as used in that of Step 2 in Proposition 3.1, we have u(x), v(x) → 0,
as |x| → ∞, which together with assumption (1.6) implies

αν α−2 β |λ1 |
−∆u = (|λ1 | + |u|p−2 + |u| |v| )u ≥ u > 0.
2∗ 2

28
for |x| > R0 with R0 > 0 large enough, so u is superharmonic at infinity. Then using the Hadamard three
spheres theorem [48, Chapter 2], we can proceed with the argument presented in Step 2 of Proposition 3.1 that
rN −2 m(r) ≥ R0N −2 m(R0 ) for all r ≥ R0 , where m(r) := min|x|=r u(x). Then
Z +∞ Z +∞
q q N −1
kukq ≥C |m(r)| r dr ≥ C rq(2−N )+N −1 dr (4.1)
R0 R0

for some C > 0. If N = 3, 4, we choose q = 2 in (4.1) and obtain that

Z +∞
2
kuk2 ≥ C r3−N dr = +∞,
R0

which is a contradiction. If N ≥ 5, it can be observed that q can not be chosen as 2. In order to get a
contradiction, we need choose q ≤ N/(N − 2). However, u ∈ H 1 (RN ) could not directly imply that u ∈
Lq (RN ) for some 0 < q ≤ N/(N − 2). Therefore, we need such additional assumption to get a contradiction.
This completes the proof.

5 The Sobolev critical case p = α + β = 2∗

In this section, we consider the system (1.3) with Sobolev critical exponent p = α + β = 2∗ and prove the
existence Theorem 1.5 and nonexistence results Theorem 1.6.
Proof of Theorem 1.5. Recalling the definition of Φ(u,v) (t) in (1.22), for any (u, v) ∈ T (a, b), we have
 Z 
1 2t 2 2

1 2∗ t 2∗ 2∗ α β
Φ(u,v) (t) = I(t ⋆ (u, v)) = e k∇uk2 + k∇vk2 − ∗ e kuk2∗ + kvk2∗ + ν |u| |v| dx ,
2 2 RN

Obviously, the function Φ(u,v) (t) has a unique critical point t(u, v) ∈ R, which is also the unique strict maxi-
mum point, such that t(u, v) ⋆ (u, v) ∈ P(a, b). More precisely,
! 2∗1−2
t(u,v) k∇uk22 + k∇vk22
e = 2∗ 2∗ R .
kuk2∗ + kvk2∗ + ν RN |u|α |v|β dx

We claim that P + (a, b) ∪ P 0 (a, b) = ∅, implying that P(a, b) = P − (a, b). Indeed, for any (u, v) ∈ P + (a, b) ∪
P 0 (a, b), we have
 Z 

2∗ 2∗

2 k∇uk22 + k∇vk22 ≥ 2∗ kuk2∗ + kvk2∗ + ν |u|α |v|β dx = 2∗ k∇uk22 + k∇vk22 ,
RN

implying that k∇uk22 + k∇vk22 = 0. which is a contradiction to (u, v) ∈ T (a, b). Therefore, we prove that the
claim is true and
m(a, b) = inf I(u, v) = inf I(u, v).
(u,v)∈P(a,b) (u,v)∈P − (a,b)

Moreover, it is standard to show that

m(a, b) = inf I(u, v) = inf max I(t ⋆ (u, v))

(u,v)∈P − (a,b) (u,v)∈T (a,b) t∈R

= inf I(t(u, v) ⋆ (u, v))

(u,v)∈T (a,b)

2∗
1 k∇uk22 + k∇vk22 2∗ −2
= inf   2∗2−2 .
N (u,v)∈T (a,b) 2∗ 2∗ R
kuk2∗ α β
+ kvk2∗ + ν RN |u| |v| dx

Now we claim that m(a, b) = C, where C is defined in (1.13). In fact, we deduce from (1.14) that C ≤ m(a, b).
On the other hand, it follows from [31] that the ground state level C can be achieved by
   
− N 4−2
eε,y , Veε,y = x̃1 Fmax − N −2
U Uε,y , x̃2 Fmax4 Uε,y ∈ D, (5.1)

29
which is given by (1.15). Observe that the Aubin-Talenti bubble Uε,y belongs to L2 (RN ) if and only if N ≥ 5
Then for the unique choice of ε0 > 0, we have kU eε0 ,y k2 = a and kVeε0 ,y k2 = b since we assume that N ≥ 5,
x̃1 6= 0, x̃2 6= 0 and ax̃2 = bx̃1 . Therefore, by (1.14), (1.16) and (5.1)
  2∗2−2
∗

eε,y k22 + k∇Veε,y k22

k∇U 1 − N2−2 N
m(a, b) ≤   2∗2−2 = N Fmax S 2 = C.
2∗ 2∗ R
N eε,y
U + Veε,y +ν eε,y |α |Veε,y |β dx
|U
2∗ 2∗ RN

Therefore, the claim is true and m(a, b) = C. That is, the minimization of I(u, v) on P(a, b) is equivalent to  the
minimization of I(u, v) on N . It follows from (5.1) that the minimization of I(u, v) on N is U eε0 ,y , Veε0 ,y for
the unique choice of ε0 >0 such that kU eε0 ,y k2 = a and kVeε0 ,y k2 = b. Hence the system (1.3) has a normalized
ground state of the form Ueε0 ,y , Veε0 ,y solving the system (1.3) with λ1 = λ2 = 0. It completes the proof.

Proof of Theorem 1.6. Here, we give a brief proof of Theorem 1.6. Indeed, let (u, v) be a positive normalized
solution of (1.3) with N = 3, 4 and p = α + β = 2∗ , then using Pohozaev identity, one can easily obtain that

λ1 kuk22 + λ2 kvk22 = 0; (5.2)

since u, v satisfy
−∆u + λ1 u ≥ 0 and − ∆v + λ2 v ≥ 0,
by [35, Lemma A.2] we obtain immediately λ1 , λ2 > 0. It follows from (5.2) that u = v = 0, which is a
contradiction.

Acknowledgement
Houwang Li is supported by the postdoctoral foundation of BIMSA. Tianhao Liu is supported by the
Postdoctoral Fellowship Program of CPSF (GZB20240945). Wenming Zou is supported by National Key R&D
Program of China (Grant 2023YFA1010001) and NSFC(12171265).

Data availability statement

No data was used for the research described in the article.

Conflict of interest statement

On behalf of all authors, the corresponding author states that there is no conflict of interest.

References
[1] B. Abdellaoui, V. Felli and I. Peral. Some remarks on systems of elliptic equations doubly critical in the
whole RN . Calc. Var. Partial Differential Equations, 34(2009), 97-137.
[2] S. K. Adhikari. Superfluid fermi-fermi mixture: phase diagram, stability, and soliton formation. Phys. Rev.
A 76, 053609 (2007).
[3] N. Akhmediev and A. Ankiewicz. Partially coherent solitons on a finite background. Phys. Rev. Lett.,
82(1999), 2661-2664.
[4] C. O. Alves, D. C. de Morais Filho and M. A. S. Souto. On systems of elliptic equations involving sub-
critical or critical Sobolev exponents. Nonlin. Analysis, 42(2000), 771-787.
[5] A. Ambrosetti and E. Colorado. Standing waves of some coupled nonlinear Schrödinger equations. J.
Lond. Math. Soc. (2), 75(2007), No. 1, 67-82.

30
 [6] T. Bartsch, N. Dancer, Z. Q. Wang. A Liouville theorem, a priori bounds, and bifurcating branches of
positive solutions for a nonlinear elliptic system. Calc. Var. Partial Differential Equations. 37 (2010),
345-361.
[7] T. Bartsch and L. Jeanjean. Normalized solutions for nonlinear Schrödinger systems. Proc. Roy. Soc.
Edinburgh Sect. A, 148(2018), no.2, 225-242.
[8] T. Bartsch, L. Jeanjean and N. Soave. Normalized solutions for a system of coupled cubic Schrödinger
equations on R3 . J. Math. Pures Appl., 106(2016), 583-614.
[9] T. Bartsch and N. Soave. A natural constraint approach to normalized solutions of nonlinear Schrödinger
equations and systems. J. Funct. Anal., 272(2017), 4998-5037.
[10] T. Bartsch and N. Soave. Multiple normalized solutions for a competing system of Schrödinger equations.
Calc. Var. Partial Differential Equations, 58(2019), No. 1, Paper No. 22, 24 pp.
[11] T. Bartsch, X. X. Zhong and W. M. Zou. Normalized solutions for a coupled Schrödinger system. Math.
Ann., 2020. https://doi.org/10.1007/s00208-020-02000-w.
[12] T. Bartsch, H. W. Li and W. M. Zou. Existence and asymptotic behavior of normalized ground states for
Sobolev critical Schrödinger systems. Calc. Var. Partial Differential Equations 62 (2023), no. 1, Paper No.
9, 34 pp.
[13] T. Bartsch, Z. Q. Wang. Note on ground states of nonlinear Schrödinger systems. J. Partial Differ. Equat.
19(2006), 200–207.
[14] T. Bartsch, Z. Q. Wang, J. C. Wei, Bound states for a coupled Schrödinger system. J. Fixed Point Theory
Appl. 2(2007), 353–367.
[15] M. Clapp and J.Faya. Multiple solutions to a weakly coupled purely critical elliptic system in a bounded
domain. Discrete Contin. Dyn. Syst. 39(6)(2019), 3265”C3289
[16] M. Clapp and A.Pistoia. Existence and phase separation of entire solutions to a pure critical competitive
elliptic system. Calc. Var. Partial Differ. Equ. 57 (2018), no. 1, Art. 23, 20 pp
[17] M. Clapp, A. Szulkin, A simple variational approach to weakly coupled competitive elliptic systems.
Nonlinear Differ. Equ. Appl. 26, (2019).
[18] Z. J. Chen and W. M. Zou, Positive least energy solutions and phase separation for coupled Schrödinger
equations with critical exponent: higher dimensional case. Calc. Var. Partial Differential Equations. 52
(2015), 423–467.
[19] Z. J. Chen and W. M. Zou. A remark on doubly critical elliptic systems. Calc. Var. Partial Differential
Equations, 50(2014), 939-965.
[20] Z. J. Chen and W. M. Zou. Existence and symmetry of positive ground states for a doubly critical
Schrödinger system. Trans. Amer. Math. Soc., 367(2015), No. 5, 3599-3646.
[21] B. Dodson. Global well-posedness and scattering for the mass critical nonlinear Schrödinger equation with
mass below the mass of the ground state. Advances in Mathematics, 285 (2015), 1589-1618.
[22] T. Duyckaerts, J. Holmer, S. Roudenko. Scattering for the non-radial 3D cubic nonlinear Schrödinger
equation. Mathematical Research Letters 15 ( 2008), 1233–1250.
[23] B. Esry, C. Greene, J. Burke and J. Bohn. Hartree-Fock theory for double condensates. Phys. Rev. Lett.,
78(1997),3594-3597.
[24] N. Ghoussoub. Duality and perturbation methods in critical point theory, Research Monograph, Cambridge
Tracts, Cambridge University Press, (1993) 268pp.
[25] T. Gou and L. Jeanjean. Existence and orbital stability of standing waves for nonlinear Schrödinger
systems. Nonlin. Analysis, 144(2016), 10-22.

31
[26] Y. X. Guo, B. Li and J. C. Wei. Entire nonradial solutions for non-cooperative coupled elliptic system with
critical exponents in R3 . J. Differ. Equ. 256(10)(2014), 3463–3495
[27] D. Gilbarg and N. S. Trudinger, Elliptic partial differential equations of second order. Grundlehren Math.
Wiss., vol. 224 [Fundamental Principles of Mathematical Sciences] Springer-Verlag, Berlin, (1983).
[28] T. Gou and L. Jeanjean. Multiple positive normalized solutions for nonlinear Schrödinger systems. Non-
linearity, 31(2018), no.5, 2319-2345.
[29] D. Hall, M. Matthews, J. Ensher, C. Wieman and E. Cornell. Dynamics of component separation in a
binary mixture of Bose-Einstein condensates. Phys. Rev. Lett., 81(1998), 1539-1542.
[30] G. Hardy, J. E. Littlewood, and G. Polya. Inequalities. Cambridge Univ. Press, Cambridge, UK, 1934.
[31] Q. H. He and J. Yang, Quantitative properties of ground-states to an M-coupled system with critical expo-
nent in RN . Sci. China Math. 61 (2018), 709–726.
[32] T. Hmidi, S. Keraani. Blowup theory for the critical nonlinear Schrödinger equations revisited. Interna-
tional Mathematics Research Notices. 46 ( 2005), 2815–2828.

[33] J. Holmer, S. Roudenko. A sharp condition for scattering of the radial 3d cubic nonlinear Schrödinger
equation. Communications in Mathematical Physics 282 (2008), 435–467.
[34] S. Ibrahim, N. Masmoudi, K. Nakanishi. Scattering threshold for the focusing nonlinear Klein-Gordon
equation. Analysis & Partial Differential Equations 4 (3) (2011), 405–460.
[35] N. Ikoma. Compactness of minimizing sequences in nonlinear Schrödinger systems under multiconstraint
conditions. Adv. Nonlinear Stud., 14(1)(2014), 115-136.

[36] N. Ikoma and Y. Miyamoto. The compactness of minimizing sequences for a nonlinear Schr?dinger system
with potentials. arXiv:2010.14722.
[37] N. Ikoma and K. Tanaka. A note on deformation argument for L2 normalized solutions of nonlinear
Schrödinger equations and systems. Adv. Differential Equations, 24(2019), no. 11-12, 609–646.
[38] L. Jeanjean. Existence of solutions with prescribed norm for semilinear elliptic equations. Nonlin. Analy-
sis, 28(1997), no.10, 1633-1659.
[39] L. Jeanjean and S. S. Lu. A mass supercritical problem revisited. Calc. Var. Partial Differential Equations,
59(2020), no.5, 44 pp.
[40] L. Jeanjean and T. Trung Le. Multiple normalized solutions for a Sobolev critical Schrödinger equation.
Math. Ann. 384 (2022), no. 1-2, 101-134.
[41] C. E. Kenig, F. Merle. Global well-posedness, scattering and blow up for the energy-critical, focusing,
nonlinear Schrödinger equation in the radial case. Inventiones Mathematicae 166 (2006), 645–675.
[42] C. E. Kenig, F. Merle. Global well-posedness, scattering and blow up for the energy critical focusing
non-linear wave equation. Acta Mathematica 201( 2008), 147–212.
[43] H. W. Li and W. M. Zou. Normalized ground states for semilinear elliptic systems with critical and sub-
critical nonlinearities. J. Fixed Point Theory Appl. 23 (2021), no. 3, Paper No. 43, 30 pp.
[44] T. C. Lin and J. C. Wei. Ground state of N coupled nonlinear Schrödinger equations in Rn , n ≤ 3. Comm.
Math. Phys., 255(2005), No. 3, 629-653.
[45] B. Malomed. Multi-component Bose-Einstein condensates: theory. In: Kevrekidis, P.G., Frantzeskakis,
D.J., Carretero-Gonzalez, R. (eds.) Emergent nonlinear phenomena in Bose-Einstein condensation, pp.
287–305. Springer-Verlag, Berlin (2008)
[46] J. Mederski and J. Schino, Normalized solutions to Schrödinger equations in the strongly sublinear regime.
Calc. Var. Partial Differential Equations, 63 (2024). https://doi.org/10.1007/s00526-024-02729-1

32
[47] F. Merle. Determination of blow-up solutions with minimal mass for nonlinear Schrödinger equations with
critical power. Duke Mathematical Journal 69(2) (1993), 427–454.
[48] M. H. Protter and H. F. Weinberger. Maximum principles in differential equations. Prentice-Hall, Inc.,
Englewood Cliffs, N.J., (1967)
[49] P. Quittner and P. Souplet. Superlinear parabolic problems. Blow-up, global existence and steady states.
Second edition. Birkhäuser Basel, (2007).
[50] P. Raphael, J. Szeftel. Existence and uniquenes of minimal blow up solutions to an inhomogeneous mass
critical NLS. Journal of the American Mathematical Society 24 (2011), 471–546.
[51] J. Schino. Normalized ground states to a cooperative system of Schrödinger equations with generic L2 -
subcritical or L2 -critical nonlinearity. Adv. Differential Equations, 27 (2022), 467–496.
[52] B. Sirakov. Least energy solitary waves for a system of nonlinear Schrödinger equations in RN . Comm.
Math. Phys., 271(2007), No. 1, 199-221.
[53] M. Struwe. Variational Methods: Applications to Nonlinear Partial Differential Equations and Hamiltonian
systems. Springer, Berlin, (1996).
[54] N. Soave. Normalized ground states for the NLS equation with combined nonlinearities. J. Differential
Equations, 269(2020), no.9, 6941-6987.
[55] N. Soave. Normalized ground states for the NLS equation with combined nonlinearities: the Sobolev
critical case. J. Funct. Anal., 279(2020), 108610.
[56] G. Talenti. Best constant in Sobolev inequality. Ann. Mat. Pura Appl. (4), 110(1976), 353-372.

[57] J. C. Wei and Y. Z. Wu. Normalized solutions for Schrödinger equations with critical Sobolev exponent
and mixed nonlinearities. J. Funct. Anal., 283 (2022), no. 6, Paper No. 109574, 46 pp.
[58] M. I. Weinstein.Nonlinear Schrödinger equations and sharp interpolation estimates. Commun. Math. Phys.
87 (1982/1983), 567-576

33