ANALYSIS ON THE MINIMAL REPRESENTATION OF O(p, q)

– II. BRANCHING LAWS

Toshiyuki KOBAYASHI and Bent ØRSTED

RIMS Kyoto and SDU-Odense University

Abstract. This is a second paper in a series devoted to the minimal unitary representation of O(p, q). By explicit methods from conformal geometry of pseudo Riemannian manifolds, we find the branching law corresponding to restricting the minimal unitary representation to natural symmetric subgroups. In the case of purely discrete spectrum we obtain the full spectrum and give an explicit Parseval-Plancherel formula, and in the general case we construct an infinite discrete spectrum.

Contents

Introduction §4. Criterion for discrete decomposable branching laws §5. Minimal elliptic representations of O(p, q) §6. Conformal embedding of the hyperboloid §7. Explicit branching formula (discretely decomposable case) §8. Inner product on ϖ^{p,q} and the Parseval-Plancherel formula §9. Construction of discrete spectra in the branching laws

Introduction

This is the second in a series of papers devoted to the analysis of the minimal representation ϖ^{p,q} of O(p, q). We refer to [23] for a general introduction; also the numbering of the sections is continued from that paper, and we shall refer back to sections there. However, the present paper may be read independently from [23], and and its main object is to study the branching law for the minimal unitary representation ϖ^{p,q} from analytic and geometric point of view. Namely, we shall find by explicit means, coming from conformal geometry, the restriction of ϖ^{p,q} with respect to the symmetric pair

(G, G′) = (O(p, q), O(p′, q′) × O(p′′, q′′)).

If one of the factors in G′ is compact, then the spectrum is discrete (see Theorem 4.2 also for an opposite implication), and we find the explicit branching law; when both factors are non-compact, there will still (generically) be an infinite discrete spectrum, which we also construct (conjecturally almost all of it; see §9.8). We shall see that the (algebraic) situation is similar to the theta-correspondence, where the metaplectic representation is restricted to analogous subgroups.

Let us here state the main results in a little more precise form, referring to sections 8 and 9 for further notation and details.

Typeset by AMS-TEX

Theorem A (the branching law for O(p, q) ↓ O(p, q′) × O(q′′); see Theorem 7.1). If q′′ ≥ 1 and q′ + q′′ = q, then the twisted pull-back Φ̃_{1}^{∗} of the local conformal map Φ_{1} between spheres and hyperboloids gives an explicit irreducible decomposition of the unitary representation ϖ^{p,q} when restricted to O(p, q′) × O(q′′):

{ \atop (Φ_{1})^{∗}}: ϖ^{p,q}|_{O(p,q′)×O(q′′)} →^{∼} ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix}π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ H^{l}(ℝ^{q′′}).

The representations appearing in the decompositions are in addition to usual spherical harmonics H^{l}(ℝ^{q}) for compact orthogonal groups O(q), also the representations π_{+,λ}^{p,q} for non-compact orthogonal groups O(p, q). The latter ones may be thought of as discrete series representations on hyperboloids

X(p, q) := {x = (x′, x′′) ∈ ℝ^{p+q} : |x′|^{2} − |x′′|^{2} = 1}

for λ > 0 or their analytic continuation for λ ≤ 0; they may be also thought of as cohomologically induced representations from characters of certain θ-stable parabolic subalgebras. The fact that they occur in this branching law gives a different proof of the unitarizability of these modules π_{+,λ}^{p,q} for λ > −1, once we know ϖ^{p,q} is unitarizable (cf. Part I, Theorem 3.6.1). It might be interesting to remark that the unitarizability for λ < 0 (especially, λ = −\frac{1}{2} in our setting) does follow neither from a general unitarizability theorem on Zuckerman-Vogan’s derived functor modules [31], nor from a general theory of harmonic analysis on semisimple symmetric spaces.

Our intertwining operator { \atop (Φ_{1})^{∗}} in Theorem A is derived from a conformal change of coordinate (see §6 for its explanation) and is explicitly written. Therefore, it makes sense to ask also about the relation of unitary inner products between the left-hand and the right-hand side in the branching formula. Here is an answer (see Theorem 8.5): We normalize the inner product ∥ ∥_{π_{+,λ}^{p,q}} (see §8.3) such that for λ > 0,

∥f∥_{π_{+,λ}^{p,q}}^{2} = λ∥f∥_{L^{2}(X(p,q))}^{2}, for any f ∈ (π_{+,λ}^{p,q})_{K}.

Theorem B (the Parseval-Plancherel formula for O(p, q) ↓ O(p, q′) × O(q′′)).

1) If we develop F ∈ Ker Δ̃_{M} as F = ∑_{l}^{∞} F_{l}^{(1)}F_{l}^{(2)} according to the irreducible decomposition in Theorem A, then we have

∥F ∥_{ϖ^{p,q}}^{2} = ∑_{l=0}^{∞} ∥F_{l}^{(1)}∥_{π_{+,l+ \frac{q′′}{2} −1}^{p,q′}}^{2}∥ F_{l}^{(2)}∥_{L^{2}(S^{q′′−1})}^{2}.

2) In particular, if q′′ ≥ 3, then all of π_{+,l+ \frac{q′′}{2} −1}^{p,q′} are discrete series for the hyperboloid

X(p, q′) and the above formula amounts to

∥F ∥_{ϖ^{p,q}}^{2} = ∑_{l=0}^{∞}(l + \frac{q′′}{2} − 1) ∥F_{l}^{(1)}∥_{L^{2}(X(p,q′))}^{2}∥ F_{l}^{(2)}∥_{L^{2}(S^{q′′−1})}^{2}.

The formula may be also regarded as an explicit unitarization of the minimal representation ϖ^{p,q} on the “hyperbolic space model” by means of the right side (for an abstract unitarization of ϖ^{p,q}, it suffices to choose a single pair (q′, q′′)). We note that the formula was previously known in the case where (q′, q′′) = (0, q) (namely, when each summand in the right side is finite dimensional) by Kostant, Binegar-Zierau by a different approach. The formula is new and seems to be particularly interesting even in the special case q′′ = 1, where the minimal representation ϖ^{p,q} splits into two irreducible summands when restricted to O(p, q − 1) × O(1).

In Theorem 9.1, we consider a more general setting and prove:

Theorem C (discrete spectrum in the restriction O(p, q) ↓ O(p′, q′) × O(p′′, q′′)). The twisted pull-back of the locally conformal diffeomorphism also constructs

\begin{matrix}∑ & ⊕ \\ λ∈A′(p′,q′)∩A′(q′′,p′′) \end{matrix}π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} ⊕ \begin{matrix}∑ & ⊕ \\ λ∈A′(q′,p′)∩A′(p′′,q′′) \end{matrix}π_{−,λ}^{p′,q′} ⊠ π_{+,λ}^{p′′,q′′}

as a discrete spectrum in the branching law for the non-compact case.

Even in the special case (p′′, q′′) = (0, 1), our branching formula includes a new and mysterious construction of the minimal representation on the hyperboloid as below (see Corollary 7.2.1): Let W^{p,r} be the set of K-finite vectors (K = O(p) × O(r)) of the kernel of the Yamabe operator

Ker Δ̃_{X(p,r)} = \{f ∈ C^{∞}(X(p, r)) : Δ_{X(p,r)}f = \frac{1}{4}(p + r − 1)(p + r − 3)f\},

on which the isometry group O(p, r) and the Lie algebra of the conformal group O(p, r+1) act. The following Proposition is a consequence of Theorem 7.2.2 by an elementary linear algebra.

Proposition D. Let m > 3 be odd. There is a long exact sequence

0 → W^{1,m−1} ⟶^{φ_{1}} W^{2,m−2} ⟶^{φ_{2}} W^{3,m−3} ⟶^{φ_{3}} ⋯ ⟶^{φ_{m−2}} W^{m−1,1} ⟶^{φ_{m−1}} 0

such that Ker φ_{p} is isomorphic to (ϖ^{p,q})_{K} for any (p, q) such that p + q = m + 1.

We note that each representation space W^{p,q−1} is realized on a different space X(p, q−1) whose isometry group O(p, q − 1) varies according to p (1 ≤ p ≤ m). So, one may expect that only the intersections of adjacent groups can act (infinitesimally) on Ker φ_{p}. Nevertheless, a larger group O(p, q) can act on a suitable completion of Ker φ_{p}, giving rise to another construction of the minimal representation on the hyperboloid X(p, q − 1) = O(p, q − 1)/O(p − 1, q − 1) ! We note that Ker φ_{p} is roughly half the kernel of the Yamabe operator on the hyperboloid (see §7.2 for details).

We briefly indicate the contents of the paper: In section 4 we recall the relevant facts about discretely decomposable restrictions from [16] and [17], and apply the criteria to our present situation. In particular, we calculate the associated variety of ϖ^{p,q} as well as its asymptotic K-support introduced by Kashiwara-Vergne. Theorem 4.2 and Corollary 4.3 clarify the reason why we start with the subgroup G′ = O(p, q′) × O(q′′) (i.e. p′′ = 0). Section 5 contains the identification of the representations π_{+,λ}^{p,q} and π_{−,λ}^{p,q} of O(p, q) in several ways, namely as: derived functor modules, Dolbeault cohomologies, eigenspaces on hyperboloids, and quotients or subrepresentations of parabolically induced modules. In section 6 we give the main construction of embedding conformally a direct product of hyperboloids into a product of spheres; this gives rise to a canonical intertwining operator between solutions to the so-called Yamabe equation, studied in connection with conformal differential geometry, on conformally related spaces. Applying this principle in section 7 we obtain the branching law in the case where one factor in G′ is compact, and in particular when one factor is just O(1). In this case we have Corollary 7.2.1, stating that ϖ^{p,q} restricted to O(p, q − 1) is the direct sum of two representations, realized in even respectively odd functions on the hyperboloid for O(p, q − 1). Note here the analogy with the metaplectic representation. Also note here Theorem 7.2.2, which gives a mysterious extention of ϖ^{p,q} by ϖ^{p+1,q−1} - both inside the space of solutions to the Yamabe equation on the hyperboloid X(p, q − 1) = O(p, q − 1)/O(p − 1, q − 1). We also point out that the representations π_{+,λ}^{p,q} for λ = 0, −\frac{1}{2} are rather exceptional; they are unitary, but outside the usual “fair range” for derived functor modules, see the remarks in section 8.2. Section 8 contains a calculation of spectra of intertwining operators and gives the explicit normalization of a Parseval-Plancherel formula for the branching law, and finally in section 9 we use certain Sobolev estimates to construct an infinite discrete spectrum when both factors in G′ are non-compact. We also conjecture the form of the full discrete spectrum (true in the case of a compact factor). It should be interesting to calculate the full Parseval-Plancherel formula in the case of both discrete and continuous spectrum.

The first author expresses his sincere gratitude to SDU - Odense University for the warm hospitality.

§4. Criterion for discrete decomposable branching laws

\underline{4.1.} Our object of study is the discrete spectra of the branching law of the restriction ϖ^{p,q} with respect to a symmetric pair (G, G′) = (O(p, q), O(p′, q′) × O(p′′, q′′)). The aim of this section is to give a necessary and sufficient condition on p′, q′, p′′ and q′′ for the branching law to be discretely decomposable.

We start with general notation. Let G be a linear reductive Lie group, and G′ its subgroup which is reductive in G. We take a maximal compact subgroup K of G such that K′ := K ∩ G′ is also a maximal compact subgroup. Let 𝔤_{0} = 𝔨_{0} + 𝔭_{0} be a Cartan decomposition, and 𝔤 = 𝔨 + 𝔭 its complexification. Accordingly, we have a direct decomposition 𝔤^{∗} = 𝔨^{∗} + 𝔭^{∗} of the dual spaces.

Let π ∈ Ĝ. We say that the restriction π|_{G′} is G′-admissible if π|_{G′} splits into a direct Hilbert sum of irreducible unitary representations of G′ with each multiplicity finite (see [15]). As an algebraic analogue of this notion, we say the underlying (𝔤, K)-module π_{K} is discretely decomposable as a (𝔤′, K′)-module, if π_{K} is decomposed into an algebraic direct sum of irreducible (𝔤′, K′)-modules (see [17]). We note that if the restriction π|_{K′} is K′-admissible, then the restriction π|_{G′} is also G′-admissible ([15], Theorem 1.2) and the underlying (𝔤, K)-module π_{K} is discretely decomposable (see [17], Proposition 1.6). Here are criteria for K′-admissibility and discrete decomosability:

Fact 4.1 (see [16], Theorem 2.9 for (1); [17], Corollary 3.4 for (2)).

1) If AS_{K}(π) ∩ Ad^{∗}(K)(𝔨′)^{⊥} = 0, then π is K′-admissible and also G′-admissible.

2) If π_{K} is discretely decomposable as a (𝔤′, K′)-module, then pr_{𝔤→𝔤′} (V_{𝔤}(π_{K})) ⊂ N_{𝔤′}^{∗}.

Here, AS_{K}(π) is the asymptotic cone of

Supp_{K}(π) := {highest weight of τ ∈ K̂_{0} : [π|_{K_{0}} : τ] ≠ 0}

where K_{0} is the identity component of K, and (𝔨′)^{⊥} ⊂ 𝔨^{∗} is the annihilator of 𝔨′. Let N_{𝔭}^{∗} (⊂ 𝔭^{∗}) be the nilpotent cone for 𝔭. V_{𝔤}(π_{K}) denotes the associated variety of π_{K}, which is an Ad^{∗}(K_{ℂ})-invariant closed subset of N_{𝔭}^{∗}. We write the projection pr_{𝔭→𝔭′}: 𝔭^{∗} → 𝔭′^{∗} dual to the inclusion 𝔭′ ↪ 𝔭.

\underline{4.2.} Let us consider our setting where π = ϖ^{p,q} and (G, G′) = (O(p, q), O(p′, q′) × O(p′′, q′′)).

Theorem 4.2. Suppose p′ + p′′ = p (≥ 2), q′ + q′′ = q (≥ 2) and p + q ∈ 2ℕ. Then the following three conditions on p′, q′, p′′, q′′ are equivalent:

i) ϖ^{p,q} is K′-admissible. ii) ϖ_{K}^{p,q} is discretely decomposable as a (𝔤′, K′)-module. iii) min(p′, q′, p′′, q′′) = 0.

The implication (i) ⇒ (ii) holds by a general theory as we explained ([17], Proposition 1.6); (ii) ⇒ (iii) will be proved in §4.4, and (iii) ⇒ (i) in §4.5, by an explicit computation of the asymptotic cone AS_{K}(ϖ^{p,q}) and the associated variety V_{𝔤}(ϖ_{K}^{p,q}) which are used in Fact 4.1.

Remark. Analogous results to the equivalence (i) ⇔ (ii) in Theorem 4.2 were first proved in [17], Theorem 4.2 in the setting where (G, G′) is any reductive symmetric pair and the representation is any A_{𝔮}(λ) module in the sense of Zuckerman-Vogan, which may be regarded as “representations attached to elliptic orbits”. We note that our representations ϖ^{p,q} are supposed to be attached to nilpotent orbits. We refer [20], Conjecture A to relevant topics.

\underline{4.3.} The following corollary is a direct consequence of Theorem 4.2, which will be an algebraic background for the proof of the explicit branching law (Theorem 7.1).

Corollary 4.3. Suppose that min(p′, q′, p′′, q′′) = 0.

1) The restriction of the unitary representation ϖ^{p,q}|_{G′} is also G′-admissible.

2) The space of K′-finite vectors ϖ_{K′}^{p,q} coincides with that of K-finite vectors ϖ_{K}^{p,q}.

Proof. See [15], Theorem 1.2 for (1); and [17], Proposition 1.6 for (2). □

A geometric counterpart of Corollary 4.3 (2) is reflected as the removal of singularities of matrix coefficients for the discrete spectra in the analysis that we study in §6; namely, any analytic function defined on an open subset M_{+} (see §6 for notation) of M which is a K′-finite vector of a discrete spectrum, extends analytically on M if p′′ = 0. The reason for this is not only the decay of matrix coefficients but a matching condition of the leading terms for t → ±∞. This is not the case for min(p′, q′, p′′, q′′) > 0 (see §9).

\underline{4.4.} Proof of (ii) ⇒ (iii) in Theorem 4.2.

We identify 𝔭^{∗} with 𝔭 via the Killing form, which is in turn identified with M(p, q; ℂ) by

M(p, q; ℂ) →^{∼} 𝔭, X ↦ ( \begin{matrix}O & X \\ ^{t}X & O \end{matrix} ) .

Then the nilpotent cone N_{𝔭}^{∗} corresponds to the following variety.

(4.4.1)  {X ∈ M(p, q; ℂ) : both X^{t}X and ^{t}XX are nilpotent matrices}

We put

M_{0,0}(p, q; ℂ) := {X ∈ M(p, q; ℂ) : X^{t}X = O,^{t}XX = O}.

Then M_{0,0}(p, q; ℂ)\{O} is the unique K_{ℂ} ≃ O(p, ℂ)×O(q, ℂ)-orbit of dimension p+q −3. The associated variety V_{𝔤}(ϖ_{K}^{p,q}) of ϖ^{p,q} is of dimension p+q −3, which follows easily from the K-type formula of ϖ^{p,q} (see [23], Theorem 3.6.1). Thus, we have proved:

Lemma 4.4. The associated variety V_{𝔤}(ϖ_{K}^{p,q}) equals M_{0,0}(p, q; ℂ).

The projection pr_{𝔭→𝔭′}: 𝔭^{∗} → 𝔭′^{∗} is identified with the map

pr_{𝔭→𝔭′}: M(p, q; ℂ) → M(p′, q′; ℂ) ⊕ M(p′′, q′′; ℂ), ( \begin{matrix}X_{1} & X_{2} \\ X_{3} & X_{4} \end{matrix} ) ↦ (X_{1}, X_{4}).

Suppose p′p′′q′q′′ ≠ 0. If we take

X := E_{1,1} − E_{p′+1,q′+1} + \sqrt{−1}E_{p′+1,1} + \sqrt{−1}E_{1,p′+1} ∈ M_{0,0}(p, q; ℂ),

then pr_{𝔭→𝔭′}(X) = (E_{1,1}, −E_{p′+1,q′+1}). But E_{1,1} ∉ N_{𝔬(p′,q′)}^{∗} and −E_{p′+1,q′+1} ∉ N_{𝔬(p′′,q′′)}^{∗}. Thus, pr_{𝔤→𝔤′}(X) ∉ N_{𝔤′}^{∗}. It follows from Fact 4.1 (2) that ϖ_{K}^{p,q} is not discrete decomposable as a (𝔤′, K′)-module. Hence (ii) ⇒ (iii) in Theorem 4.2 is proved. □

\underline{4.5.} Proof of (iii) ⇒ (i) in Theorem 4.2.

We take an orthogonal complementary subspace 𝔨′′_{0} of 𝔨′_{0} in 𝔨_{0} ≃ 𝔬(p) + 𝔬(q). Let 𝔱_{0}^{c} be a Cartan subalgebra of 𝔨_{0} such that 𝔱′′_{0} := 𝔱_{0}^{c} ∩ 𝔨′′_{0} is a maximal abelian subspace in 𝔨′′_{0}. We choose a positive system Δ^{+}(𝔨, 𝔱^{c}) which is compatible with a positive system of the restricted root system Σ(𝔨, 𝔱′′). Then we can find a basis \{f_{i} : 1 ≤ i ≤ [\frac{p}{2}] + [\frac{q}{2}]\} on \sqrt{−1}𝔱_{0}^{∗} such that a positive root system of 𝔨 is given by

△^{+}(𝔨, 𝔱^{c}) =\{f_{i} ± f_{j} : 1 ≤ i < j ≤ [\frac{p}{2}]\}

∪ \{f_{i} ± f_{j} : [\frac{p}{2}] + 1 ≤ i < j ≤ [\frac{p}{2}] + [\frac{q}{2}]\}

∪ (\{±f_{l} : 1 ≤ l ≤ [\frac{p}{2}]\} (p:odd) )

∪ (\{±f_{l} : [\frac{p}{2}] + 1 ≤ l ≤ [\frac{p}{2}] + [\frac{q}{2}]\} (q:odd) ) ,

and such that

(4.5.1)  \sqrt{−1}(𝔱′′_{0})^{∗} = ∑_{j=1}^{min(p′,p′′)} ℝf_{i} + ∑_{j=1}^{min(q′,q′′)} ℝf_{[ \frac{p}{2} ]+j}

if we regard (𝔱′′_{0})^{∗} as a subspace of (𝔱_{0}^{c})^{∗} by the Killing form.

Suppose p′q′p′′q′′ = 0. Without loss of generality we may and do assume p′′ = 0, namely, G′ = O(p, q′) × O(q′′) with q′ + q′′ = q.

Let us first consider the case p ≠ 2. Then the irreducible O(p)-representation H^{a}(ℝ^{p}) remains irreducible when restricted to SO(p). The corresponding highest weight is given by af_{1}. It follows from the K-type formula of ϖ^{p,q} (Theorem 3.6.1) that

Supp_{K}(ϖ^{p,q}) = \{af_{1} + bf_{[ \frac{p}{2} ]+1} : a, b ∈ ℕ, a + \frac{p}{2} = b + \frac{q}{2}\}.

Therefore we have proved:

(4.5.2)  AS_{K}(ϖ^{p,q}) = ℝ_{+}(f_{1} + f_{[ \frac{p}{2} ]+1}).

Then AS_{K}(ϖ^{p,q}) ∩ \sqrt{−1}(𝔱′′_{0})^{∗} = \{0\} from (4.5.1) and (4.5.2), which implies

AS_{K}(ϖ^{p,q}) ∩ \sqrt{−1} Ad^{∗}(K)(𝔨′_{0})^{⊥} = \{0\}

because (𝔱′′_{0})^{∗} meets any Ad^{∗}(K)-orbit through (𝔨′_{0})^{⊥}. Therefore, the restriction ϖ^{p,q}|_{K′} is K′-admissible by Fact 4.1 (1).

If p = 2, then ϖ^{p,q} splits into two representations (see Remark 3.7.3), say ϖ_{+}^{2,q} and ϖ_{−}^{2,q}, when restricted to the connected component SO_{0}(2, q). Likewise, H^{a}(ℝ^{p}) is a direct sum of two one-dimensional representations when restricted to SO(2) if a ≥ 1. Then we have

AS_{K}(ϖ_{±}^{2,q}) = ℝ_{+}(±f_{1} + f_{p+1}).

Applying Fact 4.1 (1) to the identity components (G_{0}, G′_{0}) of groups (G, G′), we conclude that the restriction ϖ_{±}^{p,q}|_{K′_{0}} is K′_{0}-admissible. Hence the restriction ϖ^{p,q}|_{K′} is also K′- admissible. Thus, (iii) ⇒ (i) in Theorem 4.2 is proved.

Now the proof of Theorem 4.2 is completed. □

§5. Minimal elliptic representations of O(p, q)

\underline{5.1.} In this section, we introduce a family of irreducible representations of G = O(p, q), denoted by π_{+,λ}^{p,q}, π_{−,λ}^{p,q}, for λ ∈ A_{0}(p, q), in three different realizations. These representations are supposed to be attached to minimal elliptic orbits, for λ > 0 in the sense of the Kirillov-Kostant orbit method. Here, we set

(5.1.1)  A_{0}(p, q) := \{ \begin{matrix}\{λ ∈ ℤ + \frac{p+q}{2} : λ > −1\} & (p > 1, q ≠ 0), \\ \{λ ∈ ℤ + \frac{p+q}{2} : λ ≥ \frac{p}{2} − 1\} & (p > 1, q = 0), \\ ∅ & (p = 1, q ≠ 0) or (p = 0), \\ \{−\frac{1}{2}, \frac{1}{2}\} & (p = 1, q = 0). \end{matrix}

It seems natural to include the parameter λ = 0, −\frac{1}{2} in the definition of A_{0}(p, q) as above, although λ = −\frac{1}{2} is outside the weakly fair range parameter in the sense of Vogan [33]. Cohomologically induced representations for λ = −\frac{1}{2} and λ = −1 will be discussed in details in a subsequent paper. In particular, the case λ = −1 is of importance in another geometric construction of the minimal representation via Dolbeault cohomology groups (see Part I, Introduction, Theorem B (4)).

\underline{5.2.} Let ℝ^{p,q} be the Euclidean space ℝ^{p+q} equipped with the flat pseudo-Riemannian metric:

g_{ℝ^{p,q}} = dv_{0}^{2} + ⋯ + dv_{p−1}^{2} − dv_{p}^{2} − ⋯ − dv_{n+1}^{2}.

We define a hyperboloid by

X(p, q) := {(x, y) ∈ ℝ^{p,q} : |x|^{2} − |y|^{2} = 1}.

We note X(p, 0) ≃ S^{p−1} and X(0, q) = ∅. If p = 1, then X(p, q) has two connected components. The group G acts transitively on X(p, q) with isotropy subgroup O(p − 1, q) at

(5.2.1)  x^{o} := ^{t}(1, 0, ⋯, 0).

Thus X(p, q) is realized as a homogeneous manifold:

X(p, q) ≃ O(p, q)/O(p − 1, q).

We induce a pseudo-Riemannian metric g_{X(p,q)} on X(p, q) from ℝ^{p,q} (see [23], §3.2), and write Δ_{X(p,q)} for the Laplace-Beltrami operator on X(p, q). As in [23], Example 2.2, the Yamabe operator is given by

(5.2.2)  Δ̃_{X(p,q)} = Δ_{X(p,q)} − \frac{1}{4}(p + q − 1)(p + q − 3).

For λ ∈ ℂ, we set

C_{λ}^{∞}(X(p, q)) := \{f ∈ C^{∞}(X(p, q)) : Δ_{X(p,q)}f = (−λ^{2} + \frac{1}{4}(p + q − 2)^{2})f\}

= \{f ∈ C^{∞}(X(p, q)) : Δ̃_{X(p,q)}f = (−λ^{2} + \frac{1}{4})f\}. (5.2.3)

Furthermore, for ϵ = ±, we write

C_{λ}^{∞}(X(p, q))_{ϵ} := {f ∈ C_{λ}^{∞}(X(p, q)) : f(−z) = ϵf(z), z ∈ X(p, q)}.

Then we have a direct sum decomposition

(5.2.4)  C_{λ}^{∞}(X(p, q)) = C_{λ}^{∞}(X(p, q))_{+} + C_{λ}^{∞}(X(p, q))_{−}

and each space is invariant under left translations of the isometry group G because G commutes with Δ_{X(p,q)}. With the notation in §3.5, we note if q = 0, then C_{λ}^{∞}(X(p, 0))_{sgn(−1)^{k}} is finite dimensional and isomorphic to the space of spherical harmonics:

H^{k}(ℝ^{p}) ≃ C_{λ}^{∞}(X(p, 0)) = C_{λ}^{∞}(X(p, 0))_{sgn(−1)^{k}} (k := λ + \frac{p − 2}{2}).

\underline{5.3.} Let G = O(p, q) where p, q ≥ 1 and let θ be the Cartan involution corresponding to K = O(p) × O(q). We extend a Cartan subalgebra 𝔱_{0}^{c} of 𝔨_{0} (given in §4.5) to that of 𝔤_{0}, denoted by 𝔥_{0}^{c}. If both p and q are odd, then dim 𝔥_{0}^{c} = dim 𝔱_{0}^{c} + 1; otherwise 𝔥_{0}^{c} = 𝔱_{0}^{c}. The complexification of 𝔥_{0}^{c} is denoted by 𝔥^{c}.

We can take a basis \{f_{i} : 1 ≤ i ≤ [\frac{p+q}{2} ]\} of (𝔥^{c})^{∗} (see §4.5; by a little abuse of notation if both p and q are odd) such that the root systems of 𝔤 is given by

\begin{matrix}△(𝔤, 𝔥^{c}) = & \{±(f_{i} ± f_{j}) : 1 ≤ i < j ≤ [\frac{p + q}{2} ]\} \\ ∪ (\{±f_{l} : 1 ≤ l ≤ [\frac{p + q}{2} ]\} (p + q:odd) ) . \end{matrix}

Let {H_{i}} ⊂ 𝔥^{c} be the dual basis for {f_{i}} ⊂ (𝔥^{c})^{∗}. Set 𝔱 := ℂH_{1} (⊂ 𝔱^{c} ⊂ 𝔥^{c}). Then the centralizer L of 𝔱 in G is isomorphic to SO(2) × O(p − 2, q). Let 𝔮 = 𝔩 + 𝔲 be a θ-stable parabolic subalgebra of 𝔤 with nilpotent radical 𝔲 given by,

△(𝔲, 𝔥^{c}) := \{f_{1} ± f_{j} : 2 ≤ j ≤ [\frac{p + q}{2} ]\} ∪ (\{±f_{1}\} (p + q:odd) ) ,

and with a Levi part 𝔩 = 𝔩_{0} ⊗ ℂ given by

𝔩_{0} ≡ Lie(L) ≃ 𝔬(2) + 𝔬(p − 2, q).

Any character of the Lie algebra 𝔩_{0} (or any complex character of 𝔩) is determined by its restriction to 𝔥_{0}^{c}. So, we shall write ℂ_{ν} for the character of the Lie algebra 𝔩_{0} whose restriction to 𝔥^{c} is ν ∈ (𝔥^{c})^{∗}. With this notation, the character of L acting on ∧^{dim 𝔲}𝔲 is written as ℂ_{2ρ(𝔲)} where

(5.3.1)  ρ(𝔲) := (\frac{p + q}{2} − 1)f_{1}.

The homogeneous manifold G/L carries a G-invariant complex structure with canonical bundle ∧^{top}T^{∗}G/L ≃ G ×_{L} ℂ_{2ρ(𝔲)}. As an algebraic analogue of a Dolbeault cohomology of a G-equivariant holomorphic vector bundle over a complex manifold G/L, Zuckerman introduced the cohomological parabolic induction R_{𝔮}^{j} ≡ (R_{𝔮}^{𝔤})^{j} (j ∈ ℕ), which is a covariant functor from the category of metaplectic (𝔩, (L ∩ K)^{∼})-modules to that of (𝔤, K)- modules. Here, L̃ is a metaplectic covering of L defined by the character of L acting on ∧^{dim 𝔲}𝔲 ≃ ℂ_{2ρ(𝔲)}. In this paper, we follow the normalization in [32], Definition 6.20 which is different from the one in [30] by a ‘ρ-shift’.

The character ℂ_{λf_{1}} of 𝔩_{0} lifts to a metaplectic (𝔩, (L ∩ K)^{∼})-module if and only if λ ∈ ℤ + \frac{p+q}{2} . In particular, we can define (𝔤, K)-modules R_{𝔮}^{j}(ℂ_{λf_{1}}) for λ ∈ A_{0}(p, q). The Z(𝔤)-infinitesimal character of R_{𝔮}^{j}(ℂ_{λf_{1}}) is given by

(λ, \frac{p + q}{2} − 2, \frac{p + q}{2} − 3, …, \frac{p + q}{2} − [\frac{p + q}{2} ]) ∈ (𝔥^{c})^{∗}

in the Harish-Chandra parametrization if it is non-zero. In the sense of Vogan [33], we have

ℂ_{λf_{1}} is in the good range  ⇔ λ > \frac{p + q}{2} − 2,
ℂ_{λf_{1}} is in the weakly fair range ⇔ λ ≥ 0.

We note that R_{𝔮}^{j}(ℂ_{λf_{1}}) = 0 if j ≠ p − 2 and if λ ∈ A_{0}(p, q). This follows from a general result in [31] for λ ≥ 0; and [14] for λ = −\frac{1}{2}.

\underline{5.4.} For b ∈ ℤ, we define an algebraic direct sum of K = O(p) × O(q)-modules by

(5.4.1)  Ξ(K : b) ≡ Ξ(O(p) × O(q) : b) := ⨁_{\begin{matrix}m,n∈ℕ \\ m−n≥b \\ m−n≡b mod 2 \end{matrix}} H^{m}(ℝ^{p}) ⊠ H^{n}(ℝ^{q}).

For λ ∈ A_{0}(p, q), we put

(5.4.2)  b ≡ b(λ, p, q) := λ − \frac{p}{2} + \frac{q}{2} + 1 ∈ ℤ,
(5.4.3)  ϵ ≡ ϵ(λ, p, q) := (−1)^{b}.

We define the line bundle L_{n} over G/L by the character nf_{1} of L (see §5.3).

Here is a summary for different realizations of the representation π_{+,λ}^{p,q}:

Fact 5.4. Let p, q ∈ ℕ (p > 1).

1) For any λ ∈ A_{0}(p, q), each of the following 5 conditions defines uniquely a (𝔤, K)- module, which are mutually isomorphic. We shall denote it by (π_{+,λ}^{p,q})_{K}. The (𝔤, K)-module (π_{+,λ}^{p,q})_{K} is non-zero and irreducible.

i) A subrepresentation of the degenerate principal representation Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}) (see §3.7) with K-type Ξ(K : b). i)′ A quotient of Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{−λ}) with K-type Ξ(K : b). ii) A subrepresentation of C_{λ}^{∞}(X(p, q))_{K} with K-type Ξ(K : b). iii) The underlying (𝔤, K)-module of the Dolbeault cohomology group

Hp−2 ; ¯∂(G/L, L_{(λ+ \frac{p+q−2}{2} )})_{K}.

iii)′ The Zuckerman-Vogan derived functor module R_{𝔮}^{p−2}(ℂ_{λf_{1}}).

2) In the realization of (iii), if f ∈ (π_{+,λ}^{p,q})_{K}, then there exists an analytic function a ∈ C^{∞}(S^{p−1} × S^{q−1}) such that

f(ω cosh t, η sinh t) = a(ω, η)e^{−(λ+ρ)t}(1 + te^{−2t}O(1)) as t → ∞

Here, we put ρ = \frac{p+q−2}{2}.

For details, we refer, for example, to [11] for (i) and (i)′; to [27] for (ii) and also for a relation with (i) (under some parity assumption on eigenspaces); to [15], §6 (see also [14]) for (iii)′ ⇔ (ii); and to [34] for (iii) ⇔ (iii)′. The second statement follows from a general theory of the boundary value problem with regular singularities; or also follows from a classical asymptotic formula of hypergeometric functions (see (8.3.1)) in our specific setting.

Remark. 1) By definition, (i) and (i)′ make sense for p > 1 and q > 0; and others for p > 1 and q ≥ 0.

2) Each of the realization (i), (i)′, (ii), and (iii) also gives a globalization of π_{+,λ}^{p,q}, namely, a continuous representation of G on a topological vector space. Because all of (π_{+,λ}^{p,q})_{K} (λ ∈ A_{0}(p, q)) are unitarizable we may and do take the globalization π_{+,λ}^{p,q} to be the unitary representation of G.

3) If λ > 0 and λ ∈ A_{0}(p, q), then the realization (ii) of π_{+,λ}^{p,q} gives a discrete series representation for X(p, q). Conversely,

{π_{+,λ}^{p,q} : λ ∈ A_{0}(p, q), λ > 0}

exhausts the set of discrete series representations for X(p, q).

If (p, q) = (1, 0), then O(p, q) ≃ O(1) and it is convenient to define representations of O(1) by

π_{+,λ}^{1,0} = \{ \begin{matrix}1 & (λ = −\frac{1}{2}), \\ sgn & (λ = \frac{1}{2}), \\ 0 & (otherwise). \end{matrix}

As we defined π_{+,λ}^{p,q} in Fact 5.4, we can also define an irreducible unitary representation, denoted by π_{−,λ}^{p,q}, for λ ∈ A_{0}(q, p) such that the underlying (𝔤, K)-module has the following K-type

⨁_{\begin{matrix}m,n∈ℕ \\ m−n≤−λ+ \frac{q}{2} − \frac{p}{2} −1 \\ m−n≡−λ+ \frac{q}{2} − \frac{p}{2} −1 mod 2 \end{matrix}} H^{m}(ℝ^{p}) ⊠ H^{n}(ℝ^{q}).

Similarly to π_{+,λ}^{p,q}, the representations π_{−,λ}^{p,q} are realized in function spaces on another hyperboloid O(p, q)/O(p, q − 1).

In order to understand the notation here, we remark:

i) π_{−,λ}^{p,q} ∈ { \atop O(p, q)} corresponds to the representation π_{+,λ}^{q,p} ∈ { \atop O(q, p)} if we identify O(p, q) with O(q, p).

ii) π_{+,λ}^{p,0} ≃ H^{k}(ℝ^{p}), where k = λ − \frac{p−2}{2} and p ≥ 1, k ∈ ℕ.

\underline{5.5.} The case λ = ±\frac{1}{2} is delicate, which happens when p + q ∈ 2ℕ + 1.

First, we assume p+q ∈ 2ℕ+1. By using the equivalent realizations of π_{+,λ}^{p,q} in Fact 5.4 and by the classification of the composition series of the most degenerate principal series representation Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}) (see [11]), we have non-splitting short exact sequences of (𝔤, K)-modules:

(5.5.1)  0 →(π_{−,− \frac{1}{2}}^{p,q})_{K} →Ind_{P^{max}}^{G}((−1)^{\frac{p−q+1}{2}} ⊗ ℂ_{− \frac{1}{2}} ) →(π_{+, \frac{1}{2}}^{p,q})_{K} → 0,
(5.5.2)  0 →(π_{+,− \frac{1}{2}}^{p,q})_{K} →Ind_{P^{max}}^{G}((−1)^{\frac{p−q−1}{2}} ⊗ ℂ_{− \frac{1}{2}} ) →(π_{−, \frac{1}{2}}^{p,q})_{K} → 0.

Because π_{+,λ}^{p,q} (λ ∈ A_{0}(p, q)) is self-dual, the dual (𝔤, K)-modules of (5.5.1) and (5.5.2) give the following non-splitting short exact sequences of (𝔤, K)-modules:

(5.5.3)  0 →(π_{+, \frac{1}{2}}^{p,q})_{K} →Ind_{P^{max}}^{G}((−1)^{\frac{p−q+1}{2}} ⊗ ℂ_{\frac{1}{2}} ) →(π_{−,− \frac{1}{2}}^{p,q})_{K} → 0,
(5.5.4)  0 →(π_{−, \frac{1}{2}}^{p,q})_{K} →Ind_{P^{max}}^{G}((−1)^{\frac{p−q−1}{2}} ⊗ ℂ_{\frac{1}{2}} ) →(π_{+,− \frac{1}{2}}^{p,q})_{K} → 0.

Next, we assume p + q ∈ 2ℕ. Then, ϖ^{p,q} is realized as a subrepresentation of some degenerate principal series (see [23], Lemma 3.7.2). More precisely, we have non-splitting short exact sequences of (𝔤, K)-modules

0 → ϖ_{K}^{p,q} → Ind_{P^{max}}^{G}((−1)^{\frac{p−q}{2}} ⊗ ℂ_{−1}) → ((π_{−,1}^{p,q})_{K} ⊕ (π_{+,1}^{p,q})_{K}) → 0, (5.5.5)

0 → ((π_{−,1}^{p,q})_{K} ⊕ (π_{+,1}^{p,q})_{K}) → Ind_{P^{max}}^{G}((−1)^{\frac{p−q}{2}} ⊗ ℂ_{1}) → ϖ_{K}^{p,q} → 0, (5.5.6)

and an isomorphism of (𝔤, K)-modules:

(5.5.7)  Ind_{P^{max}}^{G}((−1)^{\frac{p−q+2}{2}} ⊗ ℂ_{0}) ≃ (π_{−,0}^{p,q})_{K} ⊕ (π_{+,0}^{p,q})_{K}.

These results will be used in another realization of the unipotent representation ϖ^{p,q}, namely, as a submodule of the Dolbeault cohomology group in a subsequent paper (cf. Part 1, Introduction, Theorem B (4)).

§6. Conformal embedding of the hyperboloid

This section prepares the geometric setup which will be used in §7 and §9 for the branching problem of ϖ^{p,q}|_{G′}. Throughout this section, we shall use the following notation:

|x|^{2} := |x′|^{2} + |x′′|^{2} = ∑_{i=1}^{p′}(x′_{i})^{2} + ∑_{j=1}^{p′′}(x′′_{j})^{2}, for x := (x′, x′′) ∈ ℝ^{p′+p′′} = ℝ^{p},

|y|^{2} := |y′|^{2} + |y′′|^{2} = ∑_{i=1}^{q′}(y′_{i})^{2} + ∑_{j=1}^{q′′}(y′′_{j})^{2}, for y := (y′, y′′) ∈ ℝ^{q′+q′′} = ℝ^{q}.

\underline{6.1.} We define two open subsets of ℝ^{p+q} by

ℝ_{+}^{p′+p′′,q′+q′′} := {(x, y) = ((x′, x′′), (y′, y′′)) ∈ ℝ^{p′+p′′,q′+q′′} : |x′| > |y′|},
ℝ_{−}^{p′+p′′,q′+q′′} := {(x, y) = ((x′, x′′), (y′, y′′)) ∈ ℝ^{p′+p′′,q′+q′′} : |x′| < |y′|}.

Then the disjoint union ℝ_{+}^{p′+p′′,q′+q′′} ∪ℝ_{−}^{p′+p′′,q′+q′′} is open dense in ℝ^{p+q}. Let us consider the intersection of ℝ_{±}^{p′+p′′,q′+q′′} with the submanifolds given in §3.2:

M ⊂ Ξ ⊂ ℝ^{p,q}.

Then, we define two open subsets of M ≃ S^{p−1} × S^{q−1} by

(6.1.1)  M_{±} := M ∩ ℝ_{±}^{p′+p′′,q′+q′′}.

Likewise, we define two open subsets of the cone Ξ by

(6.1.2)  Ξ_{±} := Ξ ∩ ℝ_{±}^{p′+p′′,q′+q′′}.

We notice that if (x, y) = ((x′, x′′), (y′, y′′)) ∈ Ξ then

|x′| > |y′| ⟺ |x′′| < |y′′|

because |x′|^{2} + |x′′|^{2} = |y′|^{2} + |y′′|^{2}. The following statement is immediate from definition:

(6.1.3)  Ξ_{+} = ∅ ⇔ M_{+} = ∅ ⇔ p′q′′ = 0.
(6.1.4)  Ξ_{−} = ∅ ⇔ M_{−} = ∅ ⇔ p′′q′ = 0.

\underline{6.2.} We embed the direct product of hyperboloids

X(p′, q′) × X(q′′, p′′) = {((x′, y′), (y′′, x′′)) : |x′|^{2} − |y′|^{2} = |y′′|^{2} − |x′′|^{2} = 1}.

into Ξ_{+} (⊂ ℝ^{p,q}) by the map

(6.2.1)  X(p′, q′) × X(q′′, p′′) ↪ Ξ_{+}, ((x′, y′), (y′′, x′′)) ↦ (x′, x′′, y′, y′′).

The image is transversal to rays (see [23], §3.3 for definition) and the induced pseudo-Riemannian metric g_{X(p′,q′)×X(q′′,p′′)} on X(p′, q′) × X(q′′, p′′) has signature (p′ − 1, q′) + (p′′, q′′ − 1) = (p − 1, q − 1). With the notation in §5.2, we have

g_{X(p′,q′)×X(q′′,p′′)} = g_{X(p′,q′)} ⊕ (−g_{X(q′′,p′′)}).

We note that if p′′ = q′ = 0, then X(p′, q′) × X(q′′, p′′) is diffeomorphic to S^{p−1} × S^{q−1}, and g_{X(p′,0)×X(q′′,0)} is nothing but the pseudo-Riemannian metric g_{S^{p−1}×S^{q−1}} of signature (p − 1, q − 1) (see [23], §3.3).

By the same computation as in (3.4.1), we have the relationship among the Yamabe operators on hyperboloids (see also (5.2.2)) by

(6.2.2)  Δ̃_{X(p′,q′)×X(q′′,p′′)} = Δ̃_{X(p′,q′)} − Δ̃_{X(q′,p′′)}.

We denote by Φ_{1} the composition of (6.2.1) and the projection Φ: Ξ → M (see [23], (3.2.4)), namely,

(6.2.3) Φ_{1} : X(p′, q′) × X(q′′, p′′) ↪ M, ((x′, y′), (y′′, x′′)) ↦ (\frac{(x′, x′′)}{|x|}, \frac{(y′, y′′)}{|y|}) .

Lemma 6.2. 1) The map Φ_{1} : X(p′, q′) × X(q′′, p′′) → M is a diffeomorphism onto M_{+}. The inverse map Φ_{1}^{−1} : M_{+} → X(p′, q′) × X(q′′, p′′) is given by the formula:

(6.2.4)  ((u′, u′′), (v′, v′′)) ↦ (\frac{(u′, v′)}{\sqrt{|u′|^{2} − |v′|^{2}}} , \frac{(v′′, u′′)}{\sqrt{|v′′|^{2} − |u′′|^{2}}} ) .

2) Φ_{1} is a conformal map with conformal factor |x|^{−1} = |y|^{−1}, where x = (x′, x′′) ∈ ℝ^{p′+p′′} and y = (y′, y′′) ∈ ℝ^{q′+q′′}. Namely, we have

Φ_{1}^{∗} (g_{S^{p−1}×S^{q−1}}) = \frac{1}{|x|^{2}} g_{X(p′,q′)×X(q′′,p′′)}.

Proof. The first statement is straightforward in light of the formula

|u′|^{2} − |v′|^{2} = |v′′|^{2} − |u′′|^{2} > 0

for (u, v) = ((u′, u′′), (v′, v′′)) ∈ M_{+} ⊂ S^{p−1} × S^{q−1}. The second statement is a special case of Lemma 3.3. □

\underline{6.3.} Now, the conformal diffeomorphism Φ_{1} : X(p′, q′) × X(q′′, p′′) →^{∼} M_{+} establishes a bijection of the kernels of the Yamabe operators owing to Proposition 2.6:

Lemma 6.3. Φ̃_{1}^{∗} gives a bijection from Ker Δ̃_{M_{+}} onto Ker Δ̃_{X(p′,q′)×X(q′′,p′′)}.

Here, the twisted pull-backs Φ̃_{1}^{∗} and { \atop (Φ_{1}^{−1})^{∗}} (see Definition 2.3), namely,

(6.3.1)  Φ̃_{1}^{∗} :  C^{∞}(M_{+})  → C^{∞}(X(p′, q′) × X(q′′, p′′)),
(6.3.2)  { \atop (Φ_{1}^{−1})^{∗}}: C^{∞}(X(p′, q′) × X(q′′, p′′)) →  C^{∞}(M_{+}),

are given by the formulae

(Φ̃_{1}^{∗}F)(x′, y′, y′′, x′′) := (|x′|^{2} + |x′′|^{2})^{− \frac{p+q−4}{4}} F (\frac{(x′, x′′)}{\sqrt{|x′|^{2} + |x′′|^{2}}} , \frac{(y′, y′′)}{\sqrt{|y′|^{2} + |y′′|^{2}}} ) ,

({ \atop (Φ_{1}^{−1})^{∗}}f)(u′, u′′, v′, v′′) := (|u′|^{2} − |v′|^{2})^{− \frac{p+q−4}{4}} f (\frac{(u′, v′)}{\sqrt{|u′|^{2} − |v′|^{2}}} , \frac{(u′′, v′′)}{\sqrt{|v′′|^{2} − |u′′|^{2}}} ) ,

respectively. We remark that { \atop (Φ_{1}^{−1})^{∗}} = (Φ̃_{1}^{∗})^{−1}.

\underline{6.4.} Similarly to §6.2, we consider another embedding

(6.4.1)  X(q′, p′) × X(p′′, q′′) ↪ Ξ_{−}, ((y′, x′), (x′′, y′′)) ↦ (x′, x′′, y′, y′′).

The composition of (6.4.1) and the projection Φ: Ξ → M is denoted by

(6.4.2) Φ_{2} : X(q′, p′) × X(p′′, q′′) ↪ M, ((y′, x′), (x′′, y′′)) ↦ (\frac{(x′, x′′)}{|x|}, \frac{(y′, y′′)}{|y|}) .

Obviously, results analogous to Lemma 6.2 and Lemma 6.3 hold for Φ_{2}. For example, here is a lemma parallel to Lemma 6.2:

Lemma 6.4. The map Φ_{2} : X(q′, p′) × X(p′′, q′′) → M_{−} is a conformal diffeomorphism onto M_{−}. The inverse map Φ_{2}^{−1} : M_{−} → X(q′, p′) × X(p′′, q′′) is given by

((u′, u′′), (v′, v′′)) ↦ (\frac{(v′, u′)}{\sqrt{|v′|^{2} − |u′|^{2}}} , \frac{(u′′, v′′)}{\sqrt{|u′′|^{2} − |v′′|^{2}}} ) .

§7. Explicit branching formula (discrete decomposable case)

If one of p′, q′, p′′ or q′′ is zero, then the restriction ϖ^{p,q}|_{G′} is decomposed discretely into irreducible representations of G′ = O(p′, q′) × O(p′′, q′′) as we saw in §4. In this case, we can determine the branching laws of ϖ^{p,q}|_{G′} as follows:

Theorem 7.1. Let p + q ∈ 2ℕ. If q′′ ≥ 1 and q′ + q′′ = q, then we have an irreducible decomposition of the unitary representation ϖ^{p,q} when restricted to O(p, q′) × O(q′′):

ϖ^{p,q}|_{O(p,q′)×O(q′′)} ≃ ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix}π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ π_{−,l+ \frac{q′′}{2} −1}^{0,q′′}

≃ ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix}π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ H^{l}(ℝ^{q′′}). (7.1.1)

We shall prove Theorem 7.1 in §7.5 after we prepare an algebraic lemma in §7.3 and a geometric lemma in §7.4.

Remark. The formula in Theorem 7.1 is nothing but a K-type formula (see Theorem 3.6.1) when q′ = 0.

\underline{7.2.} The branching law (7.1.1) is an infinite direct sum for q′′ > 1. This subsection treats the case q′′ = 1, which is particularly interesting, because the branching formula consists of only two irreducible representations (we recall H^{l}(ℝ^{1}) ≠ 0 if and only if l = 0, 1). For simplicity, we shall assume q ≥ 3 in §7.2.

It follows from Theorem 7.1 with q′ = 1 that

ϖ^{p,q}|_{O(p,q−1)×O(1)} ≃ (π_{+,− \frac{1}{2}}^{p,q−1} ⊠ π_{−,− \frac{1}{2}}^{0,1} ) ⊕ (π_{+, \frac{1}{2}}^{p,q−1} ⊠ π_{−, \frac{1}{2}}^{0,1} )

≃ (π_{+,− \frac{1}{2}}^{p,q−1} ⊠ 1) ⊕ (π_{+, \frac{1}{2}}^{p,q−1} ⊠ sgn) . (7.2.1)

This means that ϖ_{K}^{p,q} can be realized in a subspace of C∞_{\frac{1}{2}}(X(p, q −1)), namely, the kernel

of the Yamabe operator Δ̃_{X(p,q−1)} (see (5.2.3)).

More precisely, according to the direct sum decomposition (see (5.2.4)), we have

(7.2.2) Ker Δ̃_{X(p,q−1)} ≡ C∞_{\frac{1}{2}}(X(p, q − 1)) = C∞_{\frac{1}{2}}(X(p, q − 1))_{+} + C∞_{\frac{1}{2}}(X(p, q − 1))_{−}.

We recall that the central element −I_{p+q} ∈ G acts on ϖ^{p,q} with scalar δ, where

(7.2.3)  δ := (−1)^{\frac{p−q}{2}} .

In view of the composition series of eigenspaces on the hyperboloid (see [27] for the case δ = +; similar for δ = −), we have non-splitting exact sequences of Harish-Chandra modules of O(p, q − 1):

(7.2.4)  0 →(π_{+,− \frac{1}{2}}^{p,q−1})_{K} → (C∞_{\frac{1}{2}}(X(p, q − 1))_{δ})_{K} →(π_{−, \frac{1}{2}}^{p,q−1})_{K} → 0,
(7.2.5)  0 →(π_{+, \frac{1}{2}}^{p,q−1})_{K} →(C∞_{\frac{1}{2}}(X(p, q − 1))_{−δ})_{K} →(π_{−,− \frac{1}{2}}^{p,q−1})_{K} → 0,

Here is a realization of ϖ^{p,q} in a subspace of the kernel of the Yamabe operator on the hyperboloid X(p, q −1), on which O(p, q) acts as meromorphic conformal transformations.

Corollary 7.2.1. Let W_{±} be the unique non-trivial subrepresentation of O(p, q − 1) in (Ker Δ̃_{X(p,q−1)})_{±} ≡ C∞_{\frac{1}{2}}(X(p, q − 1))_{±}. Each of the underlying (𝔤, K)-module is infinites-

imally unitarizable, and we denote the resulting unitary representation by \overline{W}_{±}. Then, the irreducible unitary representation ϖ^{p,q} of O(p, q) is realized on the direct sum \overline{W_{+}} + \overline{W_{−}}.

We note that \overline{W_{−δ}} ⊂ L^{2}(X(p, q − 1)) and \overline{W_{δ}} ⊄ L^{2}(X(p, q − 1)) where δ = (−1)^{\frac{p−q}{2}} . It is interesting to note that the Laplacian Δ_{X(p,q−1)} acts on a discrete series π_{+,λ}^{p,q−1} for

the hyperboloid X(p, q − 1) as a scalar −λ^{2} + \frac{1}{4}(p + q − 3)^{2}, which attains the maximum when λ = \frac{1}{2} if p + (q − 1) ∈ 2ℕ + 1.

Taking the direct sum of two exact sequences (7.2.4) and (7.2.5), we have the following:

Theorem 7.2.2. There is a non-split exact sequence of Harish-Chandra modules for O(p− 1, q).

(7.2.6)  0 → (ϖ^{p,q})_{K} → (Ker Δ̃_{X(p,q−1)})_{K} → (ϖ^{p+1,q−1})_{K} → 0.

It is a mysterious phenomenon in (7.2.6) that ϖ^{p,q} extends to a representation of O(p, q) and (ϖ^{p+1,q−1})_{K} to that of O(p + 1, q − 1). So, different real forms of O(p + q, ℂ) act on subquotients of the kernel of the Yamabe operator on the hyperboloid X(p, q − 1) = O(p, q − 1)/O(p − 1, q − 1) !

Here, we remark that Ker Δ̃_{X(p,q−1)} ∩ L^{2}(X(p, q − 1)) ≠ {0} if and only if p + q ∈ 2ℤ, by the classification of discrete series for the hyperboloid X(p, q − 1) for p > 1.

\underline{7.3.} By Theorem 4.2, the restriction ϖ^{p,q}|_{K′} is K′-admissible, where K′ ≃ O(p)×O(q′)× O(q′′). Let us first find the K′-structure of ϖ^{p,q}. We recall a classical branching law with respect to (O(q), O(q′) × O(q′′)) (q = q′ + q′′, q′ ≥ 1, q′′ ≥ 1):

(7.3.1)  H^{n}(ℝ^{q}) ≃ ⨁_{\begin{matrix}k,l∈ℕ \\ k+l≤n \\ k+l≡n \end{matrix}} H^{k}(ℝ^{q′}) ⊠ H^{l}(ℝ^{q′′}).

Then we have isomorphisms as K′-modules:

⨁_{\begin{matrix}m,n∈ℕ \\ m−n=b \\ m−n≡b mod 2 \end{matrix}} H^{m}(ℝ^{p}) ⊠ H^{n}(ℝ^{q})|_{O(p)×O(q′)×O(q′′)}

≃ ⨁_{\begin{matrix}m,n∈ℕ \\ m−n=b \\ m−n≡b mod 2 \end{matrix}} ⨁_{\begin{matrix}k,l∈ℕ \\ k+l≤n \\ k+l≡n mod 2 \end{matrix}} H^{m}(ℝ^{p}) ⊠ H^{k}(ℝ^{q′}) ⊠ H^{l}(ℝ^{q′′})

≃ ⨁_{l∈ℕ} ⨁_{\begin{matrix}m,k∈ℕ \\ m−b≥k+l \\ m−b≡k+l mod 2 \end{matrix}} H^{m}(ℝ^{p}) ⊠ H^{k}(ℝ^{q′}) ⊠ H^{l}(ℝ^{q′′})

≃ ⨁_{l∈ℕ} Ξ(O(p) × O(q′) : b + l) ⊠ H^{l}(ℝ^{q′′})

In view of Theorem 3.6.1, we have proved:

Lemma 7.3. We have an isomorphism of K′-modules:

(7.3.2)  ϖ_{K}^{p,q} ≃ ⨁_{l∈ℕ} Ξ(O(p) × O(q′) : \frac{q − p}{2} + l) ⊠ H^{l}(ℝ^{q′′})

\underline{7.4.} By Theorem 4.2 and [17], Lemma 1.3, the underlying (𝔤, K)-module ϖ_{K}^{p,q} is decomposed into an algebraic direct sum of irreducible (𝔤′, K′)-modules:

(7.4.1)  ϖ_{K}^{p,q} ≃ ⨁_{τ} m_{τ} τ ≃ ⨁_{τ_{1},τ_{2}} m_{τ_{1},τ_{2}}τ_{1} ⊠ τ_{2},

where m_{τ} ∈ ℕ and τ runs over irreducible (𝔤′, K′)-modules or equivalently, τ_{1} runs over irreducible (𝔤′_{1}, K′_{1})-modules (with obvious notation for G′_{1} := O(p, q′)) and τ_{2} runs over irreducible O(q′′)-modules. It follows from Lemma 7.3 that for each l there exists a (𝔤′_{1}, K′_{1})- module W_{l} which is a direct sum of irreducible (𝔤′_{1}, K′_{1})-module such that W_{l} is isomorphic to Ξ(O(p) × O(q′) : \frac{q−p}{2} + l) as K′_{1}-modules. Let us prove that W_{l} is in fact irreducible as a (𝔤′_{1}, K′_{1})-module.

Lemma 7.4. W_{l} is realized in a subspace of C_{λ}^{∞}(X(p, q′)) with λ = l + \frac{q′′}{2} − 1.

Proof. In our conformal construction of ϖ^{p,q} in §3, we recall that each K-finite vector of ϖ^{p,q} is an analytic function satisfying the Yamabe equation on M ≃ S^{p−1} × S^{q−1}. By using the conformal diffeomorphism Φ_{1} : X(p, q′) × S^{q′′−1} → M_{+}, an open dense subset of M (see (6.1.4) with p′′ = 0), we can realize W_{l} × H^{l}(ℝ^{q′′}) in the space of smooth functions on X(p, q′) × S^{q′′−1} satisfying the Yamabe equation by the following diagram:

A(M) ⊂_{dense}C^{∞}(M) ↪_{restriction}C^{∞}(M_{+}) →^{∼}_{Φ_{1}^{∗}} C^{∞}(X(p, q′) × S^{q′′−1})
∪  ∪  ∪  ∪
W_{l} ⊠ H^{l}(ℝ^{q′′}) ⊂ ϖ_{K}^{p,q} ⊂_{dense} Ker Δ̃_{M} ↪_{restriction} Ker Δ̃_{M_{+}} →^{∼}_{Φ_{1}^{∗}} Ker Δ̃_{X(p,q′)×S^{q′′−1}}.

Because Δ̃_{X(p,q′)×S^{q′′−1}} = Δ̃_{X(p′,q′)} − Δ̃_{S^{q′′−1}} (see (6.2.2)) acts on W_{l} ⊠ H^{l}(ℝ^{q′′}) as 0, and

because Δ̃_{S^{q′′−1}} acts on H^{l}(ℝ^{q′′}) as a scalar \frac{1}{4} − (l + \frac{q′′}{2} − 1)^{2} (see (3.5.1)), we conclude that Δ̃_{X(p′,q′)} acts on W_{l} as the same scalar. Hence, Lemma is proved. □

\underline{7.5.} Let us complete the proof of Theorem 7.1. It follows from Lemma 7.4 together with the K′_{1}-structure of W_{l} in §7.3 that W_{l} is irreducible and isomorphic to (π_{+,l+ \frac{q′′−2}{2}}^{p,q′})_{K′_{1}} as

(𝔤′_{1}, K′_{1})-module (see Fact 5.4). Therefore, we have an isomorphism of (𝔤′, K′)-modules

ϖ_{K}^{p,q} ≃ ⨁_{l=0}^{∞}(π_{+,l+ \frac{q′′}{2} −1}^{p,q′})_{K′_{1}} ⊠ H^{l}(ℝ^{q′′}) (algebraic direct sum).

Taking the closure in the Hilbert space, we have (7.1.1). Hence Theorem 7.1 is proved. □

\underline{7.6.} So far, we have not used the irreducibility of ϖ^{p,q} in the branching law. Although the irreducibility of ϖ^{p,q} (p, q) ≠ (2, 2) is known [2], we can give a new and simple proof for it, as an application of the branching formulae in Theorem 7.1.

Theorem 7.6. Let p, q ≥ 2, p + q ∈ 2ℤ and (p, q) ≠ (2, 2). Then, ϖ^{p,q} is an irreducible representation of O(p, q).

Proof. Suppose W ≠ {0} is a closed invariant subspace of the unitary representation (ϖ^{p,q}, \overline{V^{p,q}}). We want to prove W = \overline{V^{p,q}}.

Without loss of generality (p and q play a symmetric role), we may assume q ≥ 3,

and fix q′ ≥ 1, q′′ ≥ 2 such that q′ + q′′ = q. We write \overline{V^{p,q}} = ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix} V_{l} according to the

irreducible decomposition (7.1.1) of G′ = G′_{1} × G′_{2} := O(p, q′) × O(q′′).

Because W is non-zero, W contains a K-type of the form H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q}), which we fix once for all. In view of the branching formula (7.3.1), H^{b}(ℝ^{q}) contains a non-zero O(q′′)-fixed vector. That is,

V_{0} := \overline{V^{p,q}}^{G′_{2}} ⊃ W^{G′_{2}} ⊃ H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q})^{G′_{2}} ≠ \{0\}

Because G′_{1} = O(p, q′) acts on V_{0} by π_{+, \frac{q′′}{2} −1}^{p,q′} (see Theorem 7.1) which is irreducible, and

because V_{0} ⊃ W^{G′_{2}} is stable under the action of G′_{1}, we conclude that V_{0} = W^{G′_{2}}. Hence, we have proved

(7.6.1)  W ⊃ V_{0}.

If W ≠ \overline{V^{p,q}}, the orthogonal complement W^{⊥} ⊃ V_{0} by the same argument. This would contradict to W ∩ W^{⊥} = \{0\}. Therefore, W must coincides with \overline{V^{p,q}}. Hence, ϖ^{p,q} is irreducible. □

§8. Inner product on ϖ^{p,q} and the Parseval-Plancherel formula

In this section, we prove the Parseval-Plancherel type formula for our discrete decomposable branching law given in Theorem 7.1. Our main result in this section is Theorem 8.5.

\underline{8.1.} The subsections §8.1 ∼ §8.3 review some results and notation from [21] without proof. We set

A_{+}(p, q) := A_{0}(p, q) ∩ {λ ∈ ℝ : λ > 0},
Λ_{+−}(λ) := {(λ′, λ′′) ∈ A_{+}(p′, q′) × A_{+}(q′′, p′′) : λ′ − λ′′ − λ − 1 ∈ 2ℕ},
Λ_{++}(λ) := {(λ′, λ′′) ∈ A_{+}(p′, q′) × A_{+}(p′′, q′′) : λ − λ′ − λ′′ − 1 ∈ 2ℕ}.

We recall the Jacobi function

φ_{iλ}^{(λ′′,λ′)}(t) := _{2}F_{1} (\frac{λ′ + λ′′ + 1 − λ}{2}, \frac{λ′ + λ′′ + 1 + λ}{2}; λ′′ + 1; − sinh^{2} t) ,

which satisfies

φ_{iλ}^{(λ′′,λ′)}(0) = 1,

(t(1 − t)\frac{d^{2}}{dt^{2}} + (c − (a + b + 1)t)\frac{d}{dt} − ab) φ_{iλ}(t) = 0

with

a := \frac{λ′ + λ′′ + 1 − λ}{2}, b := \frac{λ′ + λ′′ + 1 + λ}{2}, c := λ′′ + 1.

We set meromorphic functions of three variables λ, λ′ and λ′′ by

V_{+−,λ}^{(λ′,λ′′)} := \frac{(Γ(λ′′ + 1))^{2} Γ(\frac{λ′−λ′′+λ+1}{2}) Γ(\frac{λ′−λ′′−λ+1}{2})}{2λ Γ(\frac{λ′+λ′′+λ+1}{2}) Γ(\frac{λ′+λ′′−λ+1}{2})},

V_{++,λ}^{(λ′,λ′′)} := \frac{(Γ(λ′′ + 1))^{2} Γ(\frac{−λ′−λ′′+λ+1}{2}) Γ(\frac{λ′−λ′′+λ+1}{2})}{2λ Γ(\frac{−λ′+λ′′+λ+1}{2}) Γ(\frac{λ′+λ′′+λ+1}{2})}.

Then we have

∫_{0}^{∞} |φ_{iλ}^{(λ′′,λ′)}(t)|^{2}(cosh t)^{2λ′+1}(sinh t)^{2λ′′+1} dt = V_{+−,λ}^{(λ′,λ′′)} for (λ′, λ′′) ∈ Λ_{+−}(λ),
∫0^{\frac{π}{2}} |φ_{iλ}^{(λ′′,λ′)}(iθ)|^{2}(cos θ)^{2λ′+1}(sin θ)^{2λ′′+1} dθ = V_{++,λ}^{(λ′,λ′′)} for (λ′, λ′′) ∈ Λ_{++}(λ).

\underline{8.2.} We put

(8.2.1)  M ≡ M(λ, λ′, λ′′) := (−1)^{\frac{λ′−λ′′−λ−1}{2}} \frac{Γ(\frac{λ′+λ′′−λ+1}{2})Γ(λ + 1)}{Γ(\frac{λ′−λ′′+λ+1}{2})Γ(λ′′ + 1)}.

Lemma 8.2 (triangular relation of the Jacobi function). If cot θ = cosh t (0 < θ < \frac{π}{2} , 0 < t), then

φ_{iλ′}^{(λ,λ′′)}(iθ) = M(cosh t)^{λ+λ′+λ′′+1}φ_{iλ}^{(λ′′,λ′)}(t).

Furthermore, we have

(8.2.2)  λ′V_{++,λ′}^{(λ′′,λ)} = M^{2}λV_{+−,λ}^{(λ′,λ′′)}.

\underline{8.3.} Unitarizaition of π_{+,λ}^{p,q} (λ ∈ A_{0}(p, q)).

In this subsection, we give an explicit unitary inner product on π_{+,λ}^{p,q} for λ ∈ A_{0}(p, q). In view of Fact 5.4, the (𝔤, K)-module (π_{+,λ}^{p,q})_{K} is realized in C_{λ}^{∞}(X(p, q))_{K} with K-types Ξ(K : b) where b = λ − \frac{p}{2} + \frac{q}{2} + 1. Suppose f ∈ C_{λ}^{∞}(X(p, q))_{K} belongs to the K-type H^{m}(ℝ^{p}) ⊠ H^{n}(ℝ^{q}) ∈ Ξ(K : b). Then f is of the form

(8.3.1)  f(ω cosh t, η sinh t) = h_{m}(ω)h_{n}(η)(cosh t)^{m}(sinh t)^{n}φ_{iλ}^{(n+ \frac{q}{2} −1,m+ \frac{p}{2} −1)}(t),

where h_{m} ∈ H^{m}(ℝ^{p}), h_{n} ∈ H^{n}(ℝ^{q}), ω ∈ S^{p−1}, η ∈ S^{q−1}, and t > 0. We put

(8.3.2)  ∥f∥_{π_{+,λ}^{p,q}}^{2} := ∥h_{m}∥_{L^{2}(S^{p−1})}^{2} ∥h_{n}∥_{L^{2}(S^{q−1})}^{2} λV_{+−,λ}^{(m+ \frac{p−2}{2} ,n+ \frac{q−2}{2} )}.

This norm defines an inner product on (π_{+,λ}^{p,q})_{K} (λ ∈ A_{0}(p, q)) and one defines an irreducible unitary representation of G = O(p, q) on its Hilbert completion. There is an obvious inner product for λ > 0, because (π_{+,λ}^{p,q})_{K} ⊂ L^{2}(X(p, q)). The relation between our norm ∥ ∥_{π_{+,λ}^{p,q}} and L^{2}-norm ∥ ∥_{L^{2}(X(p,q))} is given by

(8.3.3)  ∥f∥_{π_{+,λ}^{p,q}}^{2} = λ∥f∥_{L^{2}(X(p,q))}^{2}, for any f ∈ (π_{+,λ}^{p,q})_{K},

if λ > 0. For λ ∈ A_{0}(p, q) such that λ ≤ 0 (this can happen if λ = 0, −\frac{1}{2}), π_{+,λ}^{p,q} is also unitarizable by this inner product ( , )_{π_{+,λ}^{p,q}} given in (8.3.2), which is still positive definite.

Remark 8.3 (Unitarity). As we explained, all of (π_{+,λ}^{p,q})_{K} are unitarizable for λ ∈ A_{0}(p, q). We explain four different approaches:

1) If λ > 0, then (π_{+,λ}^{p,q})_{K} is unitarizable because of the realization in L^{2}(X(p, q)).

2) If λ ≥ 0, then (π_{+,λ}^{p,q})_{K} is unitarizable because of the realization of Zuckerman-Vogan’s derived functor module R_{𝔮}^{p−2}(ℂ_{λ}) with the parameter λ in the weakly fair range [31]. We note that the case λ = −\frac{1}{2} is not treated in the above two methods. However,

3) Use the classification of unitarizable subquotients of Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}) given in [11], §3. At last, here is a new proof of the unitarizability of (π_{+,λ}^{p,q})_{K} for all λ ∈ A_{0}(p, q). The

idea is to use our branching formula Theorem 7.1, for which the proof does not use the unitarizability of (π_{+,λ}^{p,q})_{K}:

4) All of (π_{+,λ}^{p,q})_{K} are unitarizable because they appear as discrete spectra in the branching law of a unitary representation ϖ^{p,q+c} of a larger group O(p, q + c) to O(p, q) × O(c) for some c > 0 (see Theorem 7.1). For this purpose, c ≤ 3 will do.

\underline{8.4.} We notice that the map Φ_{1} : X(p, q′) × S^{q′′−1} → M_{+} (6.2.3) (in the case p′′ = 0) is given by

((ω cosh t, η′ sinh t), η′′) ↦ (ω, (η′ cos θ, η′′ sin θ)),

where θ and t satisfy cot θ = cosh t. Suppose f ∈ C^{∞}(X(p, q′) × S^{q′′−1}) belongs to

π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ H^{l}(ℝ^{q′′})

as an O(p′) × O(q′′)-module, and furthermore to H^{m}(ℝ^{p}) ⊠ H^{n}(ℝ^{q′}) as an O(p) × O(q′)- module in the first factor. Then f is of the form:

f((ω cosh t, η′ sinh t), η′′) = h_{m}(ω) h_{k}(η′) h_{l}(η′′)(cosh t)^{m}(sinh t)^{k}φ_{i(l+ \frac{q′′}{2} −1)}^{(k+ \frac{q′}{2} −1,m+ \frac{p}{2} −1)}(t)

where h_{m} ∈ H^{m}(ℝ^{p}), h_{k} ∈ H^{k}(ℝ^{q′}), h_{l} ∈ H^{l}(ℝ^{q′′}), ω ∈ S^{p−1}, η′ ∈ S^{q′−1}, η′′ ∈ S^{q′′−1}, and t > 0.

Lemma 8.4. The twisted pull-back { \atop (Φ_{1}^{−1})^{∗}}: C^{∞}(X(p, q′) × S^{q′′−1}) → C^{∞}(M_{+}) (see (6.3.2) for definition) is given by the formula:

(8.4.1) ({ \atop (Φ_{1}^{−1})^{∗}}f)(ω, (η′ cos θ, η′′ sin θ))

= M^{−1}h_{m}(ω) h_{k}(η′) h_{l}(η′′) (cos θ)^{k} (sin θ)^{l} φ_{i(n+ \frac{q}{2} −1)}^{(l+ \frac{q′′}{2} −1,k+ \frac{q′}{2} −1)}(iθ).

Proof. In view of the definition, we have

({ \atop (Φ_{1}^{−1})^{∗}}f)(ω, (η′ cos θ, η′′ sin θ)) = (cosh t)^{\frac{p+q−4}{2}} f((ω cosh t, η′ sinh t), η′′).

Then (8.4.1) follows from Lemma 8.2 and from m + \frac{p}{2} = n + \frac{q}{2}. □

\underline{8.5.} Let G′ = O(p, q′)×O(q′′). We define an inner product on the irreducible constituent π_{+,λ}^{p,q′} ⊠π_{−,λ}^{0,q′′} in the branching law ϖ^{p,q}|_{G′} (see Theorem 7.1) for each λ = l + \frac{q′′}{2} −1 (l ∈ ℕ) by

∥f∥ := ∥f_{1}∥_{π_{+,λ}^{p,q′}} ∥f_{2}∥_{L^{2}(S^{q′′−1})}

for f = f_{1} f_{2} with f_{1} ∈ π_{+,λ}^{p,q′} and f_{2} ∈ π_{−,λ}^{0,q′′} ≃ H^{l}(ℝ^{q′′}).

Because we use an explicit map to prove the branching law, the generalized Parseval-Plancherel formula makes sense.

Theorem 8.5. 1) If we develop F ∈ Ker Δ̃_{M} as F = ∑_{l}^{∞} F_{l}^{(1)}F_{l}^{(2)} according to the irreducible decomposition

(8.5.1)  { \atop (Φ_{1})^{∗}}: ϖ^{p,q}|_{O(p,q′)×O(q′′)} →^{∼} ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix}π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ H^{l}(ℝ^{q′′})

then we have

(8.5.2)  ∥F ∥_{ϖ^{p,q}}^{2} = ∑_{l=0}^{∞} ∥F_{l}^{(1)}∥_{π_{+,l+ \frac{q′′}{2} −1}^{p,q′}}^{2}∥ F_{l}^{(2)}∥_{L^{2}(S^{q′′−1})}^{2}.

2) In particular, if q′′ ≥ 3, then all of π_{+,l+ \frac{q′′}{2} −1}^{p,q′} are discrete series for the hyperboloid

X(p, q′) and

(8.5.3)  ∥F ∥_{ϖ^{p,q}}^{2} = ∑_{l=0}^{∞}(l + \frac{q′′}{2} − 1) ∥F_{l}^{(1)}∥_{L^{2}(X(p,q′))}^{2}∥ F_{l}^{(2)}∥_{L^{2}(S^{q′′−1})}^{2}.

Remark. The formula (8.5.3) coincides with the Kostant-Binegar-Zierau formula (see §3) in the special case where q′ = 0 (namely, where G′ is compact).

Proof. We write λ := l + \frac{q′′}{2} − 1, λ′ := m + \frac{p}{2} − 1 = n + \frac{q}{2} − 1 and λ′′ := k + \frac{q′}{2} − 1. If F is of the form of the right side of (8.4.1), then

\frac{∥{ \atop (Φ_{1})^{∗}}F ∥_{π_{+,λ}^{p,q′} ⊠H^{l}(ℝ^{q′′})}^{2}}{∥F ∥_{ϖ^{p,q}}^{2}} =\frac{M^{2} ∥h_{m}∥_{L^{2}(S^{p−1})}^{2} ∥h_{k}∥_{L^{2}(S^{q′−1})}^{2} ∥h_{l}∥_{L^{2}(S^{q′′−1})}^{2} λV_{+−,λ}^{(λ′,λ′′)}}{∥h_{m}∥_{L^{2}(S^{p−1})}^{2} ∥h_{k}∥_{L^{2}(S^{q′−1})}^{2} ∥h_{l}∥_{L^{2}(S^{q′′−1})}^{2} λ′V_{++,λ′}^{(λ′′,λ)}}
=1

because of (8.2.1). Hence the first statement is proved. The second statement follows from (8.3.2). □

§9. Construction of discrete spectra in the branching laws

\underline{9.1.} In §7, we determined explicitly the branching law ϖ^{p,q}|_{G′} of the minimal unipotent representation ϖ^{p,q} ∈ Ĝ where G′ = O(p, q′) × O(q′′). The resulting branching law has no continuous spectrum (see Theorem 4.2 and Theorem 7.1). In this section, we treat a more general case where continuous spectrum may appear, namely, the branching law with respect to the semisimple symmetric pair

(G, G′) = (O(p, q), O(p′, q′) × O(p′′, q′′)),

where p′ + p′′ = p (≥ 2), q′ + q′′ = q (≥ 2), p + q ∈ 2ℕ, and (p, q) ≠ (2, 2).

We shall construct explicitly discrete spectra by using the conformal geometry.

Retain the notation in §5.1. We set

(9.1.1)  A′(p, q) := A_{0}(p, q) ∩ {λ ∈ ℝ : λ > 1}.

Theorem 9.1. The restriction of the unitary representation ϖ^{p,q}|_{G′} contains

\begin{matrix}∑ & ⊕ \\ λ∈A′(p′,q′)∩A′(q′′,p′′) \end{matrix}π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} ⊕ \begin{matrix}∑ & ⊕ \\ λ∈A′(q′,p′)∩A′(p′′,q′′) \end{matrix}π_{−,λ}^{p′,q′} ⊠ π_{+,λ}^{p′′,q′′}

as a discrete summand. Here, π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} ∈ Ĝ′ is the outer tensor product of π_{+,λ}^{p′,q′} ∈

{ \atop O(p′, q′)} and π_{−,λ}^{p′′,q′′} ∈ { \atop O(q′′, p′′)}.

We have already established the full branching law ϖ^{p,q}|_{G′} if p′q′p′′q′′ = 0. Thus, the main part of this section will be devoted to the proof of Theorem 9.1 when p′q′p′′q′′ ≠ 0. We shall give some remarks of Theorem 9.1 at the end of this subsection.

\underline{9.2.} Let K′ = O(p′)×O(q′)×O(p′′)×O(q′′). We realize the unitary representation ϖ^{p,q} on the Hilbert space \overline{V^{p,q}}. Here we recall the notation of §3.9, briefly as follows:

\overline{V^{p,q}} = Ker(D_{p} − D_{q}) ⊂_{closed} V = Dom(D_{p}) ∩ Dom(D_{q}) ( ⊂_{dense} L^{2}(M))

M = S^{p−1} × S^{q−1} ⊃_{dense} M_{+} ∪ M_{−}.

Different from Corollary 4.3 in the discretely decomposable case (p′q′p′′q′′ = 0), a K′-finite vector of a G′-irreducible summand (i.e. a discrete spectrum) in ϖ^{p,q}|_{G′} is not necessarily a real analytic function on M if p′q′p′′q′′ ≠ 0 in our conformal construction of ϖ^{p,q}. With

this in mind, we extend { \atop (Φ_{1}^{−1})^{∗}}f ∈ C^{∞}(M_{+}) (see §6 for notation) to a function, denoted by T_{+}f, on S^{p−1} × S^{q−1} as

(9.2.1)  T_{+}f := \{ \begin{matrix}{ \atop (Φ_{1}^{−1})^{∗}}f & on M_{+}, \\ 0 & on M \ M_{+}. \end{matrix}

Here is a key lemma:

Lemma 9.2. Suppose λ′ ∈ A_{0}(p′, q′) and λ′′ ∈ A_{0}(q′′, p′′). Let f ∈ C^{∞}(X(p′, q′) × X(q′′, p′′)) be a K′-finite function which belongs to π_{+,λ′}^{p′,q′} ⊠ π_{−,λ′′}^{p′′,q′′} (see Fact 5.4 (1) (ii)).

1) If λ′ ≥ \frac{1}{2} and λ′′ ≥ \frac{1}{2}, then T_{+}f ∈ V.

2) If λ′ > 1 and λ′′ > 1, then Y (T_{+}f) ∈ L^{2}(M) and Y Y′(T_{+}f) ∈ L^{1}(M) for any smooth vector field Y, Y′ on M.

3) If λ′ = λ′′ > 1 then T_{+}f ∈ \overline{V^{p,q}}.

Before proving Lemma 9.2, we first show that Lemma 9.2 implies Theorem 9.1. In fact, Lemma 9.2 (3) constructs a non-zero (𝔤′, K′)-homomorphism

T_{+} : (π_{+,λ}^{p′,q′})_{K′_{1}} ⊠ (π_{−,λ}^{p′′,q′′})_{K′_{2}} → ϖ^{p,q},

for λ ∈ A′(p′, q′) ∩ A′(q′′, p′′). Then T_{+} is injective because (π_{+,λ}^{p′,q′})_{K′_{1}} ⊠ (π_{−,λ}^{p′′,q′′})_{K′_{2}} is irreducible. Since ϖ^{p,q} is a unitary representation of G, T_{+} extends to an isometry of unitary representations of G′:

π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} → ϖ^{p,q},

by taking the closure with respect to the inner product induced from ϖ^{p,q}. This proves Theorem 9.1 for the irreducible representation π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} that appears in the first summand. The second summand is constructed similarly by using the conformal diffeomorphism (see Lemma 6.4)

Φ_{2} : X(q′, p′) × X(p′′, q′′) →^{∼} M_{−} (⊂ M).

Hence, the proof of Theorem 9.1 is completed by assuming Lemma 9.2.

\underline{9.3.}

Remark 9.3. There exist λ′ ∈ A_{0}(p′, q′) and λ′′ ∈ A_{0}(q′′, p′′) satisfying λ′ = λ′′ > 1 if and only if p′ + q′ ≡ p′′ + q′′ mod 2 and p′ ≥ 2, q′′ ≥ 2. This implies p + q ∈ 2ℕ, p ≥ 2 and q ≥ 2, and V^{p,q} is non-zero (see §3). Of course, Lemma 9.2 (3) also implies V^{p,q} ≠ {0}.

\underline{9.4.} It follows from Lemma 3.8.1 that the first statement of Lemma 9.2 is proved if Y (T_{+}) ∈ L^{2−ϵ} for any ϵ > 0 and for any smooth vector field Y on M. Therefore, both of (1) and (2) of Lemma 9.2 are proved by studying the asymptotic behaviour of T_{+}f near the boundary of M_{+} in M. This asymptotic estimate is studied below.

Any K′-finite vector f is a finite linear combination of the form f_{1}f_{2}, where

f_{1} ∈ C^{∞}(X(p′, q′)) is an O(p′) × O(q′)-finite vector that belongs to π_{+,λ′}^{p′,q′},
f_{2} ∈ C^{∞}(X(q′′, p′′)) is an O(p′′) × O(q′′)-finite vector that belongs to π_{−,λ′′}^{p′′,q′′}.

In order to prove Lemma 9.2, we may and do assume f is of the form f_{1}f_{2}. Then we have

(9.4.1) (T_{+}(f_{1}f_{2}))(u, v)

= (|u′|^{2} − |v′|^{2})_{+}^{− \frac{p+q−4}{4}} f_{1} (\frac{(u′, v′)}{\sqrt{|u′|^{2} − |v′|^{2}}} ) f_{2} (\frac{(u′′, v′′)}{\sqrt{|u′′|^{2} − |v′′|^{2}}} ) .

Here, we have used the following notation:

r_{+}^{ν} := \{ \begin{matrix}r^{ν} & (r > 0), \\ 0 & (r ≤ 0). \end{matrix}

\underline{9.5.} In order to analyze the asymptotic behaviour of T_{+}(f_{1}f_{2}) (see (9.4.1)) near the boundary of M_{+}, we consider a change of variables on S^{p−1} × S^{q−1} by the surjective map

(S^{1} × S^{p′−1} × S^{p′′−1}) × (S^{1} × S^{q′−1} × S^{q′′−1}) → S^{p−1} × S^{q−1},
(e^{iθ}, ω′, ω′′), (e^{iφ}, η′, η′′)  ↦ (u, v)

defined by

(9.5.1)  (u, v) ≡ (u′, u′′, v′, v′′) := (ω′ cos θ, ω′′ sin θ, η′ cos φ, η′′ sin φ).

Because |u′|^{2} − |v′|^{2} = |ω′ cos θ|^{2} − |η′ cos φ|^{2} = | cos θ|^{2} − | cos φ|^{2}, M_{±} defined in (6.1.1) is rewritten as

M_{+} = {(ω′ cos θ, ω′′ sin θ, η′ cos φ, η′′ sin φ) : | cos θ| > | cos φ|}
M_{−} = {(ω′ cos θ, ω′′ sin θ, η′ cos φ, η′′ sin φ) : | cos θ| < | cos φ|}

Here is an elementary computation corresponding to the change of variables (9.5.1):

Lemma 9.5. 1) The volume element du dv on S^{p−1} × S^{q−1} is given by

du dv = | cos θ|^{p′−1} | sin θ|^{p′′−1} | cos φ|^{q′−1} | sin φ|^{q′′−1} dθ dφ dω′ dω′′ dη′ dη′′,

where dω′ is the volume element on S^{p′−1} and so on.

2) Any smooth vector field on S^{p−1} × S^{q−1} is a linear combination of

\frac{1}{cos θ} X′, \frac{1}{sin θ} X′′, \frac{1}{cos φ}Y′, \frac{1}{sin φ}Y′′, \frac{∂}{∂φ}, \frac{∂}{∂θ} ,

whose coefficients are smooth functions of (ω′, ω′′, η′, η′′, θ, φ). Here, X′, X′′, Y′ and Y′′ are smooth vector fields on S^{p′−1}, S^{p′′−1}, S^{q′−1} and S^{q′′−1}, respectively.

Fact 5.4 (2) describes the asymptotic behaviour of K-finite functions that belong to discrete series representations for a hyperboloid. Applying it to f_{1} with respect to the coordinate (9.5.1), we find a_{1} ∈ C^{∞}(S^{p′−1} × S^{q′−1}) and h_{1} ∈ C^{∞}(ℝ) such that

f_{1} (\frac{(u′, v′)}{\sqrt{|u′|^{2} − |v′|^{2}}} ) = a_{1}(ω′, η′) (\frac{cos^{2} θ + cos^{2} φ}{cos^{2} θ − cos^{2} φ})_{+}^{− \frac{2λ′+p′+q′−2}{4}} h_{1} (\sqrt{\frac{cos^{2} θ − cos^{2} φ}{cos^{2} θ + cos^{2} φ}})

Likewise, there exist a_{2} ∈ C^{∞}(S^{q′′−1} × S^{p′′−1}) and h_{2} ∈ C^{∞}(ℝ) such that

f_{2} (\frac{(v′′, u′′)}{\sqrt{|v′′|^{2} − |u′′|^{2}}} ) = a_{2}(η′′, ω′′) (\frac{sin^{2} θ + sin^{2} φ}{cos^{2} θ − cos^{2} φ})_{+}^{− \frac{2λ′′+p′′+q′′−2}{4}} h_{2} (\sqrt{\frac{cos^{2} θ − cos^{2} φ}{sin^{2} θ + sin^{2} φ}})

We treat the boundary ∂M_{+} of M_{+} ⊂ M locally in the following three cases: Case 1) cos^{2} θ − cos^{2} φ = 0, (cos θ, cos φ) ≠ (0, 0), (sin θ, sin φ) ≠ (0, 0). Case 2) cos θ = cos φ = 0. Case 3) sin θ = sin φ = 0.

We note that Case (2) or (3) happens only when p′q′p′′q′′ ≠ 0.

\underline{9.6.} (Case 1): In this subsection, we consider a generic part of the boundary ∂M_{+} corresponding to Case 1. In a local coordinate (ω′, ω′′, η′, η′′, θ, φ) ∈ S^{p′−1} × S^{p′′−1} × S^{q′−1} × S^{q′′−1} × ℝ^{2}, T_{+}(f_{1}f_{2}))(u, v) is written as

A (cos^{2} θ − cos^{2} φ)_{+}^{− \frac{p+q−4}{4} + \frac{2λ′+p′+q′−2}{4} + \frac{2λ′′+p′′+q′′−2}{4}} = A (cos^{2} θ − cos^{2} φ)+^{\frac{λ′+λ′′}{2}} .

Here, A is a smooth function of variables ω′, ω′′, η′, η′′, (cos^{2} θ − cos^{2} φ)+^{\frac{1}{2}}. Therefore, by using Lemma 9.5, we have:

λ′ + λ′′ > −1 ⇒ T_{+}f ∈ L_{loc}^{2},
λ′ + λ′′ ≥ 1 ⇒ Y (T_{+}f) ∈ L_{loc}^{2−ϵ}  for any ϵ > 0, Y ∈ 𝔛(M),
λ′ + λ′′ > 2 ⇒ Y_{1}Y_{2}(T_{+}f) ∈ L_{loc}^{1}  for any Y_{1}, Y_{2} ∈ 𝔛(M),

in a neighbourhood of the boundary point of M_{+} for Case (1).

\underline{9.7.} (Case 2): In this subsection we consider a neighbourhood of (ω′, ω′′, η′, η′′, θ, φ) satisfying the condition of Case 2. We take a polar coordinate for (cos θ, cos φ) ≒ (0, 0) as

\{ \begin{matrix}cos θ & = r cos ψ, \\ cos φ & = r sin ψ. \end{matrix}

The composition to (9.5.1) yields a new coordinate on S^{p−1} × S^{q−1} given by

(u, v) = (u′, u′′, v′, v′′) = (ω′r cos ψ, ω′′\sqrt{1 − r^{2} cos^{2} ψ}, η′r sin ψ, η′′\sqrt{1 − r^{2} sin^{2} ψ}),

where ω′ ∈ S^{p′−1}, ω′′ ∈ S^{p′′−1}, η′ ∈ S^{q′−1}, η′′ ∈ S^{q′′−1}, r ≥ 0, and ψ ∈ ℝ. Then our interest is in a neighbourhood of r = 0. In this coordinate, we have

M_{+} = {r ≠ 0, cos 2ψ > 0}

The Jacobian matrix of the transform (θ, φ) → (r, ψ) is given by

( \begin{matrix}\frac{∂r}{∂θ} & \frac{∂r}{∂φ} \\ \frac{∂ψ}{∂θ} & \frac{∂ψ}{∂φ} \end{matrix} ) = ( \begin{matrix}− cos ψ sin θ & − sin ψ sin φ \\ \frac{1}{r}sin ψ sin θ & \frac{−1}{r}cos ψ sin φ \end{matrix} ) .

Lemma 9.7. 1) The standard measure on S^{p−1} × S^{q−1} is locally represented as

smooth function of (r, ψ, ω′, η′, ω′′, η′′) × r^{p′+q′−1} drdψdω′dη′dω′′dη′′.

2) Any smooth vector field on S^{p−1} × S^{q−1} near r = 0 is a linear combination of

\frac{1}{r cos ψ} X′, X′′, \frac{1}{r sin ψ} Y′, Y′′, \frac{∂}{∂r} , \frac{1}{r} \frac{∂}{∂ψ} ,

whose coefficients are smooth functions of (ω′, ω′′, η′, η′′, r, ψ). Here, X′, X′′, Y′ and Y′′ are smooth vector fields on S^{p′−1}, S^{p′′−1}, S^{q′−1} and S^{q′′−1}, respectively.

By noting the relations

\frac{cos^{2} θ + cos^{2} φ}{cos^{2} θ − cos^{2} φ} = \frac{1}{cos 2ψ} , \frac{sin^{2} φ + sin^{2} θ}{sin^{2} φ − sin^{2} θ} = \frac{2 − r^{2}}{r^{2} cos 2ψ} ,

T_{+}(f_{1}f_{2})(u, v) is locally written as

B (r^{2} cos 2ψ)^{− \frac{p+q−4}{4}} (cos 2ψ)+^{\frac{2λ′+p′+q′−2}{4}} (r^{2} cos 2ψ)+^{\frac{2λ′′+p′′+q′′−2}{4}}

=B r+^{\frac{2λ′′−p′−q′+2}{2}} (cos 2ψ)+^{\frac{λ′+λ′′}{2}} ,

where B is a smooth function of variables ω′, ω′′, η′, η′′, r_{+}, (cos 2ψ)+^{\frac{1}{2}}. Therefore, by using Lemma 9.7, we have:

2λ′′ > −1,  λ′ + λ′′ > −1 ⇒ T_{+}f ∈ L_{loc}^{2},
2λ′′ ≥ 0,  λ′ + λ′′ ≥ 1 ⇒ Y (T_{+}f) ∈ L_{loc}^{2−ϵ}  for any ϵ > 0, Y ∈ 𝔛(M),
2λ′′ > 2 − p′ − q′,λ′ + λ′′ > 2 ⇒ Y_{1}Y_{2}(T_{+}f) ∈ L_{loc}^{1} for any Y_{1}, Y_{2} ∈ 𝔛(M),

in a neighbourhood of the boundary ∂M_{+} for Case (2).

The asymptotic estimate for Case (3) parallels to that of Case (2).

Remark 9.7. Assume λ′ = λ′′ ∈ ℤ + \frac{p′+q′}{2} . Then r+^{\frac{2λ′′−p′−q′+2}{2}} is bounded near r = 0 if and only if λ′ ≥ \frac{p′+q′}{2} − 1, equivalently, λ′ > \frac{p′+q′}{2} − 2, which means that ℂ_{λ′} is in the good range with respect to the θ-stable parabolic subalgebra defined by ℂ_{λ′} in the sense of Vogan.

\underline{9.8.} We end with some remarks and conjectures, primarily concerning the precise form of the discrete spectrum and also the continuous spectrum (where it would be very interesting to develop the complete Plancherel formula, given our explicit intertwining operator).

1) (multiplicity free property)

Each irreducible component in Theorem 9.1 occurs as multiplicity free. If p = 2, then ϖ^{p,q} is the direct sum of an irreducible highest weight module and a lowest one. It was proved in [19] that the multiplicity in the full Plancherel formula (both discrete and continuous spectrum) is at most one, in the branching law of any highest weight module of scalar type with respect to any symmetric pair.

2) (at most finitely many discrete spectrum, and full discrete spectrum)

If p′, q′, p′′ and q′′ satisfy

(9.8.1)  min(p′, q′′) ≤ 1 and min(q′, p′′) ≤ 1,

then the parameter set in Theorem 9.1 is empty, namely,

A′(p′, q′) ∩ A′(q′′, p′′) = A′(q′, p′) ∩ A′(p′′, q′′) = ∅.

We conjecture that there are at most finitely many discrete spectra in the branching law ϖ^{p,q}|_{G′} if (9.8.1) holds. We further conjecture that the full discrete spectrum is as in Theorem 9.1 with A′(p, q) replaced by A_{0}(p, q) everywhere.

3) (no discrete spectrum)

Furthermore, if we exclude the case such as G′ = O(p, q − 1) × O(1) (see §7.2), namely, if

(9.8.2)  min(p′, q′′) ≤ 1, min(q′, p′′) ≤ 1, p′ + q′ > 1, and p′′ + q′′ > 1,

then

A_{0}(p′, q′) ∩ A_{0}(q′′, p′′) = A_{0}(q′, p′) ∩ A_{0}(p′′, q′′) = ∅.

It is likely that there is no discrete spectra in the branching law ϖ^{p,q}|_{G′} if (9.8.2) is satisfied.

We note that the condition (9.8.2) is equivalent to that at least one of X(p′, q′) or X(q′′, p′′) is a non-compact Riemannian symmetric space and at least one of X(q′, p′) or X(p′′, q′′) is a non-compact Riemannian symmetric space.

4) (discretely decomposable case)

The opposite extremal case is when

(9.8.3)  min(p′, p′′, q′, q′′) = 0.

As we have proved in Theorem 4.2, the restriction ϖ^{p,q}|_{G′} is discretely decomposable without any continuous spectrum. We have obtained the full branching formula in Theorem 7.1 by using Theorem 4.2 and the K-type formula of ϖ^{p,q}.

If we employ only the method in this section to the special case (9.8.3), then we do not have to consider Cases (2) and (3) in §9.7. Then, Theorem 9.1 exhausts all discrete spectra in Theorem 7.1 in most cases, but there are some few exceptions. To be precise, we consider the case p′′ = 0 without loss of generality. Then, in view of Theorem 7.1, the right side of Theorem 9.1 exhausts all discrete spectra if q′′ ≥ 5; while at most two of (𝔤′, K′)-modules π_{+,λ}^{p,q′} ⊠ π_{−,λ}^{0,q′′} are missing in Theorem 9.1 if q′ ≤ 4. The precise missing parameters in the case p′′ = 0 and q′ = 0 are: λ = ±\frac{1}{2} (q′′ = 1); λ = 0, 1 (q′′ = 2); λ = \frac{1}{2} (q′′ = 3); and λ = 1 (q′′ = 4). In order to to cover all missing parameter by the purely geometric method of this section, we should notice a specific feature in the case p′′ = 0:

i) M_{+} is a dense subspace of M. ii) Any real analytic functions on M_{+} satisfying the Yamabe equation corresponding to

K′-finite vectors of the (𝔤′, K′)-module with the above missing parameter extend to real analytic functions on M (see Corollary 4.3 (2)).

5) (explicit continuous spectrum)

We conjecture that

L^{2}- Ind_{P_{1}^{max}}^{G′_{1}}(ϵ ⊗ ℂ_{\sqrt{−1}λ}) ⊠ L^{2}- Ind_{P_{2}^{max}}^{G′_{2}}(ϵ ⊗ ℂ_{\sqrt{−1}λ}) (λ ∈ ℝ)

is a continuous spectrum with multiplicity free if min(p′, p′′, q′, q′′) > 0.

References

[1] W. Beckner, Geometric inequalities in Fourier analysis, Essays on Fourier analysis in honor of Elias M. Stein (C. Fefferman, R. Fefferman and S. Wainger, ed.), Princeton Univ. Press, 1995, pp. 36–68.

[2] B. Binegar and R. Zierau, Unitarization of a singular representation of SO(p, q), Comm. Math. Phys. 138 (1991), 245–258.

[3] R. Brylinski and B. Kostant, Minimal representations of E_{6}, E_{7} and E_{8} and the generalized Capelli identity, Proc. Nat. Acad. Sci. U.S.A. 91 (1994), 2469–2472.

[4] , Differential operators on conical Lagrangian manifolds, Lie theory and geometry, vol. 123, Progress in Math., 1994, pp. 65–96.

[5] , Lagrangian models of minimal representations of E_{6}, E_{7} and E_{8}, Functional analysis on the eve of the 21st century, Vol. 1, vol. 131, Progress in Math., 1995, pp. 13–63.

[6] A. Erdélyi, Higher Transcendental Functions, vol. I, McGraw-Hill, New York, 1953.

[7] A. Erdélyi, Tables of Integral Transforms, vol. II, McGraw-Hill, New York, 1954.

[8] I. M. Gelfand and G. E. Shilov, Generalized Functions, I, Academic Press, 1964.

[9] S. Helgason, Differential Geometry, Lie Groups and Symmetric Spaces, Pure and Appl. Math., vol. 80, Academic Press, 1978.

[10] R. Howe, Transcending classical invariant theory, Jour. A. M. S. 2 (1989), 535–552.

[11] R. Howe and E. Tan, Homogeneous functions on light cones, Bull. A. M. S. 28 (1993), 1–74.

[12] J-S. Huang and C-B. Zhu, On certain small representations of indefinite orthogonal groups, Representation Theory 1, 190–206.

[13] S. Kobayashi, Transformation Groups in Differential Geometry, Ergebnisse, vol. 70, Springer.

[14] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽), vol. 462, Memoirs A. M. S., 1992.

[15] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205.

[16] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups II — micro-local analysis and asymptotic K-support, Ann. of Math. 147 (1998), 709–729.

[17] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups III — restriction of Harish-Chandra modules and associated varieties, Invent. Math. 131 (1998), 229–256.

[18] , Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (eds. K. Mimachi) (1997), 9–17.

[19] , Multiplicity-free theorem in branching problems of unitary highest weight modules, preprint.

[20] , Discretely decomposable restrictions of unitary representations of reductive Lie groups, Analysis on Homogeneous Spaces and Representation Theory of Lie Groups (T. Kobayashi et al, eds.), vol. 26, 2000, pp. 98–126.

[21] , Branching laws of O(p, q) associated to minimal elliptic orbits, (in preparation).

[22] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits, C. R. Acad. Sci. Paris 326 (1998), 925–930.

[23] , Analysis on the minimal representation of O(p, q) – I. Realization via conformal geometry, preprint.

[24] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4), Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory (Connes et al, eds.), vol. 92, Birkhäuser, Boston, 1990, pp. 85–124.

[25] J. M. Lee and T. H. Parker, The Yamabe problem, Bull. A. M. S. 17 (1987), 37–91.

[26] H. Sabourin, Une représentation unipotente associée à l’orbite minimale: Le cas de SO(4,3), J. Funct. Anal. 137 (1996), 394–465.

[27] H. Schlichtkrull, Eigenspaces of the Laplacian on hyperbolic spaces: composition series and integral transforms, J. Funct. Anal. 70 (1987), 194–219.

[28] W. Schmid, Boundary value problems for group invariant differential equations (1985), Asterisque, hors s’erie, 311–321, Élie Cartan et les mathématiques d’aujourd’hui.

[29] P. Torasso, Méthode des orbites de Kirillov-Dufulo et representations minimales des groupes simples sur un corps local de caractéristique nulle, preprint (1997).

[30] D. Vogan Jr., Representations of Real Reductive Lie Groups, Progress in Math., vol. 15, Birkhäuser, 1981.

[31] , Unitarizability of certain series of representations, Ann. of Math. (1984), 141–187.

[32] , Unitary Representations of Reductive Lie Groups, Ann. Math. Stud. 118, Princeton U. P., Princeton, New Jersey, 1987.

[33] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 191–221.

[34] H. Wong, Dolbeault cohomologies and Zuckerman modules associated with finite rank representations, ph. D. dissertation, Harvard University (1992).

[35] B. Ørsted, A note on the conformal quasi-invariance of the Laplacian on a pseudo-Riemannian manifold, Lett. Math. Phys. 1 (1977), 183.

[36] , Conformally invariant differential equations and projective geometry, J. Funct. Anal. 44 (1981), 1–23.

RIMS, Kyoto University, Sakyo-ku, Kyoto, 606-8502, Japan

Department of Mathematics and Computer Science, SDU-Odense University, Campusvej 55, DK-5230, Odense M, Denmark

E-mail address: toshi@kurims.kyoto-u.ac.jp; orsted@imada.ou.dk