DISCRETE SERIES REPRESENTATIONS FOR

THE ORBIT SPACES ARISING FROM TWO

INVOLUTIONS OF REAL REDUCTIVE LIE GROUPS^{†}

Toshiyuki KOBAYASHI

Graduate School of Mathematical Sciences, University of Tokyo, Komaba, Meguro, Tokyo, 153

Abstract. Let H ⊂ G be real reductive Lie groups. A discrete series representation for a homogeneous space G/H is an irreducible representation of G realized as a closed G-invariant subspace of L^{2}(G/H). The condition for the existence of discrete series representations for G/H was not known in general except for reductive symmetric spaces. This paper offers a sufficient condition for the existence of discrete series representations for G/H in the setting that G/H is a homogeneous submanifold of a symmetric space G̃/H̃ where G ⊂ G̃ ⊃ H̃. We prove that discrete series representations are non-empty for a number of non-symmetric homogeneous spaces such as Sp(2n, ℝ)/Sp(n_{0}, ℂ) × GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ) (∑ n_{j} = n) and O(4m, n)/U(2m, j) (0 ≤ 2j ≤ n).

Contents

§1. Introduction. §2. Admissible restrictions of unitary representations and restrictions of function. §3. Decay of functions on homogeneous manifolds of reductive type. §4. Discrete series for symmetric spaces and Zuckerman-Vogan’s modules. §5. Discrete series representations for the orbit spaces G^{τ} \G/G^{σ}. §6. Examples. §7. Holomorphic discrete series representations. §8. Appendix – Analysis on principal orbits.

§1. Introduction.

\underline{1.1.} Our object of study is discrete series representations for a homogeneous manifold G/H, where G is a real reductive linear Lie group and H is a closed subgroup that is reductive in G. Here, we say that an irreducible representation π

Key words and phrases. discrete series representations, harmonic analysis on homongeneous spaces, Vogan-Zuckerman derived functor modules, restriction of unitary representations, semisimple symmetric spaces, homogeneous manifolds of reductive type.

†This work is supported by the Royal Swedish Academy of Sciences.

Typeset by AMS-TEX

of G is a discrete series representation for G/H if π is realized as a closed G-invariant subspace of the Hilbert space L^{2}(G/H).

\underline{1.2.} We denote by Disc(G/H) the unitary equivalence class of discrete series representations for G/H. A natural question is:

“Which homogeneous manifold G/H admits discrete series representations?” If G/H is a group manifold G′ × G′/ diag(G′), then it is a celebrated work due

to Harish-Chandra that Disc(G/H) ≠ ∅ if and only if rank G′ = rank K′, where K′ is a maximal compact subgroup of G′. A generalization to a reductive symmetric space G/H is due to Flensted-Jensen, Matsuki and Oshima ([5], [25]) as follows: If we take a maximal compact subgroup K of G such that H ∩ K is also a maximal compact subgroup of H, then we have Disc(G/H) ≠ ∅ if and only if

(1.2)  rank G/H = rank K/H ∩ K.

Discrete series representations have played a fundamental role in L^{2}-harmonic analysis on G/H in these cases, not only “discrete spectrum” but also for “tempered representations” which are constructed as induced representations of discrete series representations for smaller “G/H”, as one can see by the Plancherel formula of a group manifold due to Harish-Chandra and by that of a semisimple symmetric space announced by Delorme [3] and Oshima. Discrete series representations for G/H also contribute to a deeper understanding of representation theory of G itself, such as the unitarizability of Zuckerman-Vogan’s derived functor modules A_{𝔮}(λ) for certain θ-stable parabolic subalgebras 𝔮 (c.f. [34], [37] for algebraic approach in a more general setting). Discrete series representations are also important in the applications to automorphic forms such as the construction of harmonic forms on locally symmetric spaces that are dual to the modular symbols defined by H (see [32]).

However, our current knowledge on discrete series representations is very poor for a more general homogeneous manifold of reductive type, in spite that we could expect the importance in L^{2}-harmonic analysis and the applications in other branches of mathematics such as automorphic forms. In fact, previous to this, discrete series representations for homogeneous spaces of reductive type have been studied only in the cases of group manifolds, reductive symmetric spaces, indefinite Stiefel manifolds ([28], [12], [21], [15]), and some other small number of spherical homogeneous manifolds ([14], Corollary 5.6). This is mostly because of the lack of powerful methods that were successful in the symmetric cases such as the Flensted-Jensen duality (in general, there is no “dual” homogeneous space G^{d}/H^{d} !) and the spectral theory of invariant differential operators (in general, the ring of invariant differential operators is not commutative).

\underline{1.3.} In this paper, we consider the existence of discrete series representations for G/H, a homogeneous manifold of reductive type in a more general setting. Our strategy is divided into the following three steps:

1) To embed G/H into a larger homogeneous manifold G̃/H̃, on which harmonic analysis is well-understood (e.g. group manifolds, symmetric spaces).

2) To take discrete series representations (π, H) for G̃/H̃.

3) To take functions belonging to H (↪ L^{2}(G̃/H̃)) and to restrict them with respect to a submanifold G/H (↪ G̃/H̃).

The main difficulty is that the restriction of L^{2}-functions to a submanifold does not make sense in general and does not always yield L^{2}-functions. This can be overcome by assuming a representation theoretic condition, that is, the admissibility of the restriction of the unitary representation with respect to a reductive subgroup (see Definition 2.6).

\underline{1.4.} Suppose that (G̃, G) and (G̃, H̃) are symmetric pairs defined by two involutions τ and σ of G̃, respectively. (We remark that our notation later is slightly different; we shall write G′ ⊂ G ⊃ H instead of G ⊂ G̃ ⊃ H̃.)

Then one of our main result (see Theorem 5.1) is briefly as follows:

Theorem. Assume that G̃/H̃ satisfies the rank assumption (1.2) and that

Cone(σ) ∩ Subsp(τ) = {0}.

Then Disc(G/H_{x}) ≠ ∅ for any x ∈ K̃, where H_{x} = G ∩ xH̃x^{−1}.

Here, Cone(σ) is a cone defined by σ and Subsp(τ) is a vector space defined by τ, both of which are subsets of a certain Cartan subalgebra (see Definition 4.2 and §4.3, respectively).

The point is that we have different homogeneous manifolds G/H_{x} (mostly, nonsymmetric) that admit discrete series representations as x ∈ K̂ varies. Recent progress due to M. Iida and T. Matsuki ([23], [24]) on the double coset space G\G̃/H̃ helps us to understand which H_{x} := G ∩ xG̃x^{−1} appears as x varies. For example, we shall prove that

Sp(2n, ℝ)/Sp(n_{0}, ℂ) × GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ) (∑ n_{j} = n)

and

O(4m, n)/U(2m, j) (0 ≤ 2j ≤ n)

admit discrete series representations. The properties of the resulting discrete series are also studied by representation theoretic methods.

\underline{1.5.} This paper is organized as follows: In §2, we consider the restriction of functions on G̃/H̃ with respect to a homogeneous submanifold G/H, and show how to single out a non-zero irreducible representation of G realized in the space of functions on G/H. In §3, we prove the decay of functions on G/H, which are obtained by the restriction of functions (after normal derivatives) on G̃/H̃. Both in §2 and §3, the crucial assumption is the admissibility of the restriction of a unitary representation (Definition 2.6). In particular, we prove a general framework in Theorem 3.7 for the existence of discrete series representations on G/H.

The assumptions of Theorem 3.7 are stated very explicitly in Theorem 5.1 in a specific setting where (G̃, G) and (G̃, H̃) are symmetric pairs, based on preliminary results given in §4. In §6, we illustrate Theorem 5.1 by an example G/H = O(2m, n)/U(m, j) (0 ≤ 2j ≤ n).

In §7, we consider homogeneous spaces that admit discrete series representations having highest weight vectors. In this case, we can check the assumption of the admissible restriction in Theorem 3.7 by much more elementary methods (see Theorem 7.4). Theorem 7.5 offers a sufficient condition that G/H admits “holomorphic discrete series representations”. As a very special case, we give a new proof that symmetric spaces of Hermitian type admit “holomorphic discrete series representations”, which were known by other methods (e.g. [9], [4]).

Our approach based on the embedding G/H ↪ G̃/H̃ becomes much easier when G/H is “a generic orbit”, or of principal type. A refinement of Theorem 3.7 is given in Theorem 8.6 under the assumption that G/H is of principal type.

\underline{1.6.} This paper is an outgrowth of the talks delivered at the conference in honor of Professor Helgason’s 65th birthday on September 1992 at Roskilde University in Denmark and at the annual meeting of unitary representation theory on November 1993 at Atagawa in Japan. Part of the result was announced in [13], Part I and [17]. The most part of the manuscript was written while the author was a guest at the Institut Mittag-Leffler of the Royal Swedish Academy of Sciences during the academic year 1995–1996. He expresses his sincere gratitude to the staff of the Institute and to the organizers, in particular, Professors Flensted-Jensen and ’Olafsson, of the special year “Analysis on Lie Groups” which presented him a very warm and wonderful atmosphere of research. He also thanks Professor T. Matsuki who explained kindly his recent work [23] and [24]. Thanks are also due to Professor T. Oshima for his interest and encouragement.

§2. Admissible restrictions of unitary representations and restrictions of function.

\underline{2.1.} The restriction of L^{2}-functions to a submanifold is not well-defined in general. In this section, we shall give a representation theoretic condition, namely, admissible restriction (see Definition 2.6) that assures the well-defined restriction of functions to a submanifold. Furthermore, we shall estimate the asymptotic behaviour of the functions belonging to an irreducible representation (and its normal derivatives) along the submanifold.

\underline{2.2.} We begin with a standard argument of normal derivatives.

Lemma 2.2. Let M be a connected real analytic manifold and M′ ↪ M a real analytic submanifold. We assume that there exist analytic vector fields X̃_{1}, …, X̃_{n} on M such that

(2.2.1)  T (M′)_{p} + ∑_{i=1}^{n} ℝX̃_{i}(p) = T M_{p}

for some point p ∈ M′. Then for any non-zero analytic function f on M, there exist i_{1}, …, i_{k} ∈ {1, 2, …, n} such that the restriction (X̃_{i_{1}} … X̃_{i_{k}}f) |_{M′} is not identically zero on the connected component of M′ containing p.

Proof. We take a local coordinate (x_{1}, …, x_{l}, y_{1}, …, y_{m}) (m ≤ n), simply denoted by (x, y), of M such that M′ is locally represented by y = 0. We note that the assumption (2.2.1) holds in a neighbourhood of p ∈ M′. We write the Taylor expansion of f(x, y) along the normal direction as

f(x, y) = ∑_{α∈ℕ^{m}} g_{α}(x)y^{α},

where g_{α}(x) is a real analytic function on M′ and y^{α} = y_{1}^{α_{1}} ⋯ y_{m}^{α_{m}} for each multiindex α = (α_{1}, …, α_{m}) ∈ ℕ^{m}. If the restriction (X̃_{i_{1}} … X̃_{i_{k}}f) |_{M′} is identically zero for all such expressions, then g_{α}(x) = 0 for any α ∈ ℕ^{m} and for any (x, 0) in a neighbourhood of p ∈ M′. Then g_{α}(x) (α ∈ ℕ^{m}) is identically zero on the connected component of M′ containing p because g_{α}(x) is real analytic. This implies that f(x, y) is identically zero because f is real analytic. Hence we have the lemma. □

\underline{2.3.} Here is the main setting that we shall use throughout this paper:

Setting 2.3. Let G be a real reductive linear Lie group, 𝔤_{0} the Lie algebra and 𝔤 its complexification. Analogous notation is used for other groups denoted by Roman uppercase letters. Let K be a maximal compact subgroup, θ the corresponding Cartan involution of G and 𝔤_{0} = 𝔨_{0} + 𝔭_{0} the Cartan decomposition.

We fix a non-degenerate symmetric Ad(G)-invariant bilinear form B on 𝔤_{0} with the following two properties: (2.3.1) B is positive definite on 𝔭_{0} × 𝔭_{0} and negative definite on 𝔨_{0} × 𝔨_{0}, (2.3.2) 𝔨_{0} and 𝔭_{0} is orthogonal with respect to B.

If G is semisimple, we can take a Killing form of 𝔤_{0} as B. Suppose that G′ and H are θ-stable closed subgroups with at most finitely many connected components. Then G′ and H are also real reductive linear Lie groups. We shall say that the homogeneous manifold G/H (also G/G′) is of reductive type. We write o := eH ∈ G/H. Let 𝔥_{0}^{⊥} be the orthogonal complement of 𝔥_{0} in 𝔤_{0} with respect to B. Then we have a direct sum decomposition

𝔤_{0} = 𝔥_{0} ⊕ 𝔥_{0}^{⊥}

because the restriction B|_{𝔥_{0}×𝔥_{0}} is also a non-degenerate symmetric bilinear form. Similarly, we have an orthogonal decomposition

𝔤_{0} = 𝔤′_{0} ⊕ 𝔤′_{0}^{⊥}.

We put H′ := G′ ∩ H, and write

ι: G′/H′ ↪ G/H

for the natural embedding.

\underline{2.4.} Suppose we are in the setting of §2.3. The left action of G on G/H defines a vector field X̃ on G/H for each element X ∈ 𝔤_{0} by the formula:

X̃(p) := \frac{d}{dt}_{|t=0}e^{tX} · p ∈ T (G/H)_{p}, p ∈ G/H.

Lemma 2.4. Retain the notation in §2.3. We take a basis X_{1}, …, X_{n} of 𝔤′_{0}^{⊥}. We put M′ := G′/H′ ⊂ M := G/H. Then the vector fields X̃_{1}, …, X̃_{n} on M satisfy the assumption of Lemma 2.2 at any point p ∈ M′.

Proof. Fix g ∈ G′. We write L_{g}: G/H → G/H, x ↦ gx for the left translation and L_{g∗}: 𝔤_{0}/𝔥_{0} ⟶^{∼} T (G/H)_{g·o} for its differential. Here we have used the identification of T (G/H)_{o} with 𝔤_{0}/𝔥_{0}. Then we have

L_{g∗}^{−1}(T (G′/H′)_{g·o}) = Ad(g^{−1})𝔤′_{0} mod 𝔥_{0},
L_{g∗}^{−1}(X̃_{i}(g · o)) = Ad(g^{−1})X_{i} mod 𝔥_{0}.

Since 𝔤_{0} = 𝔤′_{0} + 𝔤′_{0}^{⊥} = 𝔤′_{0} + ∑_{i=1}^{n} ℝX_{i}, we have

L_{g∗}^{−1}(T (G′/H′)_{g·o}) + L_{g∗}^{−1}(∑_{i=1}^{n} ℝX̃_{i}(g · o)) = Ad(g^{−1})(𝔤′_{0} + ∑_{i=1}^{n} ℝX_{i}) mod 𝔥_{0}
= 𝔤_{0} mod 𝔥_{0}
≃ T (G/H)_{o}.

Thus, we have T (G′/H′)_{g·o} + ∑_{i=1}^{n} ℝX̃_{i}(g · o) = T (G/H)_{g·o}. □

\underline{2.5.} Suppose we are in the setting of §2.3. We recall ι: G′/H′ ↪ G/H is a natural embedding. The space of C^{∞} functions, C^{∞}(G/H), is a G-module by the left translation. Then the pullback of functions ι^{∗}: C^{∞}(G/H) → C^{∞}(G′/H′) respects the actions of G′ (⊂ G). The complexified Lie algebra 𝔤 acts on C^{∞}(G/H) by the differential of the G-action, so that C^{∞}(G/H) is also a 𝔤-module. Similarly, C^{∞}(G′/H′) is a G′-module as well as a 𝔤′-module. Then ι^{∗}: C^{∞}(G/H) → C^{∞}(G′/H′), also respects the actions of 𝔤′ ⊂ 𝔤.

A vector space W over ℂ is called a (𝔤, K)-module, if W is a representation space of 𝔤 and also if W is a representation space of K, both representations denoted by π satisfying the following conditions:

1) dim ℂ-span of {π(k)v : k ∈ K} is finite for any v ∈ W.

2) lim_{t→0} \frac{π(exp(tY))v − v}{t} = π(Y)v for any v ∈ W and Y ∈ 𝔨_{0} = Lie(K).

3) π(Ad(k)Y)v = π(k)π(Y)π(k)^{−1}v for any v ∈ W, k ∈ K and Y ∈ 𝔤.

Lemma 2.5. Let (π, V_{K}) be an irreducible (𝔤, K)-module. If there is a non-zero (𝔤, K)-homomorphism i: V_{K} → C^{∞}(G/H) then ι^{∗}(i(V_{K})) ≠ {0}.

Proof. We fix v ∈ V_{K}. Let v = ∑_{τ∈K̂} v_{τ} ∈ V_{K} be a finite sum corresponding to the irreducible decomposition of K-types. Then f := i(v) is an analytic function on G/H because of the elliptic regularity theorem; the elliptic operator C − 2C_{K} acts on i(v_{τ}) by a scalar for each τ ∈ K̂, where C is the G-invariant differential operator on G/H of second order corresponding to the Casimir element of 𝔤 defined by the invariant symmetric bilinear form B and C_{K} is the K-invariant one defined by B|_{𝔨_{0}×𝔨_{0}}. It follows from Lemma 2.2 and Lemma 2.4 that we find X_{i_{1}}, …, X_{i_{k}} ∈ 𝔤′_{0}^{⊥} such that (X̃_{i_{1}} … X̃_{i_{k}}f) |_{G′/H′} is not identically zero. Since

(X̃_{i_{1}} … X̃_{i_{k}}f) |_{G′/H′} = ι^{∗}i(π(−X_{i_{k}}) ⋯ π(−X_{i_{1}})v) ∈ ι^{∗}(i(V_{K})),

we have proved ι^{∗}(i(V_{K})) ≠ 0. □

\underline{2.6.} We review the definition of the admissible unitary representations.

Definition 2.6. Let denote by Ĝ the unitary dual of a real reductive linear Lie group G. We shall say that a unitary representation (π, V) of G is G-admissible if (π, V) is decomposed into a discrete Hilbert direct sum with finite multiplicities of irreducible representations of G (see [14], §1).

We note that the restriction of (π, V) to a maximal compact subgroup K is K-admissible for any (π, V) ∈ Ĝ (Harish-Chandra). This property is usually called “admissible”, however, we say “K-admissible” in this paper by specifying the groups.

\underline{2.7.} Suppose we are in the setting of §2.3. In particular, K ⊃ K′ := K ∩ G′ are maximal compact subgroups of G ⊃ G′, respectively. Given (π, V) ∈ Ĝ, we write V_{K} for the space of K-finite vectors of V. The complexified Lie algebra 𝔤 and K naturally act on V_{K}. The (𝔤, K)-module V_{K} is called the underlying (𝔤, K)-module of V. Similarly, V_{K′} denotes the space of K′-finite vectors of V. Obviously, we have V_{K} ⊂ V_{K′}. The following lemma is a very important property of admissible restrictions:

Lemma 2.7. Assume that the restriction of (π, V) ∈ Ĝ to K′ is K′-admissible. Then we have V_{K} = V_{K′}. Furthermore, V_{K} is decomposed into an algebraic sum of irreducible (𝔤′, K′)-modules.

Proof. See [18], Proposition 1.6. □

\underline{2.8.} Suppose we are in the setting of §2.3. Then the G′-orbit through xH ∈ G/H (x ∈ G) is a submanifold of G/H that is isomorphic to G′/H′_{x} where H′_{x} = G′ ∩ xHx^{−1}. Here is a framework that we can find an irreducible representation of G′ (⊂ G) realized in the space of functions on the submanifold G′/H′_{x} (⊂ G/H), provided a representation of G is realized on G/H.

Theorem 2.8. Let (π, V) ∈ Ĝ and x ∈ K. Assume that the following two conditions hold:

i) The restriction of π to K′ is K′-admissible. ii) Hom_{𝔤,K}(V_{K}, C^{∞}(G/H)) ≠ 0.

Then there exists an irreducible (𝔤′, K′)-module W satisfying the following two conditions:

Hom_{𝔤′,K′}(W, V_{K}) ≠ 0, (2.8.1)

Hom_{𝔤′,K′}(W, C^{∞}(G′/H′_{x})) ≠ 0. (2.8.2)

Proof. First, we shall show that Hom_{𝔤,K}(V_{K}, C^{∞}(G/xHx^{−1})) ≠ 0.

Let i be a non-zero (𝔤, K)-homomorphism from V_{K} to C^{∞}(G/H). Then i is injective because V_{K} is an irreducible (𝔤, K)-module. The automorphism φ_{x}: G → G, g ↦ xgx^{−1} induces a diffeomorphism

ψ_{x}: G/H → G/xHx^{−1}, gH ↦ (xgx^{−1})(xHx^{−1}).

Then ψ_{x} respects the G-action where G acts on G/H from the left and on G/xHx^{−1} via φ_{x}, namely, we have

φ_{x}(g′)ψ_{x}(gH) = ψ_{x}(g′gH), for any g′, g ∈ G.

We write ψ_{x}^{∗}: C^{∞}(G/xHx^{−1}) ⟶^{∼} C^{∞}(G/H) for the pullback of C^{∞}-functions. Let G act on C^{∞}(G/H) by f ↦ f(g^{−1}·) and on C^{∞}(G/xHx^{−1}) by f ↦ f(φ_{x}(g)^{−1} · ). Then ψ_{x}^{∗} is a G-intertwining operator, namely, we have

(ψ_{x}^{∗}f(φ_{x}(g′)^{−1} · )) (gH) = (ψ_{x}^{∗}f)(g′^{−1}gH), for any g′, g ∈ G.

On the other hand, the linear map π(x): V → V induces a (𝔤, K)-homomorphism,

π(x): V_{K} → V_{K},

where (𝔤, K) acts on the second V_{K} via φ_{x}, namely,

dπ(dφ_{x}(Y))π(x)v = π(x)dπ(Y)v,
π(φ_{x}(k))π(x)v = π(x)π(k)v,

for any k ∈ K, Y ∈ 𝔤, and v ∈ V_{K}. Therefore, we have a non-zero (𝔤, K)- homomorphism

V_{K} → C^{∞}(G/xHx^{−1}), v ↦ (ψ_{x}^{∗})^{−1} ∘ i(π(x)^{−1}v).

Hence, Hom_{𝔤,K}(V_{K}, C^{∞}(G/xHx^{−1})) ≠ 0. Thus, in order to prove Theorem, we may and do assume x = e.

Let H′ := G′ ∩ H and we write ι: G′/H′ ↪ G/H for the natural embedding which is G′-equivariant. Then the (𝔤′, K′)-homomorphism

ι^{∗} ∘ i: V_{K} → C^{∞}(G′/H′)

is a non-zero map because of Lemma 2.5. On the other hand, it follows from Lemma 2.7 that V_{K} is decomposed into an algebraic direct sum:

V_{K} ≃ ⨁_{j} W_{j},

where W_{j} are irreducible (𝔤′, K′)-modules. Therefore, ι^{∗} ∘ i is injective at least on one of irreducible constituents W_{j}’s, say W. This (𝔤′, K′)-module W is what we wanted. □

Remark 2.9.

1) The admissibility of restriction (see the assumption (i) of Theorem 2.8) has been studied in [13], [14] and [18]. We shall review the criterion for the admissible restriction in Fact 4.3.

2) By the elliptic regularity theorem as we discussed in the proof of Lemma 2.5, the assumption (ii) in Theorem 2.8 (also in Lemma 2.5) is equivalent to Hom_{𝔤,K}(V_{K}, B(G/H)) ≠ 0, where B denotes the sheaf of hyperfunctions.

3) The advantage of the formulation here is that we can apply Theorem to homogeneous manifolds of G′ with various isotropy subgroups H′_{x} by different choices of x.

§3. Decay of functions on homogeneous manifolds of reductive type.

\underline{3.1.} Suppose G/H is a homogeneous manifold of reductive type. We recall that 𝔤_{0} = 𝔥_{0} ⊕ 𝔥_{0}^{⊥} = 𝔨_{0} ⊕ 𝔭_{0} are orthogonal decompositions of 𝔤_{0} = Lie(G) with respect to B (see §2.3). We write ∥ · ∥ for the induced norm on 𝔥_{0}^{⊥} ∩ 𝔭_{0}, on which the restriction of B is positive definite. For ξ ∈ ℝ, we define a subspace of continuous functions of “exponential decay” by

C(G/H; ξ) := {f ∈ C(G/H) : sup_{k∈K} sup_{X∈𝔭_{0}∩𝔥_{0}^{⊥}} f(k exp X) exp(ξ∥X∥) < ∞}.

Similarly, we define C^{∞}(G/H; ξ) := C(G/H; ξ) ∩ C^{∞}(G/H). We note that

C(G/H; ξ) ⊂ C(G/H; ξ′) if ξ > ξ′.

\underline{3.2.} The Cartan decomposition G = KAH for a reductive symmetric space G/H (see [6]) reduces the L^{p}-estimate of functions on G/H to that on A ≃ ℝ^{l}. However, there is no analog of a Cartan decomposition “G = KAH” of a non-symmetric homogeneous manifold G/H of reductive type in general. The notion of C(G/H; ξ) plays a crucial role in L^{p}-harmonic analysis on a homogeneous manifold G/H of reductive type, without a Cartan decomposition. Here are basic results on C(G/H; ξ).

Lemma 3.2. Suppose we are in the setting of §2.3.

1) There exists a constant ν ≡ ν_{G/H} > 0 with the following property: if 1 ≤ p ≤ ∞ and if p ξ > ν, then C(G/H; ξ) ⊂ L^{p}(G/H).

2) Let x ∈ G and we assume that xHx^{−1} is θ-stable. We put H′_{x} := G′ ∩ xHx^{−1}. Let ι_{x}: G′/H′_{x} ↪ G/H be a natural embedding induced from the mapping G′ → G/H, g ↦ gxH. Then there exist a positive constant b ≡ b(G′/H′_{x}; G/H) > 0 such that

ι_{x}^{∗}C(G/H; ξ) ⊂ C(G′/H′_{x}; b ξ) for any ξ > 0.

Proof. See [16], Corollary 3.9 for the first statement. The second one follows from [16], Theorem 5.6 with xHx^{−1} replaced by H. □

\underline{3.3.} Let G/H be a homogeneous manifold of reductive type.

Definition 3.3. We say G/H satisfies (D-∞) if there exist a sequence of irreducible (𝔤, K)-modules (π_{j}, V_{j}) and a sequence ξ_{j} ∈ ℝ (j ∈ ℕ) with the following two conditions:

i) lim_{j→∞} ξ_{j} = ∞ ii) Hom_{𝔤,K}(V_{j}, C^{∞}(G/H; ξ_{j})) ≠ 0.

\underline{3.4.} Here is a typical example of homogeneous manifolds of reductive type satisfying (D-∞).

Example 3.4. Suppose G is a real reductive linear Lie group.

1) A group manifold G×G/ diag(G) satisfies (D-∞) if and only if rank G = rank K.

2) A reductive symmetric space G/H satisfies (D-∞) if and only if rank G/H = rank K/H ∩ K.

(1) is due to Harish-Chandra. and (2) generalizes (1), which is due to Flensted-Jensen, Matsuki and Oshima (see Lemma 4.5).

\underline{3.5.} A discrete series representation for a homogeneous manifold G/H is an irreducible unitary representation (π, H) of G such that H can be realized as a closed invariant subspace of L^{2}(G/H). The following lemma enables us to consider discrete series representations on the level of (𝔤, K)-modules instead of unitary representations of G.

Lemma 3.5. Let G be a real reductive linear Lie group, H a closed unimodular subgroup, and L^{2}(G/H) the Hilbert space of square integrable functions on G/H with respect to a G-invariant measure.

1) If (π, H) ∈ Ĝ is a discrete series representation for G/H, then there is a nonzero (𝔤, K)-homomorphism ι : H_{K} → C^{∞}(G/H) such that ι(H_{K}) ⊂ L^{2}(G/H).

2) Conversely, let V be an irreducible (𝔤, K)-module. If there is a non-zero (𝔤, K)- homomorphism ι : V → C^{∞}(G/H) such that ι(V) ⊂ L^{2}(G/H), then there is an irreducible unitary representation (π, H) of G such that H is a discrete series representation for G/H and that H_{K} ≃ V.

Proof. 1) Let i: H → L^{2}(G/H) be a non-zero G-homomorphism. Let ι be the restriction of i to H_{K}, the space of K-finite vectors of H. Then ι(H_{K}) ⊂ A(G/H) (⊂ C^{∞}(G/H)) by elliptic regularity theorem as we saw in the proof of Lemma 2.5. Hence the first statement is proved.

2) We induce an inner product on V through a non-zero (therefore, injective) homomorphism ι: V → C^{∞}(G/H) ∩ L^{2}(G/H). Then V is an infinitesimally unitarizable (𝔤, K)-module. Therefore, there is a unique irreducible unitary representation H of G such that H_{K} = V and that V is dense in H (Harish-Chandra). Then the (𝔤, K)-homomorphism ι: V → C^{∞}(G/H) extends to an isometry ῑ: H → L^{2}(G/H). Because ῑ is an isometry and because H is complete, the image ῑ(H) is closed. Therefore, H is realized as a closed G-invariant subspace of L^{2}(G/H). □

\underline{3.6.} The condition (D-∞) assures the existence of discrete series representations for a homogeneous manifold of reductive type:

Lemma 3.6. Let G/H be a homogeneous manifold of reductive type satisfying (D-∞). Then we have:

1) Irreducible (𝔤, K)-modules V_{j} (see Definition 3.3) are unitarizable for sufficiently large j.

2) Fix 1 ≤ p ≤ ∞. There exist infinitely many (counted with multiplicity) irreducible (𝔤, K)-modules that belong to L^{p}(G/H) (in particular, discrete series representations for G/H).

Proof. Retain the notation in Definition 3.3. Then, for any fixed p with 1 ≤ p ≤ ∞, there exists N ≡ N(p) such that

p ξ_{j} > ν_{G/H} for any j ≥ N,

where ν_{G/H} is the constant in Lemma 3.2. Then we have

C(G/H; ξ_{j}) ⊂ L^{p}(G/H), for any j ≥ N

by Lemma 3.2 (i). It follows from the assumption on V_{j} (see Definition 3.3 (ii)) that there exists a non-zero (𝔤, K)-homomorphism ι_{j}: V_{j} → C^{∞}(G/H) such that ι_{j}(V_{j}) ⊂ C(G/H; ξ_{j}) for each j. Hence, we have ι_{j}(V_{j}) ⊂ L^{p}(G/H) for any j ≥ N. In particular, if we put p = 2, then V_{j} is unitarizable by the inner product induced from the Hilbert space L^{2}(G/H) and its closure is a discrete series representation for G/H by Lemma 3.5. Hence we have proved the lemma. □

\underline{3.7.} Here is a sufficient condition for the existence of discrete series representations on homogeneous submanifolds in a primitive form. Theorem 3.7 will be reformulated in Theorem 5.1 and Theorem 7.5 by explicit assumptions in specific settings.

Suppose we are in the setting of §2.3.

Theorem 3.7. Suppose that G/H satisfies (D-∞). We assume that we can take (𝔤, K)-modules (π_{j}, V_{j}) in Definition 3.3 such that the restriction π_{j}|_{K′} is K′- admissible for each j ∈ ℕ. Then the homogeneous manifold G′/H′_{x} satisfies (D-∞) for any x ∈ K, where we put H′_{x} := G′ ∩ xHx^{−1}. In particular, Disc(G′/H′_{x}) ≠ ∅.

Proof. Retain the notation in Definition 3.3. In particular, we have

Hom_{𝔤,K}(V_{j}, C^{∞}(G/H; ξ_{j})) ≠ 0,

where ξ_{j} → ∞ as j → ∞. Let ι_{x}: G′/H′_{x} ↪ G/H be a natural embedding induced from the mapping G′ → G/H, g ↦ gxH. By Lemma 3.2 (2), there exists b > 0 such that we have a G′-homomorphism

ι_{x}^{∗}: C^{∞}(G/H; ξ_{j}) → C^{∞}(G′/H′_{x}; b ξ_{j}),

for any j. It follows from Theorem 2.8 that there exists an irreducible (𝔤′, K′)- submodule W_{j} of V_{j} (⊂ C^{∞}(G/H; ξ_{j})) such that

ι_{x}^{∗}(W_{j}) ≠ {0}.

Namely, we have

Hom_{𝔤′,K′}(W_{j}, C^{∞}(G′/H′_{x}; b ξ_{j})) ≠ 0.

Therefore G′/H′_{x} satisfies (D-∞). The second statement follows from Lemma 3.6 (2) with G/H replaced by G′/H′_{x}. □

Remark 3.8. As we saw in the proof, the discrete series representations for G′/H′_{x} constructed in Theorem 3.7 are irreducible constituents of the restriction π_{j}|_{G′}.

§4. Discrete series for symmetric spaces and Zuckerman-Vogan’s modules.

\underline{4.1.} In the previous section, we obtained a general framework of the existence of discrete series representations for a homogeneous manifold of reductive type (see Theorem 3.7). We shall apply Theorem 3.7 to a specific setting defined by two involutions σ and τ of G, in order to obtain an explicit condition that assures the existence of discrete series representations. This section is devoted to a quick review of discrete series representations for semisimple symmetric spaces, Zuckerman-Vogan’s derived functor modules and the criterion for the admissible restrictions with respect to reductive subgroups, which will be used in §5.

Throughout this section we suppose that G is a real reductive linear Lie group contained in a connected complex Lie group G_{ℂ} with Lie algebra 𝔤 = 𝔤_{0} ⊗_{ℝ} ℂ. Let

θ be a Cartan involution of G, K = G^{θ} the fixed point group of θ and 𝔤_{0} = 𝔨_{0} + 𝔭_{0} the corresponding Cartan decomposition. We take a Cartan subalgebra 𝔱_{0}^{c} of 𝔨_{0} and fix a positive system Δ^{+}(𝔨, 𝔱^{c}). Let 𝔥_{0}^{c} be a fundamental Cartan subalgebra of 𝔤_{0} which contains 𝔱_{0}^{c}.

\underline{4.2.} We review Zuckerman-Vogan’s derived functor modules that give a vast generalization of Borel-Weil-Bott’s construction of finite dimensional representations of compact Lie groups.

Given an element X ∈ \sqrt{−1}𝔱_{0}^{c}, we define a θ-stable parabolic subalgebra

𝔮 = 𝔩 + 𝔲 ≡ 𝔩(X) + 𝔲(X) (⊂ 𝔤)

such that 𝔩 and 𝔲 are the sum of eigenspaces with 0 and positive eigenvalues of ad(X), respectively. We note that 𝔩 is the complexification of the Lie algebra of L = Z_{G}(X), the centralizer of X in G. We denote by L̃ the metaplectic covering of L defined by the character of L acting on ∧^{dim 𝔲}𝔲.

We say that 𝔮 is in a standard position for a fixed positive system Δ^{+}(𝔨, 𝔱^{c}) if X lies in a dominant chamber with respect to Δ^{+}(𝔨, 𝔱^{c}). We note that any θ-stable parabolic subalgebra is conjugate to the one in a standard position by Ad(K).

As an algebraic analogue of the Dolbeault cohomology of a G-equivariant holomorphic vector bundle over a complex manifold G/L, Zuckerman introduced the cohomological parabolic induction R_{𝔮}^{j} ≡ (R_{𝔮}^{𝔤})^{j} (j ∈ ℕ), which is a covariant functor from the category of metaplectic (𝔩, (L∩K)^{∼})-modules to that of (𝔤, K)-modules (see [33], Ch.6; [35], Ch.6; [38], Ch.6). In this paper, we follow the normalization in [35], Definition 6.20. Then R_{𝔮}^{S}(ℂ_{ρ(𝔲)}) is a non-zero irreducible (𝔤, K)-module having the same infinitesimal character with that of the trivial representation 1, where S = dim_{ℂ}(𝔲 ∩ 𝔨) and ρ(𝔲) = \frac{1}{2}Trace (ad_{|𝔲}).

We fix a positive system Δ^{+}(𝔩, 𝔥^{c}) and write ρ_{𝔩} for half the sum of positive roots of Δ(𝔩, 𝔥^{c}). Following [36] Definition 2.5, we say that a one dimensional representation ℂ_{λ} of 𝔩 is in the good range if

(4.2.1)(a)  Re⟨λ + ρ_{𝔩}, α⟩ > 0 for any α ∈ Δ(𝔲, 𝔥^{c}),

which is independent of the choice of Δ^{+}(𝔩, 𝔥^{c}). We say that ℂ_{λ} of 𝔩 is in the fair range if

(4.2.1)(b)  Re⟨λ, α⟩ > 0 for any α ∈ Δ(𝔲, 𝔥^{c}),

which is implied by (4.2.1)(a). It is weakly good (respectively, weakly fair) if the weak inequalities hold. (The good range is defined for more general representations of 𝔩, but we do not need such generalization in this paper.)

Fact 4.2 ([34]; cf. [35], Theorem 6.8). Suppose ℂ_{λ} is a one dimensional metaplectic representation of L̃.

1) If ℂ_{λ} is a metaplectic representation of L̃ in the good range, then R_{𝔮}^{S}(ℂ_{λ}) is non-zero and irreducible.

2) If ℂ_{λ} is a metaplectic unitary representation of L̃ in the weakly fair range, then R_{𝔮}^{S}(ℂ_{λ}) is an infinitesimally unitarizable (𝔤, K)-module.

Hereafter, we write Π(𝔮, λ) for the unitary representation of G obtained by the completion of the pre-Hilbert space R_{𝔮}^{S}(ℂ_{λ}) in the setting of Fact 4.2 (2).

We define a closed cone in \sqrt{−1}(𝔱_{0}^{c})^{∗} by

(4.2.2)  ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ := { ∑_{β∈Δ(𝔲∩𝔭,𝔱^{c})} n_{β} β : n_{β} ≥ 0}.

\underline{4.3.} We review the criterion that the restriction of the unitary representation Π(𝔮, λ)|_{K′} is K′-admissible.

Let τ be an involutive automorphism of G and G′ an open subgroup of the fixed point subgroup G^{τ} := {g ∈ G : τg = g}. Then (G, G′) is called a reductive symmetric pair. If G is semisimple, then (G, G′) is also called a semisimple symmetric pair.

We have already fixed K, θ, 𝔱_{0}^{c} and a positive system Δ^{+}(𝔨, 𝔱^{c}) in §4.1. We say that τ is in a standard position for Δ^{+}(𝔨, 𝔱^{c}) if the following four conditions are satisfied: (4.3.1) τθ = θτ. (4.3.2) τ(𝔱^{c}) = 𝔱^{c}. (4.3.3) Let 𝔨_{0}^{−τ} := {X ∈ 𝔨_{0} : τX = −X}. Then 𝔱_{0}^{−τ} := 𝔱_{0}^{c} ∩ 𝔨_{0}^{−τ} is a maximally abelian subspace in 𝔨_{0}^{−τ}. (4.3.4) {α_{|𝔱^{−τ}} : α ∈ Δ^{+}(𝔨, 𝔱^{c})} \ {0} gives a positive system of Σ(𝔨, 𝔱^{−τ}).

We note that any involutive automorphism of G is conjugate (by an inner automorphism) to the one that is in a standard position for Δ^{+}(𝔨, 𝔱^{c}).

In the following theorem, we shall regard (𝔱_{0}^{c})^{∗} ⊃ (𝔱_{0}^{−τ})^{∗}, according to the direct sum 𝔱_{0}^{c} = (𝔱_{0}^{c} ∩ 𝔨_{0}^{τ}) ⊕ 𝔱_{0}^{−τ} (see (4.3.2)).

Fact 4.3. Let τ be an involutive automorphism of G that is in a standard position for a fixed positive system Δ^{+}(𝔨, 𝔱^{c}). Retain the above notation. We put K′ := G′ ∩ K. Let 𝔮 = 𝔩 + 𝔲 be a θ-stable parabolic subalgebra of 𝔤 which is in a standard position for Δ^{+}(𝔨, 𝔱^{c}). Then the following two conditions are equivalent:

i) 𝔮 = 𝔩 + 𝔲 and τ satisfy

(4.3.5)  ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1} (𝔱_{0}^{−τ})^{∗} = \{0\}.

ii) The restriction of Π(𝔮, λ) with respect to K′ is K′-admissible for any metaplectic unitary representation ℂ_{λ} in the weakly fair range.

Proof. See [14], Theorem 3.2 for the implication (1) ⇒ (2). See [18], Theorem 4.2 for the implication (1) ⇐ (2). □

We note that the following condition is also equivalent to (i) (or equivalently, (ii)) (see [18]): the restriction of Π(𝔮, λ) with respect to K′ is K′-admissible for some metaplectic unitary representation ℂ_{λ} in the good range.

\underline{4.4.} General theory of discrete series representations for a semisimple symmetric space has been developed in the last two decade. Here is a brief summary of the classification of discrete series representations in terms of Π(𝔮, λ) (see §4.2).

Let σ be an involutive automorphism of G which we may assume to be in a standard position with respect to a fixed positive system Δ^{+}(𝔨, 𝔱^{c}). Let H be G^{σ} or its open subgroup. Using an analogous notation of §4.3, we choose Σ^{+}(𝔨, 𝔱^{−σ}) that is compatible with Δ^{+}(𝔨, 𝔱^{c}) (see (4.3.4)). If rank G/H = rank K/H ∩ K then 𝔱_{0}^{−σ} is a maximally abelian subspace in 𝔤_{0}^{−σ} := {X ∈ 𝔤_{0} : σX = −X}. We denote by W (𝔤, 𝔱^{−σ}) ⊃ W (𝔨, 𝔱^{−σ}) the Weyl groups of the restricted root systems Σ(𝔤, 𝔱^{−σ}) ⊃ Σ(𝔨, 𝔱^{−σ}). We fix a positive system Σ^{+}(𝔤, 𝔱^{−σ}) which contains Σ^{+}(𝔨, 𝔱^{−σ}). Fix a strictly dominant element X ∈ \sqrt{−1}𝔱_{0}^{−σ} with respect to Σ^{+}(𝔤, 𝔱^{−σ}). Then X gives rise to a θ-stable parabolic subalgebra 𝔮 ≡ 𝔮(X) = 𝔩 + 𝔲 with Σ^{+}(𝔤, 𝔱^{−σ}) = Δ(𝔲, 𝔱^{−σ}) in the manner of §4.2. Choose a representative m_{w} ∈ K for each w ∈ W (𝔨, 𝔱^{−σ})\W (𝔤, 𝔱^{−σ}) such that Ad(m_{w})X is dominant with respect to Δ^{+}(𝔨, 𝔱^{c}) and we define a θ-stable parabolic subalgebra 𝔮^{w} := Ad(m_{w})𝔮 = 𝔩 + 𝔲^{w}, where 𝔲^{w} := Ad(m_{w})𝔲. Let λ^{w} := Ad^{∗}(m_{w})λ. We note that λ is in the fair range for 𝔮 if and only if so is λ^{w} for 𝔮^{w}.

Discrete series representations for a reductive symmetric space G/H were originally constructed as a composition of the Flensted-Jensen duality and the Poisson transform of the space of hyperfunctions on the real flag variety with support in a certain algebraic subvariety. It was proved later that the underlying (𝔤, K)-modules are isomorphic to certain Zuckerman’s derived functor modules. We summarize:

Fact 4.4. ([5]; [25]; [6], Chapter VIII §2; [36], §4). Let G/H be a reductive symmetric space.

1) Disc(G/H) ≠ ∅ if and only if rank G/H = rank K/H ∩ K.

2) If rank G/H = rank K/H ∩ K, then any discrete series representation for G/H is of the form Π(𝔮^{w}, λ^{w}) where w ∈ W (𝔨, 𝔱^{−σ})\W (𝔤, 𝔱^{−σ}) and λ^{w} is in the fair range with respect to 𝔮^{w} satisfying some integral conditions determined by (G, H).

We shall denote by V_{w,λ} ⊂ L^{2}(G/H) the corresponding closed G-invariant subspace. That is, V_{w,λ} ≃ Π(𝔮^{w}, λ^{w}) as unitary representations of G and (V_{w,λ})_{K} ≃ R_{𝔮}^{S}(ℂ_{λ}) as (𝔤, K)-modules.

\underline{4.5.} We review the asymptotic behaviour of K-finite functions that belong to discrete series representations for semisimple symmetric spaces. This was the main ingredients of the proof of Fact 4.4 (1).

Retain the notation in §4.4. Suppose G/H is a reductive symmetric space with rank G/H = rank K/H ∩ K. Let {α_{1}, …, α_{m}} be the set of simple roots of Σ^{+}(𝔤, 𝔱^{−σ}). For λ ∈ (𝔱^{−σ})^{∗}, we set

(4.5.1)  ξ(λ) := min_{1≤i≤m} Re⟨λ, α_{i}⟩.

Then ξ(λ) > 0 if and only if ℂ_{λ} is in the fair range with respect to 𝔮 = 𝔩 + 𝔲 (see §4.2). We note that the underlying (𝔤, K)-module (V_{w,λ})_{K} ⊂ L^{2}(G/H) ∩ A(G/H).

Then the following lemma is a reformulation of a special case of [27], Theorem 0.2 (cf. [32], §2).

Lemma 4.5. Assume we are in the above setting. Then there exists a constant M ≡ M_{G/H} > 0 such that

(V_{w,λ})_{K} ⊂ C^{∞}(G/H; M ξ(λ)),

for any discrete series representation V_{w,λ}. In particular, G/H satisfies (D-∞) (see Definition 3.3).

We remark that the constant M depends on the normalization of Ad(G)-invariant bilinear form B on 𝔤.

§5. Discrete series representations for the orbit spaces G^{τ} \G/G^{σ}.

\underline{5.1.} In this section, we shall give an explicit condition that assures the existence of discrete series representations on certain submanifolds of reductive symmetric spaces, as an application of Theorem 3.7.

Theorem 5.1. Let σ and τ be involutive automorphisms of G which are in standard positions for a fixed positive system Δ^{+}(𝔨, 𝔱^{c}) (see §4.3). Let H := G^{σ} and G′ := G^{τ}. We assume the following two conditions:

i) rank G/H = rank K/H ∩ K. ii) There exists w ∈ W (𝔨, 𝔱^{−σ})\W (𝔤, 𝔱^{−σ}) (see §4.4) such that

ℝ_{+} ⟨𝔲^{w} ∩ 𝔭⟩ ∩ \sqrt{−1} (𝔱_{0}^{−τ})^{∗} = \{0\}.

We put H′_{x} := G′ ∩ xHx^{−1} for x ∈ K. Then the following statements hold:

1) There exist infinitely many discrete series representations for G′/H′_{x} for any x ∈ K.

2) Assume moreover that Z_{G}(𝔱^{−σ}) is compact. Then Disc(G′/H′_{x}) ∩ Disc(G′) ≠ ∅ for any x ∈ K.

Proof. We write K′ := K ∩ G′, a maximal compact subgroup of G′.

1) It follows from Fact 4.3 and from the assumption (ii) that the restriction of the unitary representation V_{w,λ} ≃ Π(𝔮^{w}, λ^{w}) with respect to K′ is K′-admissible. We take a sequence of discrete series representations V_{w,λ_{j}} (j = 1, 2, ...) such that lim_{j→∞} ξ(λ_{j}) = ∞ (see (4.5.1)). Then the assumption of Theorem 3.7 is satisfied by Lemma 4.5. Thus, (1) follows from Theorem 3.7.

2) To prove the second statement, we recall that the discrete series representations for G′/H′_{x} obtained in (1) are isomorphic to irreducible constituents of Π(𝔮^{w}, λ^{w})_{|G′} (see Remark 3.8). If Z_{G}(𝔱^{−σ}) is compact and if λ is sufficiently regular, then Π(𝔮^{w}, λ^{w}) is a discrete series representation for G (see [33]; this is an algebraic analogue of the Langlands conjecture proved in [30]). Then any irreducible constituent of Π(𝔮^{w}, λ^{w})_{|G′} is a discrete series representation for G′, as we shall see in Corollary 8.7 (1). Hence we have proved (2). □

Remark 5.2. Several remarks are in order.

1) We do not assume the commutativity of σ and τ in Theorem 5.1. In fact, the following triplet

(G, G^{σ}, G^{τ}) = (U(2p, 2q), Sp(p, q), U(i, j) × U(2p − i, 2q − j))

satisfy the assumptions (i) and (ii) in Theorem 5.1. If i or j is odd, then σ does not commute with τ (or any involution which is conjugate to τ by an inner automorphism).

2) The special case where dim H +dim G′ = dim G+dim(H ∩G′) (and x = e) was studied in [14], Corollary 5.6, where we dealt with certain non-symmetric spherical homogeneous manifolds.

3) The assumptions of Theorem 5.1 are also satisfied if the triplet (G, H, G′) ≡ (G, G^{σ}, G^{τ}) is one of the following cases:

(O(p, q), O(m) × O(p − m, q), O(p, q − r) × O(r)),
(U(p, q), U(m) × U(p − m, q), U(p, q − r) × U(r)),
(Sp(p, q), Sp(m) × Sp(p − m, q), Sp(p, q − r) × Sp(r)),

where 2m ≤ p and 0 ≤ r ≤ q. Explicit branching laws in the case m = 1 and the relation with “minimal unipotent representations” will be studied in a forthcoming paper joint with B. Ørsted [19]. Different types of examples of Theorem 5.1 are presented in §6 and §7.

4) Regarding to homogeneous manifolds of the form G′/H′_{x}, we refer to a recent study of T. Matsuki on the orbit structure of G^{τ} acting on G/G^{σ} (see [23], [24]). It seems promising to generalize our approach here to harmonic analysis on arbitrary “semisimple orbits” of G^{τ} \G/G^{σ} in the sense of [24] by relaxing our assumption x ∈ K.

§6. Examples.

\underline{6.1.} In this section we illustrate Theorem 5.1 by a specific example in details, compare known cases, and examine which homogeneous manifolds G′/H′_{x} appear when we vary x ∈ K. The discrete series representations for G′/H′_{x} obtained here (and also in examples in Remark 5.2 (3)) are not highest weight modules. In §7 we discuss discrete series representations that have highest weight modules.

\underline{6.2.} The goal of this section is to prove:

Proposition 6.2. If m ∈ 4ℕ, then the homogeneous manifold O(m, n)/U(\frac{m}{2}, j) admits discrete series representations for any j and n with 0 ≤ 2j ≤ n. Furthermore, Disc(O(m, n)/U(\frac{m}{2}, j)) ∩ Disc(O(m, n)) ≠ ∅.

\underline{6.3.} The cases n = 2j and n = 2j + 1 were previously known:

Remark. 1) Assume n = 2j. Then the homogeneous manifold O(m, n)/U(\frac{m}{2}, \frac{n}{2}) is a semisimple symmetric space and the rank assumption (see Fact 4.4) amounts to the condition m ∈ 4ℕ or n ∈ 4ℕ. Therefore, O(m, n)/U(\frac{m}{2}, \frac{n}{2}) admits discrete series representations if and only if m ∈ 4ℕ or n ∈ 4ℕ.

2) Assume n = 2j + 1. Then the homogeneous manifold O(m, n)/U(\frac{m}{2}, \frac{n−1}{2}) is not a symmetric space but so called a spherical homogeneous manifold (e.g. [20], [2]). Taking this opportunity, we would like to correct an example of [14], Corollary 5.9 (a). The assumption “if pq ∈ 2ℤ” (loc. cit.) for the existence of discrete series representations for O(2p−1, 2q)/U(p−1, q) should be replaced by “if q ∈ 2ℤ”, which corresponds to the condition m ∈ 4ℕ with the notation here. We note that there does not exist a discrete series representation for O(2p − 1, 2q)/U(p − 1, q) if pq is odd.

\underline{6.4.} Proof of Proposition 6.2. We fix a sufficiently large l (e.g. l ≥ m + n) such that l + n ∈ 2ℤ. Let

(G, H, G′) := (O(m, n + l), U(\frac{m}{2}, \frac{n + l}{2} ), O(m, n) × O(l)) .

Then both G/H and G/G′ are symmetric spaces, and we write σ and τ for the corresponding involutive automorphisms of G. We note that

rank G/H = rank K/H ∩ K = [\frac{m + l + n}{4}] .

It is convenient to put

p := \frac{m}{4}, q := [\frac{n + l}{4} ] , ε := \{ \begin{matrix}1 & if l + n + 2 ∈ 4ℤ, \\ 0 & if l + n ∈ 4ℤ. \end{matrix}

We fix a maximal abelian subspace 𝔱_{0}^{c} of 𝔨_{0} ≃ 𝔬(m) ⊕ 𝔬(l + n) and take a basis \{f_{1}, …, f_{2p+2q+ε}\} of \sqrt{−1}(𝔱_{0}^{c})^{∗} with

Δ^{+}(𝔨, 𝔱^{c}) = {±(f_{i} ± f_{j}) : 1 ≤ i < j ≤ 2p or 2p + 1 ≤ i < j ≤ 2p + 2q + ε}.

With the coordinate defined by f_{1}, …, f_{2p+2q+ε}, we can take

𝔱_{0}^{−σ} = \{(H_{1}, H_{1}, …, H_{p+q}, H_{p+q}, (0)) : H_{j} ∈ \sqrt{−1}ℝ\}  ⊂ 𝔱_{0}^{c},
𝔱_{0}^{−τ} = \{(\underbrace{0, …, 0}_{2p}, H_{1}, H_{2}, …, H_{n}, \underbrace{0, …, 0}_{2q+ε−n}) : H_{j} ∈ \sqrt{−1}ℝ\} ⊂ 𝔱_{0}^{c}.

Here (0) stands for 0 if ε = 1; for ∅ if ε = 0.

We define a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲 of 𝔤 by

X := (p + q, p + q, …, q + 1, q + 1, q, q, …, 1, 1, (0)) ∈ \sqrt{−1}𝔱_{0}^{−σ} ⊂ \sqrt{−1}𝔱_{0}^{c}

(see §4.1 and §4.4). Then we have

Δ(𝔲 ∩ 𝔭, 𝔱^{c}) = {f_{i} ± f_{j} : 1 ≤ i ≤ 2p, 2p + 1 ≤ j ≤ 2p + 2q + ε},

and therefore

ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ⊂ {(a_{1}, …, a_{2p+2q+ε}) : |a_{j}| ≤ ∑_{i=1}^{2p} a_{i}, (2p + 1 ≤ j ≤ 2p + 2q + ε)}.

Suppose a := (a_{1}, …, a_{2p+2q+ε}) ∈ ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1}(𝔱_{0}^{−τ})^{∗}. Then a_{1} = ⋯ = a_{2p} = 0 because a ∈ \sqrt{−1}(𝔱_{0}^{−τ})^{∗}, and |a_{j}| ≤ ∑_{i=1}^{2p} a_{i}, (2p + 1 ≤ j ≤ 2p + 2q + ε) because a ∈ ℝ_{+}⟨𝔲 ∩ 𝔭⟩. Therefore a = 0. Hence, ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1}(𝔱_{0}^{−τ})^{∗} = \{0\}. Applying Theorem 5.1 (1) with w = e, we have proved that there exist discrete series representations for G′/H′_{x} whenever x ∈ K.

Now the first statement of the proposition is deduced from the following two lemmas. The second statement follows from the fact that

Z_{G}(𝔱_{0}^{−σ}) ≃ U(2)^{p+q} × 𝕋^{ε}

is compact. □

Lemma 6.5 (Matsuki). If l ≥ m + n then for any j with 0 ≤ 2j ≤ n, we can find x ≡ x(j) ∈ K such that

H′_{x} = G′ ∩ xHx^{−1} ≃ U(\frac{m}{2}, j) × (compact subgroups),

where U(\frac{m}{2}, j) is contained in the first factor of G′ = O(m, n) × O(l).

Let

a(E_{1}, …, E_{[ \frac{n}{2} ]}) := (\underbrace{0, …, 0}_{2p}, E_{1}, E_{1}, …, E_{[ \frac{n}{2} ]}, E_{[ \frac{n}{2} ]}, \underbrace{0, …, 0}_{2q+ε−2[ \frac{n}{2} ]}).

With notation in the proof of Proposition 6.2, we remark that

𝔱_{0}^{−σ} ∩ 𝔱_{0}^{−τ} = \{a(E_{1}, …, E_{[ \frac{n}{2} ]}) : E_{j} ∈ \sqrt{−1}ℝ\} (⊂ 𝔱_{0}^{c}).

Fix 0 < E_{j+1} ≤ E_{j+2} ≤ ⋯ ≤ E_{[ \frac{n}{2} ]} ≤ \frac{π}{4} . Then the following x(j) (0 ≤ 2j ≤ n) is what we wanted in Lemma 6.5:

x(j) := exp (a(\underbrace{0, …, 0}_{j}, E_{j+1}, …, E_{[ \frac{n}{2} ]})) ∈ K.

Lemma 6.6. Let G = G_{1} × G_{2} be the direct product of unimodular Lie groups with G_{2} compact. Suppose H = H_{1} ×M is a unimodular closed subgroup of G such that H_{1} ⊂ G_{1} and that M is a compact subgroup of G = G_{1}×G_{2}. If Disc(G/H) ≠ ∅ then Disc(G_{1}/H_{1}) ≠ ∅.

Proof. Since M ≃ H/H_{1} is compact, we have L^{2}(G/H) ⊂ L^{2}(G/H_{1}) according to the fiber bundle G/H_{1} → G/H with compact fiber H/H_{1}. Therefore, we have Disc(G/H) ⊂ Disc(G/H_{1}). Because H_{1} ⊂ G_{1} ⊂ G = G_{1} × G_{2}, Disc(G/H_{1}) = Disc(G_{1}/H_{1}) × Ĝ_{2}. Hence lemma. □

§7. Holomorphic discrete series representations.

\underline{7.1.} “Holomorphic discrete series representations” are discrete series representation that have highest weight vectors. Holomorphic discrete series representations for a group manifold are the best understood discrete series representations which were first constructed by Harish-Chandra. More generally, “holomorphic discrete series representations” for a semisimple symmetric space G/H exist if G/H is of Hermitian type (see [26]). There are two known methods for the proof of this fact:

1) To identify discrete series representations for a semisimple symmetric space by means of the Langlands classification [29] or by means of Zuckerman’s derived functor modules (use Fact 4.4 (2) and [1]).

2) To construct Hardy spaces based on invariant cones in Lie algebras (see [4] for a survey and references).

In this section, we give a third proof of this fact based on the admissible restriction (Definition 2.6). Much more than that, we find quite a large class of new examples such that non-symmetric homogeneous manifolds G/H admit infinitely many discrete series representations which are isomorphic to holomorphic discrete series representations for G. It might be interesting to interpret the results in this section in the context of Olshanskii semigroups.

\underline{7.2.} Let G be a simple linear Lie group, θ a Cartan involution of G, K a maximal compact subgroup and 𝔤_{0} = 𝔨_{0} + 𝔭_{0} the Cartan decomposition. Throughout this section we assume that G/K is Hermitian, that is, the center 𝔠(𝔨_{0}) of 𝔨_{0} is not trivial. It is known that 𝔠(𝔨_{0}) is one dimensional, and we can take Z ∈ 𝔠(𝔨_{0}) so that

𝔤 = 𝔨 ⊕ 𝔭^{+} ⊕ 𝔭^{−}

are 0, \sqrt{−1} and −\sqrt{−1} eigenspaces of ad(Z). An irreducible (𝔤, K)-module is said to be a highest weight module if there exists a non-zero vector annihilated by 𝔭^{+}. It will be convenient to allow the term for ‘highest weight module’ to refer also to an irreducible unitary representation of G whose underlying (𝔤, K)-modules is a highest weight module. We denote by Ĝ_{h.w.} (⊂ Ĝ) the unitary equivalence class of irreducible unitary highest weight modules. Then an element of Disc(G) ∩ Ĝ_{h.w.} is called a holomorphic discrete series representation for G. Similarly, we say that an element of Disc(G/H) ∩ Ĝ_{h.w.} is a holomorphic discrete series representation for G/H. Lowest weight modules and anti-holomorphic discrete series are defined similarly with 𝔭^{+} replaced by 𝔭^{−}.

Suppose τ is an involutive automorphism of G commuting with θ. Since τ𝔠(𝔨_{0}) = 𝔠(𝔨_{0}), there are two exclusive possibilities:

(7.2.1)  τZ = Z,
(7.2.2)  τZ = −Z.

A semisimple symmetric space G/G^{τ} with τ satisfying (7.2.2) is a typical example of symmetric spaces of Hermitian type, or compactly causal symmetric space (see [4] for the terminology). Another example of semisimple symmetric space of Hermitian type is the group manifold G × G/ diag(G) if G/K is a Hermitian symmetric space.

\underline{7.3.} Here is an infinitesimal classification of the semisimple symmetric pairs (𝔤_{0}, 𝔤_{0}^{τ}) = (Lie(G), Lie(G^{τ})) with 𝔤_{0} simple that satisfy the condition (7.2.1) or (7.2.2) (see also [4] and references therein for Table 7.3 (2)):

(G, G^{τ}) satisfying 7.2 (1)

𝔤_{0}  𝔤_{0}^{τ}
𝔰𝔲(p, q) 𝔰(𝔲(i, j) + 𝔲(p − i, q − j))
𝔰𝔲(n, n)  𝔰𝔬^{∗}(2n)
𝔰𝔲(n, n)  𝔰𝔭(n, ℝ)
𝔰𝔬^{∗}(2n)  𝔰𝔬^{∗}(2p) + 𝔰𝔬^{∗}(2n − 2p)
𝔰𝔬^{∗}(2n)  𝔲(p, n − p)
𝔰𝔬(2, n)  𝔰𝔬(2, p) + 𝔰𝔬(n − p)
𝔰𝔬(2, 2n)  𝔲(1, n)
𝔰𝔭(n, ℝ)  𝔲(p, n − p)
𝔰𝔭(n, ℝ)  𝔰𝔭(p, ℝ) + 𝔰𝔭(n − p, ℝ)
𝔢_{6(−14)}  𝔰𝔬(10) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔬^{∗}(10) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔬(8, 2) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)
𝔢_{6(−14)}  𝔰𝔲(4, 2) + 𝔰𝔲(2)
𝔢_{7(−25)}  𝔢_{6(2)} + 𝔰𝔬(2)
𝔢_{7(−25)}  𝔢_{6(−14)} + 𝔰𝔬(2)
𝔢_{7(−25)}  𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)
𝔢_{7(−25)}  𝔰𝔬^{∗}(12) + 𝔰𝔲(2)
𝔢_{7(−25)}  𝔰𝔲(6, 2)

Table 7.3(1)

(G, G^{τ}) satisfying 7.2 (2)

𝔤_{0}  𝔤_{0}^{τ}
𝔰𝔲(p, q)  𝔰𝔬(p, q)
𝔰𝔲(n, n)  𝔰𝔩(n, ℂ) + ℝ
𝔰𝔲(2p, 2q)  𝔰𝔭(p, q)
𝔰𝔬^{∗}(2n)  𝔰𝔬(n, ℂ)
𝔰𝔬^{∗}(4n)  𝔰𝔲^{∗}(2n) + ℝ
𝔰𝔬(2, n)  𝔰𝔬(1, p) + 𝔰𝔬(1, n − p)
𝔰𝔭(n, ℝ)  𝔤𝔩(n, ℝ)
𝔰𝔭(2n, ℝ)  𝔰𝔭(n, ℂ)
𝔢_{6(−14)}  𝔣_{4(−20)}
𝔢_{6(−14)}  𝔰𝔭(2, 2)
𝔢_{7(−25)}  𝔢_{6(−26)} + 𝔰𝔬(1, 1)
𝔢_{7(−25)}  𝔰𝔲^{∗}(8)

Table 7.3(2)

\underline{7.4.} The restriction problem of a unitary representation π is often easier if π is a highest weight module and has been studied in different contexts such as Howe’s dual pair correspondence.

Using the property of highest weight modules, we can treat admissible conditions in a quite elementary way without using results of derived functor modules (e.g.

Fact 4.3). Here is a theorem, which we shall supply with an elementary proof for the convenience of the reader.

Theorem 7.4. Let π, π′ ∈ Ĝ be irreducible unitary highest weight modules.

1) The tensor product π ˆ⊗π′ is K-admissible, and especially G-admissible.

2) Any irreducible constituent of π ˆ⊗π′ is a unitary highest weight module of G.

3) If π is a holomorphic discrete series representation and if π′ is a unitary highest weight module, then any irreducible constituent of π ˆ⊗π′ is also a holomorphic discrete series representation.

Assume that τ is an involutive automorphism satisfying (7.2.1).

4) The restriction of π_{|K^{τ}} is K^{τ} -admissible, and especially G^{τ} -admissible.

5) Any G^{τ} -irreducible constituent is a unitary highest weight module of G^{τ}.

6) Furthermore, if π is a holomorphic discrete series representation for G then any irreducible constituent is a holomorphic discrete series representation for G^{τ}.

Proof. Let G_{ℂ} be a connected complex Lie group with Lie algebra 𝔤, P_{−} a parabolic subgroup of G_{ℂ} with Lie algebra 𝔨+𝔭^{−}. We equip G/K with a G-invariant complex structure through an open embedding (Borel embedding):

G/K ↪ G_{ℂ}/P_{−}.

Let (π, V) be an irreducible unitary highest weight representation of G, and V_{K} the underlying (𝔤, K)-module. Then,

U := (V_{K})^{𝔭^{+}} = {v ∈ V_{K} : dπ(Y)v = 0 for any Y ∈ 𝔭^{+}} (≠ {0})

is an irreducible representation, denoted by σ, of K. Let ( , )_{V} be a G-invariant inner product on V, and ( , )_{U} a K-invariant one on U. We consider the map

G × V × U → ℂ, (g, v, u) ↦ (π(g)^{−1}v, u)_{V} = (v, π(g)u)_{V} .

For fixed g ∈ G and v ∈ V, the map U → ℂ, u ↦ (π(g)^{−1}v, u)_{V} is an anti-linear function on U, and therefore there exists a unique element F_{v}(g) ∈ U such that

(F_{v}(g), u)_{U} = (π(g)^{−1}v, u)_{V} for any u ∈ U.

If g ∈ G, k ∈ K, v ∈ V and u ∈ U, then we have

(F_{v}(gk), u)_{U} = (π(gk)^{−1}v, u)_{V}

= (π(g)^{−1}v, σ(k)u)_{U} = (F_{v}(g), σ(k)u)_{U} = (σ(k)^{−1}F_{v}(g), u)_{U}.

Hence we have F_{v}(gk) = σ(k)^{−1}F_{v}(g). Similarly, we have F_{π(g_{0})v}(g) = F_{v}(g_{0}^{−1}g) for any g, g_{0} ∈ G. Thus, we have a non-zero intertwining operator between G-modules given by

F: V → C^{∞}(G ×_{K} U), v ↦ F_{v}.

Because U is annihilated by 𝔭^{+}, F_{v} is a holomorphic sections of the holomorphic vector bundle G×_{K} U → G/K, that is, F_{v} ∈ O(G×_{K} U). Because V is irreducible, the map F: V → O(G ×_{K} U) is injective. Thus, we have proved that V is realized as a G-submodule of O(G ×_{K} U).

The K-structure of the underlying (𝔤, K)-module of O(G ×_{K} U) is given by

O(G ×_{K} U)_{K} ≃ ⨁_{n=0}^{∞} S^{n}(𝔭^{−}) ⊗ U.

Here, S^{n}(𝔭^{−}) stands for the vector space of n-th symmetric tensors of 𝔭^{−}. We note that the central element Z ∈ 𝔠(𝔨) acts on S^{n}(𝔭^{−}) by the scalar −n for each n ∈ ℕ. Since τZ = Z, Z is contained in 𝔨^{τ}. Therefore, S(𝔭^{−}) = ⨁_{n=0}^{∞} S^{n}(𝔭^{−}) is K^{τ} - admissible, that is, it decomposes discretely into irreducible K^{τ} -modules with finite multiplicity. Since the K^{τ} -admissibility is preserved by taking the tensor product with a finite dimensional representation (σ, U), the restriction O(G ×_{K} U)_{K}|_{K^{τ}} is K^{τ} -admissible. Since the K^{τ} -admissibility is also preserved by taking a submodule, the restriction V |_{K^{τ}} is K^{τ} -admissible in view of V ↪ O(G ×_{K} U). Hence, the first statement of (4) is proved. If V is K^{τ} -admissible then V is also G^{τ} -admissible (see [14], Theorem 1.2). Hence, the second statement of (4) is also proved.

The proof of (1) is similar to that of (4).

Next, let us prove (5). We note that the assumption (7.2.1) gives a compatible direct sum decomposition

𝔤^{τ} = 𝔨^{τ} ⊕ (𝔭^{+} ∩ 𝔤^{τ}) ⊕ (𝔭^{−} ∩ 𝔤^{τ}).

Since v ∈ V_{K} is a non-zero vector annihilated by 𝔟_{𝔨} + 𝔭^{−}, it is also annihilated by (𝔟_{𝔨} ∩ 𝔨^{τ}) + (𝔭^{−} ∩ 𝔤^{τ}) which is a Borel subalgebra of 𝔤^{τ}. Because V_{K} is discretely decomposable as a (𝔤^{τ} , K^{τ})-module (see [18], Proposition 1.6), we can find finitely many (𝔤^{τ} , K^{τ})-modules W_{i} (⊂ V_{K}) such that v ∈ ⨁_{i} W_{i}. Then at least one of W_{i} (say, W_{1}) is a highest weight (𝔤^{τ} , K^{τ})-module. If W_{1} is a highest weight (𝔤^{τ} , K^{τ})-module then so is any irreducible constituent of W_{1} ⊗F where F is a finite dimensional representation of G^{τ}. Since any irreducible (𝔤^{τ} , K^{τ})-submodule of V_{K} is contained in W_{1} ⊗ F with a suitable choice of a finite dimensional representation F of G^{τ}, it is also a unitary highest weight module of G^{τ}. This proves (5). The proof of (2) is similar.

The statement (6) follows from (5) and from Corollary 8.7.

Finally, let us prove the statement (3). Any K-finite matrix coefficient of a constituent of the tensor product π ˆ⊗π′ is a linear combination of products ψψ′, with ψ a matrix coefficient of π and ψ′ of π′. Then ψ is L^{2} because π is a discrete series representation for G, and ψ′ is bounded because π′ is unitary (e.g. [10]). Therefore, ψψ′ is L^{2}. Hence, (3) is proved. □

Remark. 1) The statements (1) and (4) of Theorem 7.4 are known in the case where π and π′ are holomorphic discrete series representations ([22], [11]). It is also obtained as a special case of Fact 4.3 (see [14], Example 4.6).

2) It is a sharp contrast to Theorem 7.4 (4) that the restriction of a holomorphic discrete series representation with respect to K^{τ} with τ satisfying (7.2.2) instead of (7.2.1) is never K^{τ} -admissible (see [18], Theorem 5.3).

\underline{7.5.} Let G/K be an irreducible Hermitian symmetric space, σ an involutive automorphism of G satisfying (7.2.2), and x ∈ K. We consider the following two settings: Case 1: We put L_{x} := {g ∈ G : xσ(g)x^{−1} = g}. Case 2: Let τ be an involutive automorphism of G satisfying (7.2.1) and G′ := G^{τ}. We put H′_{x} := G′ ∩ xG^{σ}x^{−1}.

Theorem 7.5. The homogeneous manifolds

G/L_{x}  in Case 1
G′/H′_{x}  in Case 2

satisfy (D-∞) for any x ∈ K. Furthermore, we have

Disc(G/L_{x}) ∩ Disc(G) ∩ Ĝ_{h.w.} ≠ ∅  in Case 1,
Disc(G′/H′_{x}) ∩ Disc(G′) ∩ Ĝ′_{h.w.} ≠ ∅  in Case 2.

\underline{7.6.} Proof of Theorem 7.5. (Case 1) We define two homomorphisms:

Δ : G → G × G, g ↦ (g, g),
Δ_{σ} : G → G × G, g ↦ (g, σ(g)),

and apply Theorem 3.7 with (G′, G, H) replaced by (Δ_{σ}(G), G × G, Δ(G)). Let π be a holomorphic discrete series representation of G, and π^{∗} its dual. Then the outer tensor product π ⊠ π^{∗} ∈ { \atop G × G} is a discrete series representation for a group manifold G × G/Δ(G) regarded as a symmetric space. The restriction of π ⊠ π^{∗} to Δ_{σ}(G) is nothing but the tensor product π ˆ⊗(π^{∗} ∘ σ). On the other hand, the dual representation π^{∗} is an anti-holomorphic discrete series representation. Then π^{∗} ∘ σ is a holomorphic discrete series representation because (7.2.2) implies σ𝔭^{+} = 𝔭^{−} and σ𝔭^{−} = 𝔭^{+}. It follows from Theorem 7.4 (1) that the restriction of the tensor product π ˆ⊗(π^{∗} ∘ σ) with respect to K is K-admissible, equivalently, the restriction of π ⊠ π^{∗} with respect to Δ_{σ}(K) is Δ_{σ}(K)-admissible. Therefore, if we take a sequence π_{j} of holomorphic discrete series representations of G such that the infinitesimal character of π_{j} tends to away from the walls of Weyl chambers, then the assumption of Theorem 3.7 is satisfied with (π_{j}, V_{j}) replaced by π_{j} ⊠ π_{j}^{∗} (see Example 3.4, Lemma 4.5). Hence the homogeneous manifold

Δ_{σ}(G)/Δ_{σ}(G) ∩ (x, y)Δ(G)(x, y)^{−1}

satisfies (D-∞) for any (x, y) ∈ K × K. In particular, if we put y = e, then

Δ_{σ}(G)/Δ_{σ}(G) ∩ (x, e)Δ(G)(x, e)^{−1} = Δ_{σ}(G)/Δ_{σ}(L_{x}) ≃ G/L_{x}

satisfies (D-∞) and admits discrete series representations for any x ∈ K. Moreover, irreducible constituents of the tensor product π ˆ⊗(π^{∗} ∘ σ) are again holomorphic discrete series representations of G by Theorem 7.4 (3). This proves Theorem 7.5 in Case 1.

(Case 2) It follows from Case 1 with x = e (see also Example 7.8) that there exists a sequence of μ_{j} ∈ Ĝ satisfying the following two conditions:

i) μ_{j} is a holomorphic discrete series representation of G. ii) G/L_{e} ≃ G/G^{σ} satisfies (D-∞) with π_{j} in Definition 3.3 replaced by μ_{j}. Since the restriction of μ_{j} with respect to K^{τ} is K^{τ} -admissible by Theorem 7.4 (4), G′/H′_{x} satisfies (D-∞) for any x ∈ K by Theorem 3.7. Also, they admit infinitely many discrete series representations that are isomorphic to holomorphic discrete series representations by Theorem 7.4 (6). Hence we have proved Theorem 7.5. □

Remark 7.7.

1) Our proof relies only on the fact of holomorphic discrete series representations for a group manifold and does not depend on Fact 4.4 (we have used Lemma 4.5 only in the group manifold case). In particular, our proof for the existence of “holomorphic discrete series representations” is new even in symmetric cases (see Example 7.8).

2) If x = e then G/L is a symmetric space in Case 1. If x = e and if στ = τσ then G′/H′_{x} is a symmetric space in Case 2.

3) Theorem 7.5 gives examples of homogeneous manifolds G/H (not necessarily symmetric spaces) satisfying Disc(G/H) ∩ Disc(G) ∩ Ĝ_{h.w.} ≠ ∅. We note that there are examples of semisimple symmetric spaces G/H (of Hermitian type), for which we know the classification of discrete series representations (Fact 4.4), such that

Disc(G/H) ∩ Disc(G) ⊄ Ĝ_{h.w.},
Disc(G/H) ∩ Ĝ_{h.w.} ⊄ Disc(G).

Example 7.8. (Case 1; symmetric spaces cf. [26]). If x = e in Case 1, then L = G^{σ} and G/L = G/G^{σ} is a symmetric space of Hermitian type. See [26] for a different construction of “holomorphic discrete series” for G/G^{σ}.

Example 7.9. (Case 1). There exist infinitely many discrete series representations for

Sp(2n, ℝ)/Sp(n_{0}, ℂ) × GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ),

where n_{0} + n_{1} + ⋯ + n_{k} = n.

We note that the above homogeneous manifold is a symmetric space if n_{1} = ⋯ = n_{k} = 0.

Proof. We shall apply Case 1 of Theorem 7.5 to the symmetric pair (G, G^{σ}) = (Sp(2n, ℝ), Sp(n, ℂ)) with a suitable choice of x ∈ K. Let us compute L_{x} = {g ∈ G : xσ(g)x^{−1} = g}. We realize the Lie algebra 𝔤_{0} = 𝔰𝔭(2n, ℝ) in the space of matrices as

𝔤_{0} := \{( \begin{matrix}X & Y \\ Z & −^{t}X \end{matrix} ) : Y = ^{t}Y, Z = ^{t}Z\} ⊂ 𝔤𝔩(4n, ℝ).

We put X(a, b) := ( \begin{matrix}a & b \\ b & −a \end{matrix} ) and Y (a, b) := ( \begin{matrix}a & b \\ −b & a \end{matrix} ) for a, b ∈ M(n, ℝ). Then

\{( \begin{matrix}Y (a, b) & X(p, q) \\ X(r, u) & −^{t}Y (a, b) \end{matrix} ) : p, q, r, u are symmetric n × n matrices\} ⊂ 𝔤_{0}

is isomorphic to 𝔤_{0}^{σ} := 𝔰𝔭(n, ℂ). Let n = n_{0} + n_{1} + ⋯ + n_{k} be a partition of n, θ_{0} := 0 and θ_{1}, …, θ_{k} ∈ ℝ. We define c, s ∈ M(n, ℝ) by

c := diag(\underbrace{cos θ_{0}, …, cos θ_{0}}_{n_{0}}, \underbrace{cos θ_{1}, …, cos θ_{1}}_{n_{1}}, …, \underbrace{cos θ_{k}, …, cos θ_{k}}_{n_{k}}),
s := diag(\underbrace{sin θ_{0}, …, sin θ_{0}}_{n_{0}}, \underbrace{sin θ_{1}, …, sin θ_{1}}_{n_{1}}, …, \underbrace{sin θ_{k}, …, sin θ_{k}}_{n_{k}}).

We assume

(7.9.1)  θ_{i} + θ_{j} ≢ 0 mod 2πℤ  (0 ≤ i, j ≤ k),
(7.9.2)  θ_{i} + θ_{j} ≢ πℤ mod 2πℤ  (0 ≤ i, j ≤ k),
(7.9.3)  θ_{i} − θ_{j} ≢ πℤ mod 2πℤ  (0 ≤ i, j ≤ k).

We put x := ( \begin{matrix}Y (c, s) & O \\ O & Y (c, s) \end{matrix} ) ∈ G (⊂ GL(4n, ℝ)). Since σx = x, (𝔩_{x})_{0} :=

{Z ∈ 𝔤_{0} : Ad(x)σ(Z) = Z} that is the Lie algebra of L_{x} (see §6.4 for notation) is σ-stable. By elementary computations, we have

{Z ∈ (𝔩_{x})_{0} : σZ = Z} = 𝔰𝔭(n_{0}, ℂ) ⊕ ⨁_{j=1}^{k} 𝔤𝔩(n_{j}, ℂ) by (7.9.1),
{Z ∈ (𝔩_{x})_{0} : σZ = −Z} = {0}  by (7.9.2), (7.9.3).

Therefore, we have (𝔩_{x})_{0} = 𝔰𝔭(n_{0}, ℂ) ⊕ ⨁_{j=1}^{k} 𝔤𝔩(n_{j}, ℂ). □

Example 7.10. (Case 2). The homogeneous manifold U(2m, n)/Sp(m, j) admits discrete series representations for any j and n with 0 ≤ 2j ≤ n. Furthermore, there exist infinitely many holomorphic discrete series representations for U(2m, n) which are realized as discrete series representations for U(2m, n)/Sp(m, j).

Sketch of Proof. The proof parallels to that of Proposition 5.2 if we put G := U(2m, n + l), H := Sp(m, \frac{n+l}{2}) and G′ := U(2m, n) × U(l) where l is chosen such that l ≥ 2m + n and l + n ∈ 2ℕ. □

§8. Appendix – Analysis on principal orbits.

\underline{8.1.} Suppose a compact Lie group G acts on a connected manifold M. Then it is known that there exist a unique isotropy type H and an open dense subset M′ of M such that G_{x} is conjugate to H for any x ∈ M′. The homogeneous manifold G/H is called the principal orbit type of M (e.g. [31], Chapter 1).

An analogous statement for noncompact G is not true in general. Nevertheless, it sometimes happens that there are only finitely many isotropy types for an open dense subset of M. Harmonic analysis on such generalized “principal orbits” is much simpler than other orbits that we have discussed so far. In this section, we give a refinement of Theorem 3.7 assuming that orbits are “principal orbits” by elementary argument. A distinguishing feature in this section is that we can capture discrete series representations for principal orbits of G′ even though the restriction π|_{G′} (π ∈ Ĝ) contains both discrete and continuous spectrum (in particular, the restriction π|_{G′} is not G′-admissible).

\underline{8.2.} First, we define an analogous notion of “principal orbits” of the action of G′ on a homogeneous manifold G/H, where G′ and H are reductive subgroups of a reductive Lie group G.

Assumption 8.2. Suppose we are in the setting of §2.3. Assume that there exist closed subgroups H′_{1}, …, H′_{n} of G′, submanifolds I_{1}, …, I_{n} of G/H, and measurable maps v_{j}: I_{j} → G (1 ≤ j ≤ n) with the following properties:

i) {g ∈ G′ : g · y = y} = v_{j}(y)^{−1} H′_{j} v_{j}(y) for any y ∈ I_{j}. ii) The mapping φ_{j}: G′/H′_{j} ×I_{j} → G/H, (g, y) ↦ gv_{j}(y)·y is an open embedding. iii) The complement of ⋃_{j} φ_{j}(G′/H′_{j} × I_{j}) in G/H has measure zero. Then we say that the action of G′ on G/H admits principal orbits and that G′/H′_{j} (j = 1, …, n) are the principal orbit type of G/H.

In the following examples, we can choose I_{j} so that v_{j}(y) = e for all y ∈ I_{j}.

Example 8.3. (group manifolds). Let G be a real reductive linear Lie group, H = {e}, and G′ an arbitrary subgroup of G. Then, Assumption 8.2 is satisfied if we take:

n = 1,  H′_{1} = {e},
I_{1} (⊂ G) is a smooth section of the principal bundle G → G′\G

(the section is defined in an open dense subset of G′\G).

Example 8.4. (semisimple orbits). Suppose that G_{1} is a real reductive linear Lie group. Let θ be a Cartan involution of G_{1}. We set

G := G_{1} × G_{1} ⊃ G′ = H := diag(G_{1}).

Then the action of G′ on G/H admits principal orbits. To see this, let J_{1}, …, J_{n} be θ-stable Cartan subgroups of G_{1}, which are complete representatives of the conjugacy classes of Cartan subgroups of G_{1}. For each j, we define the Weyl group W (J_{j}) := N_{G}(J_{j})/Z_{G}(J_{j}), the open dense subset J_{j}^{reg} of J_{j} consisting of regular elements, a complete representative J_{j}^{+} of W (J_{j})\J_{j}^{reg}, I_{j} := {(x, e) : x ∈ J_{j}^{+}} ⊂ G, and H′_{j} := diag(J_{j}) ⊂ G′. Via the identification G/H ≃ G_{1}, (g_{1}, g_{2}) ↦ g_{1}g_{2}^{−1}, we define

φ_{j}: G′/H′_{j} × I_{j} → G/H ⟶^{∼} G_{1}, ((g, g), (x, e)) ↦ gxg^{−1}.

Then Assumption 8.2 is satisfied. The homogeneous manifolds G′/H′_{j} ≃ G_{1}/J_{j} (1 ≤ j ≤ n) are the principal orbit type, which are called regular semisimple orbits of the adjoint action.

This example is generalized into the following setting:

Example 8.5. (semisimple orbits in symmetric spaces). Suppose σ and τ are involutive automorphisms of G which commute with θ. We allow the case where στ ≠ τσ even though σ is replaced by its conjugation by an inner automorphism of G. Let G′ := G^{τ} and H := G^{σ}. Then the action of G′ on G/H admits principal orbits. This generalizes Example 8.4 and follows from the description of the double coset space G′\G/H in [24]. Corresponding to Cartan subgroups J_{j} of a group manifold in Example 8.4, Matsuki introduced the “Cartan subset” for the double coset space G^{τ} \G/G^{σ}, of which some open subset (corresponding to J_{j}^{+} in Example 8.4) gives a choice of I_{j} in Assumption 8.2.

\underline{8.6.} Suppose G′ is a reductive subgroup of a real reductive Lie group G. Let π ∈ Ĝ. Then the restriction π|_{G′} is decomposed into irreducible representations of G′. Let Disc(π|_{G′}) ⊂ Ĝ′ be the set of irreducible unitary representations that contribute discrete spectrum of the irreducible decomposition of π|_{G′}. Equivalently,

(8.6.1)  Disc(π|_{G′}) := {τ ∈ Ĝ′ : Hom_{G′}(τ, π|_{G′}) ≠ 0}.

Here Hom_{G′}( , ) stands for the space of continuous G′-intertwining operators. As we shall see in Corollary 8.7, Disc(π|_{G′}) can be an empty set.

Theorem 8.6. Suppose we are in the setting of §2.3. Assume that the action of G′ on G/H admits principal orbits G′/H′_{j} (j = 1, …, n) (see §8.2 for definition). Then we have

(8.6.2)  ⋃_{π∈Disc(G/H)} Disc(π|_{G′}) ⊂ ⋃_{j=1}^{n} Disc(G′/H′_{j}).

Furthermore, If π ∈ Disc(G/H) is K′-admissible, then

(8.6.3)  Disc(π|_{G′}) ⊂ ⋂_{j} Disc(G′/H′_{j}).

Before proving Theorem, we give some typical examples and applications.

\underline{8.7.} A special case corresponding to Example 8.3 yields

Corollary 8.7. Suppose rank G = rank K. Let π ∈ Disc(G) (⊂ Ĝ).

1) Any irreducible representation of G′ occurring as a discrete part of the decomposition π_{|G′} is a discrete series representation for G′.

2) If rank G′ > rank K′, then π_{|G′} is decomposed into only continuous spectrum, namely, Disc(π|_{G′}) = ∅.

Proof. Applying Theorem 8.6 with n = 1, H′_{1} = {e} and H = {e} (see Example 8.3), we have

⋃_{π∈Disc(G)} Disc(π|_{G′}) ⊂ Disc(G′).

This proves the first statement. Since rank G′ > rank K′, we have Disc(G′) = ∅. Hence, we have Disc(π|_{G′}) = ∅, proving the second statement.

It is known as a Mackey-Anh’s reciprocity theorem that if π ∈ Disc(G) then the Plancherel measure of π_{|G′} is supported on the set of tempered representations of G′. Corollary 8.7 (1) is a refinement of this fact.

Example 8.8. Here are opposite extremal cases:

1) If π ∈ Disc(SO(2p, 2q − 1)) then the restriction π_{|SO(2p−1,2q−1)} is always decomposed into only continuous spectrum.

2) There exists π ∈ Disc(SO(2p, 2q)) such that the restriction π_{|SO(2p−1,2q)} is decomposed into only discrete spectrum.

Proof. If we take G := SO(2p, 2q − 1) and G′ := SO(2p − 1, 2q − 1), then

rank G = rank K = rank G′ = p + q − 1 > rank K′ = p + q − 2.

Therefore, the first statement follows from Corollary 8.7 (2). The second statement follows from Fact 4.3 and we omit the details as the verification of the criterion (4.3.5) is similar to the computation in the proof of Proposition 6.2. □

\underline{8.9.} Proof of Theorem 8.6. 1) Let dμ be the G-invariant measure on G/H and dμ_{j} the G′-invariant measure on G′/H′_{j} (j = 1, …, n). Because dμ is G′-invariant and because φ_{j} (1 ≤ j ≤ n) are G′-equivariant maps, there exists a unique measure dν_{j} on I_{j} for each j such that the open embeddings φ_{j}: G′/H′_{j} × I_{j} ↪ G/H induce unitary equivalent maps

L^{2}(G/H; dμ) ≃ ⨁_{j=1}^{n} L^{2}(φ_{j}(G′/H′_{j} × I_{j}); dμ)

≃ ⨁_{j=1}^{n} L^{2}(G′/H′_{j}; dμ_{j}) ˆ⊗ L^{2}(I_{j}; dν_{j}).

Let us denote by

pr_{j}: L^{2}(G/H; dμ) → L^{2}(G′/H′_{j}; dμ_{j}) ˆ⊗ L^{2}(I_{j}; dν_{j})

the j-th projection. Suppose (π, V_{π}) ∈ Ĝ is a discrete series representation for G/H and (τ, W_{τ}) ∈ Disc(π_{|G′}) (⊂ Ĝ′). Then we have an isometric map

ι: W_{τ} ↪ V_{π} ↪ L^{2}(G/H),

respecting G′ actions. There exists at least one j such that pr_{j} ∘ι(W_{τ}) ≠ {0}. Then there exists f ∈ L^{2}(I_{j}; dν_{j}) such that T_{f} ∘ pr_{j} ∘ι(W_{τ}) ≠ {0}, where we define

T_{f}: L^{2}(G′/H′_{j}; dμ_{j}) ˆ⊗ L^{2}(I_{j}; dν_{j}) → L^{2}(G′/H′_{j}; dμ_{j}), F ↦ ∫_{I_{j}} F (x, y)\overline{f(y)}dν_{j}(y).

Since (τ, W_{τ}) is an irreducible unitary representation of G′ and

T_{f} ∘ pr_{j} ∘ι: W_{τ} → L^{2}(G′/H′_{j}; dμ_{j})

is a continuous map which respects G′ actions, T_{f} ∘ pr_{j} ∘ι is an isometry up to scalar. Therefore, we have (τ, W_{τ}) ∈ Disc(G′/H′_{j}), proving (1).

2) If π_{|K′} is K′-admissible, then ι(W_{τ})∩A(G/H) ≠ {0}. Hence, pr_{j} ∘ι(W_{τ}) ≠ {0} for any j because φ_{j}(G′/H′_{j} ×I_{j}) is an open set of G/H. Now the second statement follows from the same argument as in the first statement. □

References

[1] J. Adams, Unitary highest weight modules, Adv. in Math. 63 (1987), 113–137.

[2] M. Brion, Classification des espaces homogenes spheriques, Compositio Math. 63-2 (1987), 189–208.

[3] P. Delorme, Formule de Plancherel pour les espaces symétriques réductifs, preprint (1996).

[4] J. Faraut and G. Ólafsson, Causal semisimple symmetric spaces, the geometry and harmonic analysis, Semigroups in Algebra, Geometry and Analysis (1995), Walter de Gruyter, 3–32.

[5] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. of Math. 111 (1980), 253–311.

[6] , Analysis on Non-Riemannian Symmetric Spaces, Conf. Board, 61, A. M. S., 1986, ISBN 0-8218-0711-0.

[7] Harish-Chandra, Representations of semi-simple Lie groups, I, III, Trans. A. M. S. 75, (1953), 185-243; 76, (1954), 234-253; IV, Amer. J. Math. 77, (1955), 743-777.

[8] S. Helgason, Differential Geometry, Lie Groups and Symmetric Spaces, Pure and Appl. Math., vol. 80, Academic Press, 1978, ISBN 0-12-338460-5.

[9] J. Hilgert, G. Ólafsson and B. Ørsted, Hardy spaces on affine symmetric spaces, J. reine angew. Math. 415 (1991), 189–218.

[10] R. E. Howe and C. C. Moore, Asymptotic properties of unitary representations, J. Funct. Anal. 32 (1979), 72–96.

[11] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29–53.

[12] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽), vol. 462, Memoirs of Amer. Math. Soc., 1992, ISBN 0-8128-2524-0.

[13] , The restriction of A_{𝔮}(λ) to reductive subgroups, Proc. Acad. Japan 69 (1993), 262–267; Part II, ibid. 71 (1995), 24-26.

[14] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205.

[15] , Harmonic analysis on homogeneous manifolds of reductive type and unitary representation theory, Sugaku of Math. Soc. Japan 46 (1994), 124–143, (Japanese); English translation is to appear in Sugaku Exposition of Amer. Math. Soc..

[16] , Invariant measures on homogeneous manifolds of reductive type, J. reine angew. Math. (to appear).

[17] , L^{p}-analysis on homogeneous manifolds of reductive type and representation theory, Proc. Acad. Japan (to appear).

[18] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups, Part III, Invent. Math. (to appear).

[19] T. Kobayashi and B. Ørsted, Branching laws for unitary representations attached to minimal elliptic and nilpotent orbits, (in preparation).

[20] M. Krämer, Sphärische Untergruppen in kompakten zusammenhängenden Liegruppen, Compositio Math. 38-2 (1979), 129–153.

[21] J-S. Li, On the discrete series of generalized Stiefel manifolds, Trans. A. M. S. 340 (1993), 753–766.

[22] S. Martens, The characters of the holomorphic discrete series, Proc. Nat. Acad. Sci. U.S.A. 72 (1975), 3275–3276.

[23] T. Matsuki, Double coset decompositions of algebraic groups arising from two involutions I, J. Algebra 175 (1995), 865–925.

[24] , Double coset decompositions of reductive Lie groups arising from two involutions, preprint, 67 pages (1995).

[25] T. Matsuki and T. Oshima, A description of discrete series for semisimple symmetric spaces, Advanced Studies in Pure Math. 4 (1984), 331–390.

[26] G. Ólafsson and B. Ørsted, The holomorphic discrete series of an affine symmetric space, I, J. Funct. Anal. 81 (1988), 126–159.

[27] T. Oshima, Asymptotic behavior of spherical functions on semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 561–601.

[28] H. Schlichtkrull, A series of unitary irreducible representations induced from a symmetric subgroup of a semisimple Lie group, Invent. Math. 68 (1982), 497–516.

[29] , The Langlands parameters of Flensted-Jensen’s discrete series for semisimple symmetric spaces, J. Funct. Anal. 50 (1983), 133–150.

[30] W. Schmid, On a conjecture of Langlands, Ann. Math. 93 (1971), 1-42.

[31] T. tom Dieck, Transformation Groups, de Gruyter Studies in Mathematics, vol. 8, Walter de Gruyter, 1987, ISBN 3-11-009745-1.

[32] Y. L. Tong and S. P. Wang, Geometric realization of discrete series for semisimple symmetric spaces, Invent. Math. 96 (1989), 425–458.

[33] D. Vogan, Representations of Real Reductive Lie Groups, Birkhäuser, 1981, ISBN 3-7643-3037-6.

[34] , Unitarizability of certain series of representations, Ann. of Math. (1984), 141–187.

[35] , Unitary Representations of Reductive Lie Groups, Princeton University Press, Princeton, New Jersey, 1987, ISBM 0-691-08481-5.

[36] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 191–221.

[37] N. Wallach, On the unitarizability of derived functor modules, Invent. Math. 78 (1984), 131-141.

[38] , Real Reductive Groups I, Pure and Appl. Math., vol. 132, Academic Press, 1988, ISBN 0-12-732960-9.

[39] G. Warner, Harmonic Analysis on Semisimple Lie Groups I, Springer-Verlag, Berlin, 1972, ISBN 3-540-05468-5.

Graduate School of Mathematical Sciences University of Tokyo Komaba, Meguro, Tokyo 153 Japan E-mail address: toshi@ms.u-tokyo.ac.jp