DISCRETE DECOMPOSABILITY OF THE RESTRICTION OF

A_{𝔮}(λ) WITH RESPECT TO REDUCTIVE SUBGROUPS III

— restriction of Harish-Chandra modules and associated varieties

Toshiyuki KOBAYASHI

University of Tokyo

Abstract. Let H ⊂ G be real reductive Lie groups and π an irreducible unitary representation of G. We introduce an algebraic formulation (discretely decomposable restriction) to single out the nice class of the branching problem (breaking symmetry in physics) in the sense that there is no continuous spectrum in the irreducible decomposition of the restriction π|_{H}. This paper offers basic algebraic properties of discretely decomposable restrictions, especially for a reductive symmetric pair (G, H) and for the Zuckerman-Vogan derived functor module π = \overline{A_{𝔮}(λ)}, and proves that the sufficient condition [Invent. Math. ’94] is in fact necessary. A finite multiplicity theorem is established for discretely decomposable modules which is in sharp contrast to known examples of the continuous spectrum. An application to the restriction π|_{H} of discrete series π for a symmetric space G/H is also given.

§0. Introduction.

\underline{0.1.} This paper is a continuation of the work [Ko2]. Let G be a real reductive linear Lie group and denote by Ĝ the unitary dual of G. The object of study is the restriction of π ∈ Ĝ with respect to a reductive subgroup H.

\underline{0.2.} In general, the restriction of π ∈ Ĝ to a subgroup H may have a wild behavior even if H is a maximal reductive subgroup of G. For instance, (0.2.1) the tensor product of principal series representations of a complex simple Lie group G is decomposed into only continuous spectrum with infinite multiplicity except the case where G is locally isomorphic to SL(2, ℂ) (Gelfand-Graev, Williams, see [GG], [Wi]).

Here are well-known opposite extremal cases: (0.2.2) The restriction of π with respect to a maximal compact subgroup K is decomposed discretely into irreducibles with finite multiplicity for any π ∈ Ĝ (Harish-Chandra). (0.2.3) The restriction of holomorphic discrete series representations with respect

1991 Mathematics Subject Classification. Primary 22E46; Secondary 22E47. Key words and phrases. Vogan-Zuckerman derived functor modules, restriction of unitary rep-

resentations, associated varieties, discrete series representations for semisimple symmetric spaces.

Typeset by AMS-TEX

to certain symmetric pairs is discretely decomposable (Martens, Jakobsen-Vergne; see Fact 5.4). Also, the restriction of the Segal-Shale-Weil representation with respect to a reductive dual pair H_{1} × H_{2} with H_{2} compact decomposes discretely.

These are also important in their applications: (0.2.2) has enabled us to study (𝔤, K)-modules by algebraic methods instead of representations of G; the decomposition in (0.2.3) gives Howe’s correspondence (e.g. Howe, Kashiwara-Vergne, Adams; [Ho], [KV], [A1]) and interacts closely with the theory of θ-series.

\underline{0.3.} In this paper, we focus on the discretely decomposable restriction as a generalization of the feature of (0.2.2) and (0.2.3). The concept of discretely decomposable restriction might be very interesting not only in representation theory but also in other branches of mathematics. In fact, discrete decomposability of the restriction, especially for Zuckerman’s modules A_{𝔮}(λ), has recently found its applications to the vanishing theorem of middle Hodge components of modular symbols for the arithmetic quotient of Hermitian symmetric domains [KO], the construction of discrete series representations for certain non-symmetric spherical homogeneous manifolds ([Ko2], §5; [Ko4]), spherical harmonics on pseudo-Riemannian homogeneous spaces and minimal unipotent representations [KØ], and the finite dimensional theorem of global holomorphic solutions to the generalization (to higher order) of the Gauss-Aomoto-Gelfand hypergeometric system ([Se], Theorem 7.1).

\underline{0.4.} In our previous paper, we have given a sufficient condition on (G, H, π) where H ⊂ G and π ∈ Ĝ in order that the restriction π|_{H} is discretely decomposable. Results follow from algebraic methods in [Ko2] and from microlocal analysis in Part II of [Ko1]. The main theorem was stated in the case where (G, H) is a reductive symmetric pair and where π is Zuckerman’s module A_{𝔮}(λ). However, these methods provided only a sufficient condition for the discrete decomposable decomposition.

In the present paper, we shall reformulate the discretely decomposable restriction in a purely algebraic way in §1, and give a necessary condition for the discretely decomposable restriction for a reductive pair (G, H) in §3. One of our main results here is that the aforementioned sufficient condition in the symmetric case is in fact a necessary condition; that is, the restriction of A_{𝔮}(λ) is discretely decomposable as an (𝔥, H ∩ K)-module if and only if ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1}(𝔱_{0}^{−σ})^{∗} = 0, where ℝ_{+}⟨𝔲 ∩ 𝔭⟩ is a closed cone determined by a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲 and 𝔱_{0}^{−σ} is a subspace determined by a reductive symmetric pair (G, H) (see Theorem 4.2). Furthermore, because the necessary condition is proved here by a weaker assumption (namely, no assumption on multiplicities), our results also yield a finite multiplicity theorem on the level of Harish-Chandra modules generalizing the case H = K (see (0.2.2)); If (G, H) is a reductive symmetric pair, then

dim Hom_{𝔥,H∩K} (Y, A_{𝔮}(λ)) < ∞

for any irreducible (𝔥, H ∩ K)-module Y and for any Zuckerman’s derived functor module A_{𝔮}(λ) (see Corollary 4.3). This result is striking in contrast to the example of the infinite multiplicity of the continuous spectrum for a symmetric pair (e.g. (0.2.1)) (here we recall that principal series representations for complex reductive groups are written in the form A_{𝔮}(λ)). In general, dim Hom_{𝔥,H∩K} (Y, A_{𝔮}(λ)) is not uniformly bounded, but we will give a sufficient condition for the uniformly bounded multiplicities in a subsequent paper which supports the Gross-Prasad conjecture.

\underline{0.5.} In §5 and §6, in the context of discretely decomposable restrictions, we try to reveal the representation theoretic principles which were hidden in the following sharp contrast of known results:

1) The restriction π|_{K} decomposes discretely for any π ∈ Ĝ (see (0.2.2)), while there is no discrete spectrum in L^{2}(G/K) for the Riemannian symmetric space G/K (Harish-Chandra) (see Theorem 6.2 and Remark 6.6 (1)).

2) The tensor product π ⊗ π′ is discretely decomposable if both π and π′ are holomorphic discrete series representations ([Ma], [JV]), while it always contains the continuous spectrum in the irreducible decomposition if π is holomorphic and π′ is anti-holomorphic discrete series representation ([R]) (cf. Theorem 5.3 and Remark 6.6 (4)).

3) The non-vanishing theorem ([TW]) and the vanishing theorem ([KO]) of modular symbols defined by arithmetic subgroups (see Remark 6.6 (3)).

\underline{0.6.} I have omitted in this paper all applications to harmonic analysis. In the last two sections we illustrate the theorem only by some examples. Irreducible (𝔤, K)- modules are discretely decomposable as (𝔥, H ∩ K)-modules in the following cases:

1) (G, H) = (G, K) (Riemannian symmetric pair): Any π is discretely decomposable as an (𝔥, H ∩ K)-module (see (0.2.2)).

2) (G, H) = (G′_{ℂ}, G′_{ℝ}) with G′_{ℝ} a normal real form of G′_{ℂ}: No π is discretely decomposable as an (𝔥, H ∩ K)-module except for dim π < ∞ (see Theorem 8.1).

3) (G, H) = (U(2, 2), Sp(1, 1)) ≈ (SO(4, 2), SO(4, 1)): 12 series among 18 series of irreducible, infinitesimally unitary (𝔤, K)-modules with regular integral infinitesimal character are discretely decomposable as (𝔥, H ∩ K)-modules (Proposition 7.5).

\underline{0.7.} The results in §3 were announced in Part II of [Ko1]. Most of other results were obtained while the author was a guest at the Institut Mittag-Leffler supported by the Royal Swedish Academy of Sciences. He expresses his sincere gratitude to the staff of the Institute and to the organizers of the special year “Analysis on Lie Groups”. Thanks are also due to R. Donley, M. Duflo, M. Flensted-Jensen, D. Vogan, T. Ohta, G. Ólafsson and B. Ørsted for helpful discussions and their encouragement.

§1. Discretely decomposable modules

In this section, we introduce the notion of discretely decomposable modules and present some basic properties.

\underline{1.1.} Let 𝔥 be a complex Lie algebra, and X an 𝔥-module.

Definition 1.1. We say X is discretely decomposable if there is an increasing filtration {X_{m}} of 𝔥-submodules such that (1.1.1) X = ⋃_{m=0}^{∞} X_{m}, (1.1.2) X_{m} is of finite length as 𝔥-module (i.e. has finite composition series).

We note that an 𝔥-module of finite length is obviously discretely decomposable.

\underline{1.2.} Here is a set up that we shall use frequently in this paper: Let G be a real reductive linear Lie group, and H a closed subgroup of G. We assume that H has at most finitely many connected components and that there exists a Cartan involution θ of G which stabilizes H. Then H is also a reductive subgroup with maximal compact subgroup H ∩ K, where K = G^{θ} is a maximal compact subgroup of G. We write 𝔤_{0} for the Lie algebra of G and write 𝔤 := 𝔤_{0} ⊗ℂ. In what follows analogous notation will be applied to Lie groups denoted by other Roman upper case letters without comment. Let 𝔤_{0} = 𝔨_{0} + 𝔭_{0} be the Cartan decomposition corresponding to the Cartan involution θ, and 𝔤 = 𝔨+𝔭 the complexification. There is a non-degenerate symmetric Ad(G)-invariant bilinear form, say B, on 𝔤_{0}, which is positive definite on 𝔭_{0} and negative definite on 𝔨_{0} such that 𝔨_{0} is orthogonal to 𝔭_{0}.

We say X is discretely decomposable as an (𝔥, H∩K)-module if X is an (𝔥, H∩K)- module and if X is discretely decomposable as an 𝔥-module.

\underline{1.3.} Suppose that we are in the setting of §1.2. An (𝔥, H ∩ K)-module X is said to be infinitesimally unitary if there is an inner product on X such that 𝔥 acts on X by skew-Hermitian operators and that H ∩ K does unitarily on X. In this case, the terminology “discretely decomposable” is justified by the following lemma:

Lemma 1.3. Let X be an infinitesimally unitary (𝔥, H ∩K)-module. We suppose that X has at most countable basis over ℂ. Then X is discretely decomposable if and only if X decomposes into the algebraic sum of irreducible 𝔥-modules.

Proof. If X is decomposed into the algebraic sum of irreducible 𝔥-modules, say, ⨁_{i=0}^{∞} Y_{i}, then we put X_{m} := ⨁_{i=0}^{m} Y_{i} (m ∈ ℕ) which obviously satisfies the conditions (1.1.1) and (1.1.2). Conversely, assume that X is discretely decomposable. We may and do assume that H is connected in order to prove that X decomposes into the algebraic sum of irreducible 𝔥-modules. Let X = ⋃ X_{m} be a filtration as in Definition 1.1. It is convenient to put X_{m} := {0} for a negative integer m. We note that each 𝔥-submodule X_{m} is also an (𝔥, H ∩ K)-submodule. Let \overline{X_{m}} be the Hilbert space that is the completion of X_{m} with respect to the pre-Hilbert structure of X. A basic result due to Harish-Chandra (cf. [V1], Theorem 0.3.5) asserts that there is a lattice isomorphism between closed H-invariant subspaces of \overline{X_{m}} and (𝔥, H ∩ K)-invariant subspaces of X_{m} because X_{m} is of finite length as an (𝔥, H ∩ K)-module. In particular, we can take the orthogonal complementary subspace of X_{m} in X_{m+1}, which is decomposed into the finite sum of irreducible (𝔥, H ∩ K)-modules. By the induction on m starting from m = −1, X = ⋃_{m} X_{m} is decomposed into the algebraic sum of irreducible (𝔥, H ∩ K)-modules. □

\underline{1.4.} Here is an elementary result for producing a family of discretely decomposable modules from a given discretely decomposable module.

Lemma 1.4. Suppose we are in the setting of §1.2. Assume that the (𝔥, H ∩ K)- module X is discretely decomposable.

1) Any submodule or quotient of X is discretely decomposable.

2) The tensor product X ⊗F is discretely decomposable for any finite dimensional (𝔥, H ∩ K)-module F.

Proof. 1) Let X = ⋃_{m=0}^{∞} X_{m} be a filtration satisfying (1.1.2). Suppose that Y and Z are (𝔥, H ∩ K)-modules and that ι: Y → X and π: X → Z are injective and surjective (𝔥, H ∩ K)-homomorphisms, respectively. Then Y_{m} := ι^{−1}(X_{m}) and Z_{m} := π(X_{m}) give the desired filtration of Y and Z, respectively.

2) Since X_{m} is of finite length as an (𝔥, H ∩K)-module, so is X_{m} ⊗F (see the proof of [V1], Corollary 4.5.6). Therefore, X_{m} ⊗ F (m = 0, 1, 2, …) gives the desired filtration of X ⊗ F. □

\underline{1.5.} Retain the setting of §1.2. Here is another characterization of discretely decomposable 𝔥-modules:

Lemma 1.5. Suppose (π, X) is an irreducible 𝔤-module. Then X is discretely decomposable as an 𝔥-module if and only if there exists an irreducible 𝔥-module Y such that Hom_{𝔥}(Y, X) ≠ 0.

Proof. Let U(𝔤) ⊃ U(𝔥) be the enveloping algebras of 𝔤 and 𝔥, respectively. Because 𝔥 is reductive in 𝔤, the invariant form B (see §1.2) is non-degenerate when restricted to 𝔥 × 𝔥. We denote by 𝔥^{⊥} the orthogonal complement of 𝔥 in 𝔤 with respect to B. Then 𝔥^{⊥} is an (𝔥, H ∩ K)-invariant subspace such that 𝔤 = 𝔥 ⊕ 𝔥^{⊥} is a direct sum decomposition. We set

(1.5.1) U′_{k}(𝔥^{⊥}) := ℂ-span⟨Y_{1} ⋯ Y_{m} : Y_{1}, …, Y_{m} ∈ 𝔥^{⊥}, m ≤ k⟩ ⊂ U(𝔤),
(1.5.2) U′(𝔥^{⊥}) := ⋃_{k=0}^{∞} U′_{k}(𝔥^{⊥}).

Then we have U(𝔤) = U′(𝔥^{⊥}) U(𝔥) by the Poincaré-Birkhoff-Witt theorem. Assume that there exists an irreducible (𝔥, H ∩ K)-module Y such that Hom_{𝔥}(Y, X) ≠ 0. We may regard Y as a submodule of X. We set X_{k} := U′_{k}(𝔥^{⊥}) Y (⊂ X). Because X is an irreducible 𝔤-module, we have

X = U(𝔤) Y = U′(𝔥^{⊥}) U(𝔥) Y = ⋃_{k=0}^{∞} U′_{k}(𝔥^{⊥}) Y = ⋃_{k=0}^{∞} X_{k}.

Let Z_{k} (⊂ X) be the image of the following (𝔥, H ∩ K)-homomorphism:

(1.5.3)  \underbrace{𝔥^{⊥} ⊗ ⋯ ⊗ 𝔥^{⊥}}_{k} ⊗Y → X, (Y_{1} ⊗ ⋯ ⊗ Y_{k}) ⊗ w ↦ π(Y_{1}) ⋯ π(Y_{k})w.

Then X_{k} = ∑_{i=0}^{k} Z_{i} (not necessarily a direct sum). This shows that X_{k} is of finite length as an (𝔥, H ∩ K)-module, since 𝔥^{⊥} ⊗ ⋯ ⊗ 𝔥^{⊥} is a finite dimensional (𝔥, H ∩K)-module. Hence the conditions (1.1.1) and (1.1.2) are satisfied. Therefore X is discretely decomposable as an (𝔥, H ∩ K)-module.

Conversely, if X is discretely decomposable as an (𝔥, H ∩ K)-module, then X contains an (𝔥, H∩K)-submodule X_{0} of finite length. Take an irreducible (𝔥, H∩K)- submodule Y of X_{0}. Then we have Hom_{𝔥,H∩K}(Y, X) ≠ 0. □

\underline{1.6.} Here is a source of discretely decomposable (𝔥, H ∩K)-modules in the context of the restriction of (𝔤, K)-modules. Suppose (π, H) is a unitary representation of G. We say the restriction π|_{H} is H-admissible if H is decomposed into a discrete Hilbert direct sum with finite multiplicity for each irreducible representation of H (see [Ko2], §1). There are known sufficient conditions for irreducible unitary representations to be H ∩K-admissible by algebraic methods in [Ko2] and by using the singularity spectrum (the analytic wave front set) in Part II of [Ko1]. The following property is also useful in the application to harmonic analysis on nonsymmetric spaces [Ko4]:

Proposition 1.6. Let (π, V) ∈ Ĝ. Assume that the restriction π|_{H∩K} is H ∩ Kadmissible. Then we have:

1) The space of K-finite vectors V_{K} coincides with that of H ∩ K-finite vectors V_{H∩K}. In particular, V_{K} is discretely decomposable as an (𝔥, H ∩ K)-module.

2) dim Hom_{𝔥,H∩K}(τ_{H∩K}, V_{K}) = dim Hom_{H}(τ, V), for any irreducible unitary representation τ of H. Here τ_{H∩K} denotes the underlying (𝔥, H ∩ K)-module of τ.

Proof. Since H ∩K ⊂ K, the inclusion V_{K} ⊂ V_{H∩K} is obvious. For each (σ, U_{σ}) ∈ K̂, we write V (K; σ) for the σ-isotypical component of V, that is, V (K; σ) := ∑_{φ} φ(U_{σ}) (⊂ V_{K}) where φ runs over Hom_{K}(σ, V), the space of K homomorphisms U_{σ} → V. Similarly, we write V (H ∩K; τ) (⊂ V_{H∩K}) for the τ-isotypical component of V if τ ∈ { \atop H ∩ K}. Then we have

(1.6.1)  V (H ∩ K; τ) ⊂ ∑_{\begin{matrix}σ∈K̂ \\ [σ_{|H∩K}:τ]≠0 \end{matrix}}^{⊕} V (K; σ),

where [σ_{|H∩K} : τ] := dim Hom_{H∩K}(τ, σ) and the right hand side of (1.6.1) is a Hilbert direct sum. We will show that it is actually a finite sum. Because the restriction π|_{H∩K} is H ∩ K-admissible, we have dim Hom_{H∩K}(τ, V) < ∞ for any τ ∈ { \atop H ∩ K}. By Frobenius reciprocity, we have

dim Hom_{H∩K}(τ, V) = ∑_{σ∈K̂}[σ_{|H∩K} : τ] dim Hom_{K}(σ, V),

which implies that there are only finitely many σ ∈ K̂ satisfying both [σ_{|H∩K} : τ] ≠ 0 and V (K; σ) ≠ 0. Therefore the right side of (1.6.1) is in fact a finite sum. Hence V (H ∩ K; τ) ⊂ V_{K} for any τ ∈ { \atop H ∩ K}, which means V_{H∩K} ⊂ V_{K}. Thus we have proved V_{H∩K} = V_{K}.

Because (π, V) is H ∩ K-admissible, it is also H-admissible (see [Ko2], Theorem 1.2). Namely, we have a Hilbert direct sum decomposition

(1.6.2)  π ≃ ∑_{τ∈Ĥ}^{⊕}m(τ)τ,

where the multiplicity m(τ) < ∞ for any τ ∈ Ĥ. Take (τ, W) ∈ Ĥ such that m(τ) > 0. Then V contains a closed subspace, say U, which is isomorphic to the direct sum of m(τ) copies of W. Then we have

W_{H∩K} ⊕ ⋯ ⊕ W_{H∩K} ≃ U_{H∩K} ⊂ V_{H∩K} = V_{K}.

This shows that

(1.6.3)  m(τ) ≤ dim Hom_{𝔥,H∩K}(τ_{H∩K}, V_{K}).

In particular, V_{K} is discretely decomposable as an (𝔥, H ∩ K)-module from Lemma 1.4. The opposite inequality of (1.6.3) is obvious. Hence we have proved the second statement. □

\underline{1.7.} Here are some examples of discretely decomposable restrictions due to Proposition 1.6:

Example 1.7. A (𝔤, K)-module X of finite length is discretely decomposable as an (𝔥, H ∩ K)-module if one of the following assumptions is satisfied:

1) H is compact.

2) X is a highest weight (𝔤, K)-module and H contains the center of K.

3) X = A_{𝔮}(λ) and (G, G^{σ}) is a symmetric pair with ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1}(𝔱_{0}^{−σ})^{∗} = 0.

4) X appears in a subquotient of the coherent family through a (𝔤, K)-module Y, where Y is discretely decomposable as an (𝔥, H ∩ K)-module.

(1) is obvious, (2) is obtained in [Ma] and also in [L], Theorem 4.2 (see also [Ko2], Corollary 4.4 for a generalization), and (3) is in [Ko2], Theorem 3.2 (see Theorem 4.2 for the notation and the opposite implication). (4) follows from Lemma 1.2 (see [V1], Definition 7.2.5 for the definition of the coherent continuation).

§2. Associated varieties of U(𝔤)-modules.

In this section we make a quick review on known results on associated varieties of U(𝔤)-modules.

\underline{2.1.} If V is a finite dimensional complex vector space, we use the following notation: V^{∗} : the dual vector space of V over ℂ, S(V) : the symmetric algebra of V ≃ the polynomial algebra on V^{∗}, S^{k}(V): the subspace of S(V) of homogeneous elements of degree k, S_{k}(V): = ⨁_{j=0}^{k} S^{j}(V).

Let M = ⨁_{k=0}^{∞} M_{k} be a finitely generated S(V)-module. We say M is a graded S(V)-module if S^{i}(V)M_{j} ⊂ M_{i+j} (i, j ≥ 0). We define a closed cone in V^{∗} by

Supp_{S(V)}(M) := {λ ∈ V^{∗} : f(λ) = 0 for any f ∈ Ann_{S(V)}(M)},

where Ann_{S(V)}(M) := {f ∈ S(V) : f · m = 0 for any m ∈ M}.

\underline{2.2.} Let 𝔤 be a finite dimensional Lie algebra over ℂ. For each integer n ≥ 0, let U_{n}(𝔤) denote by the subspace spanned by elements of the form Y_{1} ⋯ Y_{k} with Y_{1}, …, Y_{k} ∈ 𝔤 and k ≤ n. We note that U_{0}(𝔤) = ℂ. It is convenient to put U_{−1}(𝔤) = 0. Then U(𝔤) is a filtered algebra in the sense that

U(𝔤) = ⋃_{k=1}^{∞} U_{k}(𝔤), U_{i}(𝔤)U_{j}(𝔤) ⊂ U_{i+j}(𝔤).

The associated graded algebra gr U(𝔤) := ⨁_{k=0}^{∞} U_{k}(𝔤)/U_{k−1}(𝔤) is isomorphic to the symmetric algebra S(𝔤) = ⨁_{k=0}^{∞} S^{k}(𝔤) of 𝔤, by the Poincaré-Birkhoff-Witt theorem.

\underline{2.3.} Suppose X is a finitely generated U(𝔤)-module. We take a finite dimensional subspace X_{0} which generates X as a U(𝔤)-module. We put X_{k} := U_{k}(𝔤)X_{0} (k ∈ ℕ). It is convenient to put X_{−1} := {0}. Then we have an increasing filtration {X_{k}}_{k} such that

X = ⋃_{k=0}^{∞} X_{k}, U_{i}(𝔤)X_{j} = X_{i+j} (i, j ≥ 0).

Therefore, if we put gr X := ⨁_{k=0}^{∞} \overline{X_{k}} with \overline{X_{k}} := X_{k}/X_{k−1}, then gr X is a finitely generated gr U(𝔤) ≃ S(𝔤)-module. Define the variety V(X) by

V(X) ≡ V_{𝔤}(X) = Supp_{S(𝔤)}(gr X) ⊂ 𝔤^{∗}.

Then V_{𝔤}(X) is independent of the choice of the generating subspace X_{0} and is called the associated variety of the U(𝔤)-module X.

\underline{2.4.} The following lemma is standard (e.g. [BB], Lemma 4.1 for (2)):

Lemma 2.4. Let L, M and N be 𝔤-modules of finite length.

1) If 0 → L → M → N → 0 is an exact sequence of 𝔤-modules, then V(M) = V(N) ∪ V(L).

2) V(M ⊗ F) = V(M) for any finite dimensional 𝔤-module F.

3) V(M) = {0} if and only if M is finite dimensional.

\underline{2.5.} From now on, let G be a real reductive linear Lie group, K a maximal compact subgroup of G and 𝔤_{0} = 𝔨_{0} + 𝔭_{0} the Cartan decomposition as in §1.2. We define the nilpotent cone N^{∗} by

N^{∗} ≡ N_{𝔤}^{∗} := {λ ∈ 𝔤^{∗} : f(λ) = 0, for all f ∈ S^{+}(𝔤)^{G}}.

Here S^{+}(𝔤) := ⨁_{k=1}^{∞} S^{k}(𝔤) is the maximal ideal of S(𝔤), and S^{+}(𝔤)^{G} is the ring of the G-invariant elements. Then we have

Fact 2.5 (see [V5], Corollary 5.4). If X is a 𝔤-module of finite length, then the associated variety V_{𝔤}(X) is contained in N_{𝔤}^{∗}.

\underline{2.6.} Given an element X ∈ \sqrt{−1}𝔨_{0}, we define 𝔲 ≡ 𝔲(X), 𝔩 ≡ 𝔩(X) and 𝔲^{−} ≡ 𝔲^{−}(X) to be the sum of eigenspaces with positive, 0 and negative eigenvalues of ad(X) ∈ End_{ℂ}(𝔤), respectively. Then 𝔮 := 𝔩 + 𝔲 is said to be a θ-stable parabolic subalgebra of 𝔤. We note that 𝔩 is the complexified Lie algebra of L := Z_{G}(X). The elliptic orbit Ad(G)X ≃ G/L carries a G-invariant complex structure, with the canonical line bundle Ω := ∧^{top}T^{∗}(G/L) ≃ G×_{L}ℂ_{2ρ(𝔲)}. Here, 2ρ(𝔲) := det (Ad_{|𝔲}) is a character of L written in an additive way. Let S := dim_{ℂ}(𝔲 ∩ 𝔨). Given a character ℂ_{λ} of L, we write A_{𝔮}(λ) ≡ A_{𝔮}^{G}(λ) for the underlying (𝔤, K)-module of the Dolbeault cohomology group H^{S}(G/L, Ω⊗ℂ_{λ}) with coefficients in the associated holomorphic line bundle Ω ⊗ (G×_{L}ℂ_{λ}) ≃ G×_{L}ℂ_{λ+2ρ(𝔲)} (cf. [V1], Chapter 6; [Wo]). We note that A_{𝔮}(λ) ≃ R_{𝔮}^{S}(ℂ_{λ+ρ(𝔲)}) with the notation in [V3], Definition 6.20 (cf. [VZ], §5). We take a fundamental Cartan subalgebra 𝔥_{0}^{c} (⊂ 𝔩_{0}). Then 𝔥_{0}^{c} contains the center 𝔷_{0} of 𝔩_{0} and 𝔱_{0}^{c} : = 𝔥_{0}^{c} ∩ 𝔨_{0} is a Cartan subalgebra of 𝔨_{0}. A_{𝔮}(λ) has the Z(𝔤)- infinitesimal character γ := λ + ρ ∈ (𝔥^{c})^{∗} in the Harish-Chandra parametrization, where ρ := ρ(𝔲) + ρ_{𝔩} and ρ_{l} is half the sum of positive roots of 𝔩. Following [V4], Definition 2.5, we say λ is in the good range if

(2.6.1)(a)  Re⟨γ, α⟩ > 0 for any α ∈ Δ(𝔲, 𝔥^{c}),

and in the fair range if

(2.6.1)(b)  Re⟨γ_{|𝔷}, α⟩ > 0 for any α ∈ Δ(𝔲, 𝔥^{c}),

which is implied by (2.6.1)(a). It is weakly good (respectively, weakly fair) if the weak inequalities hold.

We recall some important results on A_{𝔮}(λ). See [V2] and [Wa] for the first statement; and Part III of [BB], Corollary 1.9 and Proposition 2.8; [HMSW] and [V4], Proposition 6.8 for the second. Let us identify 𝔤 and 𝔤^{∗} by a non-degenerate Ad(G)-invariant symmetric bilinear form on 𝔤.

Fact 2.6. Retain the notation as above.

1) If ℂ_{λ} is infinitesimally unitary and if λ is in the weakly fair range, then the (𝔤, K)-module A_{𝔮}(λ) is infinitesimally unitary.

2) If λ is in the good range, then A_{𝔮}(λ) is non-zero and irreducible. The associated variety is given by V_{𝔤}(A_{𝔮}(λ)) = Ad(K_{ℂ})(𝔲^{−} ∩ 𝔭).

We shall write \overline{A_{𝔮}(λ)} for the unitary representation of G obtained as a Hilbert completion of A_{𝔮}(λ) with respect to a pre-Hilbert structure in Fact 2.6(1).

\underline{2.7.} The Zuckerman module A_{𝔮}(λ) may vanish and may be reducible in the weakly fair range of parameters. Nevertheless, the associated variety of A_{𝔮}(λ) does not change as long as A_{𝔮}(λ) is non-zero. The usage of the adjointness of the translation functor was suggested by David Vogan, to whom the author is grateful.

Lemma 2.7. If λ is in the weakly fair range and if A_{𝔮}(λ) is non-zero, then we have V_{𝔤}(A_{𝔮}(λ)) = Ad(K_{ℂ})(𝔲^{−} ∩ 𝔭).

Proof. It follows from [V4], Proposition 4.7 that there exists a character ℂ_{μ} of L in the good range such that the translation functor ψ_{μ+ρ}^{λ+ρ} sends Z := A_{𝔮}(μ) to X := A_{𝔮}(λ). Here we define the Jantzen-Zuckerman translation functor by ψ_{μ+ρ}^{λ+ρ}(Z) = P_{λ+ρ}(F_{λ−μ} ⊗ P_{μ+ρ}(Z)), where P_{ξ} denotes the projection to (𝔤, K)- modules with the generalized infinitesimal character ξ ∈ (𝔥^{c})^{∗} and F_{λ−μ} denotes the finite dimensional representation of G with extremal weight λ − μ. In particular, we have a surjective (𝔤, K)-homomorphism F_{μ−ρ} ⊗ Z to X, and therefore

V_{𝔤}(X) ⊂ V_{𝔤}(F_{λ−μ} ⊗ Z) = V_{𝔤}(Z)

by Lemma 2.4. On the other hand, by the adjointness of the translation functor (e.g. [V1], Proposition 4.5.8) we have

Hom_{𝔤,K}(X, X) ≃ Hom_{𝔤,K}(X, ψ_{μ+ρ}^{λ+ρ}(Z)) ≃ Hom_{𝔤,K}(ψ_{λ+ρ}^{μ+ρ}(X), Z).

If X is non-zero, then the left side contains the identity map. Hence there exists a non-zero (𝔤, K)-homomorphism φ: ψ_{λ+ρ}^{μ+ρ}(X) → Z. Because Z is irreducible, φ is surjective. Using Lemma 2.4 again, we have

V_{𝔤}(Z) ⊂ V_{𝔤}(ψ_{λ+ρ}^{μ+ρ}(X)) ⊂ V_{𝔤}(F_{λ−μ}^{∗} ⊗ X) = V_{𝔤}(X).

Hence, we have V_{𝔤}(X) = V_{𝔤}(Z) which coincides with Ad(K_{ℂ})(𝔲^{−} ∩ 𝔭) by Fact 2.6 (2). □

§3. Theorems in the general case.

For discretely decomposable modules with respect to subalgebras, the algebraic approach turns out to be a powerful tool for the study of the restriction. In this section, we give a basic estimate on the associated variety of a 𝔤-module when restricted to a reductive subalgebra 𝔥. The results here will be a main tool for the study of the restriction with respect to reductive symmetric pairs in §4, §5 and §6.

\underline{3.1.} Suppose that 𝔤 is a complex reductive Lie algebra and 𝔥 is a subalgebra which is reductive in 𝔤. Write the projection pr_{𝔤→𝔥}: 𝔤^{∗} → 𝔥^{∗} dual to the inclusion of complexified Lie algebras 𝔥 ↪ 𝔤.

Theorem 3.1. Suppose X is an irreducible 𝔤-module and Y is an irreducible 𝔥module. If Hom_{𝔥}(Y, X) ≠ 0 then the associated varieties V_{𝔤}(X) ⊂ 𝔤^{∗} and V_{𝔥}(Y) ⊂ 𝔥^{∗} satisfy the following relation:

pr_{𝔤→𝔥} (V_{𝔤}(X)) ⊂ V_{𝔥}(Y).

\underline{3.2.} For the proof of Theorem 3.1, we may regard Y as a submodule of X. Fix a finite dimensional vector subspace F (⊂ Y ⊂ X). Then {Y_{n} := U_{n}(𝔥)F; (n = 0, 1, 2, …)} forms an increasing filtration of Y. We denote by gr Y the graded S(𝔥)-module, and by Ann_{S(𝔥)}(gr Y) ⊂ S(𝔥) the annihilator of gr Y. Similarly, an increasing filtration {X_{n} := U_{n}(𝔤)F; (n = 0, 1, 2, …)}, the graded S(𝔤)-module gr X, and the annihilator Ann_{S(𝔤)}(gr X) ⊂ S(𝔤) are defined. We should remark that the filtration {X_{n}} is different from the one in the proof of Lemma 1.5. In light of the isomorphism of graded rings

(3.2.1)  S(𝔥) ⊗ S(𝔥^{⊥}) →^{∼} S(𝔤),

we have:

Lemma 3.2. Ann_{S(𝔥)}(gr Y) ⊗ S(𝔥^{⊥}) ⊂ Ann_{S(𝔤)}(gr X)

Proof. Let U′(𝔥^{⊥}) = ⋃_{k=0}^{∞} U′_{k}(𝔥^{⊥}) be as in (1.5.1) and (1.5.2). Then the Poincaré-Birkhoff-Witt theorem leads to

U_{n}(𝔤) = ∑_{i+j≤n} U′_{i}(𝔥^{⊥}) U_{j}(𝔥) for n ∈ ℕ.

Thus, we have

(3.2.2)  X_{n} = U_{n}(𝔤) F = ∑_{i+j≤n} U′_{i}(𝔥^{⊥}) U_{j}(𝔥) F = ∑_{i+j≤n} U′_{i}(𝔥^{⊥}) Y_{j}.

We fix homogeneous elements u ∈ S^{a}(𝔥) ∩ Ann_{S(𝔥)}(gr Y) and v ∈ S^{b}(𝔥^{⊥}). Let ũ ∈ U_{a}(𝔥) be the symmetrization of u and let ṽ ∈ U′_{b}(𝔥^{⊥}) be that of v. It follows from u ∈ Ann_{S(𝔥)}(gr Y) that ũY_{j} ⊂ Y_{a+j−1} for any j ∈ ℕ. Then we have

(3.2.3) ũṽ(U′_{i}(𝔥^{⊥})Y_{j}) ⊂ ũ(U′_{i+b}(𝔥^{⊥})Y_{j}) ⊂ U′_{i+b}(𝔥^{⊥})(ũY_{j}) ⊂ U′_{i+b}(𝔥^{⊥})Y_{a+j−1}.

Here the second inclusion follows from the fact that ad(𝔥) stabilizes 𝔥^{⊥}. By (3.2.2) and (3.2.3), we have

ũṽX_{n} ⊂ ∑_{i+j≤n} ũṽU′_{i}(𝔥^{⊥})Y_{j} ⊂ ∑_{i+j≤n} U′_{i+b}(𝔥^{⊥})Y_{a+j−1} ⊂ X_{a+b+n−1}.

Since the natural map U_{a+b}(𝔤) → U_{a+b}(𝔤)/U_{a+b−1}(𝔤) ≃ S^{a+b}(𝔤) sends ũṽ to u ⊗ v, we have

(u ⊗ v) · (X_{n}/X_{n−1}) = (ũṽ · X_{n})/X_{a+b+n−1} = 0

in the graded module gr X for any n ∈ ℕ. Hence we have u ⊗ v ∈ Ann_{S(𝔤)}(gr X), which we wanted to prove. □

\underline{3.3.} Proof of Theorem 3.1. For V = 𝔥 or 𝔤, if f ∈ S(V) then we regard f as a polynomial over V^{∗} and denote the evaluation of f at ν ∈ V^{∗} by ⟨f, ν⟩. Then we want to prove

⟨u, pr_{𝔤→𝔥}(λ)⟩ = 0

for any u ∈ Ann_{S(𝔥)}(gr Y) and for any λ ∈ V_{𝔤}(X) ⊂ 𝔤^{∗}, where u is regarded as a polynomial over 𝔥^{∗}. It follows from Lemma 3.2 that u⊗1 (∈ S(𝔥) ⊗ S(𝔥^{⊥}) ≃ S(𝔤)) annihilates gr X, so we have ⟨u ⊗ 1, λ⟩ = 0, where u ⊗ 1 is regarded as a polynomial over 𝔤^{∗}. Now, Theorem 3.1 follows from the formula ⟨u, pr_{𝔤→𝔥}(λ)⟩ = ⟨u ⊗ 1, λ⟩. □

Theorem 3.1 gives rise to a necessary condition for a 𝔤-module to be discretely decomposable as an 𝔥-module. We recall that N_{𝔥}^{∗} ⊂ 𝔥^{∗} is the nilpotent cone for 𝔥.

Corollary 3.4. Let X be a 𝔤-module of finite length. Assume that X is discretely decomposable as an 𝔥-module. Then

pr_{𝔤→𝔥} (V_{𝔤}(X)) ⊂ N_{𝔥}^{∗}.

Proof. We may assume that X is irreducible as a 𝔤-module by using Lemma 1.2 (1) and Lemma 2.4 (1). We take an irreducible 𝔥-submodule Y of X (see Lemma 1.5). It follows from Theorem 3.1 that pr_{𝔤→𝔥} (V_{𝔤}(X)) ⊂ V_{𝔥}(Y). By Fact 2.5 applied to 𝔥, we have V_{𝔥}(Y) ⊂ N_{𝔥}^{∗}. Hence, Corollary follows. □

Applying Corollary 3.4 to X = A_{𝔮}(λ) and using Proposition 1.6, we have:

Corollary 3.5. Let us identify 𝔤^{∗} with 𝔤 as usual. We assume that a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲 of 𝔤 = 𝔨 + 𝔭 satisfies

pr_{𝔤→𝔥} (Ad(K_{ℂ})(𝔲^{−} ∩ 𝔭)) ⊄ N_{𝔥}^{∗}.

If ℂ_{λ} is in the weakly fair range and if A_{𝔮}(λ) ≠ {0}, then the (𝔤, K)-module A_{𝔮}(λ) is not discretely decomposable as an 𝔥-module. In particular, the restriction of the unitary representation \overline{A_{𝔮}(λ)} to H ∩ K is not H ∩ K-admissible.

Proof. The first assertion follows from Lemma 2.7 and Corollary 3.4. The latter is immediate from Proposition 1.6. □

Remark 3.6. Let X be an irreducible (𝔤, K)-module. If H = K, then X is always discretely decomposable as an (𝔥, H ∩ K)-module (for example, use Lemma 1.5). In this special case, Theorem 3.1 implies a well-known result (see [V5], Corollary 5.13):

(3.6.1)  pr_{𝔤→𝔨} (V_{𝔤}(X)) = {0}, namely, V_{𝔤}(X) ⊂ (𝔤/𝔨)^{∗},

because the associated variety of a finite dimensional representation is zero and because any irreducible representation of K is finite dimensional.

\underline{3.7.} The following Theorem gives a useful information on Ĥ occurring as direct summands of the restriction.

Theorem 3.7. Let X be an irreducible (𝔤, K)-module and let Y_{i} be irreducible (𝔥, H ∩ K)-modules such that Hom_{𝔥,H∩K}(Y_{i}, X) ≠ 0 (i = 1, 2). Then we have

V_{𝔥}(Y_{1}) = V_{𝔥}(Y_{2}).

Proof. Using the notation of the proof of Lemma 1.5 with Y replaced by Y_{1}, we set X_{m} := U′_{m}(𝔥^{⊥})Y_{1} = ∑_{j=0}^{m} Z_{j} for m ∈ ℕ, where Z_{j} is the image of the map of

\underbrace{𝔥^{⊥} ⊗ ⋯ ⊗ 𝔥^{⊥}}_{j} ⊗Y_{1} → X as in (1.5.3). Hence V_{𝔥}(Z_{j}) ⊂ V_{𝔥}(𝔥^{⊥} ⊗ ⋯ ⊗ 𝔥^{⊥} ⊗ Y_{1}) =

V_{𝔥}(Y_{1}) for all j because of Lemma 2.4. Again using Lemma 2.4 (1), we have

(3.7.1)  V_{𝔥}(X_{m}) ⊂ ⋃_{j=0}^{m} V_{𝔥}(Z_{j}) ⊂ V_{𝔥}(Y_{1}) for any m.

Because X = ⋃_{m=0}^{∞} X_{m}, there exists m such that X_{m} ∩ Y_{2} ≠ {0}. Since Y_{2} is an irreducible 𝔥-module, we have Y_{2} ⊂ X_{m}. Using Lemma 2.4 (1) again, we have

(3.7.2)  V_{𝔥}(Y_{2}) ⊂ V_{𝔥}(X_{m}).

By (3.7.1) and (3.7.2), we have V_{𝔥}(Y_{2}) ⊂ V_{𝔥}(Y_{1}). The opposite inclusion V_{𝔥}(Y_{1}) ⊂ V_{𝔥}(Y_{2}) is similar. □

\underline{3.8.} In [Ko1]; [Ko2], §6, we have observed a phenomenon in the restriction formula: It can happen that the unitary representation \overline{A_{𝔮}(λ)} of G is decomposed into the Hilbert direct sum of different series of irreducible unitary representations of H such as

(3.8.1)  \overline{A_{𝔮}^{G}(λ)}_{|H} = ⨁_{j=1}^{m}_{\begin{matrix}∑ & ⊕ \\ ν_{i}^{(j)}∈Λ_{j} \end{matrix}} \overline{(A_{𝔮′_{j}}^{H}(ν_{i}^{(j)}))}

where ℂ_{ν_{i}^{(j)}} is weakly fair with respect to θ-stable parabolic subalgebras 𝔮′_{j} = 𝔩′_{j} +𝔲′_{j}

(1 ≤ j ≤ m) of 𝔥 for any ν_{i}^{(j)} ∈ Λ_{j}. Theorem 3.7 implies that

Ad((H ∩ K)_{ℂ})(𝔲′_{1} ∩ 𝔭) = Ad((H ∩ K)_{ℂ})(𝔲′_{2} ∩ 𝔭) = ⋯ = Ad((H ∩ K)_{ℂ})(𝔲′_{m} ∩ 𝔭).

Example 3.8. Let (G, H) = (SO(4, 4), U(2, 2)). For representations and associated varieties for H = U(2, 2) we shall use the notation of §7.1 and 7.3. If π is a discrete series representation for the symmetric space SO(4, 4)/SO(4, 3), then the underlying (𝔤, K)-module π_{K} is discretely decomposable as an (𝔥, H ∩ K)-module. If the infinitesimal character is sufficiently regular, the the decomposition formula of π|_{U(2,2)} (see [Ko2], Theorem 6.1) contains “different series” of representations A_{𝔮(Y_{2})}(λ), A_{𝔮(X_{3})}(λ′), and A_{𝔮(Y_{5})}(λ′′) with each parameter in the weakly fair range (namely, m = 3 in (3.8.1)). As we shall see in Figure 7.1.2 and Lemma 7.4, these modules have the same associated variety O_{1110}(5).

\underline{3.9.} Theorem 3.7 is mainly intended to the case where dim Y_{1} = ∞. But, we mention here that Theorem 3.7 with dim Y_{1} = 1 implies a weaker form of Moore’s theorem [Mo]. Here, we recall briefly that Moore’s theorem asserts that “a unitary representation π of G satisfying Hom_{H}(1, π) ≠ 0 must be finite dimensional, if H is a non-compact closed subgroup of a simple Lie group G”.

Corollary 3.9. Retaining the setting of §1.2, we suppose H is semisimple without compact factors. We denote by 1 the trivial representation of H. Let π be an irreducible, infinitesimally unitary (𝔤, K)-module which is H ∩ K-admissible. If Hom_{𝔥,H∩K}(1, π) ≠ 0 then π is finite dimensional.

Proof. By Lemma 1.5 and Lemma 1.3, π is decomposed into an algebraic direct sum of irreducible (𝔥, H ∩ K)-modules:

(3.9.1)  π ≃ ∑_{τ}m(τ)τ.

Here, the multiplicity m(τ) < ∞ for each irreducible (𝔥, H ∩ K)-module τ, because π is H ∩ K-admissible (see Proposition 1.6). If an irreducible (𝔥, H ∩ K)-module τ occurs in (3.9.1), we have V_{𝔥}(τ) = V_{𝔥}(1) = {0} by Theorem 3.7. Hence τ is finite dimensional by Lemma 2.4 (3). Because there are only finitely many equivalence classes of finite dimensional unitary representations of H, π must be finite dimensional by (3.9.1).

§4. Restriction with respect to a symmetric pair

\underline{4.1.} Let σ be an involutive automorphism of G, and H an open subgroup of the fixed point subgroup G^{σ} := {g ∈ G : σg = g}. Then (G, H) is called a reductive symmetric pair. There exists a Cartan involution θ of G which commutes with σ. Let K = G^{θ} be a maximal compact subgroup of G and 𝔤_{0} = 𝔨_{0} + 𝔭_{0} the Cartan decomposition. The differential of σ or its complexification will be also denoted by σ. We set 𝔤_{0}^{−σ} := {X ∈ 𝔤_{0} : σX = −X} and 𝔨_{0}^{−σ} := 𝔨_{0} ∩ 𝔤_{0}^{−σ}. We fix a Cartan subalgebra 𝔱_{0}^{c} of 𝔨_{0} such that 𝔱_{0}^{−σ} := 𝔱_{0}^{c} ∩ 𝔨_{0}^{−σ} is a maximal abelian subspace of 𝔨_{0}^{−σ}. We fix compatible positive systems Δ^{+}(𝔨, 𝔱^{c}) and Σ^{+}(𝔨, 𝔱^{−σ}), namely, {α_{|𝔱^{−σ}} : α ∈ Δ^{+}(𝔨, 𝔱^{c})} \ {0} equals Σ^{+}(𝔨, 𝔱^{−σ}). Without loss of generality, we may assume that a θ-stable parabolic subalgebra 𝔮 ≡ 𝔮(X) is defined by a dominant element X ∈ \sqrt{−1}𝔱_{0}^{c} with respect to Δ^{+}(𝔨, 𝔱^{c}) (see §2.6). Define a closed cone in \sqrt{−1}(𝔱_{0}^{c})^{∗} by

(4.1.1)  ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ := { ∑_{β∈Δ(𝔲∩𝔭,𝔱^{c})} n_{β} β : n_{β} ≥ 0}.

Theorem 4.2. Suppose that (G, H) is a reductive symmetric pair and that 𝔮 = 𝔩 + 𝔲 is a θ-stable parabolic subalgebra defined by a dominant element of \sqrt{−1}(𝔱_{0}^{c})^{∗}. In what follows, ℂ_{λ} is a unitary character of L and \overline{A_{𝔮}(λ)} is a unitary representation of G obtained by the Hilbert completion of a Vogan-Zuckerman (𝔤, K)-module A_{𝔮}(λ) (see §2.6). Then the following eight conditions on (G, H, 𝔮) are equivalent:

1) \overline{A_{𝔮}(λ)}|_{H∩K} is H ∩ K-admissible for any ℂ_{λ} in the weakly fair range.

2) \overline{A_{𝔮}(λ)}|_{H∩K} is H ∩ K-admissible for some ℂ_{λ} in the weakly fair range such that A_{𝔮}(λ) ≠ {0}.

3) A_{𝔮}(λ) is discretely decomposable as an (𝔥, H ∩ K)-module for any ℂ_{λ} in the weakly fair range.

4) A_{𝔮}(λ) is discretely decomposable as an (𝔥, H ∩ K)-module for some ℂ_{λ} in the weakly fair range such that A_{𝔮}(λ) ≠ {0}.

5) pr_{𝔤→𝔥}(𝔲 ∩ 𝔭) ⊂ N_{𝔥}^{∗}.

6) σ𝔲 ∩ 𝔲^{−} ⊂ 𝔨.

7) σ (Δ(𝔲 ∩ 𝔭, 𝔱^{c})) ∩ Δ(𝔲^{−} ∩ 𝔭, 𝔱^{c}) = ∅.

8) ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1}(𝔱_{0}^{−σ})^{∗} = 0.

Here, we have identified 𝔤 with 𝔤^{∗} in (5) and have regarded (𝔱_{0}^{−σ})^{∗} as a subspace of 𝔱_{0}^{∗} by using a non-degenerate symmetric Ad(G)-invariant bilinear form on 𝔤.

It follows from [Ko2], Corollary 1.3 that the above equivalent eight conditions imply that the unitary representation \overline{A_{𝔮}(λ)} of G decomposes discretely as a representation of H:

\overline{A_{𝔮}(λ)}|_{H} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈Ĥ \end{matrix}}m(τ)τ, m(τ) < ∞.

Examples of the multiplicity free decomposition, namely, m(τ) ≤ 1 for any τ ∈ Ĥ, are given in [JV], §4; [Ko1], Theorem 3.4, Theorem 3.5 and Theorem 3.6; [Ko2], Theorem 6.1, Theorem 6.4; and [GW], §6.

\underline{4.3.} Each K-type occurring in any irreducible unitary representation of G is of finite multiplicity. On the other hand, it often occurs that the multiplicity of the continuous spectrum in the irreducible decomposition of π|_{H} is infinite where (G, H) is a symmetric pair and π ∈ Ĝ. The following Corollary gives the finite multiplicity theorem for the discrete part on the level of Harish-Chandra modules, generalizing the case with H = K.

Corollary 4.3. Suppose (G, H) is a reductive symmetric pair. Then

dim Hom_{𝔥,H∩K} (Y, A_{𝔮}(λ)) < ∞

for any irreducible (𝔥, H ∩ K)-module Y and for any Zuckerman’s derived functor module A_{𝔮}(λ) with ℂ_{λ} a unitary character of L in the weakly fair range.

Proof of Corollary 4.3. If Hom_{𝔥,H∩K} (Y, A_{𝔮}(λ)) = {0} for any irreducible (𝔥, H ∩ K)-module Y, then there is nothing to prove. If Hom_{𝔥,H∩K} (Y, A_{𝔮}(λ)) ≠ {0} for some irreducible (𝔥, H ∩K)-module Y, then A_{𝔮}(λ) is discretely decomposable as an (𝔥, H∩K)-module by Lemma 1.5. By Theorem 4.2, the restriction \overline{A_{𝔮}(λ)}|_{H∩K} is H∩ K-admissible. By Theorem 1.2 of [Ko2], the restriction \overline{A_{𝔮}(λ)}|_{H} is H-admissible; namely, the multiplicity of each irreducible representation of H occurring in the irreducible decomposition of the restriction \overline{A_{𝔮}(λ)}|_{H} is finite. □

\underline{4.4.} Strategy of Proof of Theorem 4.2. The implication (8) ⇒ (1) is proved in [Ko2], Theorem 3.2. The implications (1) ⇒ (2) and (3) ⇒ (4) are obvious. The implications (1) ⇒ (3) and (2) ⇒ (4) are proved in Proposition 1.6. The implication (4) ⇒ (5) follows from Corollary 3.5 because the condition (4) leads to

pr_{𝔤→𝔥}(𝔲^{−} ∩ 𝔭) ⊂ pr_{𝔤→𝔥} (Ad(K_{ℂ})(𝔲^{−} ∩ 𝔭)) ⊂ N_{𝔥}^{∗}.

Here, we note that pr_{𝔤→𝔥}(𝔲^{−} ∩ 𝔭) ⊂ N_{𝔥}^{∗} if and only if pr_{𝔤→𝔥}(𝔲 ∩ 𝔭) ⊂ N_{𝔥}^{∗}.

For the rest of this section, we shall give a proof of the implications (5) ⇒ (6) ⇒ (7) ⇒ (8).

\underline{4.5.} (5) ⇒ (6): For W ∈ 𝔤 = 𝔤_{0} ⊗ ℂ, we denote by W ↦ \overline{W} the complex conjugation with respect to 𝔤_{0}. Let X ∈ (𝔲 ∩ 𝔭) ∩ σ(𝔲^{−} ∩ 𝔭). Assuming (5), we will show X = 0. We first note that \overline{X} ∈ 𝔲^{−} ∩ 𝔭, σX ∈ 𝔲^{−} ∩ 𝔭 and σ\overline{X} = \overline{σX} ∈ 𝔲 ∩ 𝔭. We put Y := X + σ\overline{X} ∈ 𝔲 ∩ 𝔭. Then Y + σY = (X + \overline{X}) + σ(X + \overline{X}) ∈ 𝔭 ∩ 𝔤_{0} = 𝔭_{0}. By using the assumption (5), we have

Y + σY ∈ pr_{𝔤→𝔥}(𝔲 ∩ 𝔭) ∩ 𝔭_{0} ⊂ N_{𝔥}^{∗} ∩ 𝔭_{0}.

Here, we note that pr_{𝔤→𝔥}(Y) = \frac{1}{2}(Y + σY) under the identification of 𝔤 with 𝔤^{∗} and of 𝔥 with 𝔥^{∗}. Because any element of 𝔭_{0} is semisimple, N_{𝔥}^{∗} ∩ 𝔭_{0} = {0}. Since Y ∈ 𝔲 and σY ∈ 𝔲^{−}, Y + σY = 0 implies Y = 0. Similarly, if we put Z := X − σ\overline{X}, then Z + σZ = (X − \overline{X}) + σ(X − \overline{X}) ∈ \sqrt{−1}𝔭_{0} is also a semisimple element. Hence we have Z = −σZ ∈ 𝔲 ∩ 𝔲^{−} = \{0\}. Therefore X = \frac{1}{2}(Y + Z) = 0. Hence we have proved the implication (5) ⇒ (6).

\underline{4.6.} (6) ⇒ (7): This is easy. In fact, if there is an element α ∈ σ (Δ(𝔲 ∩ 𝔭, 𝔱^{c})) ∩ Δ(𝔲^{−} ∩ 𝔭, 𝔱^{c}), then we take a non-zero root vector X_{α}. Then X_{α} ∈ 𝔲^{−} ∩ 𝔭 and σX_{α} ∈ 𝔲 ∩ 𝔭. Therefore, σ(𝔲 ∩ 𝔭) ∩ (𝔲^{−} ∩ 𝔭) ≠ {0}. Thus we have proved (6) ⇒ (7).

\underline{4.7.} (7) ⇒ (8): Suppose A ∈ \sqrt{−1}𝔱_{0}^{c} defines the parabolic subalgebra 𝔮 = 𝔩 + 𝔲. This means that α(A) > 0 for any α ∈ Δ(𝔲, 𝔱^{c}) and that α(A) = 0 for any α ∈ Δ(𝔩, 𝔱^{c}). We assume (7), equivalently,

(4.7.1)  σ (Δ(𝔲 ∩ 𝔭, 𝔱^{c})) ⊂ Δ(𝔲 ∩ 𝔭, 𝔱^{c}) ∪ Δ(𝔩 ∩ 𝔭, 𝔱^{c}).

Let β := ∑_{α∈Δ(𝔲∩𝔭,𝔱^{c})} a_{α}α ∈ ℝ_{+}⟨𝔲 ∩ 𝔭⟩ with a_{α} ≥ 0 for all α. If β belongs to \sqrt{−1}(𝔱_{0}^{−σ})^{∗}, then we have σβ + β = 0. By (4.7.1), we have (σβ)(A) ≥ 0. Therefore β(A) ≤ 0, which occurs only if a_{α} = 0 for all α, namely, β = 0. Therefore, we have proved (7) ⇒ (8).

This completes the proof of Theorem 4.2.

§5. Restriction of holomorphic discrete series representations

\underline{5.1.} In this section, we assume that G is a simple linear Lie group such that G/K is Hermitian, namely, the center 𝔠(𝔨_{0}) of 𝔨_{0} is not trivial. It is known that 𝔠(𝔨_{0}) is one dimensional and that there exists Z ∈ 𝔠(𝔨_{0}) so that 𝔤 = 𝔨 ⊕ 𝔭^{+} ⊕ 𝔭^{−} are 0, \sqrt{−1} and −\sqrt{−1} eigenspaces of ad(Z). A (𝔤, K)-module is said to be a highest weight module (resp. lowest weight module) if there exists a non-zero vector annihilated by 𝔭^{+} (resp. 𝔭^{−}). It will be convenient to allow ‘ highest weight module’ to refer also to an irreducible unitary representation of G whose underlying (𝔤, K)- module is a highest weight module. A discrete series representation for G is said to be a holomorphic discrete series representation (resp. anti-holomorphic discrete representation) if it is a highest weight module (resp. a lowest weight module).

\underline{5.2.} Suppose σ is an involutive automorphism of G commuting with the Cartan involution θ. Since σ𝔠(𝔨_{0}) = 𝔠(𝔨_{0}), there are two exclusive possibilities:

(5.2.1)  σZ = Z,
(5.2.2)  σZ = −Z.

In this section, we suppose H = G^{σ}, the subgroup of G consisting of fixed points by σ.

The geometric meaning of (5.2.1) and (5.2.2) is the following: H/H ∩ K ⊂ G/K is a complex submanifold in the case (5.2.1) and is a totally real submanifold in the case (5.2.2)

Theorem 5.3. Suppose G is a non-compact simple Lie group. Let π be a holomorphic discrete series representation of G. If σ satisfies (5.2.2), then the underlying (𝔤, K)-module π_{K} is not discretely decomposable as an (𝔥, H ∩ K)-module. In particular, the restriction π|_{H∩K} is not H ∩ K-admissible.

\underline{5.4.} The above theorem is in sharp contrast with the following fact (cf. [Ma], [JV], [Ko4]):

Fact 5.4. Let π ∈ Ĝ be an irreducible unitary highest weight module. If σ satisfies (5.2.1), then the restriction π|_{H∩K} is H ∩ K-admissible. In particular, the underlying (𝔤, K)-module π_{K} is discretely decomposable as an (𝔥, H ∩ K)-module.

\underline{5.5.} Proof of Theorem 5.3. Any holomorphic discrete series representation π is of the form \overline{A_{𝔮}(λ)} such that ℂ_{λ} is in the good range with respect to a θ-stable parabolic subalgebra (actually, Borel subalgebra) 𝔮 = 𝔩 + 𝔲 with 𝔲 ∩ 𝔭 = 𝔭_{−}. On the other hand, the assumption (5.2.2) implies σ𝔭_{−} = 𝔭_{+}. Therefore, if the condition (6) of Theorem 4.2 holds, then we have {0} = σ𝔭_{−} ∩𝔭_{+}. Hence 𝔭_{+} = {0}. Then 𝔭 = 𝔭_{−} + 𝔭_{+} = σ𝔭_{+} + 𝔭_{+} = {0}, which implies that G is compact, yielding a contradiction. Therefore, the condition (6) of Theorem 4.2 fails. Thus, the underlying (𝔤, K)- module of π is not discretely decomposable as an (𝔥, H ∩ K)-module. □

Remark 5.6. In view of Fact 5.4, one might expect that no unitary highest weight module of infinite dimension is discretely decomposable as an (𝔥, H ∩ K)-module if H = G^{σ} with σ satisfying (5.2.2). However, this is not always the case. For instance, with the notation of §7.3 where (G, H) = (U(2, 2), Sp(1, 1)), A_{𝔮(Z_{i})} (1 ≤ i ≤ 4) are unitary highest weight modules due to [A2], while they are discretely decomposable as an (𝔥, H ∩ K)-modules (see §7).

§6. Discrete series for semisimple symmetric spaces and restrictions.

In this section we give another application of our main theorem in the case of discrete series representations for semisimple symmetric spaces G/H and mention some related topics in our context.

\underline{6.1.} Let (G, H) be a reductive symmetric pair defined by an involution σ of G. We shall use the notation in §4.1. There is a G-invariant measure on a reductive symmetric space G/H. Then we have a natural unitary representation (a regular representation) on the Hilbert space of square integrable functions L^{2}(G/H). An irreducible unitary representation π is said to be a discrete series representation for G/H if there is a non-zero unitary G-homomorphism from π into L^{2}(G/H). We denote by Disc(G/H) ⊂ Ĝ the set of discrete series representations for G/H.

\underline{6.2.} Here is a main theorem in this section.

Theorem 6.2. Suppose that G/H is a non-compact reductive symmetric space. Then the underlying (𝔤, K)-module of any discrete series representation for G/H is not discretely decomposable as an (𝔥, H ∩ K)-module. In particular, no discrete series representation for G/H is H ∩ K-admissible.

\underline{6.3.} With the notation in §4.1, we review the following result due to Flensted-Jensen, Matsuki-Oshima and Vogan.

Fact 6.3 ([FJ1]; [MO]; [FJ2], Ch.VIII, §2; [V4], §4). Let G/H be a reductive symmetric space. Then Disc(G/H) ≠ ∅ if and only if rankG/H = rankK/H ∩ K. Furthermore, if the rank condition is satisfied, then any discrete series π ∈ Disc(G/H) is of the form \overline{A_{𝔮}(λ)}, where 𝔮 = 𝔩 + 𝔲 is defined by a generic element in \sqrt{−1}𝔱_{0}^{−σ} and ℂ_{λ} is a unitary character of L = Z_{G}(𝔱_{0}^{−σ}) in the fair range satisfying some integral conditions determined by (G, H).

\underline{6.4.} Before proving Theorem 6.2, we prepare the following lemma:

Lemma 6.4. Retain the notation in Fact 6.3. Suppose that G/H is a reductive symmetric space with rankG/H = rankK/H ∩ K. If 𝔭 ⊂ 𝔩, then G/H is compact.

Proof. The rank condition implies that 𝔱^{−σ} is a maximal abelian subspace in 𝔤^{−σ}, and so we have 𝔩 ∩ 𝔤^{−σ} = 𝔱^{−σ}. Assume 𝔭 ⊂ 𝔩. Then 𝔭 ∩ 𝔤^{−σ} ⊂ 𝔩 ∩ 𝔤^{−σ} = 𝔱^{−σ} ⊂ 𝔨, which implies 𝔭 ∩ 𝔤^{−σ} = {0}. Hence we have 𝔭_{0} ∩ 𝔤_{0}^{−σ} = {0}. Since G/H is diffeomorphic to a vector bundle over a compact manifold K/H ∩ K with typical fiber 𝔭_{0} ∩ 𝔤_{0}^{−σ}, it follows that G/H is compact. □

\underline{6.5.} Proof of Theorem 6.2. Suppose there exists π ∈ Disc(G/H) such that the underlying (𝔤, K)-module π_{K} is discretely decomposable as an (𝔥, H ∩ K)-module. Then π_{K} is of the form A_{𝔮}(λ), with 𝔮 = 𝔩 + 𝔲 is defined by a generic element of \sqrt{−1}𝔱_{0}^{−σ} as in Fact 6.3. In particular, we have σ𝔲 = 𝔲^{−}. Then the equivalent condition (6) in Theorem 4.2 yields 𝔲∩𝔭 = {0}. In view of the direct sum decomposition

𝔭 = (𝔲 ∩ 𝔭) + (𝔩 ∩ 𝔭) + σ(𝔲 ∩ 𝔭)

we have 𝔭 = 𝔩 ∩ 𝔭, that is, 𝔭 ⊂ 𝔩. By Lemma 6.4, G/H must be compact, which contradicts to our assumption. Hence we have proved Theorem. □

Remark 6.6. Related topics are in order:

1) Let K be a maximal compact subgroup of G as usual. Any irreducible unitary representation is K-admissible (Harish-Chandra). Therefore, Theorem 6.2 in the special case where H = K is equivalent to the well-known fact that there is no discrete series representation for the non-compact Riemannian symmetric space G/K. This example does not give a new proof of this classical result, but rather a different perspective.

2) There are a number of examples of π ∈ Disc(G/H) such that π|_{L} is L ∩ Kadmissible where σ and τ are involutive automorphisms of G and H = G^{σ} and L = G^{τ} (see [Ko2], §5 and [Ko4] for the application to L^{p}-analysis). For instance, this is the case if σ satisfies (5.2.2) and if τ satisfies (5.2.1) in the setting of §5.2. On the other hand, Theorem 6.2 asserts that such examples can be found only if L ≠ H, namely, σ ≠ τ in the non-compact case.

3) Suppose Γ is a torsion free discrete subgroup of G such that both Γ\G and (H ∩Γ)\H are compact. Then the image of the natural map (H ∩Γ)\H/(H ∩K) → Γ\G/K defines a cycle which is called a modular symbol. The Poincaré dual of modular symbols are studied in the context of the Matsushima–Murakami formula ([MM], see also [BW]): H_{de Rham}^{j}(Γ\G/K; ℂ) ≃ ⊕_{π∈Ĝ} Hom_{G}(π, L^{2}(Γ\G)) ⊗ H^{j}(𝔤, K; π_{K}). Y. Tong and S. Wang proved the non-vanishing theorem of the modular symbols for the component of π ∈ Disc(G/H) ([TW]), while T. Oda and the author recently proved the vanishing theorem for the component of π ∈ Ĝ whose restriction to H is H-admissible. Thus, one could expect that no π ∈ Ĝ satisfies both assumptions. This was a motivation of Theorem 6.2. We note that it requires some elaboration to deduce Theorem 6.2 from [TW] and [KO] because the proof of the non-vanishing theorem of [TW] is not valid for the constant sheaf, whose vanishing theorem is given in [KO].

4) Let us consider the group manifold case G/H ≃ G′ × G′/ diag(G′). Then any discrete series representation for G/H is of the form π ⊠ π^{∗} where π ∈ Disc(G′) and π^{∗} is the dual of π. Its restriction to H = diag(G′) is nothing but the tensor product π ˆ⊗π^{∗}. Theorem 6.2 means that π_{K} ⊗π_{K}^{∗} is not discretely decomposable as a (𝔤, K)-module. This fact was known more explicitly in the special setting where π is a holomorphic discrete series representation. That is, the tensor product of holomorphic and anti-holomorphic discrete series representations is unitarily equivalent to the quasi-regular representation on the space of L^{2}-sections of a certain vector bundle over a Riemannian symmetric space by a result of Repka [R] (see also [Ko3], §6.2 for some more references), of which the irreducible decomposition involves continuous spectra and at most finitely many discrete series representations.

5) Generalizing the result of Repka, Ólafsson and Ørsted announced the irreducible decomposition of the restriction π|_{H} under the assumption that G/H satisfies (5.2.2) and that π is a holomorphic discrete series representation for G/H having a one-dimensional minimal K-type [OO]. Only continuous spectra appear in the irreducible decomposition of π|_{H} in this case. We note that the decomposition formula for π|_{H} is not known for a non-holomorphic discrete series representation π in general.

§7. Example 1 : the restriction from U(2, 2) to Sp(1, 1).

In this section, we illustrate our results by an example of a symmetric pair (G, H) = (U(2, 2), Sp(1, 1)) ≈ (SO(4, 2), SO(4, 1)). We shall give a classification of all irreducible unitary representations of G with regular integral infinitesimal character, whose underlying (𝔤, K)-modules are discretely decomposable as an (𝔥, H ∩ K)-modules (Proposition 7.5). Among 18 series of irreducible infinitesimally unitary (𝔤, K)-modules, 12 will be proved to be discretely decomposable as (𝔥, H ∩ K)-modules. Proposition 7.5 itself follows from Theorem 4.2 immediately by computation of the root system, however, we also present some elementary computations of associated varieties to illustrate the ingredients of §3 (cf. Example 3.8 and Remark 5.6).

\underline{7.1.} Let G = U(2, 2) be the indefinite unitary group realized as \{g ∈ GL(4, ℂ) : tḡI_{2,2}g = I_{2,2}}, where I_{2,2} := diag(1, 1, −1, −1). We take a maximal compact group K = G ∩ U(4) ≃ U(2) × U(2) and then K_{ℂ} ≃ GL(2, ℂ) × GL(2, ℂ). We begin with the description of Ad(K_{ℂ})-orbits on N_{𝔤}^{∗} ∩ (𝔤/𝔨)^{∗}. We identify

(7.1.1)  (𝔤/𝔨)^{∗} ≃ 𝔭 ≃ \{( \begin{matrix}O & A \\ B & O \end{matrix} ) : A, B ∈ M(2, ℂ)\} ≃ M(2, ℂ) ⊕ M(2, ℂ),

on which K_{ℂ} acts by

(A, B) ↦ (g_{1}Ag_{2}^{−1}, g_{2}Bg_{1}^{−1}), (g_{1}, g_{2}) ∈ K_{ℂ} ≃ GL(2, ℂ) × GL(2, ℂ).

Then the nilpotent cone N_{𝔭}^{∗} := N_{𝔤}^{∗} ∩ (𝔤/𝔨)^{∗} is given by

{(A, B) ∈ M(2, ℂ) ⊕ M(2, ℂ) : AB and BA are nilpotent matrices}.

The variety N_{𝔭}^{∗} splits into 10 Ad(K_{ℂ})-orbits:

O_{1211}(6){X_{2}} O_{2111}(6){X_{5}}

O_{1101}(5){X_{4}, Y_{3}, Y_{4}} O_{1110}(5){X_{3}, Y_{2}, Y_{5}}

O_{0200}(4){X_{1}, Y_{1}}  O_{1100}(4){W }  O_{2000}(4){X_{6}, Y_{6}}

O_{0100}(3){Z_{1}, Z_{2}} O_{1000}(3){Z_{3}, Z_{4}}

O_{0000}(0){U}

Figure 7.1.2

Here, we have put

O_{ijkl} := {(A, B) ∈ N_{𝔭}^{∗} : rankA = i, rankB = j, rankAB = k, rankBA = l}.

Since rankA, rankB, rankAB and rankBA are invariants of the Ad(K_{ℂ})-action, O_{ijkl} are K_{ℂ}-invariant sets. In fact, each O_{ijkl} in Figure 7.1.2 is a single Ad(K_{ℂ})- orbit. For the reader’s convenience, O_{ijkl} is also written as O_{ijkl}(N) if dim_{ℂ} O_{ijkl} = N. The segment denotes the closure relation. The brace followed by O_{ijkl}(N) (e.g. {X_{2}} followed by O_{1211}(6)) describes the associated variety which we will explain in §7.3. Figure 7.1.2 is obtained as follows. For instance, if rankA = 1 then we can assume A = E_{12}, after conjugating by an element of K_{ℂ}, where E_{ij} ∈ M(2, ℂ) stands for the matrix unit. Because AB and BA are nilpotent, the (2, 1)-component of B must be 0. Then B is conjugate to one of E_{11} + E_{22}, E_{11}, E_{22}, E_{12} or O by the subgroup of K_{ℂ} stabilizing A. The corresponding orbits are O_{1211}(6), O_{1101}(5), O_{1110}(5), O_{1100}(4) or O_{1000}(3), respectively. Other cases are similar or trivial.

\underline{7.2.} Let H = Sp(1, 1). Corresponding to the embedding H ↪ G, we have the projection pr_{𝔤→𝔥}: 𝔭^{∗} → (𝔭 ∩ 𝔥)^{∗} which respects the action of (H ∩ K)_{ℂ} ↪ K_{ℂ}. The projection is given in the matrix spaces by

pr_{𝔤→𝔥}: 𝔭^{∗} ≃ M(2, ℂ) ⊕ M(2, ℂ) → (𝔭 ∩ 𝔥)^{∗} ≃ M(2, ℂ), (A, B) ↦ A + τB,

where τ ( \begin{matrix}a & b \\ c & d \end{matrix} ) := ( \begin{matrix}−d & b \\ c & −a \end{matrix} ) and the action of (H ∩ K)_{ℂ} ≃ SL(2, ℂ) ×

SL(2, ℂ) ∋ (g_{1}, g_{2}) on (𝔭 ∩ 𝔥)^{∗} is given by Z ↦ g_{1}Zg_{2}^{−1}. The nilpotent variety N_{𝔥}^{∗} ∩ (𝔭 ∩ 𝔥)^{∗} is identified with {X ∈ M(2, ℂ) : rankX ≤ 1}.

Lemma 7.2. The following conditions on a K_{ℂ}-orbit O in N_{𝔭}^{∗} are equivalent:

1) pr_{𝔤→𝔥}(O) ⊄ N_{𝔥}^{∗}.

2) pr_{𝔤→𝔥}(\overline{O}) ⊄ N_{𝔥}^{∗}. Here, \overline{O} denotes the closure of O.

3) O is one of O_{2111}(6), O_{1211}(6), O_{2000}(4) or O_{0200}(4).

Proof. By the closure relation described in Figure 7.1.2, it suffices to show (1) ⇔ (3). Since O_{2111}(6) contains (A, B) := (E_{11} + E_{22}, E_{12}) (see §7.1), pr_{𝔤→𝔥}(A, B) = A + τB ∉ N_{𝔥}^{∗} because rank(A + τB) = 2 > 1. Hence pr_{𝔤→𝔥}(O_{2111}(6)) ⊄ N_{𝔥}^{∗}. Similarly, O_{2000}(4) contains (E_{11} + E_{22}, O) and thus pr_{𝔤→𝔥}(O_{2000}(4)) ⊄ N_{𝔥}^{∗}. Changing the role of A and B for other cases, we have (3) ⇒ (1). To prove (1) ⇒ (3), we note that

pr_{𝔤→𝔥} (Ad(K_{ℂ})(A, B)) = Ad((H ∩ K)_{ℂ}){(aA + bτB) : a, b ∈ ℂ^{×}}.

For example, O_{1101}(5) contains (E_{12}, E_{11}) and det(aE_{12} + bτE_{11}) = 0 for any a, b ∈ ℂ^{×}. Hence we have pr_{𝔤→𝔥}(O_{1101}(5)) ⊂ N_{𝔥}^{∗}. Other cases are similar. □

\underline{7.3.} It follows from a result of S. Salamanca Riba (see [Sa], Theorem 1.2) that any irreducible unitary representation of G = U(2, 2) whose underlying (𝔤, K)-module has regular integral infinitesimal character is of the form A_{𝔮}(λ) where 𝔮 is a θ-stable parabolic subalgebra and λ is in the good range. Because the associated variety and the discrete decomposability are preserved by coherent continuation in the good range of parameters, we shall assume λ = 0 without loss of generality. Let 𝔱_{0}^{c} be a Cartan subalgebra of 𝔨_{0} ≃ 𝔲(2) + 𝔲(2). We choose a coordinate in \sqrt{−1}𝔱_{0}^{c} so that Δ^{+}(𝔨, 𝔱^{c}) = {e_{1} − e_{2}, e_{3} − e_{4}}. With this basis, we put X_{1} = (4, 3, 2, 1), X_{2} = (4, 2, 3, 1), X_{3} = (4, 1, 3, 2), X_{4} = (3, 2, 4, 1), X_{5} = (3, 1, 4, 2), X_{6} = (2, 1, 4, 3), Y_{1} = (2, 1, 1, 0), Y_{2} = (2, 0, 1, 0), Y_{3} = (2, 1, 2, 0), Y_{4} = (1, 0, 2, 0), Y_{5} = (2, 0, 2, 1), Y_{6} = (1, 0, 2, 1), Z_{1} = (1, 0, 0, 0), Z_{2} = (1, 1, 1, 0), Z_{3} = (0, 0, 1, 0), Z_{4} = (1, 0, 1, 1), W = (1, 0, 1, 0), U = (0, 0, 0, 0) ∈ \sqrt{−1}𝔱_{0−}^{c}. Then the set of (𝔤, K)-modules

{A_{𝔮} : 𝔮 = 𝔮(X_{i}), 𝔮(Y_{i})(1 ≤ i ≤ 6), 𝔮(Z_{i})(1 ≤ i ≤ 4), 𝔮(W), 𝔮(U)}

is the totality of irreducible, infinitesimally unitarizable (𝔤, K)-modules with trivial infinitesimal character. We note that \overline{A_{𝔮(X_{i})}} (1 ≤ i ≤ 6) is Harish-Chandra’s discrete series for a group manifold G and A_{𝔮(U)} = ℂ. We refer to the figure in [Ko2], Example 3.7 for the geometric description of these modules in the context of the Beilinson-Bernstein correspondence between irreducible Harish-Chandra modules and irreducible K-equivariant sheaves of D-modules on the flag variety of G_{ℂ}.

\underline{7.4.} Let us compute the associated variety V(A_{𝔮}) = Ad(K_{ℂ})(𝔲^{−} ∩ 𝔭) for each θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲.

Lemma 7.4. The correspondence X ↦ A_{𝔮(X)} ↦ V_{𝔤}(A_{𝔮(X)}) =: \overline{O} is given in the Figure 7.1.2 described as O\{X\}. Here \overline{O} stands for the closure of some Ad(K_{ℂ})- orbit O = O_{ijkl}(N), and X stands for one of X_{m}, Y_{m}, Z_{m}, W or U.

For example, the above lemma means that V_{𝔤}(A_{𝔮(X_{1})}) = \overline{O_{0200}(4)}, V_{𝔤}(A_{𝔮(X_{2})}) = \overline{O_{1211}(6)}, V_{𝔤}(A_{𝔮(X_{5})}) = \overline{O_{2111}(6)}, and so on. Let us sketch the computation for V_{𝔤}(A_{𝔮(X_{5})}) = \overline{O_{2111}}. Corresponding to Δ(𝔲^{−}(X_{5}) ∩ 𝔭) = \{e_{1} − e_{3}, −e_{1} + e_{4}, e_{2} − e_{3}, e_{2} − e_{4}}, we have

𝔲^{−}(X_{5}) ∩ 𝔭 ≃ \{( \begin{matrix}a & 0 \\ c & d \end{matrix} ) , ( \begin{matrix}0 & 0 \\ b & 0 \end{matrix} ) : a, b, c, d ∈ ℂ\} ⊂ M(2, ℂ) ⊕ M(2, ℂ).

Hence, Ad(K_{ℂ})(𝔲^{−}(X_{5}) ∩ 𝔭) = \{(A, B) : rankA ≤ 2, rankB ≤ 1\} = \overline{O_{2111}(6)}.

\underline{7.5.} Here is a classification of discretely decomposable modules.

Proposition 7.5. Let G = U(2, 2) ⊃ H = Sp(1, 1). Let π be the irreducible unitary representation of G with regular integral infinitesimal character. Then the following three conditions are equivalent:

1) The restriction π|_{H∩K} is H ∩ K-admissible.

2) The underlying (𝔤, K)-module π_{K} is discretely decomposable as an (𝔥, H ∩ K)- module.

3) π is isomorphic to \overline{A_{𝔮}(λ)} where λ is in the good range of parameter and where 𝔮 is one of 𝔮(X_{3}), 𝔮(X_{4}), 𝔮(Y_{2}), 𝔮(Y_{3}), 𝔮(Y_{4}), 𝔮(Y_{5}), 𝔮(Z_{i}) (1 ≤ i ≤ 4), 𝔮(W), 𝔮(U).

Proof. (3) ⇒ (1) is given in Theorem 3.2 and Example 3.7 of [Ko2]. (1) ⇒ (2) is proved in Proposition 1.6. (2) ⇒ (3) follows from Corollary 3.5 and the above computations. □

Remark 7.6. The above proof of (3) ⇒ (1) is divided into four steps: Step 1. Computation of the K_{ℂ}-orbits in the nilpotent cone N^{∗} ∩ (𝔤/𝔨)^{∗}. Step 2. Computation of the projection pr_{𝔤→𝔥} of the orbits in Step 1. Step 3. Description of Ĝ with regular integral infinitesimal character. Step 4. Description of the associated varieties for the representations in Step 3. More general combinatorial results may be found in the literature for some of the above steps: the description of K_{ℂ}-orbits on N_{𝔭}^{∗} in terms of ab-diagrams due to H. Kraft-C. Procesi, T. Ohta and J. Schwartz for Step 1; the description of K_{ℂ}- orbit Q on the flag variety F by T. Matsuki – T. Oshima for a part of Step 3; and the description of the image of the moment map from the conormal bundle T_{Q}^{∗}F to 𝔤^{∗} (e.g. G = U(p, q)) by a recent work of A. Yamamoto for Step 4. As we mentioned in §7.1, Proposition 7.5 itself can be easily proved by Theorem 4.2 instead of Corollary 3.5 because Theorem 4.2 enables us to avoid a direct computation of associated varieties. The classification for the triplet (G, H, 𝔮) with (G, H) a classical symmetric pair which satisfies the equivalent conditions in Theorem 4.2 will be reported elsewhere.

§8. Example 2 : the restriction to normal real forms.

\underline{8.1.} It is well-known that π is discretely decomposable as a (𝔤, K)-module for any irreducible (𝔤̃, K̃)-module π if (G̃, G) is a Riemannian symmetric pair, namely if G ≃ K̃. In this section we present an opposite extremal case:

Theorem 8.1. Let G̃ be a complex reductive linear Lie group, and G a normal real form of G̃. Then, no irreducible infinite dimensional (𝔤̃, K̃)-module is discretely decomposable as a (𝔤, K)-module.

We recall that a closed subgroup G of a complex reductive linear Lie group G̃ is a normal real form if 𝔤_{0} is a real form of 𝔤̃_{0} satisfying ℝ-rankG = rankG. For example, (G̃, G) = (GL(n, ℂ), GL(n, ℝ)), (SO(2n+1, ℂ), SO(n+1, n)), (Sp(n, ℂ), Sp(n, ℝ)), (SO(2n, ℂ), SO(n, n)) are the cases. Exceptional cases are given by (𝔤̃_{0}, 𝔤_{0}) = (𝔢_{n}, 𝔢_{n(n)}) (n = 6, 7, 8), (𝔣_{4}, 𝔣_{4(4)}) and (𝔤_{2}, 𝔤_{2(2)}) (see [He], Chapter X for notation).

The rest of this section will be devoted to the proof of Theorem 8.1.

\underline{8.2.} Suppose we are in the setting of Theorem 8.1. Let 𝔤̃ = 𝔨̃+𝔭̃ be a complexified Cartan decomposition of 𝔤̃ compatible with 𝔤 = 𝔨 + 𝔭. If X is a (𝔤̃, K̃)-module of finite length, then the associated variety V𝔤̃(X) is a union of Ad(K̃_{ℂ})-orbits in N ∗ ; 𝔤̃ ∩ (𝔤̃/𝔨̃)^{∗} (see Remark 3.6). Furthermore, if dim X = ∞ then V𝔤̃(X) ≠ {0}. Since N ∗ ; 𝔤̃ consists of nilpotent elements, Theorem 8.1 follows from Corollary 3.4 if one proves the following lemma:

Lemma 8.2. If O is a non-zero Ad(K̃_{ℂ})-orbit on N ∗ ; 𝔤̃ ∩ (𝔤̃/𝔨̃)^{∗} then pr_{𝔤̃→𝔤}(O) (⊂ 𝔤^{∗}) contains a non-zero semisimple element.

\underline{8.3.} We will rewrite Lemma 8.2 by using the fact that G̃ is a complex Lie group. In view of the isomorphism of complex vector spaces 𝔤̃ = Lie(G̃) ⊗_{ℝ} ℂ = 𝔨̃+ 𝔭̃ ≃ 𝔤 + 𝔤, we can identify

(𝔤̃/𝔨̃)^{∗} ≃ 𝔭̃^{∗} ≃ 𝔭̃ ≃ 𝔤,
(𝔤/𝔨)^{∗} ≃ 𝔭^{∗} ≃ 𝔭.

Then, the projection pr_{𝔤̃→𝔤}: 𝔤̃^{∗} → 𝔤^{∗} restricted to (𝔤̃/𝔨̃)^{∗} → (𝔤/𝔨)^{∗} is identified with

pr_{𝔤→𝔭}: 𝔤 → 𝔭, X ↦ \frac{1}{2}(X − θX),

where θ is a Cartan involution of 𝔤_{0} extended to a ℂ-linear endomorphism of 𝔤 as usual. We note that the above isomorphism induces a bijection between nilpotent orbits of the identity component of K̃_{ℂ} on N ∗ ; 𝔤̃ ∩(𝔤̃/𝔨̃)^{∗} and nilpotent orbits of Int(𝔤) on N_{𝔤}^{∗}. Hence, Lemma 8.2 is equivalent to the following lemma:

Lemma 8.3. Suppose that G is a real reductive linear Lie group with ℝ-rankG = rankG. If O is a non-zero nilpotent orbit of Int(𝔤) in 𝔤, then pr_{𝔤→𝔭}(O) contains a non-zero semisimple element.

\underline{8.4.} Before proving Lemma 8.3, we need:

Lemma 8.4. Suppose that G is a real reductive linear Lie group with ℝ-rankG = rankG. Then any nilpotent orbit O of Int(𝔤) in 𝔤 meets 𝔤_{0}.

Proof. First, we recall the proof of the Dynkin-Kostant classification of complex nilpotent orbits. By the Jacobson-Morozov theorem, there exists an 𝔰𝔩_{2} triple {H, X, Y } (⊂ 𝔤) such that [H, X] = 2X, [H, Y] = −2Y, [X, Y] = H and that Int(𝔤) · X = O. We put

𝔤_{0}(H; λ) := {Z ∈ 𝔤_{0} : [H, Z] = λZ} (λ ∈ ℂ),
𝔤(H; λ) := {Z ∈ 𝔤 : [H, Z] = λZ} (λ ∈ ℂ),
P  := {Z ∈ 𝔤(H; 2) : Ker (ad Z: 𝔤(H; −2) → 𝔤(H; 0)) = {0}},
L  := the identity component of {g ∈ Int(𝔤) : Ad(g)H = H}.

We note that 𝔤(H; λ) ≠ {0} only if λ ∈ ℤ by the representation theory of 𝔰𝔩_{2}. Because 𝔤_{0} is a normal real form of 𝔤, we have

(8.4.1)  𝔤(H; λ) = 𝔤_{0}(H; λ) ⊗_{ℝ} ℂ (λ ∈ ℂ).

A theorem of Malcev says that P = Ad(L) · X and that P is a Zariski open set of 𝔤(H; 2). It follows from (8.4.1) that P ∩ 𝔤_{0}(H; 2) ≠ ∅. Therefore, we find X′ ∈ P ∩ 𝔤_{0}(H; 2) (⊂ 𝔤_{0}) and l ∈ L such that Ad(l)X = X′. Thus, O ∩ 𝔤_{0} ≠ ∅. □

\underline{8.5.} Now, let us complete the prove of Theorem 8.1.

All what we need is to show Lemma 8.3. In the setting of Lemma 8.3, we can take a non-zero element X ∈ O ∩ 𝔤_{0} owing to Lemma 8.4. Then pr_{𝔤→𝔭}(X) = \frac{1}{2}(X − θX) ∈ 𝔭_{0}, which shows pr_{𝔤→𝔭}(X) is a semisimple element. On the other hand, pr_{𝔤→𝔭}(X) ≠ 0 because any element of 𝔨_{0} = 𝔤_{0} ∩ Ker(pr_{𝔤→𝔭}: 𝔤 → 𝔭) is semisimple. This proves Lemma 8.3.

Hence, we have completed the proof of Theorem 8.1.

References

[A1] J. Adams, Discrete spectrum of the dual reductive pair (O(p, q), Sp(2m)), Invent. Math. 74 (1984), 449-475.

[A2] , Unitary highest weight modules, Adv. in Math. 63 (1987), 113-137.

[BW] A. Borel and N. Wallach, Continuous Cohomology, Discrete Subgroups, and Representations of Reductive Groups, Ann. Math. Stud. 94, Princeton U.P., 1980.

[BB] W. Borho and J. L. Brylinski, Differential operators on homogeneous spaces I, Invent. Math. 69 (1982), 437-476, Part III, 80 (1985), 1-68.

[FJ1] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. of Math. 111 (1980), 253-311.

[FJ2] , Analysis on Non-Riemannian Symmetric Spaces, Conference Board, No.61, A.M.S., 1986.

[GG] I. M. Gelfand and M. I. Graev, Geometry of homogeneous spaces, representations of groups in homogeneous spaces, and related questions of integral geometry, Transl. II. Ser., A. M. S. 37 (1964), 351-429.

[GW] B. H. Gross and N. R. Wallach, A distinguished family of unitary representations for the exceptional groups of real rank = 4, Lie Theory and Geometry, In honor of Bertram Kostant, vol. 123, Birkhäuser, 1994, pp. 289-304.

[HMSW] H. Hecht, D. Miličic, W. Schmid and J. A. Wolf, Localization and standard modules for real semisimple Lie groups, Invent. Math. 90 (1987), 297-332.

[He] , Groups and Geometric Analysis, Pure and Appl. Math. 113, vol. 113, Academic Press, 1984.

[Ho] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math. 33 (1979), A. M. S., 275-285.

[JV] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29-53.

[KV] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math. 44 (1978), 1-47.

[Ko1] T. Kobayashi, The restriction of A_{𝔮}(λ) to reductive subgroups, Proc. Acad. Japan 69 (1993), 262-267, Part II, 71, (1995), 24-26.

[Ko2] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181-205.

[Ko3] , Harmonic analysis on homogeneous manifolds of reductive type and unitary representation theory, Sugaku of Math. Soc. Japan 46 (1994), 124-143; A. M. S. Translations (in press).

[Ko4] , Discrete series representations for the orbit spaces arising from two involutions of real reductive Lie groups, preprint UTMS 96-52 (1996), to appear in J. Funct. Anal..

[KO] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on locally symmetric spaces, preprint (1996).

[KØ] T. Kobayashi and B. Ørsted, Branching laws for unitary representations attached to

minimal elliptic and nilpotent orbits, (in preparation).

[L] R. Lipsman, Restrictions of principal series to a real form, Pacific J. of Math. 89 (1980), 367-390.

[Ma] S. Martens, The characters of the holomorphic discrete series, Proc.Nat.Acad.Sci. USA 72 (1975), 3275-3276.

[MO] T. Matsuki and T. Oshima, A description of discrete series for semisimple symmetric spaces, Advanced Studies in Pure Math. 4 (1984), 331-390.

[MM] Y. Matsushima and S. Murakami, On vector bundle valued harmonic forms and automorphic forms on symmetric spaces, Ann. Math. 78 (1963), 365-416.

[Mo] C. C. Moore, Ergodicity of flows on homogeneous spaces, Amer. J. Math. 88 (1966), 154-178.

[OO] G. Ólafsson and B. Ørsted, Generalizations of the Bargmann transform, Proceedings of Workshop on Lie Theory and its application in physics, Clausthal, 1995 (to appear).

[R] J. Repka, Tensor products of holomorphic discrete series representations, Can. J. Math. 31 (1979), 836-844.

[Sa] S. Salamanca Riba, On the unitary dual of some classical Lie groups, Compo. Math. 68 (1988), 251-303.

[Se] H. Sekiguchi, The Penrose Transform for certain non-compact homogeneous manifolds of U(n, n), ph. D. dissertation, University of Tokyo, January (1995).

[TW] Y. L. Tong and S. P. Wang, Geometric realization of discrete series for semisimple symmetric spaces, Invent. Math. 96-2 (1989), 425–458.

[V1] D. Vogan, Jr., Representations of real reductive Lie groups, Birkhäuser, 1981.

[V2] , Unitarizability of certain series of representations, Ann. of Math. (1984), 141-187.

[V3] , Unitary Representations of Reductive Lie Groups, Ann. Math. Stud. 118, Princeton U. P., 1987.

[V4] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 191-221.

[V5] , Associated varieties and unipotent representations, Harmonic Analysis on Reductive Lie Groups, vol. 101, Birkhäuser, 1991, pp. 315-388.

[VZ] D. Vogan and G. Zuckerman, Unitary representations with non-zero cohomology, Compo. Math. 53 (1984), 51-90.

[Wa] N. Wallach, On the unitarizability of derived functor modules, Invent.Math. 78 (1984), 131-141.

[Wi] F. Williams, Tensor products of principal series representations, Lecture Notes in Mathematics, vol. 358, Springer-Verlag, Berlin, 1973.

[Wo] H. Wong, Dolbeault cohomologies and Zuckerman modules associated with finite rank representations, Ph.D. dissertation, Harvard University (1991).

Department of Mathematical Sciences, University of Tokyo, Meguro, Komaba, 153, Tokyo, Japan E-mail address: toshi@ms.u-tokyo.ac.jp