THE RESTRICTION OF A_{𝔮}(λ) TO REDUCTIVE SUBGROUPS II

Toshiyuki KOBAYASHI University of Tokyo

§1. Introduction

In this paper we continue the investigation of the restriction of irreducible unitary representations of real reductive groups, with emphasis on the discrete decomposability. We recall that a representation π of a reductive Lie group G on a Hilbert space V is G-admissible if (π, V) is decomposed into a discrete Hilbert direct sum with finite multiplicities of irreducible representations of G. The same terminology is used for a (𝔤, K)-module on a pre-Hilbert space, if its completion is G-admissible.

Let H be a reductive subgroup of a real reductive Lie group G, and (π, V) an irreducible unitary representation of G. The restriction (π_{|H}, V) is decomposed uniquely into irreducible unitary representations of H, which may involve a continuous spectrum if H is noncompact. In [5], [6], we have posed a problem to single out the triplet (G, H, π) such that the restriction of (π_{|H}, V) is H-admissible, together with some application to harmonic analysis on homogeneous spaces. The purpose of this paper is to give a new insight of such a triplet (G, H, π) from view points of algebraic analysis. In particular, we will give a sufficient condition on the triplet (G, H, π) for the H-admissible restriction as a generalization of [5], [6] to arbitrary H, and also present an obstruction for the H-admissible restriction.

§2. A sufficient condition for discrete decomposability

Let K be a compact Lie group. We write 𝔨_{0} for the Lie algebra of K, and 𝔨 for its complexification. Analogous notation is used for other groups. Take a Cartan subalgebra 𝔱_{0}^{c} of 𝔨_{0}. The weight lattice L in \sqrt{−1}(𝔱_{0}^{c})^{∗} is the additive subgroup of \sqrt{−1}(𝔱_{0}^{c})^{∗} consisting

Typeset by AMS-TEX

of differentials of the weights of finite dimensional representations of K. Let \overline{C} ⊂ \sqrt{−1}(𝔱_{0}^{c})^{∗} be a dominant Weyl chamber. We write K_{0} for the identity component of K, and K̂_{0} for the unitary dual of K_{0}. The Cartan-Weyl theory of finite dimensional representations establishes a bijection:

L ∩ \overline{C} →^{∼} K̂_{0}, λ ↦ F (K_{0}, λ).

Suppose X is a K-module (possibly, of infinite dimension) which carries an algebraic action of K. The K_{0}-multiplicity function of X is given by

m ≡ m_{X}: L ∩ \overline{C} → ℕ ∪ ∞, m(λ) := dim Hom_{K_{0}}(F (K_{0}, λ), X).

The asymptotic K-support T (X) ⊂ \overline{C} was introduced in [3] as follows:

S(X) := \{λ ∈ L ∩ \overline{C} : m_{X}(λ) ≠ 0\},
T (X) := \{λ ∈ \overline{C} : V ∩ S(X) is not relatively compact for any open cone V containing λ\}.

Hereafter we assume a growth condition on m_{X}: there are constants A, R > 0 such that

(2.1)  m_{X}(λ) ≤ A exp(R |λ|) for any λ ∈ L ∩ \overline{C}.

This condition assures that the character of the representation X is a hyperfunction on K, whose singularity spectrum we can estimate in terms of T (X).

Suppose H is a closed subgroup of K. Let pr_{K→H}: 𝔨^{∗} → 𝔥^{∗} be the projection dual to the inclusion of Lie algebras 𝔥 ↪ 𝔨. Put 𝔥^{⊥} := Ker (pr_{K→H}: 𝔨^{∗} → 𝔥^{∗}). We set

(2.2)  \overline{C}(𝔥) := \overline{C} ∩ Ad^{∗}(K)𝔥^{⊥} ⊂ \sqrt{−1}(𝔱_{0}^{c})^{∗}.

Note that \overline{C}(𝔨) = \{0\} and \overline{C}(0) = \overline{C}. Theorem 2.3. Let X be a K-module satisfying (2.1). If a closed subgroup H of K satisfies

T (X) ∩ \overline{C}(𝔥) = \{0\},

then the restriction X_{|H} is H-admissible.

Now, let us apply Theorem (2.3) to some standard (𝔤, K)-modules. Suppose that G is a real reductive linear Lie group and that K is a maximal compact subgroup of G. A dominant element a ∈ \sqrt{−1}𝔱_{0}^{c} defines a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲, where 𝔩, 𝔲 are the sum of eigenspaces of ad(a) with 0, positive eigenvalues, respectively. Let L be the centralizer of a in G. Zuckerman introduced the cohomological parabolic induction R_{𝔮}^{j} ≡ (R_{𝔮}^{𝔤})^{j} (j ∈ ℕ), which is a covariant functor from the category of metaplectic (𝔩, (L∩K)^{∼})-modules to that of (𝔤, K)-modules, as a generalization of the Borel-Weil-Bott construction of finite dimensional representations of compact groups. In particular, we write A_{𝔮}(λ) := (R_{𝔮}^{𝔤})^{S} (ℂ_{λ}) for a metaplectic unitary character ℂ_{λ} in the good range of parameter (see [8] Definition 2.5), where S := dim_{ℂ}(𝔲 ∩ 𝔨). Then A_{𝔮}(λ) is an irreducible unitarizable (𝔤, K)-module (see [7] Theorem 6.8), and we write \overline{A_{𝔮}(λ)} ∈ Ĝ for its completion.

The K-module structure of the alternating sum ∑_{j}(−1)^{j} (R_{𝔮}^{𝔤})^{j} (W)_{|K} is known as a generalized Blattner formula (see [7] Theorem 6.34). Its proof also gives an upper estimate of each term (R_{𝔮}^{𝔤})^{j} (W)_{|K}, which leads us to: Theorem 2.4. If W is a finite dimensional metaplectic (𝔩, (L ∩ K)^{∼})-module, then the restriction (R_{𝔮}^{𝔤})^{j} (W)_{|K} satisfies (2.1) and

T ((R_{𝔮}^{𝔤})^{j} (W)_{|K}) ⊂ ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ ∩ \overline{C} (j ∈ ℕ).

Here, we recall 𝔮 = 𝔩 + 𝔲 (Levi decomposition) and 𝔤 = 𝔨 + 𝔭 (Cartan decomposition), and we define a closed cone by

ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ := \{ ∑_{β∈Δ(𝔲∩𝔭,𝔱^{c})} n_{β} β : n_{β} ≥ 0\} ⊂ \sqrt{−1}(𝔱_{0}^{c})^{∗}.

Corollary 2.5. In the setting of Theorem (2.3), if

\overline{C}(𝔥) ∩ ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ = \{0\},

then the restriction (R_{𝔮}^{𝔤})^{j} (W)_{|H} is H-admissible for any finite dimensional metaplectic (𝔩, (L∩K)^{∼})-module W and for any j ∈ ℕ. In particular, \overline{A_{𝔮}(λ)}_{|H} is decomposed discretely into irreducible unitary representations of H.

As a special case of Corollary (2.5), we obtain a new and unified proof of some of the main results in [5], [6], where we imposed some assumptions on a subgroup H.

First, suppose that H ⊂ K is a symmetric pair defined by an involution σ ∈ Aut(K). Take a maximal abelian subspace 𝔞_{0} in {Y ∈ 𝔨_{0} : σ(Y) = −Y } and extend it to a Cartan subalgebra 𝔱_{0}^{c} of 𝔨_{0}. We take a dominant Weyl chamber \overline{C} so that \sqrt{−1}𝔞_{0}^{∗} ∩ \overline{C} is also a dominant Weyl chamber for the restricted root system Σ(𝔨, 𝔞).

Corollary 2.6 (cf. [5] Theorem 1.2; [6] Theorem 3.2). Retain the above setting. If

\sqrt{−1}𝔞_{0}^{∗} ∩ \overline{C} ∩ ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ = \{0\},

then the restriction (R_{𝔮}^{𝔤})^{j} (W)_{|H} (j ∈ ℕ) is H-admissible for any finite dimensional metaplectic (𝔩, (L ∩ K)^{∼})-module W.

Next, suppose K is (locally) isomorphic to a direct product K_{1} × K_{2}. We note that the Cartan subalgebra 𝔱_{0}^{c} is also decomposed into a direct sum 𝔱_{0}^{c} = (𝔱_{1}^{c})_{0} + (𝔱_{2}^{c})_{0}.

Corollary 2.7 (cf. [6] Corollary 4.4; [4] Proposition 4.1.3). In the setting as above, if a θ-stable parabolic subalgebra 𝔮 is given by a ∈ \sqrt{−1}(𝔱_{1}^{c})_{0}, then the restriction (R_{𝔮}^{𝔤})^{j} (W)_{|K_{1}} (j ∈ ℕ) is K_{1}-admissible for any finite dimensional metaplectic (𝔩, (L ∩ K)^{∼})-module W.

We note that Corollaries (2.6), (2.7) are deduced from Corollary (2.5) by using \overline{C}(𝔥) = \sqrt{−1}𝔞_{0}^{∗} ∩ \overline{C}, \overline{C}(𝔨_{1}) = \sqrt{−1}(𝔱_{2}^{c})_{0}^{∗} ∩ \overline{C}, respectively.

Remark 2.7. The above corollaries (2.4),(2.5),(2.6) are valid if we replace H by any reductive subgroup H′ containing H, because of Corollary (1.3) in [6].

§3. A necessary condition for discrete decomposability

In §2, we have given a sufficient condition that the restriction of a (𝔤, K)-module X has an H-admissible restriction with respect to a subgroup H. Conversely, we will find a necessary condition in terms of associated varieties of 𝔤-modules in this section.

We recall that the associated variety of a (𝔤, K)-module X of finite length is defined by

V(X) ≡ V_{G}(X) = Supp_{S(𝔤)}(gr(X)) ⊂ 𝔤^{∗},

as the support in 𝔤^{∗} of the associated graded module gr(X) over the symmetric algebra S(𝔤), with respect to a good filtration (see [1]). It is known that V(X) is a subset of the nilpotent cone N^{∗} ≡ N^{∗}(𝔤) ⊂ 𝔤^{∗}.

Let H be a closed subgroup that is reductive in G. We fix a Cartan involution θ of G which makes H stable so that H ∩ K is a maximal compact subgroup of H. Write the projection pr_{G→H}: 𝔤^{∗} → 𝔥^{∗} as before. Theorem 3.1. Suppose X is a (𝔤, K)-module of finite length. Assume that the restriction X_{|H} is H-admissible. Let Y be any (𝔥, H ∩ K)-module occurring as a direct summand of X. Then we have

pr_{G→H} (V_{G}(X)) ⊂ V_{H}(Y).

This theorem gives rise to an obstruction for the admissibility of the restriction of a unitary representation. Corollary 3.2. Suppose X is a (𝔤, K)-module of finite length. Assume that the restriction X_{|H} is H-admissible. Then

pr_{G→H} (V_{G}(X)) ⊂ N^{∗}(𝔥).

Applying Corollary (3.2) to X = A_{𝔮}(λ), we have: Corollary 3.3. Let us identify 𝔤^{∗} with 𝔤 via the Killing form. Assume a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲 of 𝔤 satisfies

pr_{G→H} (Ad(K_{ℂ})(𝔲 ∩ 𝔭)) ⊄ N^{∗}(𝔥).

Then the restriction of \overline{A_{𝔮}(λ)} ∈ Ĝ to H is not H-admissible. Remark 3.4. If H = K, then the assumption of Theorem (3.1) is always satisfied. In this special case, Theorem (3.1) implies a well-known result pr_{G→K} (V_{G}(X)) = {0} (see [9] Corollary 5.13) because the associated variety of a finite dimensional representation is zero. In a general case where H is non-compact, pr_{G→H} (V_{G}(X)) is not necessarily {0}.

Finally, we mention a useful information about Ĥ occurring as direct summands of the restriction X_{|H}, as an elementary application of associated varieties. This helps us to understand a strange phenomenon about the direct summands occurring in the restriction

of A_{𝔮}(λ)_{|H}, which was pointed out in [6] Introduction.

Theorem 3.5. Suppose X is an irreducible (𝔤, K)-module. Assume that X is H-admissible.

Let Y_{1}, Y_{2} be any (𝔥, H ∩ K)-module occurring as a direct summand of the restriction X to

(𝔥, H ∩ K). Then we have

V_{H}(Y_{1}) = V_{H}(Y_{2}).

References

[1] W.Borho and J.L.Brylinski, Differential operators on homogeneous spaces I, Invent. Math. 69 (1982), 437-476.

[2] M.Kashiwara, T.Kawai and T.Kimura, Foundations of Algebraic Analysis, Princeton Math. Series 37, 1986.

[3] M.Kashiwara and M.Vergne, K-types and singular spectrum, Lect. Notes. in Math. 728 (1979), 177-200.

[4] T.Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽), vol. 462, Memoirs A.M.S. 462, 1992.

[5] , The restriction of A_{𝔮}(λ) to reductive subgroups, Proc. Acad. Japan 69 (1993), 262-267.

[6] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181-205.

[7] D.Vogan, Unitary Representations of Reductive Lie Groups, Ann. Math. Stud. 118, Princeton U.P., 1987.

[8] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 191-221.

[9] , Associated varieties and unipotent representations, Harmonic Analysis on Reductive Lie Groups, vol. 101, Birkhäuser, 1991, pp. 315-388.

Department of Mathematical Sciences, University of Tokyo, Meguro, Komaba, 153, Tokyo, Japan

E-mail address: toshi@tansei.cc.u-tokyo.ac.jp