RESTRICTION OF UNITARY REPRESENTATIONS

OF REDUCTIVE LIE GROUPS

Toshiyuki KOBAYASHI

Graduate School of Mathematical Sciences, University of Tokyo, Meguro, Komaba, 153, Tokyo, Japan E-mail: toshi@ms.u-tokyo.ac.jp

Let G be a real reductive linear Lie group, and G′ a subgroup that is reductive in G. We write Ĝ (resp. Ĝ′) for the unitary dual of G (resp. G′). If π ∈ Ĝ, then the restriction π|_{G′} is decomposed into the direct integral of irreducible unitary representations of G′:

π|_{G′} ≃ ∫ ⊕ ; Ĝ′ m_{G′}(τ : π|_{G′}) τ dμ(τ),

where dμ is a Borel measure on Ĝ′ and m_{G′}(· : π|_{G′}): Ĝ′ → ℕ ∪ {∞} is the multiplicity defined almost everywhere with respect to dμ. Our point of view is to single out a nice class of the restriction problem.

Definition. We say that the restriction π|_{G′} is G′-admissible if the measure dμ is discrete and if the multiplicity m_{G′}(τ : π|_{G′}) < ∞ for all τ ∈ Ĝ′.

We consider a criterion for the G′-admissibility of π|_{G′} in the typical case:

π  : Zuckerman-Vogan’s derived functor modules A_{𝔮}(λ) (see [3]),
(G, G′) : a reductive symmetric pair (see Berger’s classification)

Let K be a maximal compact subgroup of G, and 𝔱 a Cartan subalgebra of the Lie algebra of K. Then Cone(π) is a cone determined by π and Subsp(G′) is a vector space determined by G′, both of which are (computable) subsets of \sqrt{−1}𝔱^{∗}.

Theorem. Cone(π) ∩ Subsp(G′) = {0} ⇒ The restriction π|_{G′} is G′-admissible.

Classically, the restriction π|_{G′} was known to be G′-admissible in the following: Case 1) G′ = K (Harish-Chandra). In this case, Subsp(G′) = {0}. Case 2) π is a highest weight module (e.g. the Segal-Shale-Weil representation, holomorphic discrete series) and G′ is a certain reductive subgroup (Howe, Martens, Jakobsen-Vergne; see [1], [2]).

A distinguishing feature in the setting of Theorem is that G′ is not necessarily compact and A_{𝔮}(λ) is not always a highest weight module. Surprisingly, there is still a rich family of the triplet (G, G′, π) such that the restriction π|_{G′} is G′-admissible.

Typeset by AMS-TEX

Example ([9], §7). If (G, G′) = (U(2, 2), Sp(1, 1)) ≈ (SO(4, 2), SO(4, 1)), then 12 series among 18 series in Ĝ with regular integral infinitesimal character are G′- admissible when restricted to G′.

The setting of Theorem can be generalized to an arbitrary π ∈ Ĝ and to an arbitrary G′ (by a similar criterion) ([5], Part II); and the assumption in Theorem is proved to be also necessary for the discretely decomposability ([9]).

Theorem gives a number of new branching laws ([6], [13]) such as an analog of Howe’s dual pair correspondence of Kostant’s minimal unipotent representation of SO(p, q) (p + q even). Theorem plays also a main role in the following applications:

1) Vanishing theorem of modular symbols defined by the arithmetic quotients of Riemannian symmetric spaces. In particular, a conjecture of T. Oda was solved affirmatively for the middle Hodge components of certain totally real modular symbols on arithmetic quotients of bounded symmetric domain of type IV ([12]).

2) Construction of irreducible L^{p}-representations for non-symmetric homogeneous manifolds with non-compact isotropy subgroups. For example, spherical homogeneous manifolds such as G_{2}(ℝ)/SL(3, ℝ) and SU(n, n + 1)/Sp(n, ℝ) admit non-empty discrete series representations, while SL(2n+1, ℝ)/Sp(n, ℝ) has no discrete series ([5], Corollary 5.6). Theorem gives new discrete series for more general non-symmetric homogeneous manifolds (see [8]) such as

Sp(2n, ℝ)/(Sp(n_{0}, ℂ) × GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ)) (∑ n_{j} = n).

References

[1] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math. 33 (1979), A. M. S., 275-285.

[2] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29-53.

[3] A. Knapp and D. Vogan, Cohomological Induction and Unitary Representations, Princeton University Press, 1995, ISBN 0-691-03756-6.

[4] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽), vol. 462, Memoirs of A.M.S., 1992, ISBN 0-8128-2524-0.

[5] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181-205; Part II (to appear).

[6] , The Restriction of A_{𝔮}(λ) to reductive subgroups. I, II, Proc. Acad. Japan 69 (1993), 262-267; 71 (1995) 24-26.

[7] , Harmonic analysis on homogeneous spaces of reductive type and unitary representation theory, Translations of Amer. Math. Soc., Ser. 2 (to appear).

[8] , Discrete series representations for homogeneous spaces and the restriction of irreducible unitary representations, J. Funct. Anal. (to appear).

[9] , Discrete decomposability of the restriction of A_{𝔮}(λ), III — restriction of Harish-Chandra modules and associated varieties, Invent. Math. (to appear).

[10] , Invariant measures on homogeneous manifolds of reductive type, J. reine und angew. Math. (to appear).

[11] , L^{p}-analysis on homogeneous manifolds of reductive type and representation theory, Proc. Acad. Japan 73 (1997), 62–66.

[12] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on locally symmetric spaces, preprint (1996).

[13] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits (to appear).

[14] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4) (1990), no. 92 Progress in Mathematics, Birkhäuser, 85–124.