MONASTIR SEMINAR ON THE RESTRICTION OF UNITARY

REPRESENTATIONS AND THEIR APPLICATIONS

Toshiyuki KOBAYASHI

Department of Mathematical Sciences, University of Tokyo, Meguro, Komaba, 153, Tokyo, Japan E-mail: toshi@ms.u-tokyo.ac.jp

Let G be a real reductive linear Lie group, and G′ a subgroup that is reductive in G. We write Ĝ (resp. Ĝ′) for the unitary dual of G (resp. G′). If π ∈ Ĝ, then the restriction π|_{G′} is decomposed into the direct integral of irreducible unitary representations of G′:

π|_{G′} ≃ ∫ ⊕ ; Ĝ′ m_{G′}(τ : π|_{G′}) τ dμ(τ),

where dμ is a Borel measure on Ĝ′ and m_{G′}(· : π|_{G′}): Ĝ′ → ℕ ∪ {∞} is the multiplicity defined almost everywhere with respect to dμ. Our point of view is to single out a nice class of the restriction problem.

Definition. We say that the restriction π|_{G′} is G′-admissible if the measure dμ is discrete and if the multiplicity m_{G′}(τ : π|_{G′}) < ∞ for all τ ∈ Ĝ′.

We consider a criterion for the G′-admissibility of π|_{G′} in the typical case:

π  : Zuckerman-Vogan’s derived functor modules A_{𝔮}(λ) (see [5]),
(G, G′) : a reductive symmetric pair (see [1] for the classification).

Let K be a maximal compact subgroup of G, and 𝔱 a Cartan subalgebra of the Lie algebra of K. Then Cone(π) is a cone determined by π and Subsp(G′) is a vector space determined by G′, both of which are (computable) subsets of \sqrt{−1}𝔱^{∗}.

Theorem. Cone(π) ∩ Subsp(G′) = {0} ⇒ The restriction π|_{G′} is G′-admissible.

Classically, the restriction π|_{G′} was known to be G′-admissible in the following: Case 1) G′ = K (Harish-Chandra). In this case, Subsp(G′) = {0}. Case 2) π is a highest weight module (e.g. the Segal-Shale-Weil representation, holomorphic discrete series) and G′ is a certain reductive subgroup (Howe, Martens, Jakobsen-Vergne; see [3], [4]).

A distinguishing feature in the setting of Theorem is that G′ is not necessarily compact and A_{𝔮}(λ) is not always a highest weight module. Surprisingly, there is still a rich family of the triplet (G, G′, π) such that the restriction π|_{G′} is G′-admissible.

Typeset by AMS-TEX

Example. If (G, G′) = (U(2, 2), Sp(1, 1)) ≈ (SO(4, 2), SO(4, 1)), then 12 series among 18 series in Ĝ with regular integral infinitesimal character are G′-admissible when restricted to G′ ([7], Part III, Proposition 7.5).

The setting of Theorem can be generalized to an arbitrary π ∈ Ĝ and to an arbitrary G′ (by a similar criterion) ([7], Part II); and the assumption in Theorem is proved to be also necessary for the discretely decomposability ([7], Part III).

Theorem gives rise to new branching laws that is an analog of Howe’s dual pair correspondence of the minimal unipotent representation of SO(p, q) [12]. It also gives new applications in other branches of mathematics such as:

1) Vanishing theorem of modular symbols defined by the arithmetic quotients of Riemannian symmetric spaces. In particular, a conjecture of T. Oda in connection with the theory of automorphic forms was solved affirmatively for the middle Hodge components of certain totally real modular symbols on arithmetic quotients of bounded symmetric domain of type IV ([11]).

2) Construction of new discrete series representations for non-symmetric homogeneous manifolds of reductive type. For example, spherical homogeneous manifolds (cf. [2]) such as G_{2}(ℝ)/SL(3, ℝ) and SU(n, n + 1)/Sp(n, ℝ) have non-empty discrete series representations, while SL(2n + 1, ℝ)/Sp(n, ℝ) has empty discrete series ([7], Corollary 5.6). More general homogeneous manifolds are treated in [10].

3) The finite dimensionality theorem of the global solutions of the system of P. D. E.’s (a generalization of the Gauss-Aomoto-Gelfand hypergeometric equations to higher order) by using the (generalized) Penrose construction (see [13], §7).

I would like to express my sincere gratitude to Professor P. Torasso for inviting me the stimulating CIMPA summer school on Invariant Theory in Monastir, 1996.

References

[1] M. Berger, Les espaces symétriques non compacts, Ann. Sci. Eco. No. Sup. 74 (1957), 85-177.

[2] M. Brion, Classification des espaces homogènes sphériques, Comp. Math. 63 (1987), 189–208.

[3] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math. 33 (1979), A. M. S., 275-285.

[4] H.P.Jakobsen and M.Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29-53.

[5] A. Knapp and D. Vogan, Cohomological Induction and Unitary Representations, Princeton University Press, 1995.

[6] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽), vol. 462, Memoirs of Amer. Math. Soc., 1992.

[7] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Inv. Math. 117 (1994), 181-205; Part II, Part III (to appear).

[8] , The Restriction of A_{𝔮}(λ) to reductive subgroups. I, II, Proc. Acad. Japan 69 (1993), 262-267; 71 (1995) 24-26.

[9] , Harmonic analysis on homogeneous spaces of reductive type and unitary representation theory, Sugaku Exposition of Amer. Math. Soc. (1996), (in press).

[10] , Discrete series representations for homogeneous spaces and the restriction of irreducible unitary representations, preprint (1996).

[11] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on locally symmetric spaces, preprint (1996).

[12] T. Kobayashi and B. Ørsted, Branching laws for unitary representations attached to minimal elliptic and nilpotent orbits, (in preparation).

[13] H. Sekiguchi, The Penrose Transform for certain non-compact homogeneous manifolds of U(n, n), ph.D. Thesis, University of Tokyo (1995).