2002 年 · 秋の表現論シンポジウム

Branching Problems of Unitary Representations

小林俊行 Toshiyuki Kobayashi*

RIMS, Kyoto University

Abstract

The irreducible decomposition of a unitary representation often contains continuous spectrum when restricted to a non-compact subgroup. The author singles out a nice class of branching problems where each irreducible summand occurs discretely with finite multiplicity (admissible restrictions). Basic theory and new perspectives of admissible restrictions are presented from both analytic and algebraic view points. We also discuss some applications of admissible restrictions to modular varieties and L^{p}-harmonic analysis.

2000 Mathematics Subject Classification: 22E46,43A85,11F67,53C50,53D20. Keywords and Phrases:unitary representation, branching law, reductive Lie group.

1. Introduction

Let π be an irreducible unitary representation of a group G. A branching law is the irreducible decomposition of π when restricted to a subgroup G′:

π|_{G′} ≃ ∫ ⊕ ; Ĝ′ m_{π}(τ)τ dμ(τ)  (a direct integral).  (1.1)

Such a decomposition is unique, for example, if G′ is a reductive Lie group, and the multiplicity m_{π} : Ĝ′ → ℕ ∪ {∞} makes sense as a measurable function on the unitary dual Ĝ′.

Special cases of branching problems include (or reduce to) the followings: Clebsch-Gordan coefficients, Littlewood-Richardson rules, decomposition of tensor product representations, character formulas, Blattner formulas, Plancherel theorems for homogeneous spaces, description of breaking symmetries in quantum mechanics, theta-lifting in automorphic forms, etc. The restriction of unitary representations

秋の表現論シンポジウム 2002 年 11 月 12 日-15 日 富士ハイツ（世話人：佐

野茂、飯田正敏、本田龍央). この原稿は、 Proceedings of ICM2002, Beijing, vol 2, (2002)

615–627 に若干の加筆修正を行ったものである. serves also as a method to study discontinuous groups for non-Riemannian homogeneous spaces (e.g. [Mg, Oh]).

Our interest is in the branching problems for (non-compact) reductive Lie groups G ⊃ G′. In this generality, there is no known algorithm to find branching laws. Even worse, branching laws usually contain both discrete and continuous spectrum with possibly infinite multiplicities (the multiplicity is infinite, for example, in the decomposition of the tensor product of two principal series representations of SL(n, ℂ) for n ≥ 3, [Ge-Gr]).

The author introduced the notion of admissible restrictions and infinitesimal discrete decomposability in [Ko_{5}] and [Ko_{9}], respectively, seeking for a good framework of branching problems, in which we could expect especially a simple and detailed study of branching laws, which in turn might become powerful methods in other fields as well where restrictions of representations naturally arise.

The criterion in Theorem B indicates that there is a fairly rich examples of admissible restrictions; some are known and the others are new. In this framework, a number of explicit branching laws have been newly found (e.g. [D-Vs, Gr-W_{1,2}, Hu-P-S, Ko_{1,3,4,8}, Ko-Ø_{1,2}, Li_{2}, Lo_{1,2}, X]). The point here is that branching problems become accessible by algebraic techniques if there is no continuous spectrum.

The first half of this article surveys briefly a general theory of admissible restrictions both from analytic and algebraic view points (§2, §3). For the simplicity of exposition, we restrict ourselves to unitary representations, although a part of the theory can be generalized to non-unitary representations. The second half discusses some applications of discretely decomposable restrictions. The topics range from representation theory itself (§4) to some other fields such as L^{p}-analysis on nonsymmetric homogeneous spaces (§5) and topology of modular varieties (§6).

2. Admissible Restrictions to Subgroups

Let G′ be a subgroup of G, and π ∈ Ĝ. In light of (1.1), we introduce:

Definition 2.1. We say the restriction π|_{G′} is G′-admissible if it decomposes discretely and the multiplicity m_{π}(τ) is finite for any τ ∈ Ĝ′.

One can easily prove the following assertion:

Theorem A ([Ko_{5}, Theorem 1.2]). Let G ⊃ G′ ⊃ G′′ be a chain of groups, and π ∈ Ĝ. If the restriction π|_{G′′} is G′′-admissible, then π|_{G′} is G′-admissible.

Throughout this article, we shall treat the setting as below:

Definition 2.2. We say (G, G′) is a pair of reductive Lie groups if 1) G is a real reductive linear Lie group or its finite cover, and

2) G′ is a closed subgroup, and is reductive in G. Then, we shall fix maximal compact subgroups K ⊃ K′ of G ⊃ G′, respectively.

A typical example is a reductive symmetric pair (G, G′), by which we mean that G is as above and that G′ is an open subgroup of the set G^{σ} of the fixed points of an involutive automorphism σ of G. For example, (G, G′) = (GL(n, ℂ), GL(n, ℝ)), (SL(n, ℝ), SO(p, n − p)) are the cases.

Let (G, G′) be a pair of reductive Lie groups. Here are previously known examples of admissible restrictions:

Example 2.3. The restriction π|_{G′} is G′-admissible in the following cases: 1) (Harish-Chandra’s admissibility theorem) π ∈ Ĝ is arbitrary and G′ = K.

2) (Howe, [Ho_{1}]) π is the Segal-Shale-Weil representation of the metaplectic group G, and its subgroup G′ = G′_{1}G′_{2} forms a dual pair with G′_{1} compact.

In these examples, either the subgroup G′ or the representation π is very special, namely, G′ is compact or π has a highest weight. Surprisingly, without such assumptions, it can happen that the restriction π|_{G′} is G′-admissible. The following criterion asserts that the “balance” of G′ and π is crucial to the G′-admissibility.

Theorem B (criterion for admissible restrictions, [Ko_{7}]). Let G ⊃ G′ be a pair of reductive Lie groups, and π ∈ Ĝ. If

Cone(G′) ∩ AS_{K}(π) = {0},  (2.1)

then the restriction π|_{K′} is K′-admissible. In particular, the restriction π|_{G′} is G′-admissible, namely, decomposes discretely with finite multiplicity.

A main tool of the proof of Theorem B is the microlocal study of characters by using the singularity spectrum of hyperfunctions. The idea goes back to Atiyah, Howe, Kashiwara and Vergne [A, Ho_{2}, Ks-Vr] in the late ’70s. The novelty of Theorem B is to establish a framework of admissible restrictions with a number of new examples of interest, which rely on a deeper understanding of the unitary dual developed largely in the ’80s (see [Kn-Vo] and references therein).

Let us briefly explain the notation used in Theorem B. We write 𝔨′_{0} ⊂ 𝔨_{0} for the Lie algebras of K′ ⊂ K, respectively. Take a Cartan subalgebra 𝔱_{0} of 𝔨_{0} and fix a positive system Δ^{+}(𝔨, 𝔱) with dominant Weyl chamber \sqrt{−1}(𝔱_{0}^{∗})_{+}. Then, AS_{K}(π) is the asymptotic K-support of π ([Ks-Vr]), and Cone(G′) is defined as

Cone(G′) := \sqrt{−1}((𝔱_{0}^{∗})_{+} ∩ Ad^{∗}(K)(𝔨′_{0}^{⊥})).  (2.2)

By definition, both AS_{K}(π) and Cone(G′) are closed cones in \sqrt{−1}𝔱_{0}^{∗}.

Example 2.4. If G′ = K, then the assumption (2.1) is automatically fulfilled because Cone(G′) = {0}. The conclusion of Theorem B in this special case is nothing but Harish-Chandra’s admissibility theorem (Example 2.3 (1)).

To apply Theorem B for non-compact G′, we rewrite the assumption (2.1) more explicitly in specific settings. On the part Cone(G′), we mention:

Example 2.5. Cone(G′) is a linear subspace \sqrt{−1}(𝔱_{0}^{∗})^{−σ} (modulo the Weyl group) if (G, G′) is a reductive symmetric pair given by an involution σ. Here, we have chosen a Cartan subalgebra 𝔱_{0} to be maximally σ-split.

On the part AS_{K}(π), let us consider a unitary representation π_{λ} which is “attached to” an elliptic coadjoint orbit O_{λ} := Ad^{∗}(G)λ, in the orbit philosophy due to Kirillov-Kostant. This representation is a unitarization of a Zuckerman-Vogan module A_{𝔮}(λ) after some ρ-shift, and can be realized in the Dolbeault cohomology group on O_{λ} by the results of Schmid and Wong. (Here, we adopt the same polarization and normalization as in a survey [Ko_{4}, §2], for the geometric quantization O_{λ} ⇒ π_{λ}.) We note that π_{λ} ∈ Ĝ for “most” λ. Let 𝔤 = 𝔨 + 𝔭 be the complexification of a Cartan decomposition of the Lie algebra 𝔤_{0} of G. We set

Δ_{λ}^{+}(𝔭) := \{α ∈ Δ(𝔭, 𝔱) : ⟨λ, α⟩ > 0\}, for λ ∈ \sqrt{−1}𝔱_{0}^{∗}.

The original proof (see [Ko_{5}]) of the next theorem was based on an algebraic method without using microlocal analysis. Theorem B gives a simple and alternative proof.

Theorem C ([Ko_{5}]). Let π_{λ} ∈ Ĝ be attached to an elliptic coadjoint orbit O_{λ}. If

ℝ_{≥0} -span Δ_{λ}^{+}(𝔭) ∩ Cone(G′) = {0},  (2.2)

then the restriction π_{λ}|_{G′} is G′-admissible.

Let us illustrate Theorem C in Examples 2.6 and 2.7 for non-compact G′. For this, we note that a maximal compact subgroup K is sometimes of the form K_{1} × K_{2} (locally). This is the case if G/K is a Hermitian symmetric space (e.g. G = Sp(n, ℝ), SO^{∗}(2n), SU(p, q)). It is also the case if G = O(p, q), Sp(p, q), etc.

Example 2.6 (K ≃ K_{1} × K_{2}). Suppose K is (locally) isomorphic to the direct product group K_{1} × K_{2}. Then, the restriction π_{λ}|_{G′} is G′-admissible if λ|_{𝔱∩𝔨_{2}} = 0 and G′ ⊃ K_{1}. So does the restriction π|_{G′} if π is any subquotient of a coherent continuation of π_{λ}. This case was a prototype of G′-admissible restrictions π|_{G′} (where G′ is non-compact and π is a non-highest weight module) proved in 1988 by the author ([Ko_{1}; Ko_{2}, Proposition 4.1.3]), and was later generalized to Theorems B and C. Special cases include:

(1) K_{1} ≃ 𝕋, then π is a unitary highest weight module. The admissibility of the restrictions π|_{G′} in this case had been already known in ’70s (see Martens [Mt], Jakobsen-Vergne [J-Vr]).

(2) K_{1} ≃ SU(2), then π_{λ} is a quaternionic discrete series. Admissible restrictions π|_{G′} in this case are especially studied by Gross and Wallach [Gr-W_{1}] in ’90s.

(3) K_{1} ≃ O(q), U(q), Sp(q). Explicit branching laws of the restriction π_{λ}|_{G′} for singular λ are given in [Ko_{3}, Part I] with respect to the vertical inclusions of the diagram below (see also [Ko_{1}, Ko_{5}] for branching laws to horizontal inclusions).

O(4p, 4q)  ⊃  U(2p, 2q)  ⊃  Sp(p, q)
∪  ∪  ∪
O(4r) × O(4p − 4r, 4q) ⊃ U(2r) × U(2p − 2r, 2q) ⊃ Sp(r) × Sp(p − r, q)

Example 2.7 (conformal group). There are 18 series of irreducible unitary representations of G := U(2, 2) with regular integral infinitesimal characters. Among them, 12 series (about “67% ” !) are G′-admissible when restricted to G′ := Sp(1, 1).

The assumption in Theorem B is in fact necessary. By using the technique of symplectic geometry, the author proved the converse statement of Theorem B:

Theorem D ([Ko_{13}]). Let G ⊃ G′ be a pair of reductive Lie groups, and π ∈ Ĝ. If the restriction π|_{K′} is K′-admissible, then Cone(G′) ∩ AS_{K}(π) = {0}.

3. Infinitesimal Discrete Decomposability

The definition of admissible restrictions (Definition 2.1) is “analytic”, namely, based on the direct integral decomposition (1.1) of unitary representations. Next, we consider discrete decomposable restrictions by a purely algebraic approach.

Definition 3.1 ([Ko_{9}, Definition 1.1]). Let 𝔤 be a Lie algebra. We say a 𝔤-module X is discretely decomposable if there is an increasing sequence of 𝔤-submodules of finite length:

X = ⋃_{m=0}^{∞} X_{m}, X_{0} ⊂ X_{1} ⊂ X_{2} ⊂ · · · .  (3.1)

We note that dim X_{m} = ∞ in most cases below.

Next, consider the restriction of group representations.

Definition 3.2. Let G ⊃ G′ be a pair of reductive Lie groups, and π ∈ Ĝ. We say that the restriction π|_{G′} is infinitesimally discretely decomposable if the underlying (𝔤, K)-module π_{K} is discretely decomposable as a 𝔤′-module.

The terminology “discretely decomposable” is named after the following fact:

Theorem E ([Ko_{9}]). Let (G, G′) be a pair of reductive Lie groups, and π_{K} the underlying (𝔤, K)-module of π ∈ Ĝ. Then (i) and (ii) are equivalent:

i) The restriction π|_{G′} is infinitesimally discretely decomposable.

ii) The (𝔤, K)-module π_{K} has a discrete branching law in the sense that π_{K} is isomorphic to an algebraic direct sum of irreducible (𝔤′, K′)-modules.

Moreover, the following theorem holds:

Theorem F (infinitesimal ⇒ Hilbert space decomposition; [Ko_{11}]). Let π ∈ Ĝ. If the restriction π|_{G′} is infinitesimally discretely decomposable, then the restriction π|_{G′} decomposes without continuous spectrum:

π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈Ĝ′ \end{matrix}}m_{π}(τ)τ (a discrete direct sum of Hilbert spaces).  (3.2)

At this stage, the multiplicity m_{π}(τ) := dim Hom_{G′}(τ, π|_{G′}) can be infinite. However, for a reductive symmetric pair (G, G′), it is likely that the multiplic-

ity of discrete spectrum is finite under the following assumptions, respectively. (3.3) π is a discrete series representation for G. (3.4) The restriction π|_{G′} is infinitesimally discretely decomposable.

Conjecture 3.3 (Wallach, [X]). m_{π}(τ) < ∞ for any τ ∈ Ĝ′ if (3.3) holds.

Conjecture 3.4 ([Ko_{11}, Conjecture C]). m_{π}(τ) < ∞ for any τ ∈ Ĝ′ if (3.4) holds.

We note that Conjecture 3.4 for compact G′ corresponds to Harish-Chandra’s admissibility theorem. A first affirmative result for general non-compact G′ was given in [Ko_{9}], which asserts that Conjecture 3.4 holds if π is attached to an elliptic coadjoint orbit. A special case of this assertion is:

Theorem G ([Ko_{9}]). m_{π}(τ) < ∞ for any τ ∈ Ĝ′ if both (3.3) and (3.4) hold.

In particular, Wallach’s Conjecture 3.3 holds in the discretely decomposable case. We note that an analogous finite-multiplicity statement fails if continuous spectrum occurs in the restriction π|_{G′} for a reductive symmetric pair (G, G′):

Counter Example 3.5 ([Ko_{11}]). m_{π}(τ) can be ∞ if neither (3.3) nor (3.4) holds.

Recently, I was informed by Huang and Vogan that they proved Conjecture 3.4 for any π [Hu-Vo].

A key step of Theorem G is to deduce the K′-admissibility of the restriction π|_{K′} from the discreteness assumption (3.4), for which we employ Theorem H below. Let us explain it briefly. We write V_{𝔤}(π) for the associated variety of the underlying (𝔤, K)-module of π (see [Vo]), which is an algebraic variety contained in the nilpotent cone of 𝔤^{∗}. Let pr_{𝔤→𝔤′} : 𝔤^{∗} → (𝔤′)^{∗} be the projection corresponding to 𝔤′ ⊂ 𝔤. Here is a necessary condition for infinitesimal discrete decomposability:

Theorem H (criterion for discrete decomposability [Ko_{9}, Corollary 3.4]). Let π ∈ Ĝ. If the restriction π|_{G′} is infinitesimally discretely decomposable, then pr_{𝔤→𝔤′}(V_{𝔤}(π)) is contained in the nilpotent cone of (𝔤′)^{∗}.

We end this section with a useful information on irreducible summands.

Theorem I (size of irreducible summands, [Ko_{9}]). Let π ∈ Ĝ. If the restriction π|_{G′} is infinitesimally discretely decomposable, then any irreducible summand has the same associated variety, especially, the same Gelfand-Kirillov dimension.

Here is a special case of Theorem I:

Example 3.6 (highest weight modules, [N-Oc-T]). Let G be the metaplectic group, and G′ = G′_{1}G′_{2} is a dual pair with G′_{1} compact. Let θ(σ) be an irreducible unitary highest weight module of G′_{2} obtained as the theta-correspondence of σ ∈ Ĝ′_{1}. Then the associated variety of θ(σ) does not depend on σ, but only on G′_{1}.

An analogous statement to Theorem I fails if there exists continuous spectrum in the branching law π|_{G′} (see [Ko_{11}] for counter examples).

4. Applications to Representation Theory

So far, we have explained basic theory of discretely decomposable restrictions of unitary representations for reductive Lie groups G ⊃ G′. Now, we ask what discrete decomposability can do for representation theory. Let us clarify advantages of admissible restrictions, from which the following applications (and some more) have been brought out and seem to be promising furthermore.

1) Study of Ĝ′ as irreducible summands of π|_{G′}.

2) Study of Ĝ by means of the restrictions to subgroups G′.

3) Branching laws of their own right.

4.1. From the view point of the study of Ĝ′ (smaller group), one of advantages of admissible restrictions is that each irreducible summand of the branching law π|_{G′} gives an explicit construction of an element of Ĝ′.

Historically, an early success of this idea (in ’70s and ’80s) was the construction of irreducible highest weight modules (Howe, Kashiwara-Vergne, Adams, ⋯). A large part of these modules can be constructed as irreducible summands of discrete branching laws of the Weil representation (see Examples 2.3 (2) and 3.6).

This idea works also for non-highest weight modules. As one can observe from the criterion in Theorem B, the restriction π|_{G′} tends to be discretely decomposable, if AS_{K}(π) is “small”. In particular, if π is a minimal representation in the sense that its annihilator is the Joseph ideal, then a result of Vogan implies that AS_{K}(π) is one dimensional. Thus, there is a good possibility of finding subgroups G′ such that π|_{G′} is G′-admissible. This idea was used to construct “small” representations of subgroups G′ by Gross-Wallach [Gr-W_{1}]. In the same line, discretely decomposable branching laws for non-compact G′ are used also in the theory of automorphic forms for exceptional groups by J-S. Li [Li_{2}].

4.2. From the view point of the study of Ĝ (larger group), one of advantages of admissible restrictions is to give a clue to a detailed study of representations of G by means of discrete branching laws.

Needless to say, an early success in this direction is the theory of (𝔤, K)- modules (Lepowsky, Harish-Chandra, ⋯). The theory relies heavily on Harish-Chandra’s admissibility theorem (Example 2.3 (1)) on the restriction of π to K.

Instead of a maximal compact subgroup K, this idea applied to a non-compact subgroup G′ still works, especially in the study of “small” representations of G. In particular, this approach makes sense if the K-type structure is complicated but the G′-type structure is less complicated. Successful examples in this direction include:

1) To determine an explicit condition on λ such that a Zuckerman-Vogan module A_{𝔮}(λ) is non-zero, where we concern with the parameter λ outside the good range. In the setting of Example 2.6 (3), the author found in [Ko_{2}] a combinatorial formula on K_{1}-types of A_{𝔮}(λ) and determined explicitly when A_{𝔮}(λ) ≠ 0. The point here is that the computation of K-types of A_{𝔮}(λ) is too complicated to carry out because a lot of cancellation occurs in the generalized Blattner formula, while K_{1}-type formula (or G′-type formula for some non-compact subgroup G′) behaves much simpler in this case.

2) To study a fine structure of standard representations. For example, Lee and Loke [Le-Lo] determined the Jordan-Hölder series and the unitarizability of subquotients of certain degenerate non-unitary principal series representations π, by using G′-admissible restrictions for some non-compact reductive subgroup G′. Their method works successfully even in the case where K-type multiplicity of π is not one.

4.3. From the view point of finding explicit branching law, an advantage of admissible restrictions is that one can employ algebraic techniques because of the lack of continuous spectrum. A number of explicit branching laws are newly found (e.g. [D-Vs, Gr-W_{1,2}, Hu-P-S, Ko_{1,3,4,8}, Ko-Ø_{1,2}, Li_{2}, Lo_{1,2}, X]) in the context of admissible restrictions to non-compact reductive subgroups. A mysterious feature is that “different series” of irreducible representations may appear in discretely decomopsable branching laws (see [Ko_{5}, p.184] for a precise meaning), although all of them have the same Gelfand-Kirillov dimensions (Theorem I).

5. New Discrete Series for Homogeneous Spaces

Let G ⊃ H be a pair of reductive Lie groups. Then, there is a G-invariant Borel measure on the homogeneous space G/H, and one can define naturally a unitary representation of G on the Hilbert space L^{2}(G/H).

Definition 5.1. We say π is a discrete series representation for G/H, if π ∈ Ĝ is realized as a subrepresentation of L^{2}(G/H).

A discrete series representation corresponds to a discrete spectrum in the Plancherel formula for the homogeneous space G/H. One of basic problems in non-commutative harmonic analysis is:

Problem 5.2. 1) Find a condition on the pair of groups (G, H) such that there exists a discrete series representation for the homogeneous space G/H.

2) If exist, construct discrete series representations.

Even the first question has not found a final answer in the generality that (G, H) is a pair of reductive Lie groups. Here are some known cases:

Example 5.3. Flensted-Jensen, Matsuki and Oshima proved in ’80s that discrete series representations for a reductive symmetric space G/H exist if and only if

rank G/H = rank K/(H ∩ K).  (5.1)

This is a generalization of Harish-Chandra’s condition, rank G = rank K, for a group manifold G × G/ diag(G) ≃ G ([FJ, Mk-Os]).

Our strategy to attack Problem 5.2 for more general (non-symmetric) homogeneous spaces G/H consists of two steps:

1) To embed G/H into a larger homogeneous space G̃/H̃, on which harmonic analysis is well-understood (e.g. symmetric spaces).

2) To take functions belonging to a discrete series representation H (↪ L^{2}(G̃/H̃)), and to restrict them with respect to a submanifold G/H (↪ G̃/H̃). If G/H is “generic”, namely, a principal orbit in G̃/H̃ in the sense of Richard-

son, then it is readily seen that discrete spectrum of the branching law π|_{G} gives a discrete series for G/H ([Ko_{10}, §8]; see also [Hu, Ko_{1,5}, Li_{1}] for concrete examples).

However, some other interesting homogeneous spaces G/H occur as nonprincipal orbits on G̃/H̃, where the above strategy does not work in general. A remedy for this is to impose the admissibility of the restriction of π, which justifies the restriction of L^{p}-functions to submanifolds, and then gives rise to many non-symmetric homogeneous spaces that admit discrete series representations. For example, let us consider the case where G = G̃^{τ} and H = G̃^{σ} for commuting involutive automorphisms τ and σ of G̃ such that G̃/H̃ satisfies (5.1). Then by using Theorem C and an asymptotic estimate of invariant measures [Ko_{6}], we have:

Theorem J (discrete series for non-symmetric spaces, [Ko_{10}]). Assume that there is w ∈ W_{σ} such that

ℝ_{+} -span Δ^{+}(𝔭)_{σ,w} ∩ \sqrt{−1}(𝔱_{0}^{∗})^{−τ} = \{0\}.  (5.2)

Then there exist infinitely many discrete series representations for any homogeneous space of G that goes through xH̃ ∈ G̃/H̃ for any x ∈ K̃.

We refer to [Ko_{10}, Theorem 5.1] for definitions of a finite group W_{σ} and Δ^{+}(𝔭)_{σ,w}. The point here is that the condition (5.2) can be easily checked.

For instance, if G ≃ Sp(2n, ℝ) ≃ G̃/H̃ (a group manifold), then Theorem J implies that there exist discrete series on all homogeneous spaces of the form:

G/H = Sp(2n, ℝ)/(Sp(n_{0}, ℂ) × GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ)), (∑ n_{i} = n).

The choice of x in Theorem J corresponds to the partition (n_{0}, n_{1}, …, n_{k}). We note that the above G/H is a symmetric space if and only if n_{1} = n_{2} = ⋯ = n_{k} = 0.

The restriction of unitary representations gives new methods even for symmetric spaces where harmonic analysis has a long history of research. Let us state two results that are proved by the theory of discretely decomposable restrictions.

Theorem K (holomorphic discrete series for symmetric spaces). Suppose G/H is a non-compact irreducible symmetric space. Then (i) and (ii) are equivalent:

i) There exist unitary highest weight representations of G that can be realized as subrepresentations of L^{2}(G/H).

ii) G/K is Hermitian symmetric and H/(H ∩ K) is its totally real submanifold.

This theorem in the group manifold case is a restatement of Harish-Chandra’s well-known result. The implication (ii) ⇒ (i) was previously obtained by a different geometric approach (’Olafsson-Ørsted [Ol-Ø]). Our proof uses a general theory of discretely decomposable restrictions, especially, Theorems B, H and J.

Theorem L (exclusive law of discrete spectrum for restriction and induction). Let G/G′ be a non-compact irreducible symmetric space, and π ∈ Ĝ. Then both (1) and (2) cannot occur simultaneously.

1) The restriction π|_{G′} is infinitesimally discretely decomposable.

2) π is a discrete series representation for the homogeneous space G/G′.

We illustrate Theorems K and L by G = SL(2, ℝ). In this special case, the examples below are well-known results on harmonic analysis, however, the point here is that we can give a new proof of them by a simple idea coming from restrictions of unitary representations.

Example 5.4. 1) Holomorphic discrete series exist for G/H = SL(2, ℝ)/SO(1, 1) (a hyperboloid of one sheet). This is explained by Theorem K because the geodesic H/(H ∩ K) is obviously totally real in the Poincaré disk G/K = SL(2, ℝ)/SO(2).

2) There is no discrete series for the Poincaré disk G/K = SL(2, ℝ)/SO(2). This fact is explained by Theorem L because any representation of G is obviously discretely decomposable when restricted to a compact K.

6. Modular Varieties, Vanishing Theorem

Retain the setting as in Definition 2.2. Let Γ′ ⊂ Γ be cocompact torsion-free discrete subgroups of G′ ⊂ G, respectively. For simplicity, let G′ be a semisimple Lie group without compact factors. Then, both of the double cosets X := Γ\G/K and Y := Γ′\G′/K′ are compact, orientable, locally Riemannian symmetric spaces. Then, the inclusion G′ ↪ G induces a natural map ι : Y → X. The image ι(Y) defines a totally geodesic submanifold in X. Consider the induced homomorphism of the homology groups of degree m := dim Y,

ι_{∗} : H_{m}(Y; ℤ) → H_{m}(X; ℤ).

The modular symbol is defined to be the image ι_{∗}[Y] ∈ H_{m}(X; ℤ) of the fundamental class [Y] ∈ H_{m}(Y; ℤ). Though its definition is simple, the understanding of modular symbols is highly non-trivial.

Let us first recall some results of Matsushima-Murakami and Borel-Wallach on the de Rham cohomology group H^{∗}(X; ℂ) summarized as:

H^{∗}(X; ℂ) = ⨁_{π∈Ĝ} H^{∗}(X)_{π}, H^{∗}(X)_{π} := Hom_{G}(π, L^{2}(Γ\G))⊗H^{∗}(𝔤, K; π_{K}). (6.1)

The above result describes the topology of a single X by means of representation theory. For the topology of the pair (Y, X), we need restrictions of representations:

Theorem M (vanishing theorem for modular symbols, [Ko-Od]). If

AS_{K}(π) ∩ Cone(G′) = {0}, π ≠ 1,

then the modular symbol ι_{∗}[Y] is annihilated by the π-component H^{m}(X)_{π} in the perfect paring H^{m}(X; ℂ) × H_{m}(X; ℂ) → ℂ.

Theorem M determines, for example, the middle Hodge components of totally real modular symbols of compact Clifford-Klein forms of type IV domains.

The discreteness of irreducible decompositions plays a crucial role both in Matsushima-Murakami’s formula (6.1) and in a vanishing theorem for modular varieties (Theorem M). In the former L^{2}(Γ\G) is G-admissible (Gelfand and Piateski-Shapiro), while the restriction π|_{G′} is G′-admissible (cf. Theorem B) in the latter.

References

[A] M. F. Atiyah, The Harish-Chandra character, London Math. Soc. Lecture Note Series 34 (1979), 176–181.

[D-Vs] M. Duflo and J. Vargas, in preparation.

[FJ] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Annals of Math. 111 (1980), 253–311.

[Ge-Gv] I. M. Gelfand and M. I. Graev, Geometry of homogeneous spaces, representations of groups in homogeneous spaces, and related questions of integral geometry, Transl. II. Ser., A. M. S. 37 (1964), 351–429.

[Gr-W_{1}] B. Gross and N. Wallach, A distinguished family of unitary representations for the exceptional groups of real rank = 4, Progress in Math. 123 (1994), Birkhäuser, 289–304.

[Gr-W_{2}] B. Gross and N. Wallach, Restriction of small discrete series representations to symmetric subgroups, Proc. Sympos. Pure Math. 68 (2000), A.M.S., 255–272.

[Ho_{1}] R. Howe, θ-series and invariant theory, Proc. Sympos. Pure Math. 33

(1979), A.M.S., 275–285.

[Ho_{2}] R. Howe, Wave front sets of representations of Lie groups, Automorphic

forms, representation theory, and arithmetic (1981), Tata, 117–140.

[Hu] J-S. Huang, Harmonic analysis on compact polar homogeneous spaces, Pacific J. Math. 175 (1996), 553—569.

[Hu-P-S] J-S. Huang, P. Pandžić, and G. Savin, New dual pair correspondences,

Duke Math. 82 (1996), 447–471.

[Hu-Vo] J-S. Huang and D. Vogan, personal communications (2001). [J-Vr] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomor-

phic representations, J. Funct. Anal. 34 (1979), 29–53.

[Ks-Vr] M. Kashiwara and M. Vergne, K-types and singular spectrum, Lect.

Notes in Math., vol. 728, Springer, 1979, pp. 177–200.

[Kn-Vo] A. Knapp and D. Vogan, Jr., Cohomological Induction and Unitary Rep-

resentations, Princeton U.P., 1995.

[Ko_{1}] T. Kobayashi, Unitary representations realized in L^{2}-sections of vector bundles over semi-simple symmetric spaces, Proc. of the 27-28th Symp. of Funct. Anal. and Real Anal. (1989), Math. Soc. Japan, 39–54. (Japanese)

[Ko_{2}] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p−m, q; 𝔽), Memoirs of A.M.S., vol. 462, 1992.

[Ko_{3}] T. Kobayashi, The restriction of A_{𝔮}(λ) to reductive subgroups, Part I,

Proc. Japan Acad. 69 (1993), 262–267; Part II, ibid. 71 (1995), 24–26.

[Ko_{4}] T. Kobayashi, Harmonic analysis on homogeneous manifolds of reductive type and unitary representation theory, Transl., Series II, Selected Papers on Harmonic Analysis, Groups, and Invariants 183 (1998), A.M.S., 1–31.

[Ko_{5}] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205.

[Ko_{6}] T. Kobayashi, Invariant measures on homogeneous manifolds of reductive

type, J. reine und angew. Math. 490 (1997), 37–53.

[Ko_{7}] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups II — micro-local analysis and asymptotic K-support, Annals of Math. 147 (1998), 709–729.

[Ko_{8}] T. Kobayashi, Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu (K. Mimachi, ed.), 1997, pp. 7–13.

[Ko_{9}] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups III — restriction of Harish-Chandra modules and associated varieties, Invent. Math. 131 (1998), 229–256.

[Ko_{10}] T. Kobayashi, Discrete series representations for the orbit spaces arising from two involutions of real reductive Lie groups, J. Funct. Anal. 152 (1998), 100–135.

[Ko_{11}] T. Kobayashi, Discretely decomposable restrictions of unitary representations of reductive Lie groups —examples and conjectures, Adv. Stud. Pure Math. 26 (2000), 99-127.

[Ko_{12}] T. Kobayashi, Theory of discrete decomposable branching laws of unitary representations of semisimple Lie groups and some applications, Sugaku

Exposition, Transl. Ser., A.M.S. (to appear).

[Ko_{13}] T. Kobayashi, Admissible restrictions of unitary representations of reductive Lie groups —— symplectic geometry and discretely decomposable branching laws, in preparation.

[Ko-Od] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on

locally symmetric spaces, Comment. Math. Helvetici 73 (1998), 45–70.

[Ko-Ø_{1}] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits, C. R. Acad. Sci. Paris 326 (1998), 925–930.

[Ko-Ø_{2}] T. Kobayashi and B. Ørsted, Analysis on the minimal representation of

O(p,q), I, II, III, Adv. Math. (to appear).

[Le-Lo] S-T. Lee and H-Y. Loke, Degenerate principal series of U(p,q) and Spin(p,q),

Compositio Math. (to appear).

[Li_{1}] J-S. Li, On the discrete series of generalized Stiefel manifolds, Trans.

A.M.S. 340 (1993), 753–766.

[Li_{2}] J-S. Li, Two reductive dual pairs in groups of type E, Manuscripta Math.

91 (1996), 163–177.

[Lo_{1}] H-Y. Loke, Restrictions of quaternionic representations, J. Funct. Anal.

172 (2000), 377–403.

[Lo_{2}] H-Y. Loke, Howe quotients of unitary characters and unitary lowest

weight modules, preprint.

[Mg] G. Margulis, Existence of compact quotients of homogeneous spaces, measurably proper actions, and decay of matrix coefficients, Bul. Soc. Math. France 125 (1997), 1–10.

[Mt] S. Martens, The characters of the holomorphic discrete series, Proc. Nat. Acad. Sci. USA 72 (1975), 3275-3276.

[Mk-Os] T. Matsuki and T. Oshima, A description of discrete series for semisim-

ple symmetric spaces, Adv. Stud. Pure Math. 4 (1984), 331–390.

[N-Oc-T] K. Nishiyama, H. Ochiai, and K. Taniguchi, Bernstein degree and associated cycles of Harish-Chandra modules — Hermitian symmetric case, Asterisque 273 (2001), 13–80.

[Oh] H. Oh, Tempered subgroups and representations with minimal decay of matrix coefficients, Bull. Soc. Math. France 126 (1998), 355–380.

[Ol-Ø] G. Ólafsson and B. Ørsted, The holomorphic discrete series of an affine

symmetric space, I, J. Funct. Anal. 81 (1988), 126–159.

[Ø-Vs] B. Ørsted and J. Vargas, Restriction of square integrable representations:

discrete spectrum, preprint (2002).

[Vo] D. Vogan, Jr., Associated varieties and unipotent representations, Progress in Math. 101 (1991), Birkhäuser, 315–388.

[Vo-Z] D. Vogan, Jr. and G. Zuckerman, Unitary representations with non-zero

cohomology, Compositio Math. 53 (1984), 51–90.

[X] J. Xie, Restriction of discrete series of SU(2, 1) to S(U(1) × U(1, 1)), J.

14 T. Kobayashi Funct. Anal. 122 (1994), 478–518, ph.D. dissertation, Rutgers University.

RIMS, Kyoto University, Kyoto, 606-8502 JAPAN. E-mail address: toshi@kurims.kyoto-u.ac.jp