THEORY OF DISCRETELY DECOMPOSABLE

RESTRICTIONS OF UNITARY REPRESENTATIONS OF

SEMISIMPLE LIE GROUPS AND SOME APPLICATIONS^{∗}

Toshiyuki KOBAYASHI

RIMS, Kyoto University

Abstract. Branching problems ask how an irreducible representation of a group decomposes when restricted to a subgroup. This exposition surveys new aspects on branching problems of unitary representations of reductive Lie groups.

The first half is written from the representation theoretic viewpoint. After an observation of wild feature of branching problems to non-compact subgroups in a general setting, we introduce the notion of admissible restrictions as a good framework that enjoys two properties: finiteness of multiplicities and discreteness of spectrum. A criterion for admissible restrictions is presented, of which the idea of proof stems from microlocal analysis and algebraic geometry. In this framework, we present a finite multiplicity theorem. Furthermore, an exclusive law of discrete spectrum is formulated for inductions and restrictions.

The second half deals with applications. Once we know the non-existence of continuous spectrum in the restrictions, we could expect an algebraic approach to branching problems. In this framework, new branching formulas have been recently obtained in various settings, among which we present an example, namely, a generalization of the Kostant-Schmid formula to non-compact subgroups. Finally, we mention some applications of discretely decomposable branching laws to other fields of mathematics. The topics include

(1) topological properties of modular varieties in locally symmetric spaces,

(2) a construction of new discrete series representations for non-Riemannian nonsymmetric homogeneous spaces. We end the exposition by a brief discussion on the mystery between tessellation

of non-Riemannian homogeneous spaces and branching problems of unitary representations.

2000 Mathematics Subject Classification: 22E46, 43A85, 11F67, 53C50, 53D20. Keywords and Phrases: unitary representation, branching law, reductive Lie group, restriction, discrete series, homogeneous space, semisimple Lie group.

∗ Plenary lecture at the Mathematical Society of Japan on the occasion of the Spring Prize of 1999. The original article was published in “Sugaku”, Mathematical Society of Japan (1999) vol. 51, pp. 337–356.

Typeset by AMS-TEX

Table of Contents

§0. Introduction §1. Analytic theory of admissible restrictions 1.A. Branching laws of unitary representations 1.B. Discretely decomposable restrictions as a “nice framework” 1.C. A sufficient condition for discretely decomposable restrictions §2. Algebraic theory of discretely decomposable restrictions 2.A. Algebraic reformulation of discrete decomposability 2.B. A necessary condition for discretely decomposable restrictions

— an approach by associated varieties

2.C. Three more theorems on discretely decomposable restrictions §3. New aspect of representation theory related to branching problems 3.A. Finite multiplicity conjectures 3.B. A generalization of the Kostant-Schmid formula to semisimple symmetric

pairs 3.C. Unipotent representations and discrete branching laws 3.D. Quantization of elliptic orbits and discrete branching laws §4. Applications of admissible restrictions 4.A. Branching laws and geometry 4.B. A vanishing theorem for modular varieties 4.C. Construction of new discrete series representations for non-symmetric spaces 4.D. Discontinuous groups versus restrictions of unitary representations

§0. Introduction

In my opinion, one of the most fascinating features in representation theory of Lie groups arises from the “outside”, namely, through various interactions with different fields of mathematics and physics, including partial differential equations, differential geometry, algebraic geometry, functional analysis, combinatorics, and number theory, etc. Furthermore, such interactions are still growing actively and sometimes show up unexpectedly.

However, if we look at the “inside” of representation theory itself by forgetting interactions with other branches of mathematics, what remains as central problems? From the view point of “analysis and synthesis”, we may emphasize the following two problems: Problem 1.

Understand irreducible representations.

Find and classify “smallest” objects. Problem 2.

Decompose a given representation into irreducible ones.

How is a given representation built from “smallest” objects? In traditional chemistry or physics of condensed matter, Problem 1 would cor-

respond to the “classification of atoms” [or elementary particles, ⋯] (the level depends on what we regard as “smallest”); while Problem 2 would correspond to the “analysis and synthesis” of molecules [or of atoms, ⋯].

Let us consider Problems 1 and 2 for Lie groups and their representation theory.

First, the “smallest objects” for Lie groups should be simple Lie groups such as SL(n, ℝ) and SU(p, q), and one dimensional Abelian Lie groups such as ℝ and S^{1}. Simple Lie groups (or slightly more generally, reductive Lie groups) are the groups that we shall deal with throughout this article. Simple Lie groups were infinitesimally classified by É. Cartan (1894 for complex Lie groups, 1914 for real Lie groups) after a pioneering work of Killing from 1888 to 1890. We recall that semisimple Lie groups are locally isomorphic to the direct product of simple Lie groups; reductive Lie groups are locally isomorphic to the direct product of semisimple Lie groups and Abelian Lie groups.

Next, let us consider Problems 1 and 2 for “representations”. Then, the smallest objects should be irreducible representations (we need to use an appropriate category of representations because there are subtle topological problems in dealing with infinite dimensional representations). Problem 1 asks the classification of irreducible (unitary) representations, which contains the following subproblems:

· Construction of irreducible representations.

· Finding a complete set of invariants of representations, so that they can separate different irreducible representations from one another.

· Understanding these invariants.

A classical example of invariants of representations π is the character Trace(π). If π is infinite dimensional, then the character Trace(π) is no more a continuous function on a group G in general. Harish-Chandra justified it as a distribution for a suitable class of representations π (e.g. π is an irreducible unitary representation of a reductive Lie group G). The asymptotic K-support AS_{K}(π) and the associated variety V_{𝔤}(π) are also useful invariants of representations of a reductive Lie group G, which we shall explain and apply in branching problems in §1 and §2, respectively.

The classification of irreducible unitary representations of simple Lie groups has been a long-standing problem. More than half a century has past since a pioneering work of Bargmann and the Gel’fand school in 1940s, and there has been a large development by Vogan and some others, particularly in 1980s (we refer to the textbook [26] by Knapp and Vogan for a guide of some recent literatures). The unitary dual has been classified for some groups such as GL(n, 𝔽) (𝔽 = ℝ, ℂ, ℍ), however, has not been classified for some other groups such as O(p, q) and Sp(n, ℝ) (p, q, n ≥ 3).

Second, let us consider Problem 2. We begin with some examples of the decomposition of representations. They are closely related to classical mathematical problems such as:

1) Spectral theory of unitary operators. This is equivalent to the irreducible decomposition of a unitary representation of ℤ on a Hilbert space.

2) The theory of reduction of matrices to Jordan’s normal forms of matrices. This corresponds to the decomposition of finite dimensional representations of ℤ on ℂ^{n}.

3) The Fourier transform. We may regard this as the irreducible decomposition of the regular representation of the Abelian Lie group ℝ on L^{2}(ℝ).

4) The Fourier series expansion. We may regard this as the irreducible decomposition of the regular representation of the torus group S^{1} on L^{2}(S^{1}).

All of the above examples correspond to the decompositions of representations of Abelian groups, ℤ, ℝ and S^{1}. How about non-Abelian groups, such as SL(n, ℝ)?

We consider two important settings where questions of decomposing representations arise naturally: Let G be a group, and G′ its subgroup.

Problem 2-A (Decomposition of induction) Given an irreducible representation τ of a subgroup G′, decompose the induced representation Ind_{G′}^{G} τ into irreducibles of G. Problem 2-B (Decomposition of restriction) Given an irreducible representation π of G, decompose the restriction π|_{G′} into irreducibles of a subgroup G′.

For a compact G, these two problems are related to each other by the Frobenius reciprocity. For a non-compact G, we do not know a strong analog of the Frobenius reciprocity, however, the comparison of these problems may help us to get a feeling of the current status on them. (So, we shall compare these problems occasionally in this article.)

Problem 2-A corresponds to the Plancherel type theorem for the homogeneous space G/G′ if τ = 1 (the one dimensional trivial representation), namely, to find the formula of the irreducible decomposition of the regular representation L^{2}(G/G′). For general τ, Problem 2-A deals with L^{2}-harmonic analysis on a Gequivariant vector bundle over the homogeneous space G/G′.

The formula of the irreducible decomposition in Problem 2-B is called a branching law. The decomposition of the tensor product of two representations is an example of branching laws. Branching laws for certain groups arise in quantum mechanics as a description of breaking symmetries.

In this paper, we shall focus on Problem 2-B, namely, on branching problems. We are interested in the branching problem of the restriction π|_{G′} in a general setting where both G and G′ are reductive Lie groups and π is an irreducible unitary representation. This setting contains many important cases indeed but is perhaps too general to expect strong results (at least, now). Our initial project is to single out a nice category of branching problems, in which we could study deeply and explicitly the restriction of unitary representations. For this purpose, we shall observe some of major difficulties at present in branching problems in a general setting (see §1.A).

In order to clarify our viewpoint of this article, we begin with an elementary example of branching laws of finite dimensional representations: Let ℂ[x, y, z] be the polynomial ring of three variables. We write P_{k} for its subspace consisting of homogeneous polynomials of degree k. For instance, dim_{ℂ} P_{3} = 10. We arrange elements of P_{3} in the descending order of z^{j} as follows:

P_{3} = ℂ{z^{3}} + ℂ{xz^{2}, yz^{2}} + ℂ{x^{2}z, xyz, y^{2}z} + ℂ{x^{3}, x^{2}y, xy^{2}, y^{3}} (0.1)

≃ ℂ{z^{3}} + ℂ{x, y} ⊗ ℂ{z^{2}} + ℂ{x^{2}, xy, y^{2}} ⊗ ℂ{z} + ℂ{x^{3}, x^{2}y, xy^{2}, y^{3}}.

Then, G := GL(3, ℂ) acts on the left side of (0.1) naturally and irreducibly, and so does G′ := GL(2, ℂ) × GL(1, ℂ) on each summand of the right side. Thus, we can regard (0.1) as the branching law of the restriction of an irreducible representation of G with respect to the subgroup G′.

Let us pin down the dimensions of (0.1), by forgetting explicit representation spaces:

10 = 1 + 2 + 3 + 4
= (1 × 1) + (1 × 2) + (1 × 3) + (1 × 4).

The second equality indicates that each irreducible representation of G′ occurs with multiplicity free. We shall compare this formula with branching laws of infinite dimensional representations (see (0.4), (0.5) and (0.6) below).

Next, consider the direct sum decomposition of the Hilbert space L^{2}(S^{1}) by the Fourier series expansion:

(0.2)  L^{2}(S^{1}) ≃ \begin{matrix}∑ & ⊕ \\ n∈ℤ \end{matrix}ℂe^{\sqrt{−1}nx}, f(x) ↦ \{f̂(n)\}_{n∈ℤ}.

Let G = SL(2, ℝ). Then one can define an action of G on the left side of (0.2) as an irreducible unitary representation (a principal series representation) by

π(g) : L^{2}(ℝ) → L^{2}(ℝ), f(x) ↦ |cx + d|^{−1} f(\frac{ax + b}{cx + d})

where g^{−1} = ( \begin{matrix}a & b \\ c & d \end{matrix} ) via the identification

L^{2}(S^{1}) →^{∼} L^{2}(ℝ), F ↦ | cos \frac{θ}{2}|^{−1}F (tan \frac{θ}{2}).

Then we may interpret (0.2) as a branching law when restricted from G = SL(2, ℝ) to G′ = SO(2) (see [50], for detailed formulas).

Similarly, let us consider the direct integral decomposition of the Hilbert space L^{2}(ℝ) by the Fourier transform:

(0.3)  L^{2}(ℝ) ≃ ∫_{ℝ} ℂe^{\sqrt{−1}ξx}dξ f(x) ↦ f̃(ξ).

Then, we may interpret this formula as the branching law when restricted from SL(2, ℝ) to a unipotent subgroup G′ (consisting of strictly upper triangular matrices) which is isomorphic to ℝ. Namely, G acts on the left side L^{2}(ℝ) as an irreducible unitary representation, and the subgroup G′ acts irreducibly on each one dimensional representation ℂe^{\sqrt{−1}ξx} on the right side.

As a “coarse” information of (0.2) and (0.3), we pin down the multiplicities and the dimensions of irreducible components, respectively as follows:

Dimensions  Spectrum
(0.4) ∞ = ⋯ + (1 × 1) + (1 × 1) + (1 × 1) + ⋯  (purely discrete)
(0.5) ∞ = ∫_{ℝ}(1 × 1)dξ  (purely continuous)

Since our concern in this article is with non-Abelian and non-compact groups, we need to deal mostly with infinite dimensional representations. Then, the feature of the dimension formula (0.4) will be loosely stated in the following generalization:

(0.6)  ∞ = ⋯ + (finite × ∞) + (finite × ∞) + (finite × ∞) + · · · .

Here, the left side of (0.6) is controlled by G, and the right side indicates that each irreducible (infinite dimensional) representation of G′ occurs discretely with finite multiplicity. A branching formula with this feature will be called a G′- admissible restriction (see Definition 1.1).

The main theme of this paper is on admissible restrictions of unitary representations, namely, on branching laws without continuous spectrum and with discrete spectrum of finite multiplicity. We shall ask:

When does the restriction π|_{G′} become G′-admissible?

This is the case if G′ is a maximal compact subgroup (a fundamental theorem of Harish-Chandra; see Example 1.2). This is also often the case if π is a unitary highest weight representation (see Definition 3.5). Here, among irreducible unitary representations of a reductive group, unitary highest weight representations are rather special and have been studied extensively and understood best. Typical examples are holomorphic discrete series representations (see §3.B). The Segal-Shale-Weil representation splits into two irreducible representations of the metaplectic group Mp(n, ℝ) (the double covering group of the symplectic group Sp(n, ℝ)), and each of them is also a unitary highest weight representation. These representations are infinite dimensional, but it turns out that they are relatively “small” compared to non-highest weight representations. They have a nature of “one-sided infinity” something like the half line [0, ∞) which has the “bottom” 0. This is in contrast to a “both-sided infinity” (−∞, ∞). By this one-sided property, unitary highest weight modules π tend to be discretely decomposable when restricted to a subgroup (see Example 1.3, Theorem 3.6). We note that the minimum element (such as 0 in [0, ∞)) corresponds to a highest weight vector of π, which may be interpreted as a vacuum vector in quantum mechanics.

On the other hand, most of irreducible unitary representations are not “onesided”, namely, there are no highest weight vectors. In other words, unitary highest weight representations are rather rare among the unitary dual Ĝ. What shall we expect for the spectrum in the branching laws of “general” infinite dimensional representations ? Does it happen that the restriction π|_{G′} is G′-admissible?

As we have seen the irreducible decomposition of the regular representation of ℝ on L^{2}(ℝ) in (0.3), branching laws usually contain continuous spectrum, when restricted to non-compact subgroups. It is no wonder that most people did not pay attention on the possibility of the non-existence of continuous spectrum in the branching law of the restriction π|_{G′} in a general case where π is a non-highest weight representation and G′ is non-compact.

In 1988, inspired by the theory of discontinuous groups for pseudo-Riemannian homogeneous spaces (see [41], [84] for an exposition), I found explicit branching laws of some (non-holomorphic) discrete series representations with respect to noncompact subgroups. The branching laws are not very complicated and still discretely decomposable, and I was curious about a mysterious phenomenon of discrete decomposability even in such a general setting ([28]). My proof of the branching laws was based on the theory of harmonic analysis on semisimple symmetric spaces ([14], [68]) and vector bundles on them ([31]), and on algebraic theory of Zuckerman-Vogan’s derived functor modules ([74], [75], [76]), both of which developed largely in 1980s (see also the references in [26], [36]).

These new branching laws became the first breakthrough in our study of discretely decomposable restrictions ([31], [35], [36], [37], [44], [49], [50]). We shall explain its flavor in §4.D.5.

Different from our original methods in [28], we shall adopt in this article the approach of [37] and [46] where global analysis on homogeneous spaces has a relatively small role. That is, we shall study discretely decomposable restrictions as a problem inside representation theory (see §1, §2 and §3), and then apply the theory of restrictions as a method to study global analysis on homogeneous spaces (see §4.C).

This exposition is organized as follows. First, we formulate and give basic results on discrete decomposable branching laws from an analytic aspect (§1) and from an algebraic aspect (§2). Next, we explain some more results on discretely decomposable restrictions, possible directions for further developments and new perspectives of unitary representation theory relevant to branching problems in §3. In the latter half of this paper, we give an outline of some applications of discretely decomposable branching laws to other areas of mathematics. Most results here have been developed in the last five years. The applications explained in §4 range from number theory to discontinuous groups and to global analysis on homogeneous spaces. Each application in §4 can be read independently.

§1. Analytic theory of admissible restrictions

Throughout this paper, we shall assume that a reductive Lie group G is a linear group or its finite covering. Without loss of generality, we shall assume that a real reductive linear group G is realized as a closed subgroup of GL(N, ℝ) satisfying the following two conditions:

i) The number of connected components of G is at most finite.

ii) G is stable under the transpose operation of matrices (namely, ^{t}G = G). Here are classical examples of linear reductive Lie groups G:

G =GL(n, ℝ), SL(n, ℝ), O(p, q), U(p, q), Sp(p, q), Sp(n, ℝ),
SU^{∗}(2n), SO^{∗}(2n), GL(n, ℂ), SL(n, ℂ), SO(n, ℂ), Sp(n, ℂ).

Here, we note that GL(n, ℂ) can be realized in GL(2n, ℝ) such that it is stable under the transpose operation of 2n × 2n matrices.

Suppose G is a linear reductive Lie group satisfying the above conditions (i) and (ii). We put

K := G ∩ O(N).

Then, K is a maximal compact subgroup of G. We write 𝔤, 𝔨 for the Lie algebras of G, K, respectively. A reductive Lie group G is a semisimple Lie group if the center of 𝔤 is {0}.

Let Ĝ be the set of the (unitary) equivalence classes of irreducible unitary representations of G. Then Ĝ is called the unitary dual of G.

1.A. Branching laws of unitary representations.

Given π ∈ Ĝ and a subgroup G′ of G, we consider the problem of finding the branching law of the restriction π|_{G′}, namely, the decomposition formula of π into irreducible representations of G′.

For compact G, any irreducible unitary representation π is finite dimensional. Then, the branching problem is theoretically solvable in the sense that any particular case can be done, because there exists an algorithm to obtain branching laws, based on Weyl’s character formula. Of course, such an algorithm often involves complicated combinatorial problems.

On the other hand, for the branching law π|_{G′} of an infinite dimensional unitary representation π, even an algorithm has not been known in general if G ⊃ G′ are (non-compact) reductive Lie groups.

As a matter of fact, branching laws of unitary representations of semisimple Lie groups have not been studied systematically except for some special cases (it is another thing that some of special cases are already rich and very interesting). Here is an observation about difficulties to find branching laws of infinite dimensional unitary representations.

1) If π ∈ Ĝ is constructed as a usual induced representation (e.g. a principal series representation), then by using the classical theory of Mackey ([58]) the branching law is reduced to another (usually difficult) problem of harmonic analysis, that is, to find the Plancherel-type theorem for a homogeneous space (see Problem 2-A). Only recently, the latter problem has been solved under the assumption that the homogeneous space is a semisimple symmetric space^{1} (see [12], [68]). But, the homogeneous spaces arising from the branching problems are usually much more general than semisimple symmetric spaces and the Plancherel-type theorem for such spaces is far from being understood. (Even its subproblem such as Problem 4.C.1 is very difficult.)

2) Some of irreducible unitary representations of semisimple Lie groups cannot be realized as usual induced representations. Discrete series representations are the case. In this case, there is no known general method to find branching laws of the restriction π|_{G′} for non-compact G′. (We note that the Mackey theory does not work in this case.)

3) Branching laws when restricted to non-compact subgroups often contain both discrete and continuous spectrum. Usually, purely algebraic methods do not work if continuous spectrum occurs.

4) To be worse, multiplicities of irreducible unitary representations of G′ occurring in branching laws can be infinite even if G′ is a maximal reductive subgroup of G (e.g. (G, G′) is a semisimple symmetric pair), as we paid much attention in [28]. Infinite multiplicities in branching laws can happen even in the decomposition of tensor product representations, that is, the restriction with respect to a diagonally embedded group G_{1} in the direct product group G_{1} × G_{1}.

5) Fix a group G and its subgroup G′. By the Frobenius reciprocity, the following two problems are equivalent if G is compact:

i) To find branching laws of the restriction π|_{G′} for all π ∈ Ĝ. ii) To find irreducible decompositions of the induced representation Ind_{G′}^{G}(τ)

1Irreducible semisimple symmetric spaces were classified by M. Berger on the level of Lie algebras [6].

for all τ ∈ Ĝ′. Loosely, we might regard that the above two problems (i) and (ii) have compara-

ble difficulties also for non-compact G′. If so, let us compare our current knowledge on the problems (i) and (ii). As for (ii), very little has been studied in the case dim τ = ∞ (see [28], §1.3), although there are some successful cases, namely, where dim τ = 1 and (G, G′) is a symmetric pair, as we have already mentioned. In summary, (ii) is still far from being solved because we need to deal with all τ ∈ Ĝ′ which are mostly infinite dimensional representations. Likewise, (i) is far from being solved in general. Thus, we believe it is reasonable to attack branching problems (i) by limiting ourselves to more special and nicer cases.

1.B. Discretely decomposable restrictions as a “nice framework”.

As we have discussed so far, general branching problems involve too many, and too much different type of difficulties when dealing with infinite dimensional representations of reductive Lie groups.

So, our strategy is first to find a good framework among general branching problems, and then to initiate deeper and detailed study in this framework. Such a framework should cover at least some important cases of branching problems where Mackey’s classical theory does not apply (e.g. π is a discrete series representation of G). Furthermore, the following nature is desirable:

a) The framework is rich in new interesting examples, which are also useful in some applications.

b) In such a framework, we could find branching laws explicitly, or at least there exists an algebraic algorithm to find branching laws.

From this viewpoint, the author proposed the following Definition 1.1 in [35] and [37] with emphasis on the case of non-compact subgroups: Let G ⊃ G′ be reductive Lie groups, and π ∈ Ĝ. We define the multiplicity of τ ∈ Ĝ′ in the discrete spectrum of the restriction π|_{G′} by the dimension of continuous G′-intertwining operators:

m_{π}(τ) := dim Hom_{G′}(τ, π|_{G′}).

Definition 1.1 (analytic definition of discretely decomposable restriction). We say that the restriction π|_{G′} is G′-admissible if the restriction π|_{G′} splits into a discrete direct sum of irreducible unitary representations of G′ and if m_{π}(τ) < ∞ for any τ ∈ Ĝ′.

If π|_{G′} is G′-admissible, then we have a unitary equivalence of G′-modules:

(1.1.1)  π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈Ĝ′ \end{matrix}}m_{π}(τ)τ  (a discrete direct sum of Hilbert spaces).

Here, ^{∑⊕} denotes the Hilbert completion of an algebraic direct sum. In particular,

the formula (1.1.1) means that there is no continuous spectrum in the branching law of π|_{G′}.

The significance of Definition 1.1 is that there are “new” examples. However, we shall start with “old” examples of admissible restrictions.

The following Harish-Chandra’s theorem is fundamental in representation theory of reductive Lie groups, which has enabled us to study unitary representations by purely algebraic methods (so called the theory of (𝔤_{ℂ}, K)-modules or Harish-Chandra modules). This theorem may be regarded as a special example of Definition 1.1 (the case G′ = K):

Example 1.2 (Harish-Chandra [17], see also [78], Theorem 3.4.1). For any π ∈ Ĝ, the restriction π|_{K} is K-admissible^{2}.

In Introduction, we have seen an example of branching laws which is given by the Fourier series expansion (see (0.2)). This is a special case of Example 1.2 applied to (G, K) = (SL(2, ℝ), SO(2)) (multiplicity is free in this case).

The theta-correspondence plays an important role in number theory of automorphic forms. The following result due to Howe [20] presents another example of G′-admissible restrictions, where G′ is non-compact:

Example 1.3 (discrete decomposability in the theta-correspondence). Let G be the metaplectic group Mp(n, ℝ). Suppose that (G, G′) is a reductive dual pair, namely, G′ = G′_{1}G′_{2} is a reductive subgroup in G such that G′_{1} and G′_{2} are mutually the centralizer of the other in G. If G′_{1} or G′_{2} is compact and if π is the Segal-Shale-Weil representation of G, then the restriction π|_{G′} decomposes discretely with multiplicity free, in particular, it is G′-admissible. The branching laws produce a lot of irreducible unitary highest weight representations (see Kashiwara-Vergne [24], for example, for some explicit branching laws).

In Definition 1.1, we have formulated analytically the condition that there is no continuous spectrum in the branching law. We shall also formulate the notion of “discrete decomposability” algebraically in terms of (𝔤_{ℂ}, K)-modules (Definition 2.3). There is a slight difference between these two definitions of discrete decomposable restrictions, especially, we allow the multiplicity to be infinite in our definition of algebraic discrete decomposability in §2. Then, the study of this difference gives rise to a finite multiplicity theorem ([49], see Theorem 3.2 in §3.A):

“discreteness (in the spectrum) ⇒ finiteness (of multiplicity)”

for the restriction of discrete series representations with respect to semisimple symmetric pairs (see also Conjecture 3.4 in §3.A in a more general setting).

So far, we have given two examples of G′-admissible restrictions, namely, Examples 1.2 and 1.3. In both cases, we made strong assumptions: in Example 1.2 G′ is compact; while in Example 1.3 π has a non-zero highest weight vector (we called this property “one-sided infinity” in Introduction; such representations are very special among Ĝ).

Our formulation (Definition 1.1) was intended to seek for new settings where the branching law π|_{G′} is G′-admissible, beyond Examples 1.2 and 1.3. The criterion below (see Theorem 1.5) assures that there are quite rich examples of G′-admissible restrictions π|_{G′} even though G′ is non-compact and π is not a highest weight representation. Thus, a number of branching problems in this framework are newly obtained, which should to be accessible by purely algebraic methods. Examples of explicit (discretely decomposable) branching laws of discrete series representations (or more generally, Zuckerman-Vogan derived functor modules A_{𝔮}(λ)) have been found with respect to symmetric pairs such as (O(p, q), O(p − r, q) × O(r)), (O(2p, 2q), U(p, q)) and so on, in this new “nice” framework (see [35] Part I, [37]).

2This property is usually called “admissible”. The terminology in Definition 1.1 is named after it.

1.C. A sufficient condition for discretely decomposable restrictions.

Let G be a real reductive linear group. For simplicity, we assume that G is connected. Then, a maximal compact subgroup K of G is also connected. Since K is compact, any irreducible unitary representation of K is finite dimensional. As usual, we write K̂ for the set of equivalence classes of irreducible unitary representations of K. Let 𝔨 be the Lie algebra of K, and take a maximal Abelian subspace 𝔱 of 𝔨. We fix a positive system Δ^{+}(𝔨, 𝔱). Then K̂ is identified with the set of dominant integral weights by the Cartan-Weyl highest weight theory. Hereafter, K̂ will be regarded as a subset of \sqrt{−1}𝔱^{∗}.

Definition 1.4 (asymptotic K-support [25]). For π ∈ Ĝ, we define two subsets of \sqrt{−1}𝔱^{∗} as follows:

Supp_{K}(π) := {τ ∈ K̂ : Hom_{K}(τ, π|_{K}) ≠ {0}},
AS_{K}(π) := Supp_{K}(π)∞.

Here, for a subset S of the Euclidean space, the limit cone S∞ (see [23]) is a closed cone defined by:

S∞ := { lim_{n→∞} ϵ_{n}y_{n} : y_{n} ∈ S, ϵ_{n} → 0}.

The closed cone AS_{K}(π) is called the asymptotic K-support of the representation π.

Let G ⊃ G′ be a pair of reductive Lie groups in the sense that both G and G′ satisfy the conditions (i) and (ii) stated at the beginning of §1. Then, the adjoint representation Ad : G → GL(𝔤) is completely reducible when restricted to G′. If we set K′ := K ∩ G′, then K′ is a maximal compact subgroup of G′.

Example.

1) For (G, G′) = (GL(n, ℂ), GL(n, ℝ)), we have (K, K′) = (U(n), O(n)).

2) For (G, G′) = (GL(n, ℂ), U(p, q)), we have (K, K′) = (U(n), U(p) × U(q)). Here, p + q = n.

We write 𝔤′, 𝔨′ for the Lie algebras of G′, K′, respectively. We fix a K-invariant inner product on 𝔨. With respect to this inner product, we define (𝔨′)^{⊥} as the orthogonal complement of 𝔨′ in 𝔨. We also identify 𝔱^{∗} with 𝔱, and then regard 𝔱^{∗} as a subspace of 𝔨. In particular, we can regard AS_{K}(π) as a subset of \sqrt{−1}𝔨.

Here is a sufficient condition for the restriction π|_{G′} to be G′-admissible:

Theorem 1.5 (a sufficient condition for G′-admissible restriction; [44]). Let G ⊃ G′ be a pair of reductive Lie groups, and π ∈ Ĝ. If

(1.5.1)  AS_{K}(π) ∩ \sqrt{−1} Ad(K)(𝔨′)^{⊥} = \{0\},

then the restriction π|_{K′} is K′-admissible. In particular, the restriction π|_{G′} is G′- admissible, namely, π|_{G′} decomposes discretely with finite multiplicities (Definition 1.1).

Assumption (1.5.1) is obviously satisfied if AS_{K}(π) = {0} or if Ad(K)(𝔨′)^{⊥} = {0}. First of all, let us explain these two special cases in Example 1.6 and Example 1.7, respectively.

Example 1.6. If G′ = K, then 𝔨′ = 𝔨 and therefore (𝔨′)^{⊥} = {0}. Hence, we have

Ad(K)(𝔨′)^{⊥} = {0}.

(In fact, it is easy to see that Ad(K)(𝔨′)^{⊥} = {0} if and only if G′ ⊃ K.) Then, Assumption (1.5.1) is automatically fulfilled for any π ∈ Ĝ. The conclusion of Theorem 1.5 in this special case is the admissibility theorem due to Harish-Chandra as stated in Example 1.2.

Example 1.7. For π ∈ Ĝ, we have

AS_{K}(π) = {0} if and only if dim π < ∞.

Thus, if dim π < ∞ then Assumption (1.5.1) is obviously satisfied for any subgroup G′. The conclusion of Theorem 1.5 in this special case follows also from an easy complete reducibility result of finite dimensional unitary representations.

Here is a non-trivial example of Theorem 1.5:

Example 1.8. Let (G, G′) = (U(2, 2), Sp(1, 1)). We note that the pair (G, G′) is locally isomorphic to (S^{1} × SO(4, 2), SO(4, 1)). There are 18 series of irreducible unitary representations of G with regular and integral infinitesimal character by a result of Salamanca Riba. Among 18 series, 6 can be realized in closed subspaces of L^{2}(G), namely, they are Harish-Chandra’s discrete series representations. Among 18 series, there are 12 series (containing 2 discrete series) of irreducible unitary representations of G that satisfy the condition (1.5.1) (see [37] for details). In particular, there is no continuous spectrum in the branching laws of the restriction π|_{G′} if π belongs to these 12 series. Conversely, the remaining 18 − 12 = 6 series of irreducible unitary representations of G are not algebraically discretely decomposable (Definition 2.3) when restricted to G′ (see [49]).

For some important representations π such as discrete series representations, we can compute AS_{K}(π) in terms of the root data [44]. See [42] for a knack of actual computations to apply Theorem 1.5.

The converse direction of Theorem 1.5 will be discussed in §2.B.

The idea of the proof of Theorem 1.5 is to capture the existence of continuous spectrum as a “size^{3} of infinite dimensional representations”, by looking at the asymptotic behavior of K-types. In [44], this was carried out by extending the work of Kashiwara-Vergne and Howe in the 70s on the microlocal study of characters of representations. Here, the asymptotic K-support AS_{K}(π) plays a role of “size” of π by means of the wave front set of the character Trace(π). Before finding a general method in [44], we took a different and more algebraic approach in proving the G′-admissibility of the restriction π|_{G′} in a special case where π is a Zuckerman-Vogan derived functor module. See [31], Proposition 4.1.3 in the case where K′ is of the form of the direct product K′_{1} × K′_{2}; see also [37], Theorem 1.5 in the case where (G, G′) is a semisimple symmetric pair. Gross and Wallach [16] also studied G′-admissible restrictions in the case where K′ is of the form K′_{1} × K′_{2}, especially when K′_{1} ≃ SU(2).

In §2 below, we shall reformulate Definition 1.1 in terms of (𝔤_{ℂ}, K)-modules, and obtain a necessary condition for the discrete decomposability. Namely, we introduce the notion of infinitesimally discrete decomposability of the restriction (see Definition 2.3), and estimate the size of infinite dimensional representations by their associate varieties [77], which originally arose from the D-module theory. If the restriction π|_{G′} is discretely decomposable, then we may consider analogous notion of (𝔤_{ℂ}, K)-modules à la Harish-Chandra and Lepowski in a more general setting, namely, an algebraic direct sum of infinite dimensional representations (symbolically, a notion of (𝔤_{ℂ}, G′)-modules).

We note that Theorem 1.5 gives not only a sufficient condition for the G′- admissibility of the restriction π|_{G′} but also for the infinitesimally discrete decomposability of the restriction π|_{G′}, which is the object of §2.

§2. Algebraic theory of discretely decomposable restrictions

2.A. Algebraic reformulation of discrete decomposability.

Let us start with an example where there is only continuous spectrum in the irreducible decomposition. As an opposite extremal case, this example (Example 2.1) serves us as a hint to find an algebraic definition of discretely decomposable restrictions.

Example 2.1 (Wiener subspace). Let V be a subspace of L^{2}(ℝ). We say V is ℝ-invariant if V satisfies:

f(x − a) ∈ V for any f(x) ∈ V and any a ∈ ℝ.

For a measurable set E of ℝ, we write L^{2}(E) for the closed subspace of L^{2}(ℝ), consisting of all L^{2}-functions supported on E. Then, the image of the Fourier transform of L^{2}(E), denoted by F(L^{2}(E)), is a closed ℝ-invariant subspace. Conversely, it is known that any closed ℝ-invariant subspace is of the form F(L^{2}(E)) for some measurable set E. Then, given any non-zero closed ℝ-invariant subspace V, there exists an infinite decreasing sequence of closed ℝ-invariant subspaces {V_{j}}:

3The representation space is an infinite dimensional, separable Hilbert space, which is unique as a topological vector space. However, the size of representations may be “different” if we take group actions into account. Associated varieties (in §2) and their dimensions (Gel’fand-Kirillov dimensions) are also useful to measure ‘size’ of representations.

V ⫌ V_{1} ⫌ V_{2} ⫌ ⋯,

(to see this, it is enough to take a sequence of measurable sets E_{j} of E such that E ⫌ E_{1} ⫌ E_{2} ⫌ ⋯, and then to define V_{j} := F(L^{2}(E_{j}))). This property is equivalent to the fact that there is no discrete spectrum in the irreducible decomposition of the regular representation L^{2}(ℝ) of ℝ. (Of course, the latter property follows also from the irreducible decomposition of L^{2}(ℝ) by means of the Fourier transform (see (0.3)).

In summary, Example 2.1 relates the non-existence of discrete spectrum with the existence of an infinite decreasing sequence of invariant subspaces. Next, we shall relate the non-existence of continuous spectrum with the existence of an infinite increasing sequence of invariant subspaces. Here is an algebraic formulation:

Definition 2.2 ([49], Definition 1.1). Let 𝔤 be a Lie algebra, and X a 𝔤-module. We say the 𝔤-module X is discretely decomposable if there is an increasing sequence of 𝔤-submodules:

X_{0} ⊂ X_{1} ⊂ X_{2} ⊂ ⋯

such that the following two properties are satisfied: (2.2.1) X = ⋃_{m=0}^{∞} X_{m}. (2.2.2) Each X_{m} is of finite length as a 𝔤-module.

We note that irreducible representations of a finite dimensional Lie algebra are usually infinite dimensional. In the above definition, we have infinite dimensional modules X_{m} in mind.

Let (π, H) be an irreducible unitary representation of G, and X a subspace of the Hilbert space H consisting of K-finite vectors. Then, X is dense in H, and has the 𝔤-module structure (the differential representation), in addition to the K-module structure. The above 𝔤_{ℂ} ∪ K-module (π_{K}, X) is called the underlying (𝔤_{ℂ}, K)-module of π.

Next, let G ⊃ G′ be a pair of reductive Lie groups so that K′ := K ∩ G′ is a maximal compact subgroup of G′. We apply Definition 2.2 to the restriction to G′.

Definition 2.3 (algebraic definition of discretely decomposable restriction). Let π ∈ Ĝ. We say that the restriction π|_{G′} is 𝔤′-discretely decomposable or infinitesimally discretely decomposable if the underlying (𝔤_{ℂ}, K)-module π_{K} is discretely decomposable as a 𝔤′-module in the sense of Definition 2.2.

It might look strange at a first glance that Definition 2.2 gives the notion of “discrete decomposition”. In fact, the terminology is named after the following:

Theorem 2.4 (characterization of infinitesimally discretely decomposable restriction; [49]). Let (G, G′) be a pair of reductive Lie groups, π an irreducible unitary representation of G, and (π_{K}, X) its underlying (𝔤_{ℂ}, K)-module. Then the following three conditions on the triple (G, G′, π) are equivalent:

i) The restriction π|_{G′} is infinitesimally discretely decomposable (Definition 2.3). ii) The (𝔤_{ℂ}, K)-module (π_{K}, X) is isomorphic to an algebraic direct sum of irreducible (𝔤′_{ℂ}, K′)-modules (discrete branching law):

(2.4.1)  X ≃ ⨁_{Y} n_{π}(Y) Y (an algebraic direct sum).

Here, the sum is taken over all irreducible (𝔤′_{ℂ}, K′)-modules Y, and

n_{π}(Y) := dim Hom_{(𝔤′_{ℂ},K′)}(Y, X)

is the multiplicity of Y occurring in X. iii) There exists an irreducible (𝔤′_{ℂ}, K′)-module Y such that

Hom_{(𝔤′_{ℂ},K′)}(Y, X) ≠ {0}.

We note that the multiplicity n_{π}(Y) may be and may not be infinite in Theorem 2.4. The point of the condition (iii) is that only a single representation Y is used (for this, we have assumed that π is irreducible).

Moreover, the following theorem holds:

Theorem 2.5 (infinitesimal ⇒ Hilbert discrete decomposition). If the restriction π|_{G′} of π ∈ Ĝ is infinitesimally discretely decomposable, then we have the following equality for any τ ∈ Ĝ′:

(2.5.1)  dim Hom_{(𝔤′_{ℂ},K′)}(τ_{K′}, π_{K}) = dim Hom_{G′}(τ, π|_{G′}).

We put m_{π}(τ) by (2.5.1). Then, the restriction π|_{G′} of the unitary representation π is decomposed into irreducibles of G′ without continuous spectrum:

π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈Ĝ′ \end{matrix}}m_{π}(τ)τ (a discrete direct sum of Hilbert spaces).

We note that the right-hand side of (2.5.1) is the dimension of continuous G′- intertwining operators, while no topology is specified in the left-hand side of (2.5.1). In general, we have

the left-hand side of (2.5.1) ≤ the right-hand side of (2.5.1)

without the assumption of infinitesimally discrete decomposability.

We should keep in mind that we have not imposed the condition n_{π}(Y) < ∞ in the definition of infinitesimally discrete decomposability (Definition 2.3), while we imposed the finiteness of the multiplicities in the definition of G′-admissible restriction (Definition 1.1). As we have pointed out in [49], [50] (see also §1.B), the multiplicities tend to be finite if the restriction has no continuous spectrum (see Theorem 3.2 and Conjecture 3.4 in §3.A for a precise formulation).

The next (easy) example follows immediately from the equivalent definitions of infinitesimally discrete decomposability (Theorem 2.4). We note that G′ = {e} is allowed in the example below.

Example 2.6. If G′ is a compact subgroup, then the restriction π|_{G′} is infinitesimally discretely decomposable for any π ∈ Ĝ.

2.B. A necessary condition for discretely decomposable restrictions

— an approach by associated varieties. Loosely, the associated variety of a 𝔤-module π is an algebraic variety that

approximates the representation π by means of the graded ring gr U(𝔤_{ℂ}). Since gr U(𝔤_{ℂ}) is isomorphic to a polynomial ring, one can use a standard technique of algebraic geometry. It turns out that associated varieties are useful for the study of infinitesimally discretely decomposable restrictions ([35] Part II, [49]).

Let us recall briefly the definition of associated varieties of 𝔤-modules (see a treatise of Vogan [77] for more details). Let X be a finitely generated module (over ℂ) of a Lie algebra 𝔤. The associated variety of X, denoted by V_{𝔤}(X), is defined similarly to the characteristic varieties of D-modules as follows (see [9]). Let 𝔤_{ℂ} be the complexification of 𝔤. Then, X becomes a U(𝔤_{ℂ})-module by the universality of the enveloping algebra U(𝔤_{ℂ}). Take a natural filtration

ℂ = U_{0}(𝔤_{ℂ}) ⊂ U_{1}(𝔤_{ℂ}) ⊂ U_{2}(𝔤_{ℂ}) ⊂ ⋯

of U(𝔤_{ℂ}) (corresponding to degrees of partial differential operators). Let X_{0} be a finite dimensional subspace of X that generates X as a U(𝔤_{ℂ})-module and we put X_{n} := U_{n}(𝔤_{ℂ}) · X_{0}. Then X_{0} ⊂ X_{1} ⊂ X_{2} ⊂ ⋯ gives a filtration of X such that U_{i}(𝔤_{ℂ})X_{j} ⊂ X_{i+j}. We note that X = ⋃_{n} X_{n}. We put

gr U(𝔤_{ℂ}) := ⨁_{n} U_{n}(𝔤_{ℂ})/U_{n−1}(𝔤_{ℂ}), gr X := ⨁_{n} X_{n}/X_{n−1}.

Then, the graded module gr X carries naturally a gr U(𝔤_{ℂ})-module structure. The gr U(𝔤_{ℂ})-module, gr X, is regarded as an approximation of the U(𝔤_{ℂ})-module X. The enveloping algebra U(𝔤_{ℂ}) is a non-commutative algebra provided 𝔤 is noncommutative, while the graded ring gr U(𝔤_{ℂ}) is isomorphic to the symmetric algebra S(𝔤_{ℂ}) by the Poincaré-Birkhoff-Witt theorem, and then is isomorphic to the polynomial algebra over 𝔤_{ℂ}^{∗}. In particular, gr U(𝔤_{ℂ}) is commutative. The characteristic variety of an S(𝔤_{ℂ})-module gr X is called the associated variety of X, and will be written as V_{𝔤}(X). That is,

V_{𝔤}(X) := {λ ∈ 𝔤_{ℂ}^{∗} : f(λ) = 0 for any f ∈ Ann_{gr U(𝔤_{ℂ})}(gr X)}

where Ann_{gr U(𝔤_{ℂ})}(gr X) is the annihilator ideal of gr X.

So far, X is just a finitely generated module of the Lie algebra of 𝔤. From now, let (π_{K}, X) be the underlying (𝔤_{ℂ}, K)-module of an irreducible unitary representation π of a reductive Lie group G. Then we have:

Theorem 2.7 (Vogan, [77]). The associated variety V_{𝔤}(X) is a K_{ℂ}-invariant algebraic variety contained in the nilpotent cone of 𝔤_{ℂ}^{∗}. Furthermore, V_{𝔤}(X) ⊂ (𝔤_{ℂ}/𝔨_{ℂ})^{∗}

Here we recall that the nilpotent cone of 𝔤_{ℂ}^{∗} is the algebraic variety that is identified with the closed subset

{X ∈ 𝔤_{ℂ} : ad X is nilpotent}

under the isomorphism 𝔤_{ℂ} ≃ 𝔤_{ℂ}^{∗}. Suppose G′ is a subgroup of G. Let

pr_{𝔤→𝔤′} : 𝔤_{ℂ}^{∗} → (𝔤′_{ℂ})^{∗}

be the projection dual to the inclusion of Lie algebras 𝔤′_{ℂ} ↪ 𝔤_{ℂ}. Here is a lower estimate of the associated varieties of irreducible summands in the branching law:

Theorem 2.8 (associated varieties of irreducible summands, [49]). Suppose that there is an irreducible (𝔤′_{ℂ}, K′)-module Y such that Hom_{(𝔤′_{ℂ},K′)}(Y, X) ≠ {0}. Then, the following inclusive relation holds:

(2.8.1)  pr_{𝔤→𝔤′}(V_{𝔤}(X)) ⊂ V_{𝔤′}(Y).

To see the meaning of Theorem 2.8, let us consider the simplest case, that is, the case where Y is finite dimensional. Then, it follows from the definition of an associated variety that

V_{𝔤′}(Y) = {0}.

Consequently, (2.8.1) is equivalent to

V_{𝔤}(X) ⊂ (𝔤_{ℂ}/𝔤′_{ℂ})^{∗}.

Now, we divide into two cases in the following example: G′ is compact or noncompact.

Example 2.9. 1) If G′ is compact, then obviously there exists a finite dimensional irreducible (𝔤′_{ℂ}, K′)-module Y satisfying Hom_{(𝔤′_{ℂ},K′)}(Y, X) ≠ {0}. In particular, if G′ = K, then we have

V_{𝔤}(X) ⊂ (𝔤_{ℂ}/𝔨_{ℂ})^{∗} ≃ 𝔭_{ℂ}

by Theorem 2.8. This result was proved previously by Vogan (see Theorem 2.7).

2) Suppose G is a simple Lie group. If G′ is non-compact, then

pr_{𝔤→𝔤′}(Ad(K_{ℂ})v) ≠ {0}

for any non-zero nilpotent element v ∈ 𝔭_{ℂ}. If dim X = ∞, then V_{𝔤}(X) ≠ {0}, and in particular, V_{𝔤}(X) contains Ad(K_{ℂ})v for some non-zero nilpotent element v in 𝔭_{ℂ}. Hence, pr_{𝔤→𝔤′}(V_{𝔤}(X)) ≠ {0}. Therefore, it follows from Theorem 2.8 that

Hom_{(𝔤′_{ℂ},K′)}(Y, X) = {0}

for any finite dimensional irreducible (𝔤′_{ℂ}, K′)-module Y. See also [49], Corollary 3.9 for a relation to Moore’s ergodicity theorem [62].

The following criterion ([49], Corollary 3.4) is useful and is readily deduced from Theorem 2.8.

Theorem 2.10 (a necessary condition for discretely decomposable restrictions). Let (π_{K}, X) be the underlying (𝔤_{ℂ}, K)-module of π ∈ Ĝ. If the restriction π|_{G′} is infinitesimally discretely decomposable, then pr_{𝔤→𝔤′}(V_{𝔤}(X)) is contained in the nilpotent cone of (𝔤′_{ℂ})^{∗}.

2.C. Three more theorems on discretely decomposable restrictions. In this subsection, we state three direct consequences of Theorem 2.10. We recall an obvious fact that the restriction to a compact subgroup is always

discretely decomposable (Example 2.6). For example:

Example 2.11 (the restriction which is always discretely decomposable). Let (G, G′) = (SL(n, ℂ), SU(n)). Then, the restriction π|_{G′} is infinitesimally discretely decomposable for any irreducible unitary representation π of G.

Theorem 2.10 leads us to an opposite extremal case:

Theorem 2.12 (the restriction which is never discretely decomposable). Let (G, G′) = (SL(n, ℂ), SL(n, ℝ)). Then, the restriction π|_{G′} is not infinitesimally discretely decomposable for any irreducible unitary representation π of G except for π = 1.

So, with regard to infinitesimally discrete decomposability, two real forms SU(n) and SL(n, ℝ) of SL(n, ℂ) have completely different feature. Other real forms such as SU(p, n − p) (1 ≤ p ≤ n − 1) are intermediate. A real form of G′ of a complex reductive Lie group G is called normal or split if rank G = ℝ-rank G′. For example, SL(n, ℝ) is a normal real form of SL(n, ℂ), while SU(p, n − p) is not a normal real form except for the case (p, n) = (1, 2).

Combining Theorem 2.10 with the following lemma:

pr_{𝔤→𝔤′}(Ad(K_{ℂ})v) contains a non-zero semisimple element for

any non-zero nilpotent element v ∈ 𝔭_{ℂ} if G′ is a normal real form of a complex reductive Lie group G,

we can generalize Theorem 2.12 as follows (see [49], Theorem 8.1):

Theorem 2.13 (the restriction is never discretely decomposable). Let G be a complex reductive Lie group, and G′ its normal real form. Then, the restriction π|_{G′} is not infinitesimally discretely decomposable for any infinite dimensional irreducible unitary representation π of G.

A second application of Theorem 2.10 deals with the relation between the discreteness in the induced representation (discrete series representation) and the discreteness in the restriction (discrete decomposable restriction). It turns out that they cannot stand together for symmetric pairs. Here is a statement:

Theorem 2.14 (the exclusive law of discrete spectrum for the restriction and the induction). Let (G, G′) be an irreducible symmetric pair such that G is noncompact. Let π ∈ Ĝ. Then both (i) and (ii) cannot occur simultaneously.

i) The restriction π|_{G′} is infinitesimally discretely decomposable . ii) π is a discrete series representation for the homogeneous space G/G′ (i.e. Hom_{G}(π, L^{2}(G/G′)) ≠ {0}).

We refer to [49] for the proof. At a first glance, this result might look strange, but it is another thing that one might expect as a Frobenius reciprocity-type theorem for infinite dimensional representations.

Here is a very special example of Theorem 2.14.

Example 2.15. Let (G, G′) be a Riemannian symmetric pair, namely, G′ is a maximal compact subgroup K. Then, we recall the following two well-known results:

1) The restriction π|_{K} is infinitesimally discretely decomposable for any π ∈ Ĝ.

2) (Harish-Chandra, Helgason) There is no discrete series representation for the Riemannian symmetric space G/K (see §4.C for the definition of discrete series representations for a homogeneous space).

(1) is obvious (see Example 2.6), but (2) is non-trivial. (2) is equivalent to the fact that discrete series representations for the group manifold G do not have (nonzero) K-fixed vectors. The point here is that Theorem 2.14 asserts a non-trivial implication (1) ⇒ (2).

As a simplest case of Example 2.15, let us consider the setting

(G, K) = (SL(2, ℝ), SO(2)).

Then, (1) corresponds to the discreteness of the Fourier series expansion (e.g. (0.2) in §0), and (2) means the fact that the Laplace-Bertram operator on the Poincaré upper half plane has no L^{2}-spectrum. Thus, even for SL(2, ℝ), Theorem 2.14 gives a new relationship between these two results.

Our proof of Theorem 2.14 uses the estimate of the associated varieties for infinitesimally discretely decomposable restrictions (Theorem 2.8). A more direct approach to Theorem 2.8 would be preferable, which might give a better understanding of this mysterious relation.

Another aspect of Theorem 2.14 is that it clarifies a representation theoretic background for the following antithesis between vanishing and non-vanishing theorems of modular symbols in arithmetic quotients of Riemannian symmetric spaces induced from the morphism Γ′\G′/K′ → Γ\G/K (see §4.B for details):

1) A non-vanishing theorem [73] due to Tong and Wang (some twisted case) by using a discrete series representation π for a semisimple symmetric space G/G′. (Discrete spectrum in the induction.)

2) A vanishing theorem [52] due to Kobayashi and Oda by using the discrete decomposability of the restriction π|_{G′} (cf. Theorem 1.5). (Discrete decomposability of the restriction.)

A third application of Theorem 2.10 is about the comparison of irreducible constituents of the restriction π|_{G′}. As we shall mention in Remark 2.17, an analogous result fails if there exists continuous spectrum in the branching law.

Theorem 2.16 (irreducible summands have the same associated varieties). Let X be an irreducible (𝔤_{ℂ}, K)-module. For any irreducible (𝔤′_{ℂ}, K′)-modules Y_{1} and Y_{2} such that

Hom_{(𝔤′_{ℂ},K′)}(Y_{j}, X) ≠ {0} (j = 1, 2),

their associated varieties are the same:

(2.16.1)  V_{𝔤′}(Y_{1}) = V_{𝔤′}(Y_{2}).

Remark 2.17. If the branching law contains continuous spectrum, then representations of different Gel’fand-Kirillov dimensions may occur in the restriction π|_{G′} as discrete spectrum. For example, the Plancherel theorem for the semisimple symmetric space Sp(n, ℝ)/GL(n, ℝ) is equivalent to the decomposition of the tensor product of two degenerate principal series representations of Sp(n, ℝ) with suitable parameters by the Mackey theory (for example, see [36], Proposition 6.1). In particular, this is a special case of branching laws. On the other hand, by using the Flensted-Jensen construction ([14]), one can prove that there exist discrete series representations for the semisimple symmetric space Sp(n, ℝ)/GL(n, ℝ) with different associated varieties. Hence, (2.16.1) fails. In other words, we have given a counter example of the following wrong statement:

False “Theorem” 2.16′. Let π ∈ Ĝ. If τ_{1}, τ_{2} ∈ Ĝ′ satisfy

Hom_{G′}(τ_{j}, π|_{G′}) ≠ {0} (j = 1, 2),

then

V_{𝔤′}(Y_{1}) = V_{𝔤′}(Y_{2}),

where Y_{j} (j = 1, 2) are the underlying (𝔤′_{ℂ}, K′)-modules of τ_{j}.

§3. New aspect of representation theory related to branching problems

It turns out that there exist fairly rich examples of discretely decomposable restrictions owing to the criterion in Theorem 1.5. Consequently, many branching problems arise, to which much attention has not been paid before, and on which we can now expect a deeper and explicit study by algebraic methods.

In this section, we shall explain briefly some of recent topics related to discrete branching laws.

3.A. Finite multiplicity conjecture.

Conjecture 3.1 (Wallach, see [79]). Let (G, G′) be a semisimple symmetric pair. If π is a discrete series representation for G, then

(3.1.1)  dim Hom_{G′}(τ, π|_{G′}) < ∞  for any τ ∈ Ĝ′.

For example, the admissibility theorem of Harish-Chandra (Example 1.2) asserts that Conjecture 3.1 holds if G′ is compact. On the other hand, if G′ is compact, then the restriction π|_{G′} is obviously infinitesimally discretely decomposable. It is proved in [49] that (3.1.1) still holds by assuming only the condition that π|_{G′} is infinitesimally discretely decomposable :

Theorem 3.2 (discreteness ⇒ finite multiplicity). Let (G, G′) be a semisimple symmetric pair. For any Zuckerman-Vogan derived functor (𝔤_{ℂ}, K)-module^{4} X (more precisely, cohomologically induced from a finite dimensional representation in the good range of parameters), and for any irreducible (𝔤′_{ℂ}, K′)-module Y, we have

4These representations are algebraic analog of a generalized Borel-Weil-Bott theorem on (possibly non-compact) complex homogeneous manifolds (see §3.D). They are often denoted by A_{𝔮}(λ).

See [26], [75] for an algebraic explanation, and [36] for a survey of geometric approach due to Schmid and Wong.

dim Hom_{(𝔤′_{ℂ},K′)}(Y, X) < ∞.

Since the underlying (𝔤_{ℂ}, K)-module of any discrete series representation π is expressed as a Zuckerman-Vogan derived functor module, we have the following corollary:

Corollary 3.3 (a proof of Conjecture 3.1 in the discrete decomposable case).

If the restriction π|_{G′} is infinitesimally discretely decomposable, Conjecture 3.1 is true.

Remark (infinite multiplicity). Even if (G, G′) is a semisimple symmetric pair, the multiplicity of discrete spectrum in the restriction π|_{G′} can be infinite, namely,

dim Hom_{G′}(τ, π|_{G′}) = ∞ for some τ ∈ Ĝ′ and π ∈ Ĝ.

This can happen if the branching law of the restriction π|_{G′} contains continuous spectrum. Different from the result due to Corwin-Greenleaf in the case of nilpotent Lie groups, the situation of semisimple Lie groups is more delicate. For instance, we proved in [50] that there is an example of (G, G′, π) such that

\{ \begin{matrix}& dim Hom_{G′}(τ_{1}, π|_{G′}) = ∞ for some τ_{1} ∈ Ĝ′, \\ 0 < & dim Hom_{G′}(τ_{2}, π|_{G′}) < ∞ for some τ_{2} ∈ Ĝ′, \end{matrix}

where π is an irreducible unitary representation of SO(5, ℂ) and G′ = SO(3, 2). Building on Theorem 3.2, we proposed the following conjecture:

Conjecture 3.4 (see [50], Conjecture C). Let (G, G′) be a semisimple symmetric pair, and π ∈ Ĝ. If the restriction π|_{G′} is infinitesimally discretely decomposable, then

dim Hom_{G′}(τ, π|_{G′}) < ∞ for any τ ∈ Ĝ′.

As we saw in (2.5.1), this conjecture also implies

dim Hom_{(𝔤′_{ℂ},K′)}(Y, π_{K}) < ∞ for any irreducible (𝔤′_{ℂ}, K′)-module Y .

To end this subsection, we would like to mention an analytic aspect of Corollary 3.3: If one realizes the representation π in a geometric way, then Corollary 3.3 may give rise to an example of the following phenomenon: “In a system of non-holonomic partial differential equations, local solutions are possibly infinite dimensional, but global solutions are possibly finite dimensional.”^{5} 3.B. A generalization of the Kostant-Schmid formula to semisimple symmetric pairs.

5It is remarkable the dimension of the space of global solutions on a non-compact manifold becomes finite in this case. The relation between the dimension of global solutions and the underlying geometry might be interesting to study, as is Atiyah-Singer’s index theorem for elliptic differential operators on compact manifolds.

In the framework of discretely decomposable restrictions, an algebraic approach could work effectively in branching problems. Furthermore, if the multiplicity is free, then one could expect a simple and detailed study of branching laws. In this subsection, we shall explain such examples. More precisely, in the setting below (especially, (G, G′) is a semisimple symmetric pair), it turns out that the branching law is discrete by Theorem 1.5, and that the multiplicity of each irreducible representation is free owing to the multiplicity-one theorem in [47]. In particular, the restriction π|_{G′} is G′-admissible. Then, we shall give a new explicit branching law that generalizes the Kostant-Schmid formula [71] to the setting of non-compact subgroups. This subsection is taken from [47].

Throughout this subsection, let G be a non-compact simple Lie group of Hermitian type. This means that G is a Lie group locally isomorphic to one of

SU(p, q), SO(n, 2), Sp(n, ℝ), SO^{∗}(2n), E_{6(−14)}, E_{7(−25)}.

Then, the complexified Lie algebra 𝔤_{ℂ} := 𝔤 ⊗_{ℝ} ℂ is decomposed into irreducible modules under the adjoint action of K as follows:

𝔤_{ℂ} = 𝔨_{ℂ} ⊕ 𝔭^{+} ⊕ 𝔭^{−}.

Let 𝔱 be a maximal Abelian subspace of 𝔨, and fix a positive system Δ^{+}(𝔨, 𝔱).

Definition 3.5 (unitary highest weight representation). Let (π, V) ∈ Ĝ. We say (π, V) is an irreducible unitary highest weight representation, if V^{𝔭^{+}} ≠ {0}, where we put

V^{∞} := the set of smooth vectors of V,
V^{𝔭^{+}} := {v ∈ V^{∞} : dπ(X)v = 0 for any X ∈ 𝔭^{+}}.

Then, K acts on V^{𝔭^{+}} because Ad(K) stabilizes 𝔭^{+}. It turns out that V^{𝔭^{+}} is irreducible as a K-module. Furthermore, an irreducible highest weight representation (π, V) of G is determined uniquely by the K-module structure on V^{𝔭^{+}}. We write V^{G}(μ) for the irreducible highest weight representation (π, V) of G, if μ ∈ \sqrt{−1}𝔱^{∗} is a highest weight of the K-module V^{𝔭^{+}} with respect to the positive system Δ^{+}(𝔨, 𝔱). We use this notation also for other Lie groups G′ of Hermitian type.

We say that V is of scalar type if dim V^{𝔭^{+}} = 1. If the highest weight representation V is realized in a closed subspace of L^{2}(G), we say V is a holomorphic discrete series representation. Holomorphic discrete series representations were discovered in an early stage of unitary representation theory by Harish-Chandra and have been best-understood among discrete series representations of G.

Suppose an involution τ ∈ Aut(G) stabilizes K and acts holomorphically on the Hermitian symmetric space G/K. We define a subgroup of G by

G^{τ} := {g ∈ G : τg = g}.

Analogous notation V^{τ} will be applied to denote the set of fixed points of τ if τ acts on a vector space V. We define the subgroup

G′ := G_{0}^{τ},

by the connected component of G^{τ} containing the identity. Then G′ is a reductive subgroup, and (G, G′) forms a semisimple symmetric pair. For example, the pairs (Sp(n, ℝ), U(p, q)) and (Sp(n, ℝ), Sp(p, ℝ) × Sp(q, ℝ)) (p + q = n) are the cases.

Let

{ν_{1}, ν_{2}, …, ν_{k}}

be a maximal set of strongly orthogonal roots in Δ((𝔭^{+})^{−τ} , 𝔱^{τ}). Then one can show

k = ℝ- rank G/G^{τ} ,

the real rank of a semisimple symmetric space G/G^{τ}, or equivalently, that of G/G′. For example, k = min(p, q) if (G, G′) = (Sp(n, ℝ), Sp(p, ℝ) × Sp(q, ℝ)).

Theorem 3.6 (a generalization of the Kostant-Schmid formula; [47]). In the above setting, let V^{G}(μ) ∈ Ĝ be a holomorphic discrete series representation of scalar type. Then we have the following branching law of the restriction V^{G}(μ)|_{G′}:

(3.6.1)  V^{G}(μ)|_{G′} ≃ _{^{∑⊕} ; \begin{matrix}a_{1}≥···≥a_{k}≥0 \\ a_{j}∈ℕ \end{matrix}}V^{G′}(μ|_{𝔱^{τ}} − ∑_{j=1}^{k} a_{j}ν_{j}),

where the right side is a discrete direct sum of irreducible unitary representations of G′.

In the above theorem, if G′ is non-compact, then each V^{G′}(∗) is infinite dimensional.

Theorem 3.6 includes the following known results as special cases:

i) The formula due to Hua(classical)-Kostant(unpublished)-Schmid [71] corresponds to the case where G′ is a maximal compact group K. Then V^{K}(∗) is finite dimensional.

ii) G′ is non-compact. Some special cases have been known, including the cases G = SU(2, 2), SU(2, 1). See, for example, Jakobsen and Vergne ([22]) and Xie ([79]).

3.C. Unipotent representations and discrete branching laws.

There is another special case where the restriction has been studied extensively, that is, the local theta-correspondence (cf. Example 1.3). Then the restriction π|_{G′} concerns with the case where π is the Segal-Shale-Weil representation of the metaplectic group G = Mp(n, ℝ) and (G, G′) is a reductive dual pair. The Weil representation is an example of the minimal unipotent representation of the split group G of type C. The study of branching laws of unipotent representations of other groups has been studied in the last decade. Some explicit branching laws of unipotent representations include:

•) G is of type D, due to Kobayashi and Ørsted (discretely decomposable branching laws, 1991). See also [53], [88] for a more general case with continuous spectrum.

•) G is an exceptional group with real rank 4, due to Gross and Wallach (dis-

cretely decomposable branching law, [16]), •) G is of type E, due to J-S. Li (discretely decomposable branching law, [57]). As above, these branching laws have been studied mainly in the case where they

are discretely decomposable. Here are some advantages of discrete decompositions:

(3.C.1) From the view point of finding explicit branching laws, branching laws are less difficult to find, if there is no continuous spectrum, because one can use algebraic techniques. (3.C.2) From the view point of the study of Ĝ′ (smaller group), discrete spectrum is useful because it gives an explicit construction of irreducible unitary representations of the subgroup G′. (3.C.3) From the view point of the study of Ĝ (larger group), discrete branching laws give a clue to study representations of G in terms of G′ (e.g. a special case G′ = K gives a theory of (𝔤_{ℂ}, K)-modules).

The study of branching laws of unipotent representations are still in an early stage, and there seems a large room for further developments. For example, the following directions of research may be considered:

a) A finer study of unipotent representations by means of branching laws (see (3.C.3)).

b) Construction of singular unitary representations as irreducible summands (see (3.C.2)).

c) Global analysis on manifolds (especially, on homogeneous spaces) arising from branching laws of unipotent representations (e.g. [88]).

d) Combinatorial problems arising from algebraic study of discretely branching laws (cf. (3.C.1)).

3.D. Representations as a quantization of elliptic orbits and discrete branching laws.

An elliptic orbit is an adjoint orbit of G through an element X such that ad(X) is diagonalizable with purely imaginary eigenvalues. Any elliptic orbit carries a G-invariant pseudo-Kähler structure, and its “geometric quantization” gives an irreducible unitary representation of G, as was suggested by the Kirillov-Kostant orbit method, and as was proved by Schmid and Wong combined with algebraic results due to Vogan, Wallach and Zuckerman, under certain regular and integral conditions of X. We note that its Harish-Chandra module is expressed as Zuckerman-Vogan’s derived functor (𝔤_{ℂ}, K)-modules (sometimes, called A_{𝔮}(λ), see [26]).

In the previous exposition [36], we gave a survey on the construction of these representations in details from the view points of geometric quantization and discussed the discrete decomposability of the restriction to subgroups.

We also wrote in [35], Part I and in [37], some explicit branching laws of (small) discrete series and some more general representations A_{𝔮}(λ) for classical symmetric pairs (G, G′) such as

(G, G′) = (SO(p, q), SO(m) × SO(p − m, q)), (O(2p, 2q), U(p, q))

in the framework of G′-admissible restrictions. So we do not repeat here.

Let us just mention some new progress after [36] was written. By using Theorem 1.5 and Theorem 2.10, one have now a necessary and sufficient condition for the restriction π|_{G′} to be infinitesimally discretely decomposable if π is Zuckerman-Vogan’s derived functor module. This was proved in [49], which strengthens the result in the previous exposition ([36], Theorem 6.5).

Discretely decomposability of the restriction of Zuckerman-Vogan’s derived functor modules are particularly important in applications in §4.B and §4.C below.

§4. Applications of admissible restrictions

4.A. Branching laws and geometry.

Historically, branching problems of unitary representations have been motivated, not only by representation theory itself, but also by other fields, for instance, mathematical description of breaking symmetries in quantum mechanics, theta correspondence in automorphic forms and so on.

In this section, we shall discuss new interactions between branching problems and related fields, which have been discovered in 1990s, especially, connected with discrete branching laws. The following principle was advocated in [37]:

If representations help in the understanding of objects, so do branching laws of representations in that of morphisms.

By simplifying settings for the exposition here, this principle may be explained as follows. First, without group actions, let us consider the correspondences:

geometry of X  ⇔  function space Γ(X)
map f : Y → X ⇔ pullback f^{∗} : Γ(X) → Γ(Y).

Next, let G′ be a subgroup of G. Suppose that G acts on X and G′ on Y so that f is G′-equivariant. Then, the above correspondences are enriched by group actions:

geometry of G-space X ⇔  representation of G on Γ(X)
G′-equivariant map f : Y → X ⇔ restriction of the representation Γ(X) to G′
+ G′-intertwining map f^{∗} : Γ(X) → Γ(Y).

Thus, the knowledge of the restriction of representations of G to G′ should be transferred to some information on the original map f.

In this section, we shall illustrate this principle without technical details in the settings where restrictions of unitary representations appear in a somewhat unexpected way. We shall also try to explain how and why discretely decomposable restrictions play a crucial role there.

4.B. A vanishing theorem for modular varieties.

In this subsection, we explain an application of our criterion for discretely decomposable restrictions (Theorem 1.5) to a differential geometric problem on modular varieties. Roughly speaking, a modular symbol is the homology class in a locally Riemannian symmetric space (sometimes called a Clifford-Klein form) determined by the cycle induced by a subgroup. For example, a geodesic cycle in a closed Riemann surface with genus ≥ 2 represents a modular symbol.

More generally, we consider the following setting:

G′ ⊂ G: a pair of connected linear reductive Lie groups,
K′ ⊂ K: maximal compact subgroups of G′ ⊂ G, respectively,
Γ′ ⊂ Γ : cocompact torsion-free discrete subgroups of G′ ⊂ G, respectively,

such that K′ = K ∩ G′ and Γ′ = Γ ∩ G′. Then, both of the double cosets

X := Γ\G/K and Y := Γ′\G′/K′

are compact, orientable, locally Riemannian symmetric spaces. The inclusion G′ ↪ G induces a natural map:

ι : Y → X.

The image ι(Y) is called a modular variety. It is totally geodesic in the Riemannian manifold X because the subgroup G′ is reductive in G. We put

m = dim Y (= dim G′/K′).

Then, the fundamental class [Y] generates the homology group H_{m}(Y; ℤ) of degree m. Consider the induced homomorphism of homology groups of degree m:

ι_{∗} : H_{m}(Y; ℤ) → H_{m}(X; ℤ).

The modular symbol is defined to be the image ι_{∗}[Y] ∈ H_{m}(X; ℤ) (see Ash-Borel [2]). Though its definition is simple, the understanding of modular symbols is usually difficult.

In order to see how the discrete decomposability of the restriction G ↓ G′ (see §1, §2) affects a topological property of modular varieties, we consider a special example where

(G, G′) = (SO_{0}(2n, 2), SO_{0}(2n, 1)).

Then, dim G′/K′ = 2n, and the modular symbol ι_{∗}[Y] is an element of the homology group of degree 2n, of a 4n-dimensional locally Riemannian symmetric space

X = Γ\SO_{0}(2n, 2)/(SO(2n) × SO(2)),

of which the universal covering is the bounded symmetric domain of type IV. In particular, X is a Kähler manifold. Since X is compact and orientable, we can regard the modular symbol as an element M(Y) of the cohomology group H^{2n}(X; ℂ) via the Poincaré duality. Let

M^{p,q}(Y) ∈ H^{p,q}(X; ℂ)

be its Hodge component of type (p, q) such that p + q = 2n. Then, since X is Kähler, we have

M(Y) = ∑_{p+q=2n} M^{p,q}(Y).

Thus, the Hodge components {M^{p,q}(Y) : p + q = 2n} give a finer structure of modular symbols. Takayuki Oda conjectured a vanishing theorem of the middle Hodge component M^{n,n}(Y), from a view point of automorphic forms (see [66]). This conjecture has been solved in [52] by using our criterion of discrete decomposable restrictions (Theorem 1.5).

Theorem 4.1 (Hodge component of the modular symbol). There exists a universal element η in the cohomology group H^{n,n}(X; ℂ), such that

M^{n,n}(Y) = \frac{volume of Y}{volume of X} η.

Here, “universal” means that the element η is given in terms of Lie algebras explicitly, and does not depend on Γ. In particular, η is contained in the image of the canonical map

H^{n,n}(𝔤, K; ℂ) → H^{n,n}(X, ℂ).

Theorem 4.1 follows from a general vanishing theorem [52] of modular symbols for a pair (G, G′) of reductive groups. The key point there is that the integration of a harmonic form ω over Y becomes zero if ω comes from an infinite dimensional irreducible unitary representation π of G such that the restriction π|_{G′} is discretely decomposable. Then, for the latter condition, we can use a criterion given in Theorem 1.5.

More precisely, what we need in the above special case is the following representation theoretic result: the restriction π|_{K′} is K′-admissible for any π ∈ Ĝ if H^{n,n}(𝔤, K; π_{K}) ≠ 0. This statement follows from an easy computation of Theorem 1.5.

Instead of an explanation of further technical details, we mention a flavor of the proof by the following diagram that compares relevant results where the discreteness of spectrum plays an important role in the understanding of topology in other settings (See [42] for more details).

Discreteness of spectrum ⇒ Geometry ⇔ Function spaces

(representations)

X: compact Riemannian manifold Laplacian Δ_{X} has only discrete spectrum in L^{2}(X) ⇒

Hodge theory

topology ⇔ harmonic forms

⇓ refinement

L^{2}(Γ\G) is decomposed into a discrete direct sum of Ĝ (Gel’fand-Piateski-Shapiro, [15]) ⇒ Matsushima-Murakami formula [8], [61] (1960s)

⇓ (object ⇒ morphism)

Criterion for discrete decomposable restrictions of A_{𝔮}(λ) to subgroups (Kobayashi, [44]) ⇒ Vanishing theorem of a modular symbol (Kobayashi-Oda, [52])

Here, Matsushima-Murakami’s formula describes the cohomology groups of a locally Riemannian symmetric space in a representation theoretic way (study of objects), and Oda-Kobayashi’s vanishing theorem concerns with maps between locally Riemannian symmetric spaces (study of morphisms).

4.C. Application to non-commutative harmonic analysis

—construction of new discrete series representations

for non-symmetric spaces. Suppose a homogeneous space G/H carries a G-invariant Borel measure. This

is the case if (G, H) is a pair of reductive Lie groups in the sense that G is a reductive Lie group and H is a closed subgroup which is reductive in G. Then we have a natural unitary representation of G on the Hilbert space L^{2}(G/H). We say π is a discrete series representation for G/H, if π ∈ Ĝ is realized in a closed subspace of L^{2}(G/H), equivalently, if the space of continuous intertwining operators Hom_{G}(π, L^{2}(G/H)) ≠ {0}. Harish-Chandra’s discrete series representation π is the case where H = {e}. A discrete series representation corresponds to a discrete spectrum in the Plancherel formula, namely, the irreducible decomposition of the unitary representation L^{2}(G/H).

Discrete series representations are regarded as realizations of irreducible representations, through which infinite dimensional representation theory interacts lively with global analysis. Discrete series representations are not only a fundamental object in non-commutative harmonic analysis, but also play important roles in the following topics.

(1) Construction of tempered series representations. Plancherel theorems for group manifolds and semisimple symmetric spaces are known by Harish-Chandra, Delorme, van den Ban, Oshima, Schlichtkrull and others. The support of the Plancherel formula consists of tempered representations which are obtained as the cuspidal parabolic induction of discrete series representations for smaller symmetric spaces ([12], [18], [68]).

(2) Topology of locally Riemannian symmetric spaces (e.g. a non-vanishing theorem of modular symbol due to Tong and Wang [73]).

(3) Obstruction of the injectivity of L^{p}-Pompeiu problem in integral geometry; see [11] for simply connected solvable Lie groups; see [72] for SL(2, ℝ); see [45], Theorem 1.2.17 for general reductive Lie groups and their homogeneous spaces.

(4) A part of “isolated” irreducible unitary representations in the Fell topology

(for example, [70]). 4.C.1. A current status on discrete series representations.

As the above observation indicates, the following is one of fundamental problems in non-commutative harmonic analysis:

Problem 4.C.1.

1) Find a condition on the pair of groups (G, H) such that there exists a discrete series representation for the homogeneous space G/H.

2) If exist, construct and classify discrete series representations for G/H.

These problems have not been solved in the general setting where (G, H) is a pair of real reductive Lie groups. Here are some special cases where answers are known:

i) The case where G/H is a semisimple Lie group, namely, H = {e}. (Discrete series representations exist if and only if rank G = rank K. All discrete series representations are classified by Harish-Chandra. Geometric constructions of discrete series representations were studied by Atiyah, Hotta, Langlands, Okamoto, Parthasarathy, Schmid and others. Algebraic constructions were also studied by Enright, Vogan, Wallach, Zuckerman and others. ii) The case where G/H is a semisimple symmetric space. Discrete series representations exist if and only if rank G/H = rank K/(H ∩ K). They are constructed by Flensted-Jensen and Oshima-Matsuki. The classification has been almost done, but certain subtle problems such as non-vanishing conditions and multiplicity-one conjecture have not been completed at least in the literature.

The group case (i) can be regarded as a special case of (ii) by putting (G, H) = (G_{1} × G_{1}, diag G_{1}). We refer to a survey [36], §4 and the references therein for (i) and (ii), and for more general results (e.g. vector bundle cases). 4.C.2. Relation to the discrete branching law.

The known methods used in (i) or (ii) are powerful on the one hand, but they valid only in limited cases on the other hand, namely, only for semisimple symmetric spaces. For instance, one of the key methods for the analysis on symmetric spaces is the Flensted-Jensen duality [14], which can be defined only when G/H is a symmetric space. Consequently, we need to a completely new machinery for the analysis on a more general homogeneous space.

To construct new discrete series representations, our idea here is based on the restriction of unitary representations. We will explain a rough idea of how the proof goes (see [37] and [46] for precise formulation and some concrete examples):

\underline{Step a)} Suppose G/H is a homogeneous space, on which we want to construct discrete series representations. We embed G/H into a larger space G̃/H̃ for which harmonic analysis is well-understood (e.g. G̃/H̃ is a group manifold or a symmetric space):

ι : G/H ↪ G̃/H̃.

Then, we consider the pullback of functions (or possibly, after taking finitely many normal derivatives):

ι^{∗} : C^{∞}(G̃/H̃) → C^{∞}(G/H), f ↦ f ∘ ι.

\underline{Step b)} Take an irreducible G̃-representation π in C^{∞}(G̃/H̃) and pick up a function f on G̃/H̃ that belongs to a representation space of π. We expand the restriction ι^{∗}f into irreducible components (say, (ι^{∗}f)_{λ} for λ ∈ Ĝ) as representations of the subgroup G according to the branching law of the restriction π|_{G}, and then estimate the asymptotic behavior of each component (ι^{∗}f)_{λ} along the submanifold G/H at infinity (see [46]). \underline{Step c)} Find the asymptotic behavior of the measure on G/H, and compare the asymptotic behavior of measures at infinity between G/H and G̃/H̃ (see [43]).

If the image of ι is a generic orbit (e.g. a principal type orbit in the sense of Richardson), we do not need the steps (b) and (c), and we can construct discrete series representations by an elementary argument (this case was previously carried out for some special homogeneous spaces [21], [35], [37], [56]). For instance, by this approach, one can reduce the classification problem of discrete series representations of some non-symmetric spherical homogeneous spaces G/H, such as

G/H = SU(n + 1, n)/Sp(n, ℝ), G_{2}(ℝ)/SL(3, ℝ), ⋯

to that of discrete spectrum of some branching laws of discrete series representations of a (larger) space G̃/H̃ ([37]). However, it often happens that non-generic orbits give more interesting examples (in other words, some homogeneous spaces G/H can be embedded into G̃/H̃ only as non-generic orbits, and then the steps (b) and (c) become necessary.

In the step (b), if the branching law is discretely decomposable when restricted from G̃ to G, then one can prove that the asymptotic behavior of each irreducible component (ι^{∗}f)_{λ} has a nice decay inherited from that of the function f (see [46], §3). Consequently, if f satisfies an appropriate asymptotic decay on G̃/H̃, then each non-zero irreducible component (ι^{∗}f)_{λ} generates a discrete series representation for G/H. We remark that the assumption of discrete decomposability is crucial because such a nice decay cannot be expected if π|_{G} contains continuous spectrum.

Let us give a few comments on the step (c). We note that a reductive homogeneous space G/H does not always have a polar coordinate (for a special G/H such as a symmetric space, there is a polar coordinate G = KAH and accordingly we have an integration formula on G/H essentially on the Abelian group A). The absence of polar coordinates causes difficulties in describing explicitly the asymptotic behavior of the G-invariant measure on G/H at infinity. The idea of [47] for the step (c) for a non-symmetric space G/H is to avoid working on the pseudo-Riemannian manifold G/H (right coset space) itself, but to lift it to a larger dimensional nonhomogeneous space K × (𝔭/𝔭 ∩ 𝔥) where we can employ a comparison theorem of negatively curved Riemannian manifold K\G (left coset space).

By using the steps (a), (b) and (c), we can prove that there exist infinitely many discrete series representations on homogeneous spaces such as:

G/H = Sp(2n, ℝ)/(Sp(n_{0}, ℂ) × GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ)),

where (n_{0}, n_{1}, ⋯, n_{k}) is an arbitrary partition of n. We note that the above G/H is a symmetric space if and only if

n_{1} = n_{2} = ⋯ = n_{k} = 0,

in the above example. The existence of discrete series representations were previously known by Flensted-Jensen [14] only in this special case.

It should be noted that the above approach deals with a family of homogeneous spaces of G simultaneously, rather than a single homogeneous space alone. The point here is that different homogeneous spaces of G can arise as G-orbits on G̃/H̃. For instance, the above homogeneous space

G/H = Sp(2n, ℝ)/(Sp(n_{0}, ℂ) × GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ)),

for an arbitrary partition

n = n_{0} + ⋯ + n_{k}

arises as a G-orbit on

G̃/H̃ = (Sp(2n, ℝ) × Sp(2n, ℝ))/ diag(Sp(2n, ℝ))

(see [46]). Then we can “treat simultaneously” these homogeneous spaces by using branching laws of unitary representations form G̃ to G. For instance, this explains a phenomenon that the same representations can occur as discrete series representations on certain different homogeneous spaces.

The orbit decomposition G\G̃/H̃ has been recently studied by Iida and Matsuki [60]. Their description is useful in finding how G/H is embedded into a larger space G̃/H̃.

The above approach gives not only new discrete series representations on nonsymmetric homogeneous spaces as stated, but also gives some new viewpoint even for analysis on symmetric spaces, where there have been already extensive works in the literature. For example, we can prove (without using serious results on semisimple symmetric spaces) the following new geometric theorem, by making use of the criterion of discretely decomposable restrictions (Theorem 1.5 and Theorem 2.4):

Theorem 4.2 (a necessary and sufficient condition for the existence of holomorphic discrete series representation on symmetric spaces). Suppose G/H is a noncompact irreducible symmetric space. Then the following two conditions on (G, H) are equivalent:

i) There exist unitary highest weight representations of G that can be realized as discrete series representations for G/H. ii) H/(H ∩ K) is a real form of the complex manifold G/K.

Example 4.3. Let G/H = (SL(2, ℝ) × SL(2, ℝ))/ diag(SL(2, ℝ)). We put D := {z ∈ ℂ : |z| < 1}, the unit disc. Then the natural embedding H/(H ∩ K) ↪ G/K is realized as the following map:

D ↪ D × D, z ↦ (z, z̄).

In particular, H/(H ∩ K) is a real form of G/K, that is, the geometric condition (ii) is satisfied. Then, Theorem 4.2 gives a new explanation of the well-known fact that there exist holomorphic discrete series representations of a group manifold G/H ≃ SL(2, ℝ).

The result (i) ⇒ (ii) in Theorem 4.2 is new. The opposite direction (ii) ⇒ (i) was previously proved by a completely different method (’Olafsson-Ørsted [67]).

4.D. Discontinuous groups versus restrictions of unitary representations. 4.D.1. Discretely decomposable restrictions revisited.

Let U(H) be the group of unitary operators on a Hilbert space H, and we consider an irreducible unitary representation of G realized on H, namely, a group homomorphism

π : G → U(H).

The restriction to a subgroup G′ of G is nothing but the composition of the following group homomorphisms:

(4.4.1)  G′ ⊂ G →^{π} U(H).

If G′ is compact, then the restriction π|_{G′} is always discretely decomposable. We have seen in §1 that the restriction π|_{G′} can be discretely decomposable even when G′ is non-compact. We may discuss the discrete decomposability of the restriction π|_{G′} from the following viewpoint:

“In the infinite dimensional group U(H), the image of a non-compact Lie group of G′ may behave as if it were a compact group.”

4.D.2. Properly discontinuous actions of discrete groups.

Let us consider a different setting. Let Γ be a topological group acting continuously on a manifold M. We define a subset of Γ by

Γ_{S} := {γ ∈ Γ : γ · S ∩ S ≠ ∅},

for a subset S of M. The action of Γ on M is said to be proper [69] if Γ_{S} is compact for any compact subset S in M; properly discontinuous if Γ_{S} is finite for any compact subset S in M. We note that Γ acts properly discontinuously if and only if Γ is discrete and acts properly on M.

A typical example of properly discontinuous actions is the covering transformation of the fundamental group on the universal covering space. Conversely, if a torsion-free discrete group Γ acts on a manifold M properly discontinuously, then there is a natural manifold structure on the space of Γ-orbits, denoted by Γ\M, and the quotient map M → Γ\M becomes a covering map.

We observe that the action of Γ on M is written as a homomorphism from Γ to the group Diffeo(M) of diffeomorphisms of M, or the group Homeo(M) of homeomorphisms of M:

Γ → Homeo(M).

The action of a finite group are always properly discontinuous. Then, again, properly discontinuous actions may be discussed from the following viewpoint

“In the infinite dimensional group Homeo(M), the image of a discrete group Γ

may behave as if it were a compact (or finite) group. ”

4.D.3. Restriction of unitary representations and proper actions. The above two cases are summarized as follows: Actions of finite groups are obviously properly discontinuous. It can happen

that actions of infinite groups are still properly discontinuous, such as covering transformations.

On the other hand, branching laws of compact groups are always discretely decomposable, and it can happen that branching laws with respect to non-compact subgroups are still discretely decomposable.

Comparing these two examples, one might ask the following question:

Question. Is there any relation between “discrete decomposability of branching laws of unitary representations of Lie groups” and “properly discontinuous actions of discrete groups”?

As a special case, let us consider the setting where a Lie group G acts transitively on M. Let H be the isotropy subgroup at a point x_{0} of M. Then we have a natural homeomorphism G/H ≃ M, gH ↦ gx_{0}. Then a discrete subgroup Γ of G acts on M by the left translation. This means that we have a group homomorphism:

(4.4.2)  Γ ⊂ G → Diffeo(G/H).

We shall compare the two settings (4.4.1) and (4.4.2) below. 4.D.4. Criteria— discretely decomposable representations

and properly discontinuous actions. Without proof and precise formulation, we give a brief summary of the following

known criteria: i) In the setting (4.4.1), a sufficient condition for the discrete decomposability of

the restriction π|_{G′} was given in Theorem 1.5, roughly in the following form:

(the cone determined by G′) ∩ (the cone determined by π) = {0}.

ii) In the setting (4.4.2) (with Γ replaced by G′), a necessary and sufficient condition for a reductive subgroup G′ on the homogeneous space G/H to act properly was proved by the author in 1989 [27]:

(the linear subspace determined by G′)

∩ (the linear subspace determined by H)

= {0} modulo the action of a finite group (Weyl group).

iii) In the setting (4.4.2), a necessary and sufficient condition for the action of a discrete subgroup Γ on the homogeneous manifold G/H to be properly discontinuous is given in the following form:

{a subset determined by Γ} ∩ {a tube determined by H} is relatively compact modulo the action of a finite group (Weyl group).

This criterion generalizes [27] and was proved independently by Benoist [4] and Kobayashi [39].

4.D.5. Actions of discrete groups and branching laws

of unitary representations. In the previous subsection, (i) concerns with an analytic representation theory

(discreteness of spectrum), and (ii) and (iii) concern with topological problems (proper actions). Accordingly, the objects and methods employed there are completely different. However, the criteria themselves are apparently similar to one another, and may suggest a relationship in the following diagram:

(iii) Discrete version:

Properly discontinuous actions

of discrete groups

↓↑

(ii) Continuous version:

Proper actions of connected Lie groups

↓↑

(i) Representation theory:

Discrete decomposable restrictions of unitary representations of Lie groups

In fact, a first non-trivial example (see [28]) of the discrete branching laws of the restriction π|_{G′} (i.e. π is not a highest weight representation, and G′ is not compact) was inspired by the above diagram (see also §0). More precisely, the idea of [28] consists of the following four steps:

Step a) First, take a uniform lattice^{6} Γ for the semisimple symmetric space G/H (see [27] for the construction). Step b) Take the Zariski closure of Γ in G, for which we write G′. Step c) Take a discrete series representation π(∈ Ĝ) for G/H (see [14] for the construction). Step d) Find branching laws of the restriction π|_{G′}.

A simplest example is given in the following setting:

G/H = SU(2, 2)/U(2, 1) (an open subset of P^{3}ℂ),

G′ = Sp(1, 1) ≃ Spin(4, 1) (de Sitter group),

Γ is a uniform lattice of G′ = Sp(1, 1),

π is a discrete series representation for G/H.

We note that the above representation π has the Gel’fand-Kirillov dimension 5 and is not a highest weight representation. Then the restriction π|_{G′} turns out to be discretely decomposable (see [36], Example 3.3). As we mentioned (see Theorem 1.5), the discrete decomposability of the restriction was later formulated in terms of representation theory, apart from the above setting on global analysis on homogeneous spaces.

The above setting also gives an example of an interesting Riemannian structure g on a 6-dimensional simply connected manifold M = G/H: (M, g) is a non-compact covering of a compact Riemannian manifold, and there exists L^{2}-eigenfunctions of the Laplace-Beltram operator (related problems have been studied by Sunada [29], [64]).

With regard to Step (a), the following has been studied intensively in the last decade:

Problem. “Does a pseudo-Riemannian homogeneous manifold admits a uniform lattice ?”

Various approaches to this problem include the structural results of Lie groups, a criterion of proper actions of non-compact Lie groups, cohomology groups of discrete groups, characteristic classes, symplectic geometry, ergodic theory, and the restriction of unitary representations, etc. by Benoist, Corllette, Labourie, Margulis, Oh, Ono, Witte, Zimmer and the author after a general theory [27] in 1989 (see [4], [5], [30], [32], [33], [34], [39], [41], [48], [59], [65], [80]). In particular, a recent method due to Margulis [59] is based on the asymptotic behavior of matrix coefficients of the restrictions of unitary representations, which should merit further study, in the point that it might strengthen a tie between unitary representations and discontinuous groups.

6In contrast to Borel’s theorem [7] on the existence of a uniform lattice of an arbitrary Riemannian symmetric space G/H, there does not always exist a uniform lattice for a non-Riemannian homogeneous manifold (for example, a semisimple symmetric space). Even the so-called Calabi-Markus phenomenon [10] occurs if H is non-compact. We refer to [41] for an exposition to discontinuous groups on non-Riemannian homogeneous manifolds, developed rapidly in the last decade.

In the above diagram, the last ↓↑ is just a guiding principle, and a rigorous formulation is not known. It is mysterious to me if there is an intrinsic interaction between properly discontinuous actions and branching problems of unitary representations, especially discretely decomposable restrictions.

⟨Acknowledgement⟩

The translation of this article was mostly done during my visit to Harvard University 2000-2001. I express my hearty thanks to the staff there, in particular, Professor W. Schmid who gave me a wonderful atmosphere of research.

References

[1] J. Adams, Discrete spectrum of the dual reductive pair (O(p, q), Sp(2m)), Invent. Math. 74 (1984), 449–475.

[2] A. Ash and A. Borel, Generalized modular symbols, Cohomology of arithmetic groups and automorphic forms, Lecture Notes in Math., vol. 1447, Springer, 1990, pp. 57-76.

[3] M. F. Atiyah, The Harish-Chandra character, London Math. Soc. Lecture Note Series 34 (1979), 176–181.

[4] Y. Benoist, Actions propres sur les espaces homogenes reductifs, Annals of Math. 144 (1996), 315–347.

[5] Y. Benoist and F. Labourie, Sur les espaces homogenes modeles de varietes compactes, I. H. E. S. Publ. Math. (1992), 99–109.

[6] M. Berger, Les espaces symétriques non compacts, Ann. Sci. Ecole Norm. Sup. (3) 74 (1957), 85–177.

[7] A. Borel, Compact Clifford-Klein forms of symmetric spaces, Topology 2 (1963), 111-122.

[8] A. Borel and N. Wallach, Continuous Cohomology, Discrete Subgroups, and Representations of Reductive Groups, Ann. Math. Stud. 94, Princeton U.P., 1980.

[9] W. Borho and J. L. Brylinski, Differential operators on homogeneous spaces I, Invent. Math. 69 (1982), 437-476, Part III, 80 (1985), 1-68.

[10] E. Calabi and L. Markus, Relativistic space forms, Annals of Math. 75 (1962), 63–76.

[11] A. L. Carey, E. Kaniuth and W. Moran, The Pompeiu problem for groups, Math. Proc. Camb. Phil. Soc. 109 (1991), 45-58.

[12] P. Delorme, Formule de Plancherel pour les espaces symétriques réductifs, Annals of Math. 147 (1998), 412–452.

[13] T. Enright, R. Howe and N. Wallach, A classification of unitary highest weight modules, Representation theory of reductive groups, Progress in Math., vol. 40, Birkhäuser, 1983, pp. 97–143.

[14] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Annals of Math. 111 (1980), 253–311.

[15] I. M. Gel’fand, M. I. Graev and I. Piateski-Shapiro, Representation Theory and Automorphic Functions, Saunders Math. Books, 1969.

[16] B. H. Gross and N. R. Wallach, A distinguished family of unitary representations for the exceptional groups of real rank = 4, Lie Theory and Geometry, In honor of Bertram Kostant, Progress in Math., vol. 123, Birkhäuser, 1994, pp. 289–304.

[17] Harish-Chandra, Representations of semi-simple Lie groups, I, III, Trans. A. M. S. 75, (1953), 185–243; 76, (1954), 234–253; IV, Amer. J. Math. 77, (1955), 743–777.

[18] Harish-Chandra, Harmonic analysis on real reductive groups, I, J. Funct. Anal., 19, (1975), 104–204; II, Invent. Math., 36, (1976), 1–55; III, Annals of Math., 104, (1976), 117–201.

[19] H. Hecht and W. Schmid, A proof of Blattner’s conjecture, Invent. Math. 31 (1976), 129–154.

[20] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math. 33 (1979), Amer. Math. Soc., 275–285.

[21] J-S. Huang, Harmonic analysis on compact polar homogeneous spaces, Pacific J. Math. 175 (1996), 553—569.

[22] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29–53.

[23] M. Kashiwara, T. Kawai and T. Kimura, Foundations of Algebraic Analysis, Princeton Math. Series, vol. 37, Princeton Univ. Press, 1986.

[24] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math. 44 (1978), 1–47.

[25] M. Kashiwara and M. Vergne, K-types and singular spectrum, Lect. Notes in Math., vol. 728, Springer, 1979, pp. 177–200.

[26] A. Knapp and D. Vogan, Cohomological Induction and Unitary Representations, Princeton U.P., 1995.

[27] T. Kobayashi, Proper action on a homogeneous space of reductive type, Math. Ann. 285 (1989), 249–263.

[28] T. Kobayashi, Unitary representations realized in L^{2}-sections of vector bundles over semisimple symmetric spaces, Proceedings at the 27-28th Symp. of Functional Analysis and Real Analysis (1989), Math. Soc. Japan, 39–54.

[29] T. Kobayashi, K. Ono and T. Sunada, Periodic Schrödinger operators on a manifold, Forum Mathematicum 1 (1989), 69–79.

[30] T. Kobayashi and K. Ono, Note on Hirzebruch’s proportionality principle, J. Fac. Sci. Univ. Tokyo 37-1 (1990), 71–87.

[31] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p−m, q; 𝔽), vol. 462, Memoirs of Amer. Math. Soc., 1992, (106 pages).

[32] T. Kobayashi, A necessary condition for the existence of compact Clifford-Klein forms of homogeneous spaces of reductive type, Duke Math. J. 67 (1992), 653–664.

[33] T. Kobayashi, Discontinuous groups acting on homogeneous spaces of reductive type, Proc. of ICM-90 Satellite Conference on “Representation Theory of Lie Groups and Lie Algebras” at Fuji-Kawaguchiko, World Scientific, 1992, pp. 59–75.

[34] T. Kobayashi, On discontinuous groups acting on homogeneous spaces with non-compact isotropy subgroups, J. Geom. Physics 12 (1993), 133–144.

[35] T. Kobayashi, The Restriction of A_{𝔮}(λ) to reductive subgroups, Part I, Proc. Japan Acad. 69 (1993), 262–267; Part II, ibid. 71 1995, 24–26.

[36] T. Kobayashi, Harmonic analysis on homogeneous manifolds of reductive type and unitary representation theory, Sugaku 46 (1994), Math. Soc. Japan (in Japanese), 124–143; Translations, Series II, Selected Papers on Harmonic Analysis, Groups, and Invariants (K. Nomizu, ed.) 183 (1998), A.M.S, 1–31.

[37] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205.

[38] T. Kobayashi, Introduction to harmonic analysis on spherical homogeneous spaces, Proceedings of the 3rd summer school on number theory, Homogeneous Spaces and Automorphic Forms (F. Sato, eds.), 1995, pp. 22–41. (Japanese)

[39] T. Kobayashi, Criterion of proper actions on homogeneous spaces of reductive groups, J. Lie Theory 6 (1996), 147–163.

[40] T. Kobayashi, On the restriction of unitary representations and their applications, Proceedings of the Symposium of Representation Theory held at Mikawa-Heights (November 19–22, 1996) (Takenaka, eds.), 1996, pp. 131–141.

[41] T. Kobayashi, Discontinuous groups and Clifford-Klein forms of pseudo-Riemannian homogeneous manifolds, Algebraic and Analytic Methods in Representation Theory (H. Schlichtkrull and B. Ørsted, ed.), Perspectives in Mathematics 17, Academic Press, 1996, pp. 99–165, Lecture Notes of the European School on Group Theory 1994.

[42] T. Kobayashi, Vanishing theorem for modular symbols on locally symmetric spaces (Lecture notes in Japanese, taken by S. Ishikawa), Proceedings of “Representation theory and related topics” (held at Kurashiki, November 1996) (N. Shimeno, ed.), 1996, pp. 1–16, http://www.merry.xmath.ous.ac.jp/math /Nov5th1996.html.

[43] T. Kobayashi, Invariant measures on homogeneous manifolds of reductive type, J. reine und angew. Math. 490 (1997), 37–53.

[44] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups II — micro-local analysis and asymptotic K-support, Annals of Math. 147 (1998), 709–729.

[45] T. Kobayashi, Convex domains and Fourier transform on spaces of constant curvature, Travaux en Cours (P. Torasso, ed.), Hermann, Paris, 112 pages (to appear).

[46] T. Kobayashi, Discrete series representations for the orbit spaces arising from two involutions of real reductive Lie groups, J. Funct. Anal. 152 (1998), 100–135.

[47] T. Kobayashi, Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu (K. Mimachi, ed.), 1997, pp. 7–13.

[48] T. Kobayashi, Deformation of compact Clifford-Klein forms of indefinite-Riemannian homogeneous manifolds, Math. Ann. 310 (1998), 394–408.

[49] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups III — restriction of Harish-Chandra modules and associated varieties, Invent. Math. 131 (1998), 229–256.

[50] T. Kobayashi, Discretely decomposable restrictions of unitary representations of reductive Lie groups —examples and conjectures., Analysis on homogeneous spaces and representation theory of Lie groups, Okayama–Kyoto (1997) (T. Kobayashi et.al., eds.), vol. 26, Math. Soc. Japan, Tokyo, 2000, pp. 99-127.

[51] T. Kobayashi, Theory of discrete decomposable branching laws of unitary representations of semisimple Lie groups and its applications, (in Japanese), Proceedings of the annual meeting of Mathematical Society of Japan, (Gakushuin University, March 25–28, 1999) (1999), Math. Soc. Japan, 1–19.

[52] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on locally symmetric spaces, Comment. Math. Helvetici 73 (1998), 45–70.

[53] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits, C. R. Acad. Sci. Paris 326 (1998), 925–930.

[54] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4), Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory (Connes et al, eds.), vol. 92, Birkhäuser, 1990, pp. 85–124.

[55] R. Langlands, On the classification of irreducible representations of real algebraic groups, mimeographed notes (1973).

[56] J-S. Li, On the discrete series of generalized Stiefel manifolds, Trans. A. M. S. 340 (1993), 753–766.

[57] J-S. Li, Two reductive dual pairs in groups of type E, Manuscripta Math. 91 (1996), 163–177.

[58] G. W. Mackey, Induced representations of locally compact groups I, Annals of Math. 55 (1952), 101–139.

[59] G. Margulis, Existence of compact quotients of homogeneous spaces, measurably proper actions, and decay of matrix coefficients, Bul. Soc. math. France 125 (1997), 1–10.

[60] T. Matsuki, Double coset decompositions of algebraic groups arising from two involutions I, J. Algebra 175 (1995), 865–925.

[61] Y. Matsushima and S. Murakami, On vector bundle valued harmonic forms and automorphic forms on symmetric spaces, Annals of Math. 78 (1963), 365-416.

[62] C. C. Moore, Ergodicity of flows on homogeneous spaces, Amer. J. Math. 88 (1966), 154-178.

[63] Y. A. Neretin and G. I. Ol’shanskii, Boundary values of holomorphic functions, special unitary representations of O(p, q), and their limits, Zapiski Nauchnykh Seminarov POMI 223 (1995), 9–91.

[64] T. Ochiai, On the spectrum of the Laplacian on Riemannian manifolds (On the work of Toshikazu Sunada, winner of the 15th Mathematical Society of Japan Prize), Sugaku 39 (1987), Math. Soc. Japan, 237–242.

[65] H. Oh and D. Witte, Compact Clifford-Klein forms of homogeneous spaces of SO(2, n), preprint (1999).

[66] T. Oda, Real analysis for automorphic forms, Sugaku 50 (1998), Math. Soc. Japan, 350–357.

[67] G. Ólafsson and B. Ørsted, The holomorphic discrete series of an affine symmetric space, I, J. Funct. Anal. 81 (1988), 126–159.

[68] T. Oshima, Harmonic analysis on semisimple symmetric spaces, Sugaku 37 (1985), Math. Soc. Japan, 97–112.

[69] R. S. Palais, On the existence of slices for actions of noncompact Lie groups, Annals of Math. 73 (1961), 295–323.

[70] H. Schlichtkrull, The Langlands parameters of Flensted-Jensen’s discrete series for semisimple symmetric spaces, J. Funct. Anal. 50 (1983), 133–150.

[71] W. Schmid, Die Randwerte holomorphe Funktionen auf hermetisch symmetrischen Raumen, Invent. Math. 9 (1969-70), 61–80.

[72] A. Sitaram, Some remarks on measures on noncompact semisimple Lie groups, Pacific J. Math. 110 (1984), 429–434.

[73] Y. L. Tong and S. P. Wang, Geometric realization of discrete series for semisimple symmetric spaces, Invent. Math. 96-2 (1989), 425–458.

[74] D. A. Vogan, Jr., The algebraic structure of the representations of semisimple Lie group I, Annals of Math. 109 (1979), 1–60.

[75] D. A. Vogan, Jr., Representations of Real Reductive Lie Groups, Progress in Math. 15, Birkhäuser, 1981.

[76] D. A. Vogan, Jr., Unitarizability of certain series of representations, Annals of Math. (1984), 141–187.

[77] D. A. Vogan, Jr., Associated varieties and unipotent representations, Harmonic Analysis on Reductive Lie Groups, vol. 101, Birkhäuser, 1991, pp. 315–388.

[78] N. Wallach, Real Reductive Groups I, II, Pure and Appl. Math., vol. 132, Academic Press, 1988, 1992.

[79] J. Xie, Restriction of discrete series of SU(2, 1) to S(U(1) × U(1, 1)), J. Funct. Anal. 122 (1994), 478–518, ph.D dissertation, Rutgers University.

[80] R. J. Zimmer, Discrete groups and non-Riemannian homogeneous spaces, J. Amer. Math. Soc. 7 (1994), 159–168.

References added on the occasion of translation.

[81] B. Gross and N. Wallach, Restriction of small discrete series representations to symmetric subgroups, Proc. Sympos. Pure Math. 68 (2000), A.M.S., 255–272.

[82] J-S. Huang, P. Pandžić, and G. Savin, New dual pair correspondences, Duke Math. 82 (1996), 447–471.

[83] T. Kobayashi, Conformal geometry and analysis on minimal representations, Lecture Notes of the Winter School 2002 on geometry and physics, Czech Republic, to appear a special voluve of Rendiconti del Circolo Matematico di Palermo.

[84] T. Kobayashi, Discontinuous groups for non-Riemannian homogeneous spaces, Mathematics Unlimited —2001 and Beyond (B. Enquist and W. Schmid, eds.), Springer-Verlag, 2000, pp. 723-728.

[85] T. Kobayashi, Branching problems of unitary representations, Proc. of the I.C.M. 2002, at Beijing 2 (2002), 615–627.

[86] T. Kobayashi, Admissible restrictions of unitary representations of reductive Lie groups – discrete decomposability and symplectic geometry, in preparation.

[87] T. Kobayashi, Restrictions of unitary representations of real reductive groups, Lecture notes of the European School on Group Theory 2000 (to appear).

[88] T. Kobayashi and B. Ørsted, Analysis on the minimal representation of O(p,q), I, II, III, (math.RT/0111083, math.RT/0111085, math.RT/0111086) (to appear in Adv. Math.).

[89] S-T. Lee and H-Y. Loke, Degenerate principal series of U(p,q) and Spin(p,q), Compositio Math. 132 (2002), 311–348.

[90] H-Y. Loke, Restrictions of quaternionic representations, J. Funct. Anal. 172 (2000), 377–403.

[91] H-Y. Loke, Howe quotients of unitary characters and unitary lowest weight modules, preprint.

[92] B. Ørsted and J. Vargas, Restriction of square integrable representations: discrete spectrum, preprint.

[93] S. A. Salamanca-Riba and D. A. Vogan, Jr., On the classification of unitary representations of reductive Lie groups, Ann. of Math. 148 (1998), 1067–1133.

[94] W. Schmid and K. Vilonen, Characteristic cycles and wave front cycles of representations of reductive Lie groups, Ann. of Math. 151 (2000), 1071–1118.

RIMS, Kyoto University, Sakyo-ku, Kyoto, 606-8502, Japan E-mail address: toshi@kurims.kyoto-u.ac.jp