Discretely decomposable restrictions of unitary

representations of reductive Lie groups — examples and conjectures

Toshiyuki Kobayashi

Table of Contents

§1. Breaking symmetry and hidden symmetry §2. Analytic and algebraic notion of discretely decomposable restrictions §3. Compact-like actions in infinite dimensional groups §4. Criterion for discretely decomposable restrictions §5. Conjectures on discrete branching laws

§1. Breaking symmetry and hidden symmetry

\underline{1.1.} The purpose of this exposition is to give a survey of the recent study of discrete decomposable restrictions of unitary representations of real reductive Lie groups ([24], [30], [31]). We shall also give some perspectives by examples and open problems.

There have been recent developments in connection with restrictions of unitary representations such as:

i) finding explicit branching laws ([8], [10], [16], [23], [24], [33], [36], [37], [40], [44], [57]), ii) estimate of multiplicities in the branching laws ([30], [35]), iii) topology of modular varieties in locally Riemannian symmetric spaces ([34]), iv) construction of new discrete series representations for non-symmetric homogeneous spaces ([15], [22], [24], [27], [28], [32], [39]),

v) finding explicit parameters λ for which Zuckerman-Vogan’s derived functor modules A_{𝔮}(λ) are non-zero with λ singular in some special settings ([22], §4), vi) existence problem of compact Clifford-Klein forms of non-Riemannian homogeneous spaces ([21], [25], [26], [42]).

Some of these attempts have been successful especially in the framework of discretely decomposable restrictions. In this article, with these applications in mind, we shall concentrate on a theoretical aspect of discrete decomposable restrictions.

\underline{1.2.} Let G be a group and G′ its subgroup. Suppose that π : G → GL(H) is a representation of G. The restriction of π to G′ defines a representation of G′, denoted by (π|_{G′}, H), or simply by π|_{G′}. Conversely, a representation of a smaller group G′ is sometimes extended to a larger group G. In quantum physics, the former arises as breaking symmetry, and the latter is sometimes called hidden symmetry.

Hidden Symmetry
⌢⟩
G′  ⊂  G →^{π} GL(H)
⟨⌣
Breaking Symmetry

Suppose that G′ ⊂ G are Lie groups. We denote by Ĝ the unitary dual of G, that is, the set of equivalence classes of irreducible unitary representations of G. Likewise, Ĝ′ for G′.

The restriction π|_{G′} is not necessarily irreducible. If G′ is of type I in the sense of von Neumann algebras (any reductive Lie group G′ is the case), then the restriction π|_{G′} is uniquely decomposed into the direct integral of irreducible unitary representations of G′:

(1.2.1)  π|_{G′} ≃ ∫ ⊕ ; _{G′} m_{π}(τ) τ dμ(τ),

where dμ is a Borel measure on Ĝ′ with the Fell topology and

m_{π}(·): Ĝ′ → ℕ ∪ {∞}

is the multiplicity defined almost everywhere with respect to dμ.

The formula (1.2.1) is called the branching law for the restriction π|_{G′}.

\underline{1.3.} As an introduction of a hidden symmetry and its branching law, we consider the Fourier series expansion for L^{2}(S^{1}) and the Fourier transform for L^{2}(ℝ). The Parseval-Plancherel formula gives the isometry between L^{2}(S^{1}) (respectively, L^{2}(ℝ)) and the direct integral of one dimensional Hilbert spaces ℂe^{\sqrt{−1}nx} over a countable measure on ℤ (respectively, over the Lebesgue measure on ℝ):

(1.3.1)(a) L^{2}(S^{1}) ≃ \begin{matrix}∑ & ⊕ \\ n∈ℤ \end{matrix}ℂe^{\sqrt{−1}nx}, F ↦ F̂(n) = ∫_{0}^{2π} F (x)e^{−\sqrt{−1}nx}dx

(1.3.1)(b) L^{2}(ℝ) ≃ ∫_{−∞}^{∞} ℂe^{\sqrt{−1}ζx}dζ, f ↦ f̃(ζ) = ∫_{−∞}^{∞} f(x)e^{−\sqrt{−1}ζx}dx

with inversion formulae

(1.3.2)(a) F (x) = \frac{1}{2π} ∑_{n∈ℤ} F̂(n)e^{\sqrt{−1}nx}  (discrete),

(1.3.2)(b) f(x) = \frac{1}{2π} ∫_{−∞}^{∞} f̃(ζ)e^{\sqrt{−1}ζx} dζ  (continuous).

\underline{1.4.} We consider a naive question:

Question 1.4. Why the Fourier series expansion (1.3.2)(a) is discrete, and the Fourier transform (1.3.2)(b) is continuous in the inversion formula (also in the L^{2}-correspondence) ?

Of course, there are many explanations from various viewpoints. For instance,

Explanation 1.4.1. By the explicit inversion formula as above.

Explanation 1.4.2. It is explained by the knowledge of the square integrability of harmonic oscillator (e^{\sqrt{−1}nx} ∈ L^{2}(S^{1}); e^{\sqrt{−1}ζx} ∉ L^{2}(ℝ)).

These explanations are concrete answers for the Fourier expansion on ℝ or S^{1} indeed, but a more abstract argument could lead us to a further generalization:

Explanation 1.4.3. From the view point of analysis, the formulae (1.3.2) can be regarded as the expansion into eigenspaces of the Laplacian Δ = \frac{d^{2}}{dx^{2}} . The elliptic self-adjoint differential operator Δ has only discrete spectrum for a compact manifold M (in our case, M = S^{1}), which explains the discreteness in (1.3.2)(a).

Explanation 1.4.4. From the view point of representation theory, which goes back to H. Weyl, the formula (1.3.1) can be regarded as the irreducible decomposition of the regular representation L^{2}(G) of an abelian Lie group G = S^{1} or ℝ. Then, the discreteness of the Fourier series expansion is explained in a more general framework, namely, in the Peter-Weyl theorem which gives a (discrete) irreducible decomposition of L^{2}(G) for any compact group G.

Explanation 1.4.5. Another representation theoretic explanation is given by hidden symmetry (using the action of SL(2, ℝ)).

This is our viewpoint in this article. We shall explain the details in §1.5 ∼ §1.8. The main point here, different from Explanation 1.4.4, is that we use the non-commutative action of SL(2, ℝ) behind S^{1} or ℝ, so that we may extract more information of the Fourier expansion by using the representation theory of SL(2, ℝ).

\underline{1.5.} We consider SL(2, ℝ) and its subgroups:

G = SL(2, ℝ) ⊃ G′ := \{ \begin{matrix}K := \{( \begin{matrix}cos θ & − sin θ \\ sin θ & cos θ \end{matrix} ) : θ ∈ ℝ/2πℤ\} & ≃ S^{1}, \\ N := \{( \begin{matrix}1 & b \\ 0 & 1 \end{matrix} ) : b ∈ ℝ\} & ≃ ℝ, \\ A := \{( \begin{matrix}e^{s} & 0 \\ 0 & e^{−s} \end{matrix} ) : s ∈ ℝ\} & ≃ ℝ. \end{matrix}

We define a principal series representation π of G = SL(2, ℝ) on L^{2}(ℝ) given by

(1.5.1)  π(g): L^{2}(ℝ) → L^{2}(ℝ), f(x) ↦ |cx + d|^{−1}f(\frac{ax + b}{cx + d}),

where g^{−1} = ( \begin{matrix}a & b \\ c & d \end{matrix} ) ∈ G. It is well-known that the representation π

is irreducible as a G-module (Bargman, 1947).

\underline{1.6.} (Restriction G ↓ N) First, we consider the restriction of π with respect to G ↓ N. If a = d = 1 and c = 0 then (1.5.1) reduces to

(π(g)f)(x) = f(x + b).

This means that the restriction π|_{N} is given by the regular representation of N ≃ ℝ on L^{2}(ℝ). Thus, the branching law π|_{N} is nothing but the ℝ-irreducible decomposition of L^{2}(ℝ), which is given by the Fourier transform (1.3.1)(b).

Conversely, G = SL(2, ℝ) is regarded as a hidden symmetry for the regular representation of ℝ on L^{2}(ℝ). Then, we can explain that the spectrum of the Fourier expansion for L^{2}(ℝ) is purely continuous (in particular, it is not discretely decomposable), by using the action of SL(2, ℝ) (the subgroup AN will do) by the following lemma with σ := π|_{AN}.

Lemma 1.6. Let σ be a unitary representation of the group AN on a countable Hilbert space H such that the space of N-fixed vectors H^{N} = {0}. Then, there is no discrete spectrum in the restriction σ|_{N}, namely, Hom_{N}(τ, σ|_{N}) = {0} for any τ ∈ N̂.

Sketch of Proof. The abelian group A ≃ ℝ_{+}^{×} normalizes N ≃ ℝ, and A acts on N̂ ≃ \sqrt{−1}ℝ with three orbits:

N̂ ≃ \{e^{\sqrt{−1}ζx} : ζ < 0\} ∪ \{e^{\sqrt{−1}ζx} : ζ = 0\} ∪ \{e^{\sqrt{−1}ζx} : ζ > 0\}.

Because σ(a)∘σ∘σ(a)^{−1} is unitarily equivalent to σ for each fixed a ∈ A, the support of the measure in the branching law σ|_{N} is A-invariant, whence the lemma. □

\underline{1.7.} (Restriction G ↓ A) The restriction π with respect to G ↓ A is decomposed into only continuous spectrum with multiplicity 2. In order to see this, we consider the following map:

L^{2}(ℝ) = L^{2}(ℝ_{+}) ⊕ L^{2}(ℝ_{−}) →^{∼}_{T_{+}+T_{−}}L^{2}(ℝ) ⊕ L^{2}(ℝ),

where T_{+} (similarly, T_{−}) is defined by

T_{+} : L^{2}(ℝ_{+}) → L^{2}(ℝ), f(x) ↦ (T f)(t) := e^{\frac{1}{2} t}f(e^{t}).

Then, (π|_{A}, L^{2}(ℝ_{+})) (similarly, (π|_{A}, L^{2}(ℝ_{−}))) is unitarily equivalent to the regular representation of A (≃ ℝ) because

T_{+}(π(g)f)(t) = (T_{+}f)(t − 2s), for g = ( \begin{matrix}e^{s} & 0 \\ 0 & e^{−s} \end{matrix} ) .

This gives a concrete proof that the branching law of the restriction π|_{A} consists of only continuous spectrum. However, different from the case G ↓ N in §1.6, we cannot apply an elementary abstract argument such as Lemma 1.6. Instead, we shall give criteria for the discrete decomposability of the restriction in a general setting (see Theorem A and Theorem B in §4), by using both micro-local analysis and algebraic representation theory. In order to give a flavor, we write their criteria in this specific example of the restriction G ↓ A (notation will be explained in §4):

The criterion in Theorem A (admissibility in the branching law) fails for the restriction π|_{A} in the following manner:

Cone(A) ∩ AS_{K}(π) = ℝ ∩ ℝ ≠ {0}.

On the hand, the criterion in Theorem B for the algebraic discrete decomposability of the restriction π|_{A} fails, namely, pr_{𝔤→𝔞} (V_{𝔤}(π_{K})) ⊄ N_{𝔞}^{∗}, which is written in a suitable coordinate as

pr_{1} ({(x, y, 0) ∈ ℂ^{3} : x^{2} + y^{2} = 0}) = ℂ ⊄ {0},

where we put pr_{1} : ℂ^{3} → ℂ, (x, y, z) ↦ x.

\underline{1.8.} (Restriction G ↓ K) The restriction π with respect to G ↓ K is decomposed discretely with multiplicity free, which corresponds to the Fourier series expansion (1.3.2)(a) of S^{1}. This map can be written explicitly as follows: We define

T : L^{2}(ℝ) → L^{2}(S^{1}), f ↦ (T f)(ψ) = \frac{|f(tan \frac{ψ}{2} )|}{| cos \frac{ψ}{2} |}.

Then T is a unitary operator (up to a scalar constant) which intertwines the K-action by

T (π ( \begin{matrix}cos φ & sin φ \\ − sin φ & cos φ \end{matrix} ) f) (ψ) = (T f)(ψ − 2φ).

Thus, the restriction π|_{K} is decomposed discretely into irreducible representations of K̂, according to L^{2}(S^{1}) ≃ \begin{matrix}∑ & ⊕ \\ n∈ℤ \end{matrix}ℂe^{\sqrt{−1}nθ}.

The criterion of Theorem A (admissibility in the branching law) holds in this example in the following manner:

Cone(K) ∩ AS_{K}(π) = {0} ∩ ℝ = {0}.

On the hand, the criterion for the algebraic discrete decomposability (Theorem B) holds, namely, pr_{𝔤→𝔨} (V_{𝔤}(π_{K})) ⊂ N_{𝔨}^{∗}, which is written in a suitable coordinate as

pr_{3} ({(x, y, 0) ∈ ℂ^{2} : x^{2} + y^{2} = 0}) = {0} ⊂ {0},

where we put pr_{3} : ℂ^{3} → ℂ, (x, y, z) ↦ z.

\underline{1.9.} We have interpreted the naive question (1.4) as the discreteness (or continuity) of the spectrum in some branching laws π|_{G′} in §1.6 and §1.8.

As is easy to see, the compactness of G′ is a sufficient condition for the discrete decomposition like the Fourier series expansion (1.3.2)(a).

The starting point here is a surprising example (see Example 3.3) that the compactness of G′ is not a necessary condition for the discrete branching law.

§2. Analytic and algebraic notion of discretely decomposable restrictions

\underline{2.1.} In this article, we consider the restriction π|_{G′} in the following setting:

Setting 2.1.

G: a real reductive linear Lie group or its finite covering group. G′: a closed subgroup of G which is reductive in G. (π, H): an irreducible unitary representation of G.

There exists a Cartan involution θ of G such that θG′ = G′. We define

(2.1.1)  K := G^{θ} = {g ∈ G : θg = g}, K′ := K ∩ G′.

Then K is a maximal compact subgroup of G and K′ is that of G′. A typical example of Setting 2.1 is:

(G, G′) = (GL(n, ℝ), O(p, n − p)),
(K, K′) ≃ (O(n), O(p) × O(n − p)).

The purpose of this section is to formulate “discreteness” of the branching law of unitary representations both in an analytic way (Definition 2.3) and in an algebraic way (Definition 2.6).

\underline{2.2.} The branching law of the restriction π|_{G′} may have a wild behavior in Setting 2.1, in particular, when G′ is non-compact. It can involve both continuous and discrete spectrum, possibly with infinite multiplicity, even if (G, G′) is a semisimple symmetric pair (see [24], §0; see also Remark 5.4 for a more delicate case).

On the other hand, there have been certainly successful achievements in special settings of branching problems:

Example 2.2. 1) The case where G is compact. A classical (but still active) branching problems are to find explicit irreducible decompositions of the restrictions to various subgroups of a compact Lie group G. We have at least a procedure to obtain branching laws because π is finite dimensional and because we know an explicit character formula due to H.Weyl.

2) The case where G′ = K, a maximal compact subgroup of G. The branching law π|_{K} is called the K-type formula. A knowledge on the Ktype formula, even if it is partial, sometimes reveals the crucial property of the representation π (e.g. the minimal K-type theory due to Vogan). Also, branching laws are known for some important cases such as the generalized Blattner formula where π is a Zuckerman-Vogan’s derived functor module (see [12], [48] Theorem 6.3.12).

3) The case where π is a unitary highest weight module.

(3-a) If π is a holomorphic discrete series, an explicit decomposition formula of π|_{G′} has been found (eg. [16], [33], [43], [47]) for a symmetric pair (G, G′).

(3-b) If π is the Segal-Shale-Weil representation of the metaplectic group, and if G′ = G′_{1}G′_{2} forms a reductive dual pair, there has been an extensive study of branching problems π|_{G′} in connection to the θ-correspondence ([1], [14], [18]).

One of the motivations to introduce “discrete branching laws” is to find a nice framework of branching problems, which generalize the settings in Example 2.2.

\underline{2.3.} Here is an analytic formulation for discrete branching laws:

Definition 2.3 (analytic definition; see [23], [24]). We say that the restriction π|_{G′} is G′-admissible if the restriction π|_{G′} splits into a discrete sum of irreducible unitary representations:

π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈_{G′} \end{matrix}}n_{π}(τ)τ, (discrete Hilbert sum),

with multiplicity n_{π}(τ) ∈ ℕ = {0, 1, 2, … }.

The point here is that there is no continuous spectrum in the branching law and that each multiplicity is finite.

\underline{2.4.} Here is a special but important example of G′-admissible restrictions:

Fact 2.4 (Harish-Chandra [11]). Any irreducible unitary representation π of G is K-admissible when restricted to K.

In a usual terminology, “K-admissible” is simply called admissible. Fact 2.4 is sometimes called a basic theorem of Harish-Chandra (see [55], Theorem 3.4.1), which has played a fundamental role in establishing algebraic methods of representation theory of real reductive Lie groups (the study of (𝔤_{ℂ}, K)-modules).

\underline{2.5.} Next, we consider an algebraic formulation. Let 𝔥 be a Lie algebra, and X an 𝔥-module.

Definition 2.5 ([31], Definition 1.1). We say X is discretely decomposable if there is an increasing filtration {X_{m}} of 𝔥-submodules such that (2.5.1) X = ⋃_{m=0}^{∞} X_{m}, (2.5.2) X_{m} is of finite length as an 𝔥-module (i.e. has composition series of finite length).

Obviously, it is the case if X itself is an 𝔥-module of finite length.

\underline{2.6.} Suppose we are in Setting 2.1. We write π_{K} for the underlying (𝔤_{ℂ}, K)-module of π ∈ Ĝ.

Definition 2.6 (algebraic definition; [31]). We say that the restriction π|_{G′} is 𝔤′-discretely decomposable (or algebraically discretely decomposable), if π_{K} is discretely decomposable as a 𝔤′-module. This is equivalent to the condition that we have an isomorphism as (𝔤′_{ℂ}, K′)- modules:

(2.6.1)  π_{K} ≃ ⨁_{τ_{K′}} n_{π}(τ)τ_{K′} (as (𝔤′_{ℂ}, K′)-modules)

because π is unitary (see [31], Lemma 1.3).

The right side of (2.6.1) is an algebraic direct sum of (𝔤′_{ℂ}, K′)- modules, τ_{K′} runs over irreducible (𝔤′_{ℂ}, K′)-modules, and the multiplicity n_{π}(τ) ∈ ℕ ∪ {∞}.

The following is easy from definition:

Example 2.6.2. Suppose G′ is compact. Then, for any π ∈ Ĝ, the restriction π|_{G′} is 𝔤′-discretely decomposable.

\underline{2.7.} Algebraically discrete decomposability implies analytic discrete decomposability as follows:

Theorem 2.7. Suppose we are in Setting 2.1. If the restriction π|_{G′} is 𝔤′-discretely decomposable, then we have

dim Hom_{G′}(τ, π|_{G′}) = dim Hom_{(𝔤′_{ℂ},K′)}(τ_{K′}, π_{K}) for any τ ∈ Ĝ′,

for which we write n_{π}(τ) ∈ ℕ ∪ {∞}. Furthermore, the restriction π|_{G′} is decomposed discretely into the direct sum of unitary representations:

π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈_{G′} \end{matrix}}n_{π}(τ)τ (discrete Hilbert sum).

\underline{2.8.} There is a slight difference between Definition 2.3 (analytic definition) and Definition 2.6 (algebraic definition). The main difference is that the multiplicity is allowed to be infinite in Definition 2.6. We shall discuss this point in §5.

§3. Compact-like actions in infinite dimensional groups

\underline{3.1.} In this section, we propose to compare discrete branching laws with actions of discrete groups.

Suppose we are in Setting 2.1. The restriction π|_{G′} deals with the group homomorphism

π : G′ → U(H),

where U(H) is the group of unitary operators on H. If G′ is compact, then the restriction π|_{G′} is discretely decomposable. The notion of the discrete decomposable restriction π|_{G′} means that the image π(G′) in the group U(H) behaves somehow like a compact group.

Let Γ be a discrete group acting on a locally compact topological space X. Then we have a group homomorphism

ϖ : Γ → Homeo(X),

where Homeo(X) is the group of homeomorphisms of X. The notion of the properly discontinuous action of Γ means that the image ϖ(Γ) in the group Homeo(X) behaves somehow like a finite group.

The point in this observation is that both U(H) and Homeo(X) are “infinite dimensional groups” and that (non-compact) subgroups π(G) and ϖ(Γ) may behave like compact groups inside such huge groups.

\underline{3.2.} Our guiding philosophy is in the following diagram:

1) Discrete version :  properly discontinuous actions
↓↑
2) Continuous version :  proper actions (see R. Palais [45])
↓↑
3) Representation version 1 :  π|_{G′} is G′-admissible
(analytic definition, §2.3)  (no continuous spectrum)
↓↑
4) Representation version 2 :  algebraically discrete decomposable

(algebraic definition, §2.6)

Surprisingly, there is a mysterious similarity between properly discontinuous actions and discrete branching laws. The similarity appears not only in the observation on “compact-like” actions in both of the definitions, but also in both of the criteria; one is for the action of a discrete group Γ on a homogeneous space X := G/H to be properly discontinuous where Γ and H are subgroups of a reductive group G (see [3] and [25], which generalize the criterion of proper actions of reductive subgroups [21]), and the other is for the restriction of the unitary representation π|_{G′} to be discretely decomposable in Setting 2.1 (see Theorem A). Both of the criteria are given in terms of the intersection of certain invariants of Γ and G/H (G′ and (π, H), respectively).

\underline{3.3.} More than the above mentioned similarity, the study of properly discontinuous actions leads us to an interesting example of discrete decomposable restrictions:

Example 3.3 (Kobayashi, 1988). We take G′ ⊂ G ⊃ H to be

(G′, G, H) := (Sp(n, 1), SU(2n, 2), U(2n, 1)).

Then we have,

1) For any discrete subgroup Γ of G′, Γ acts properly discontinuously on G/H.

2) For any closed subgroup L of G′, L acts properly on G/H.

3) For any π ∈ Ĝ satisfying Hom_{G}(π, L^{2}(G/H)) ≠ {0}, the restriction π|_{G′} is G′-admissible.

4) For any π ∈ Ĝ satisfying Hom_{G}(π, L^{2}(G/H)) ≠ {0}, the restriction π|_{G′} is 𝔤′-discretely decomposable.

Example 3.3 (1) and (2) are special cases of [21], Theorem 4.1. Example (3) and (4) are generalized in [24], Theorem 3.2 (Theorem A in §4 is a further generalization).

We note that the representation π in Example 3.3 (3) is not a highest weight module. One can write π in terms of a Zuckerman-Vogan’s module A_{𝔮}(λ) (see §4.10).

§4. Criterion for discretely decomposable restrictions

\underline{4.1.} In this section, we give a sufficient condition (see Theorem A) for the restriction π|_{G′} to be G′-admissible in terms of two cones Cone(G′) (see Definition 4.2) and AS_{K}(π) (see Definition 4.4); the former depends on a subgroup G′ of G, and the latter on an irreducible representation π of G. Then, we shall discuss about a necessary condition (see Theorems B and C) for the branching law π|_{G′} to be 𝔤′-discretely decomposable.

Let G be a connected real reductive Lie group with a maximal compact subgroup K. We write 𝔤 = 𝔨 + 𝔭 for the corresponding Cartan decomposition of the Lie algebra 𝔤 of G. We fix a Cartan subalgebra 𝔱 of 𝔨, and a positive system Δ^{+}(𝔨_{ℂ}, 𝔱_{ℂ}). We write \sqrt{−1}𝔱_{+}^{∗} (⊂ \sqrt{−1}𝔱^{∗}) for the dominant Weyl chamber, and Λ (⊂ \sqrt{−1}𝔱^{∗}) for the weight lattice of K. We put

(4.1.1)  Λ_{+} := Λ ∩ \sqrt{−1}𝔱_{+}^{∗}.

The highest weight theory due to Cartan-Weyl establishes a bijection:

(4.1.2)  K̂ ≃ Λ_{+},  τ_{λ} ↔ λ.

We fix an Ad^{∗}(K)-invariant inner product on \sqrt{−1}𝔨^{∗}, and then regard \sqrt{−1}𝔱^{∗} as a subspace of \sqrt{−1}𝔨^{∗}.

\underline{4.2.} Suppose that G′ is a closed subgroup of G which is reductive in G. As in Setting 2.1, we can assume that K′ := K∩G′ is a maximal compact subgroup of G′. Then we have a Cartan decomposition 𝔤′ = 𝔨′ + 𝔭′ compatible with 𝔤 = 𝔨 + 𝔭. Let 𝔨′^{⊥} be the annihilator of 𝔨′ in 𝔨^{∗}.

Definition 4.2. For a θ-stable subgroup G′ of G, we define a cone in \sqrt{−1}𝔱^{∗} by

Cone(G′) := \sqrt{−1}𝔱_{+}^{∗} ∩ \sqrt{−1} Ad^{∗}(K)𝔨′^{⊥}.

\underline{4.3.} We can describe Cone(G′) explicitly if (G, G′) is a reductive symmetric pair.

Definition 4.3.1. Suppose σ is an involutive automorphism of a real reductive Lie group G. Let G^{σ} := {g ∈ G : σg = g}, and G_{0}^{σ} be the identity component of G^{σ}. If a subgroup G′ of G satisfies G_{0}^{σ} ⊂ G′ ⊂ G^{σ}, we say (G, G′) is a reductive symmetric pair. The Lie algebra 𝔤′ is given by 𝔤^{σ} := {X ∈ 𝔤 : σX = X}.

For example, (SL(n, ℝ), SO(p, n−p)) is a reductive symmetric pair. After a conjugation by inner automorphisms of G if necessary, we

may and do assume that σ commutes with θ and that

𝔱^{−σ} := 𝔨^{−σ} ∩ 𝔱

is a maximal abelian subspace of 𝔨^{−σ} := {X ∈ 𝔨 : σX = −X}. Furthermore, we can take Δ^{+}(𝔨_{ℂ}, 𝔱_{ℂ}) to be compatible with a positive system of the restricted root system Σ(𝔨_{ℂ}, 𝔱_{ℂ}^{−σ}). Then we have:

Proposition 4.3.2. Let (G, G′) be a reductive symmetric pair. Retain the above notation. Then we have

Cone(G′) = \sqrt{−1}𝔱_{+}^{∗} ∩ \sqrt{−1}(𝔱^{−σ})^{∗}.

Here, we regard (𝔱^{−σ})^{∗} as a subspace of 𝔱^{∗} by the direct sum decomposition 𝔱 = 𝔱^{σ} + 𝔱^{−σ}.

\underline{4.4.} Now, we give a quick review of the asymptotic cone AS_{K}(π) for π ∈ Ĝ. If V is a subset in the Euclidean space ℝ^{N}, we define the asymptotic cone V ∞ by

V ∞ := {y ∈ ℝ^{N} : there exists a sequence (y_{n}, ε_{n}) ∈ V × ℝ_{+}
such that lim_{n→∞} ε_{n}y_{n} = y and lim_{n→∞} ε_{n} = 0}.

Then, V ∞ is a closed cone in ℝ^{N}.

Definition 4.4 (asymptotic K-support; see [19]). For π ∈ Ĝ, we define two subsets of \sqrt{−1}𝔱^{∗} by

Supp_{K}(π) := {λ ∈ Λ_{+} : Hom_{K}(τ_{λ}, π) ≠ {0}},
AS_{K}(π) := Supp_{K}(π)∞.

Then AS_{K}(π) is a closed cone in \sqrt{−1}𝔱^{∗} satisfying
AS_{K}(π) ⊂ \sqrt{−1}𝔱_{+}^{∗}

because Supp_{K}(π) ⊂ Λ_{+} and Λ_{+}∞ = \sqrt{−1}𝔱_{+}^{∗}.

\underline{4.5.} Here is a sufficient condition for the restriction π|_{G′} to be G′- admissible.

Theorem A (see [30], Theorem 2.9 (1)). Suppose we are in Setting 2.1. If π ∈ Ĝ and the pair (G, G′) satisfy

(4.5.1)  AS_{K}(π) ∩ Cone(G′) = {0},

then the restriction π|_{K′} is K′-admissible. In particular, the restriction π|_{G′} is 𝔤′-discretely decomposable. Furthermore, the restriction π|_{G′} is also G′-admissible, that is, we have a unitary G′-equivalence:

π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈_{G′} \end{matrix}}m_{π}(τ) τ (discrete Hilbert sum),

where m_{π}(τ) := dim Hom_{G′}(τ, π|_{G′}) < ∞ for each τ ∈ Ĝ′.

We refer [30] for an upper estimate of the multiplicity m_{π}(τ).

\underline{4.6.} For an understanding of Theorem A, we consider two extremal cases.

Remark 4.6. First, we observe:

(4.6.1)  AS_{K}(π) = {0} ⇔ dim π < ∞,
(4.6.2)  Cone(G′) = {0} ⇔ G′ ⊃ K.

Of course, one of these conditions implies the assumption (4.5.1) in Theorem A. The conclusion of Theorem A in the first case (4.6.1) corresponds to the complete reducibility of a finite dimensional unitary representation π|_{G′}, while that in the second case (4.6.2) corresponds to Harish-Chandra’s fundamental theorem (see Fact 2.4).

\underline{4.7.} The proof of Theorem A is based on two lemmas:

Lemma 4.7.1 ([24], Theorem 1.2). If π|_{K′} is K′-admissible, π|_{G′} is G′-admissible.

Lemma 4.7.2 ([30], Theorem 2.8). Let π ∈ Ĝ. Assume (4.5.1) is satisfied.

1) The K-character Θ_{π} ∈ D′(K) of π|_{K} has a well-defined restriction Θ_{π}|_{K′} to K′ as a distribution.

2) The restriction π|_{K′} is K′-admissible, and Θ_{π}|_{K′} coincides with a K′-character of π|_{K′}.

The idea of Lemma 4.7.1 is similar to the proof of the theorem due to Gelfand-Piateski-Shapiro:

Fact 4.7.3 (see [9], Chapter I, §2). Let Γ be a co-compact discrete subgroup of G. Then L^{2}(G/Γ) is a Hilbert direct sum of irreducible unitary representations of G with finite multiplicity.

The first statement of Lemma 4.7.2 follows from an estimate of the singularity spectrum of a hyperfunction (or the wave front set of a distribution) (e.g. [13], [17]), but the second one does not follow from a general theory of micro-local analysis (see [30], Remark 2.8).

\underline{4.8.} Next, we consider algebraically discretely decomposable restrictions. Let us give a quick review on the associated varieties of U(𝔤)- modules (see [6], [52]). If V is a finite dimensional complex vector space, we use the following notation: V^{∗} : the dual vector space of V over ℂ, S(V) : the symmetric algebra of V ≃ the polynomial algebra on V^{∗}, S^{k}(V): the subspace of S(V) of homogeneous elements of degree k, S_{k}(V): = ⨁_{j=0}^{k} S^{j}(V).

Let M = ⨁_{k=0}^{∞} M_{k} be a finitely generated S(V)-module. We say M is a graded S(V)-module if S^{i}(V)M_{j} ⊂ M_{i+j} (i, j ≥ 0). We define the ideal of S(V) by

Ann_{S(V)}(M) := {f ∈ S(V) : f · m = 0 for any m ∈ M},

and a closed cone in V^{∗} by

Supp_{S(V)}(M) := {λ ∈ V^{∗} : f(λ) = 0 for any f ∈ Ann_{S(V)}(M)}.

Let 𝔤_{ℂ} be a finite dimensional Lie algebra over ℂ. For each integer n ≥ 0, let U_{n}(𝔤_{ℂ}) denote by the subspace spanned by elements of the form Y_{1} ⋯ Y_{k} with Y_{1}, …, Y_{k} ∈ 𝔤_{ℂ} and k ≤ n. We note that U_{0}(𝔤_{ℂ}) = ℂ. It is convenient to put U_{−1}(𝔤_{ℂ}) = 0. Then, U(𝔤_{ℂ}) is a filtered algebra in the sense that

U(𝔤_{ℂ}) = ⋃_{k=1}^{∞} U_{k}(𝔤_{ℂ}), U_{i}(𝔤_{ℂ})U_{j}(𝔤_{ℂ}) ⊂ U_{i+j}(𝔤_{ℂ}).

The associated graded algebra gr U(𝔤_{ℂ}) := ⨁_{k=0}^{∞} U_{k}(𝔤_{ℂ})/U_{k−1}(𝔤_{ℂ}) is isomorphic to the symmetric algebra S(𝔤_{ℂ}) = ⨁_{k=0}^{∞} S^{k}(𝔤_{ℂ}) of 𝔤_{ℂ}, by the Poincaré-Birkhoff-Witt theorem.

Suppose X is a finitely generated U(𝔤_{ℂ})-module. We take a finite dimensional subspace X_{0} which generates X as a U(𝔤_{ℂ})-module. We put X_{k} := U_{k}(𝔤_{ℂ})X_{0} (k ∈ ℕ). It is convenient to put X_{−1} := {0}. Then we have an increasing filtration {X_{k}}_{k} such that

X = ⋃_{k=0}^{∞} X_{k}, U_{i}(𝔤_{ℂ})X_{j} = X_{i+j} (i, j ≥ 0).

Therefore, if we put gr X := ⨁_{k=0}^{∞} \overline{X_{k}} with \overline{X_{k}} := X_{k}/X_{k−1}, then gr X is a finitely generated gr U(𝔤_{ℂ}) ≃ S(𝔤_{ℂ})-module. Define the variety V(X) by

V(X) ≡ V_{𝔤}(X) = Supp_{S(𝔤_{ℂ})}(gr X) ⊂ 𝔤_{ℂ}^{∗}.

Then V_{𝔤}(X) is independent of the choice of the generating subspace X_{0} and is called the associated variety of the U(𝔤_{ℂ})-module X.

We define the nilpotent cone N^{∗} for 𝔤_{ℂ} by

N^{∗} ≡ N_{𝔤_{ℂ}}^{∗} := {λ ∈ 𝔤_{ℂ}^{∗} : f(λ) = 0, for all f ∈ S^{+}(𝔤_{ℂ})^{G}}.

Here S^{+}(𝔤_{ℂ}) := ⨁_{k=1}^{∞} S^{k}(𝔤_{ℂ}) is the maximal ideal of S(𝔤_{ℂ}), and S^{+}(𝔤_{ℂ})^{G} is the ring of G-invariant elements. Then we have

Fact 4.8 (see [52], Corollary 5.4). If X is a 𝔤_{ℂ}-module of finite length, then the associated variety V_{𝔤}(X) is contained in N_{𝔤_{ℂ}}^{∗}.

\underline{4.9.} Suppose we are in Setting 2.1. We write the projection

pr_{𝔤→𝔤′}: 𝔤_{ℂ}^{∗} → (𝔤′_{ℂ})^{∗}

dual to the inclusion of complexified Lie algebras 𝔤′_{ℂ} ↪ 𝔤_{ℂ}. We write (π_{K}, X) for the underlying (𝔤_{ℂ}, K)-module. Here is a necessary condition for the restriction π|_{G′} to be 𝔤′-discretely decomposable:

Theorem B (see [31], Corollary 3.4). If the restriction π|_{G′} is 𝔤′-discretely decomposable (see Definition 2.6), then

(4.9.1)  pr_{𝔤→𝔤′}(V_{𝔤}(X)) ⊂ N_{𝔤′_{ℂ}}^{∗}.

If the restriction π|_{G′} is 𝔤′-discretely decomposable, then the “size” of the representation π should be relatively small and the “size” of the subgroup G′ should be relatively large. The above theorem gives a justification of this “feeling” in terms of the associated variety.

\underline{4.10.} Let us compute explicit criteria in Theorem A and Theorem B where π_{K} is a Zuckerman-Vogan’s derived functor module. (This is the most important case in applications (e.g. [32], [34]).) Then, we shall see that the necessary condition in Theorem B is also sufficient (see Theorem C). We start with a brief review of Zuckerman-Vogan’s derived functor modules. Standard references are [20], [48], [55].

We extend 𝔱 to a Cartan subalgebra 𝔥 of 𝔤. Given ν ∈ \sqrt{−1}𝔱_{+}^{∗} (⊂ \sqrt{−1}𝔱^{∗}), we define a θ-stable parabolic subalgebra of 𝔤_{ℂ} by 𝔮 ≡ 𝔮(ν) := 𝔩_{ℂ} + 𝔲, where 𝔩_{ℂ} and 𝔲 are stable under ad(𝔥) with weights given by

Δ(𝔩_{ℂ}, 𝔥_{ℂ}) := {α ∈ Δ(𝔤_{ℂ}, 𝔥_{ℂ}) : ⟨α, ν⟩ = 0},
Δ(𝔲, 𝔥_{ℂ}) := {α ∈ Δ(𝔤_{ℂ}, 𝔥_{ℂ}) : ⟨α, ν⟩ > 0}.

Let L := Z_{G}(ν), and L̃ a metaplectic covering of L defined by the character of L acting on ∧^{top}𝔲 ≃ ℂ_{2ρ(𝔲)}. Here we write the character ℂ_{2ρ(𝔲)} in an additive way. We note that 𝔩_{ℂ} is the complexified Lie algebra of L (and also of L̃). The elliptic orbit Ad(G)X ≃ G/L carries a G-invariant complex structure, with the canonical line bundle Ω := ∧^{top}T^{∗}(G/L) ≃ G×_{L}ℂ_{2ρ(𝔲)}. Suppose W is a finite dimensional metaplectic representation of L̃. Then the L-module W ⊗ ℂ_{ρ(𝔲)} defines a G-homogeneous holomorphic vector bundle W := G×_{L}(W ⊗ ℂ_{ρ(𝔲)}). As an algebraic analog of the Dolbeault cohomology group Hj ; ¯∂(G/L, W) (see [56]), Zuckerman introduced the cohomological parabolic induction R_{𝔮}^{j} ≡ (R_{𝔮}^{𝔤})^{j} (j ∈ ℕ), which is a covariant functor from the category of metaplectic (𝔩_{ℂ}, (L ∩ K)^{∼})-modules to that of (𝔤_{ℂ}, K)-modules. If the Z(𝔩_{ℂ})-infinitesimal character γ ∈ 𝔥_{ℂ}^{∗} of an irreducible metaplectic representation W of L̃ lies in the good range (see [51], Definition 2.5), namely, if Re⟨γ, α⟩ > 0 for any α ∈ Δ(𝔲, 𝔥_{ℂ}), then R_{𝔮}^{j}(W) = 0 for any j ≠ S := dim_{ℂ}(𝔲∩𝔨_{ℂ}) and R_{𝔮}^{S}(W) is an irreducible (𝔤_{ℂ}, K)-module. Furthermore, if dim W = 1, we sometimes write A_{𝔮}(λ) for R_{𝔮}^{S}(W) where ℂ_{λ} := W ⊗ℂ_{−ρ(𝔲)} is a character of L. Here, we follow the normalization in [50], Definition 6.20 for R_{𝔮}^{j}; and the one in [53], §5 for A_{𝔮} (as we did in [29], §2). We define a closed cone by

(4.10.1) ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ := \{ ∑_{β∈Δ(𝔲∩𝔭_{ℂ},𝔱_{ℂ})} n_{β} β : n_{β} ≥ 0\} (⊂ \sqrt{−1}𝔱^{∗}).

Theorem 4.10. With notation as above, suppose W is a finite dimensional metaplectic representation of L̃ in the good range. Then we have

AS_{K}(R_{𝔮}^{S}(W)) ⊂ ℝ_{+}⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1}𝔱_{+}^{∗}. (4.10.2)

V_{𝔤}(R_{𝔮}^{S}(W)) = Ad(K_{ℂ})(𝔲^{−} ∩ 𝔭_{ℂ}). (4.10.3)

Proof. 1) See [30], §3 for (4.10.2).

2) See [6] or [51] for (4.10.3) (see also [31], Lemma 2.7 in the case where W is in the weakly fair range). □

\underline{4.11.} By using Theorem 4.10, we have the following result from Theorem A and Theorem B for Zuckerman-Vogan’s derived functor modules:

Theorem C (see [31], Theorem 4.2). Suppose that (G, G′) is a reductive symmetric pair and that 𝔮 is a θ-stable parabolic subalgebra of 𝔤_{ℂ}. Retain the above notation and we suppose that positive systems are taken to be compatible in such a way that Δ(𝔲∩𝔨_{ℂ}, 𝔱_{ℂ}) ⊂ Δ^{+}(𝔨_{ℂ}, 𝔱_{ℂ}) (see § 4.3 and § 4.10). Then, the following three conditions (i), (ii) and (iii) on the triple (G′, G, 𝔮) are equivalent:

i) A_{𝔮}(λ) is 𝔤′-discrete decomposable for any λ. ii) A_{𝔮}(λ) is 𝔤′-discrete decomposable for some λ in the good range. iii) ℝ_{+} ⟨𝔲 ∩ 𝔭⟩ ∩ \sqrt{−1}(𝔱^{−σ})^{∗} = \{0\}.

If ℂ_{λ+ρ(𝔲)} is in the good range, then A_{𝔮}(λ) is unitarizable by a theorem of Vogan and Wallach ([49], [54]), and the Hilbert completion

π := \overline{A_{𝔮}(λ)} ∈ Ĝ

defines an irreducible unitary representation of G.

The following theorem means that algebraic discrete decomposability implies finite multiplicity.

Theorem D. If one of (therefore, all of) the equivalent conditions in Theorem C is satisfied, then the restriction π|_{G′} is G′-admissible. In particular, the multiplicity m_{π}(τ) is finite for any τ ∈ Ĝ′ in the discrete branching law

π|_{G′} ≃ _{\begin{matrix}∑ & ⊕ \\ τ∈_{G′} \end{matrix}}m_{π}(τ)τ (discrete Hilbert sum).

\underline{4.12.} For G ⊃ G′, we define a subset of Ĝ by

(4.12.1) DDR(Ĝ; G′) := {π ∈ Ĝ : π|_{G′} is 𝔤′-discretely decomposable}.

It is an interesting problem to determine explicitly the subset DDR(Ĝ; G′) of the unitary dual Ĝ for a reductive symmetric pair (G, G′). DDR stands for discretely decomposable restriction.

Example 4.12.1. If G′ is a maximal compact subgroup K of G, then any irreducible unitary representation π of G is 𝔨-discretely decomposable. Thus,

DDR(Ĝ; K) = Ĝ.

Example 4.12.2 (see [31], §7.1). There are 18 family of irreducible unitary representations of G = U(2, 2) with integral infinitesimal characters. All of them are of the form A_{𝔮}(λ) with a suitable choice of a θ-stable parabolic subalgebra 𝔮 and a character ℂ_{λ} (see [46]). We consider a subgroup G′ of G such that G′ ≃ Sp(1, 1) ≃ Spin(4, 1). Then, 12 family satisfy the conditions (iii) in Theorem C, so that they are 𝔤′-discretely decomposable.

In particular, we write Ĝ_{1} a (finite) subset of Ĝ with the same infinitesimal character of the trivial representation 1, then DDR(Ĝ; G′) ⊂ Ĝ has an intersection with Ĝ_{1} as follows:

♯ (DDR(Ĝ; G′) ∩ Ĝ_{1}) = 12, ♯Ĝ_{1} = 18.

Example 4.12.3 (see [31], Theorem 8.1). Suppose that G_{ℂ} is a connected simple complex Lie group and G_{ℝ} is a normal real form of G_{ℂ}. Then, no infinite dimensional unitary representation π of G_{ℂ} is 𝔤_{ℝ}-discretely decomposable. Thus,

DDR(Ĝ_{ℂ}; G_{ℝ}) = {1}.

§5. Conjectures on discrete branching laws

\underline{5.1.} Suppose we are in Setting 2.1. We compare the following three conditions: (5.1.1) The restriction π|_{K′} is K′-admissible. (5.1.2) The restriction π|_{G′} is G′-admissible. (5.1.3) The restriction π|_{G′} is 𝔤′-discretely decomposable.

We note that (5.1.1) implies (5.1.2) (see Lemma 4.7.1). The condition (5.1.1) also implies (5.1.3) (see [31], Proposition 1.6).

\underline{5.2.} The following conjecture is a generalization of both Harish-Chandra’s admissibility theorem and Theorem D for Zuckerman-Vogan’s derived functor modules.

Conjecture A. Assume that (G, G′) is a reductive symmetric pair. Then, (5.1.1) is equivalent to (5.1.3) for any π ∈ Ĝ.

Remark 5.2. 1) The non-trivial part of Conjecture A is the implication (5.1.3) ⇒ (5.1.1). Namely, 𝔤′-discrete decomposability should control the multiplicity of K′-types.

2) Conjecture A is true if G′ = K, a maximal compact subgroup of G. In fact, π|_{K} is K-admissible by Harish-Chandra’s admissibility theorem (Fact 2.4) for any π ∈ Ĝ. On the other hand, the restriction π|_{K} is always 𝔨-discretely decomposable (see Example 2.6.2).

3) Conjecture A is true if π_{K} ≃ A_{𝔮}(λ), a Zuckerman-Vogan’s derived functor module (see Theorem D in §4), especially if π is a discrete series representation of G.

4) Conjecture A is true if π is a Kostant-Binegar-Zierau’s minimal unipotent representation of G = O(p, q) (see [4], [36], [38]) and G′ = O(p′, q′) × O(p′′, q′′) where p′ + p′′ = p, q′ + q′′ = q. In fact, it is proved in [37], §4 that

(5.1.1) ⇔ min(p′, q′, p′′, q′′) = 0 ⇔ (5.1.3).

5) We need the assumption that (G, G′) is a symmetric pair. If fact, without any assumption on (G, G′), the condition (5.1.3) does not always imply (5.1.1). A trivial counter example is given by G′ = {e}, where (5.1.1) fails and (5.1.3) holds.

6) Conjecture A implies that the restriction π|_{G^{σ}} is 𝔤^{σ}-discretely decomposable if and only if the restriction π|_{G^{σθ}} is 𝔤^{σθ}-discretely decomposable.

Here, we put

G^{σθ} := {g ∈ G : σθg = g}

for an involutive automorphism σ commuting with a Cartan involution θ. The pair (G, G^{σθ}) is called an associated symmetric pair of a symmetric pair (G, G^{σ}). For example, (SL(n, ℂ), SO(n, ℂ)) is associated to (SL(n, ℂ), SL(n, ℝ)). The important property of the associated symmetric pair is that a maximal compact subgroup of G^{σ} is isomorphic to that of G^{σθ} because

G^{σ} ∩ K = {g ∈ G : σg = θg = g} = G^{σθ} ∩ K.

\underline{5.3.} By using Theorem A and Theorem B in §4, Conjecture A can be deduced from the following conjectured relationship between the associated varieties V_{𝔤}(π_{K}) and the asymptotic K-support AS_{K}(π):

Conjecture B. Assume that (G, G′) is a reductive symmetric pair. If π ∈ Ĝ satisfies

pr_{𝔤→𝔤′}(V_{𝔤}(π_{K})) ⊂ N_{𝔤′_{ℂ}}^{∗} (see (4.9.1)),

then

AS_{K}(π) ∩ Cone(G′) = {0} (see (4.5.1)).

\underline{5.4.} The crucial point in Conjecture A (or Conjecture B) is that it implies the following finite multiplicity conjecture:

Conjecture C (discreteness ⇒ finite multiplicity). Let (G, G′) be a reductive symmetric pair. If the restriction π|_{G′} is 𝔤′-discretely decomposable, then

(5.4.1)  dim Hom_{G′}(τ, π|_{G′}) < ∞  (∀τ ∈ Ĝ′).

Remark 5.4. 1) If Conjecture C is true, then (5.1.3) implies (5.1.2). This is a direct consequence of Theorem 2.7.

2) An analogous statement of Conjecture C is not true for the multiplicity of continuous spectrum ([31], §0.4). That is, the multiplicity in the continuous spectrum can be infinite almost everywhere in the branching law for a reductive symmetric pair (G, G′).

3) The assumption “𝔤′-discretely decomposable” in Conjecture C is important because there exists a counter example for Conjecture C without the assumption of 𝔤′-discrete decomposability. Namely, the multiplicity in the discrete spectrum can be also infinite in the branching law of the restriction π|_{G′} which is not 𝔤′-discretely decomposable (i.e. the below (5.4.2) can happen).

In Example 5.5, we shall give a more delicate example of a symmetric pair (G, G′) and π ∈ Ĝ such that the following (5.4.2) and (5.4.3) happen simultaneously:

(5.4.2)  dim Hom_{G′}(σ_{1}, π|_{G′}) = ∞ for some σ_{1} ∈ Ĝ′,
(5.4.3) 0 < dim Hom_{G′}(σ_{2}, π|_{G′}) < ∞ for some σ_{2} ∈ Ĝ′.

4) Here are implications among the above three conjectures:

(5.4.4)  Conjecture B ⇒ Conjecture A ⇒ Conjecture C.

\underline{5.5.} Here is an example which is mentioned in Remark 5.4 (3).

Example 5.5. Let (G, G′) be a reductive symmetric pair

(Sp(2, ℂ), Sp(2, ℝ)) ≈ (SO(5, ℂ), SO(3, 2)).

Let

π ≡ π_{χ} ∈ Ĝ

be a unitary principal series representation of G induced from a unitary character χ of a Cartan subgroup H = T A of G. Here, T ≃ 𝕋^{2} and A ≃ ℝ^{2}. Then, for any χ ∈ Ĥ and any non-holomorphic discrete series representation σ_{1} ∈ Ĝ′, we have:

(5.5.1)  dim Hom_{G′}(σ_{1}, π|_{G′}) = ∞.

Also, there exists χ ∈ Ĥ such that

(5.5.2)  0 < dim Hom_{G′}(σ_{2}, π|_{G′}) < ∞ for some σ_{2} ∈ Ĝ′.

Sketch of Proof. We may assume T is contained in G′. We write ℂ_{(a,b)} for the restriction χ|_{T} where (a, b) ∈ ℤ^{2}. There are 4 open orbits of G′ on the flag variety of G, for which each isotropy subgroup is isomorphic to 𝕋^{2}. Accordingly, the restriction π|_{G′} is unitarily equivalent to the sum of the Hilbert spaces of L^{2}-sections of G′-equivariant line bundles G′ ×_{T} ℂ_{(ϵ_{1}a,ϵ_{2}b)} → G′/T:

(5.5.3)  π|_{G′} ≃ ⨁_{ϵ_{1},ϵ_{2}=±1} L^{2}(G′/T; ℂ_{(ϵ_{1}a,ϵ_{2}b)}).

We denote by τ_{(a,b)} the irreducible (a − b + 1)-dimensional representation of U(2) with highest weight (a, b) ∈ ℤ^{2} (a ≥ b) (see (4.1.2)). For (μ_{1}, μ_{2}) ∈ ℤ^{2} with μ_{1} − 1 > μ_{2} > 0, we write σ_{(μ_{1},μ_{2})} ∈ Ĝ′ for a nonholomorphic discrete series representation with minimal K-type τ_{(μ_{1},μ_{2})} and with infinitesimal character (μ_{1} − 1, μ_{2}) ∈ 𝔱_{ℂ}^{∗} in the Harish-Chandra parametrization. Then an explicit computation of the Blattner formula (e.g. [35], Example 6.3) shows:

dim Hom_{U(2)}(τ_{(p,q)}, σ_{(μ_{1},μ_{2})}|_{U(2)})

= min (0, 1 + [\frac{p − μ_{1}}{2} ] , 1 + \frac{p − q − μ_{1} + μ_{2}}{2} )

if p − q − μ_{1} + μ_{2} ∈ 2ℤ. In view of the restriction formula U(2) ↓ 𝕋^{2}:

dim Hom_{𝕋^{2}}(ℂ_{(a,b)}, τ_{(p,q)}|_{𝕋^{2}}) = 1 if a + b = p + q, q ≤ a ≤ p,

we have

(5.5.4)  dim Hom_{𝕋^{2}}(ℂ_{(a,b)}, σ_{(μ_{1},μ_{2})}|_{𝕋^{2}}) = ∞

for any (a, b) ∈ ℤ^{2} because

{(p, q) ∈ ℤ^{2} : Hom_{𝕋^{2}}(ℂ_{(a,b)}, τ_{(p,q)}|_{𝕋^{2}}) ≠ {0},
Hom_{U(2)}(τ_{(p,q)}, σ_{(μ_{1},μ_{2})}|_{U(2)}) ≠ {0}}
= {(p, q) ∈ ℤ^{2} : p + q = a + b, q ≤ a ≤ p, p − q − μ_{1} + μ_{2} ∈ 2ℤ}

is an infinite set. It follows from (5.5.3) and (5.5.4) that we have

dim Hom_{G′}(σ_{(μ_{1},μ_{2})}, π|_{G′}) = ∞

for any (μ_{1}, μ_{2}) ∈ ℤ^{2} with μ_{1} − 1 > μ_{2} > 0.

There exists another family of non-holomorphic discrete series of G′, and we can show similarly that all of them occur in the discrete spectrum in the restriction π|_{G′} with infinite multiplicity. Thus, we have proved (5.5.1) for any non-holomorphic discrete series representation σ_{1} of G′ and for any χ ∈ Ĥ.

On the other hand, suppose σ is a holomorphic discrete series representation of G′. Then, for any (a, b) ∈ ℂ^{2}, we have

dim Hom_{𝕋^{2}}(ℂ_{(a,b)}, σ|_{𝕋^{2}}) < ∞,

because Theorem A guarantees the 𝕋^{2}-admissibility of the restriction σ|_{𝕋^{2}} (see also [16], [30], [43]). We fix (a, b) ∈ ℤ^{2} such that

dim Hom_{𝕋^{2}}(ℂ_{(a,b)}, π|_{𝕋^{2}}) ≠ 0.

Let π ≡ π_{χ} be a unitary principal series representation of G such that χ|_{T} = ℂ_{(a,b)}, where (a, b) is the above fixed one. Then, it follows from (5.5.3) that

0 < dim Hom_{G′}(σ, π|_{G′}) < ∞.

Thus, (5.5.2) is also proved. □

\underline{5.6.} Our strategy of the study of G′-admissible restrictions was to replace G′-admissibility by K′-admissibility (see Theorem A, for example). It is likely to be true that G′-admissibility is equivalent to K′- admissibility in the branching problem in Setting 2.1. Namely, without any assumption of the pair (G, G′), we pose:

Conjecture D. In Setting 2.1, (5.1.1) is equivalent to (5.1.2).

Remark 5.6. 1) The non-trivial part of Conjecture D is the implication (5.1.2) ⇒ (5.1.1).

2) Conjecture D is true if G′ is compact by a trivial reason.

3) There exists a unitary representation π′ of G′ such that π′ is G′- admissible but its restriction π′|_{K′} is not K′-admissible. In this case, Conjecture D implies that there is no hidden symmetry in π′, namely, the representation π′ cannot be extended to a larger reductive group G as an irreducible unitary representation of G.

4) Conjecture D implies that the restriction π|_{G^{σ}} is G^{σ}-admissible if and only if the restriction π|_{G^{σθ}} is G^{σθ}-admissible in the case where (G, G′) is a symmetric pair (G, G^{σ}) and (G, G^{σθ}) is an associated pair.

\underline{5.7.} It is likely that the notion of G′-admissibility restriction coincides with algebraically discrete decomposability, provided (G, G′) is a symmetric pair. For the record, we write

Conjecture E. Suppose (G, G′) is a reductive symmetric pair and π ∈ Ĝ. Then, (5.1.1), (5.1.2) and (5.1.3) are all equivalent.

Remark 5.7. The following three are equivalent (see Remark 5.4 (1)):

Conjectures A and D ⇔ Conjectures C and D ⇔ Conjecture E.

\underline{5.8.} Suppose that G ⊃ G′ are both real reductive linear Lie groups. Here is a (weak) relationship between discontinuous groups and branching problems of unitary representations (see the diagram in §3.2):

Conjecture F. If there exists a discrete subgroup Γ of G such that Γ acts properly discontinuously and co-compactly on a homogeneous space G/G′, then there exists an infinite dimensional unitary representation π ∈ Ĝ such that π|_{G′} is G′-admissible.

Remark 5.8. 1) Conjecture F is true if G′ = K, a maximal compact subgroup of G. In fact, there always exists such a discrete subgroup Γ by the theorem of Borel [5], while all π ∈ Ĝ is K-admissible by the theorem of Harish-Chandra (Fact 2.4).

2) Conjecture F is true if G′ is a normal real form of a complex reductive Lie group G. In fact, there is no such a discrete subgroup Γ by the Calabi-Markus phenomenon (see [7], [21], [26]), while there is no such π by Example 4.12.3.

3) Conjecture F is true if G is the direct product group G_{1} × G_{1} and G′ is a diagonally embedded subgroup diag G_{1} in G. In fact, the assumption of Conjecture F is satisfied by taking Γ := Γ_{1} × {e} with Γ_{1} co-compact discrete subgroup in G_{1}. The conclusion of Conjecture F holds because one can take π := π_{1} ⊠ 1 with π_{1} ∈ Ĝ_{1}. The above choice of Γ and π are more or less trivial. But non-trivial examples of Γ and π also exist for certain G_{1} such as G_{1} ≃ SU(2, 2) (see [26] for example).

4) Conjecture F is true if (G, G′) = (SU(2n, 2), Sp(n, 1)) for any n as we saw in Example 3.3.

References

[1] J. Adams, Discrete spectrum of the dual reductive pair (O(p, q), Sp(2m)), Invent. Math., 74 (1984), 449–475.

[2] M. F. Atiyah, The Harish-Chandra character, London Math. Soc. Lecture Note Series, 34 (1979), 176–181.

[3] Y. Benoist, Actions propres sur les espaces homogenes reductifs, Annals of Math., 144 (1996), 315–347.

[4] B. Binegar and R. Zierau, Unitarization of a singular representation of SO(p, q), Comm. Math. Phys., 138 (1991), 245–258.

[5] A. Borel, Compact Clifford-Klein forms of symmetric spaces, Topology, 2 (1963), 111-122.

[6] W. Borho and J. L. Brylinski, Differential operators on homogeneous spaces I, Invent. Math., 69 (1982), 437-476. Part III, 80 (1985), 1-68

[7] E. Calabi and L. Markus, Relativistic space forms, Annals of Math., 75 (1962), 63–76.

[8] G. van Dijk and S. C. Hille, Canonical representations related to hyperbolic spaces, J. Funct. Anal., 147 (1997), 109–139.

[9] I. M. Gelfand, M. I. Graev and I. Piateski-Shapiro, “Representation Theory and Automorphic Functions”, Saunders Math. Books, 1969.

[10] B. H. Gross and N. R. Wallach, A distinguished family of unitary representations for the exceptional groups of real rank = 4, in “Lie Theory and Geometry, In honor of Bertram Kostant”, Progress in Math., Birkhäuser, 1994, pp. 289–304.

[11] Harish-Chandra, Representations of semi-simple Lie groups, I, III, Trans. A. M. S. 75, (1953), 185–243; 76, (1954), 234–253; IV, Amer. J. Math. 77, (1955), 743–777.

[12] H. Hecht and W. Schmid, A proof of Blattner’s conjecture, Invent. Math., 31 (1976), 129-154.

[13] L. Hörmander, “The Analysis of Linear Partial Differential Operators I”, Grundlehren, vol. 256, Springer-Verlag, Berlin, 1983.

[14] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math., 33 (1979), 275-285, A. M. S.

[15] J-S. Huang, Harmonic analysis on compact polar homogeneous spaces, Pacific J. Math., 175 (1996), 553—569.

[16] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal., 34 (1979), 29-53.

[17] M. Kashiwara, T. Kawai and T. Kimura, “Foundations of Algebraic Analysis”, Princeton Math. Series, vol. 37, Princeton Univ. Press, New Jersey, 1986.

[18] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math., 44 (1978), 1–47.

[19] , K-types and singular spectrum, in “Lect. Notes. in Math., vol. 728”, Springer-Verlag, 1979, pp. 177–200.

[20] A. Knapp and D. Vogan, “Cohomological Induction and Unitary Representations”, Princeton University Press, 1995.

[21] T. Kobayashi, Proper action on a homogeneous space of reductive type, Math. Ann., 285 (1989), 249–263.

[22] , “Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p−m, q; 𝔽)”, vol. 462, Memoirs of A.M.S., 1992.

[23] , The Restriction of A_{𝔮}(λ) to reductive subgroups, Proc. Japan Acad., 69 (1993), 262–267. Part II, ibid. 71 1995, 24–26.

[24] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math., 117 (1994), 181-205.

[25] , Criterion of proper actions on homogeneous space of reductive groups, J. Lie Theory, 6 (1996), 147–163.

[26] , Discontinuous groups and Clifford-Klein forms of pseudo-Riemannian homogeneous manifolds, in “Algebraic and Analytic Methods in Representation Theory”, (H. Schlichtkrull and B. Ørsted, ed.), Perspectives in Mathematics 17, Academic Press, 1996, pp. 99–165. Lecture Notes of the European School on Group Theory.

[27] , Invariant measures on homogeneous manifolds of reductive type, J. reine und angew. Math., 490 (1997), 37–53.

[28] , L^{p}-analysis on homogeneous manifolds of reductive type and representation theory, Proc. Japan Acad., 73 (1997), 62–66.

[29] , Harmonic analysis on homogeneous manifolds of reductive type and unitary representation theory, in “Translations, Series II, Selected Papers on Harmonic Analysis, Groups, and Invariants”, (K. Nomizu, ed.), vol. 183, Amer. Math. Soc., 1998, pp. 1–31.

[30] , Discrete decomposability of the restriction of A_{𝔮}(λ), Part II — micro-local analysis and asymptotic K-support, Annals of Math., 147 (1998), 709–729.

[31] , Discrete decomposability of the restriction of A_{𝔮}(λ), Part III — restriction of Harish-Chandra modules and associated varieties, Invent. Math., 131 (1998), 229–256.

[32] , Discrete series representations for homogeneous spaces and the restriction of irreducible unitary representations, J. Funct. Anal., 152 (1998), 100–135.

[33] , Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (eds. K. Mimachi) (1997), 9–17.

[34] T. Kobayashi and T. Oda, Vanishing theorem of modular symbols on locally symmetric spaces, Comment. Math. Helv., 73 (1998), 45–70.

[35] T. Kobayashi, Multiplicity-free theorem in branching problems of unitary highest weight modules, preprint.

[36] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits, C. R. Acad. Sci. Paris, 326 (1998), 925–930.

[37] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws of O(p, q) attached to minimal nilpotent orbits (to appear).

[38] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4), (1990), no. 92 Progress in Mathematics, Birkhäuser, 85–124.

[39] J-S. Li, On the discrete series of generalized Stiefel manifolds, Trans. A. M. S., 340 (1993), 753–766.

[40] J-S. Li, Two reductive dual pairs in groups of type E, Manuscripta Math., 91 (1996), 163–177.

[41] R. Lipsman, Restrictions of principal series to a real form, Pacific J. Math., 89 (1980), 367–390.

[42] G. Margulis, Existence of compact quotients of homogeneous spaces, measurably proper actions, and decay of matrix coefficients, Bul. Soc. math. France, 125 (1997), 1–10.

[43] S. Martens, The characters of the holomorphic discrete series, Proc. Nat. Acad. Sci. USA, 72 (1975), 3275–3276.

[44] Y. A. Neretin and G. I. Ol’shanskii, Boundary values of holomorphic functions, special unitary representations of O(p, q), and their limits, Zapiski Nauchnykh Seminarov POMI, 223 (1995), 9–91.

[45] R. S. Palais, On the existence of slices for actions of noncompact Lie groups, Annals of Math., 73 (1961), 295–323.

[46] S. Salamanca Riba, On the unitary dual of some classical Lie groups, Compo. Math., 68 (1988), 251–303.

[47] W. Schmid, Die Randwerte holomorphe Funktionen auf hermetisch symmetrischen Raumen, Invent. Math., 9 (1969-70), 61–80.

[48] D. A. Vogan, Jr., “Representations of real reductive Lie groups”, Progress in Math., vol. 15, Birkhäuser, 1981.

[49] , Unitarizability of certain series of representations, Annals of Math., 120 (1984), 141–187.

[50] , “Unitary Representations of Reductive Lie Groups”, Annals of Math. Stud., vol. 118, Princeton University Press,, New Jersey, 1987.

[51] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math., 14 (1988), 191–221.

[52] , Associated varieties and unipotent representations, in “Harmonic Analysis on Reductive Lie Groups, vol. 101”, Birkhäuser, 1991, pp. 315-388.

[53] D. A. Vogan, Jr. and G. J. Zuckerman, Unitary representations with non-zero cohomology, Compositio Math., 53 (1984), 51–90.

[54] N. Wallach, On the unitarizability of derived functor modules, Invent. Math., 78 (1984), 131–141.

[55] , “Real Reductive Groups I, II”, Pure and Appl. Math., vol. 132, Academic Press, 1988, 1992.

[56] H. Wong, Dolbeault cohomologies and Zuckerman modules associated with finite rank representations, Ph.D. dissertation, Harvard University (1991).

[57] J. Xie, Restriction of discrete series of SU(2, 1) to S(U(1) × U(1, 1)), ph.D dissertation, Rutgers University. published in J. Funct. Anal. 122 (1994), 478–518

Graduate School of Mathematical Sciences The University of Tokyo Tokyo 153-8914, JAPAN