MULTIPLICITY FREE THEOREM IN BRANCHING

PROBLEMS OF UNITARY HIGHEST WEIGHT MODULES^{∗}

Toshiyuki KOBAYASHI

University of Tokyo

Abstract. Let π be a unitary highest weight module of a reductive Lie group G, and (G, G′) a reductive symmetric pair such that G′ ↪ G induces a holomorphic embedding of Hermitian symmetric spaces G′/K′ ↪ G/K. This paper proves that the multiplicity of irreducible representations of G′ occurring in the restriction π|_{G′} is uniformly bounded. Furthermore, we prove that the multiplicity is free if π has a one dimensional minimal K-type. Our method here also establishes an analogous result for the tensor product of unitary highest weight modules, and also for finite dimensional representations of compact groups. Finally, we give an explicit branching formula of a holomorphic discrete series representation π with respect to a semisimple symmetric pair (G, G′). This formula is a generalization of the Kostant-Schmid branching formula which deals with the case G′ = K.

§1 Introduction

\underline{1.1.} Let G be a reductive Lie group, and Ĝ the unitary dual. Suppose H is a reductive subgroup of G. If π ∈ Ĝ, then the restriction π|_{H} is no more irreducible as a representation of H in general. The irreducible decomposition formula of π|_{H} is called the branching law (breaking symmetry in physics) and is written in terms of the direct integral of unitary representations of H:

(1.1.1)  π|_{H} ≃ ∫ ⊕ ; Ĥ m_{H}(τ : π|_{H}) τ dμ(τ),

where dμ is a Borel measure on Ĥ and m_{H}(· : π|_{H}): Ĥ → ℕ∪{∞} is the multiplicity defined almost everywhere with respect to dμ.

One expects a simple and detailed study for the branching problem when no continuous spectrum arises in the decomposition (1.1.1) (discrete branching law), and the general theory for discrete branching laws has been studied in [K1], [K2], [K3], [K4], [K5]. A very special and simple setting of the discrete branching laws is when the following (a) and (b) hold:

a) π ∈ Ĝ is an irreducible unitary highest weight module (see §1.2 for definition), and

b) (G, H) is a semisimple symmetric pair satisfying (1.3.1) (see §1.3 for details).

∗ 表現論シンポジウム報告集, 佐賀県波戸岬, 1997 年 11 月 17 日–20 日

Typeset by AMS-TEX

The purpose of this note is to investigate the restriction π|_{H} in this special setting (a) and (b).

\underline{1.2.} Let G be a non-compact simple Lie group of finite center, θ a Cartan involution of G, and K := {g ∈ G : θg = g}. We write 𝔤 = 𝔨 + 𝔭 for the Cartan decomposition of the Lie algebra 𝔤 of G, corresponding to the Cartan involution θ. We assume that G is of Hermitian type, that is, the center 𝔠(𝔨) of 𝔨 is non-trivial. Then, it is well-known that 𝔠(𝔨) is one dimensional and that there exists Z ∈ 𝔠(𝔨) so that

𝔤_{ℂ} := 𝔤 ⊗ ℂ = 𝔨_{ℂ} ⊕ 𝔭^{+} ⊕ 𝔭^{−}

is the direct sum decomposition of eigenspaces of ad(Z) with eigenvalues 0, \sqrt{−1} and −\sqrt{−1}, respectively.

Definition 1.2.1. Let (π, H) be an irreducible unitary representation of G, and H_{K} the underlying (𝔤_{ℂ}, K)-module. (π, H) is called an irreducible unitary highest weight module if H_{K}^{𝔭^{+}} ≠ {0}, where we put

H_{K}^{𝔭^{+}} := {v ∈ H_{K} : dπ(Y)v = 0 for any Y ∈ 𝔭^{+}}.

Then, H_{K}^{𝔭^{+}} is an irreducible representation of K. We say that π is of scalar type (or of scalar minimal K-type) if H_{K}^{𝔭^{+}} is one dimensional. By a holomorphic discrete series representation for G, we mean that π is a unitary highest weight module that can be realized as a closed G-invariant subspace of L^{2}(G) (if G has an infinite center, then we need a slight modification as usual). Lowest weight modules and anti-holomorphic discrete series are defined similarly with 𝔭^{+} replaced by 𝔭^{−}.

\underline{1.3.} Suppose τ is an involutive automorphism of G commuting with θ. Because τ𝔠(𝔨) = 𝔠(𝔨) = ℝZ and τ^{2} = id, there are two exclusive possibilities:

(1.3.1)  τZ = Z,
(1.3.2)  τZ = −Z.

Let G^{τ} := {g ∈ G : τg = g} and K^{τ} := G^{τ} ∩ K.

Geometrically, (1.3.1) implies:

1-a) τ acts holomorphically on the Hermitian symmetric space G/K, 1-b) G^{τ} /K^{τ} ↪ G/K is a complex submanifold.

On the other hand, (1.3.2) implies:

2-a) τ acts anti-holomorphically on the Hermitian symmetric space G/K, 2-b) G^{τ} /K^{τ} ↪ G/K is a totally real submanifold.

§2 Main theorems

\underline{2.1.} Let G be a non-compact simple Lie group of Hermitian type. Here are our main results:

Theorem A. Let π_{1} and π_{2} be unitary highest weight modules of G. Then, there is a constant C(π_{1}, π_{2}) < ∞ with the following properties:

1) The tensor product π_{1} ˆ⊗π_{2} splits into a discrete Hilbert sum of irreducible unitary representations of G:

π_{1} ˆ⊗π_{2} ≃ _{\begin{matrix}∑ & ⊕ \\ μ∈Ĝ \end{matrix}}m_{π_{1},π_{2}}(μ)μ, (Hilbert direct sum),

with the multiplicity satisfying

(2.1.1)  m_{π_{1},π_{2}}(μ) ≤ C(π_{1}, π_{2}) for all μ ∈ Ĝ.

2) C(π_{1}, π_{2}) = 1 if both π_{1} and π_{2} are of scalar minimal K-types. Namely, the tensor product π_{1} ˆ⊗π_{2} is decomposed discretely into irreducible unitary representations of G with multiplicity free, for any unitary highest weight modules π_{1} and π_{2} of scalar minimal K-types.

Theorem B. Let π be a unitary highest weight module of G. Then, there is a constant C(π) < ∞ with the following properties: Suppose that τ is an involutive automorphism of G satisfying (1.3.1). Let H be an open subgroup of G^{τ}.

1) The restriction π|_{H} splits into a discrete Hilbert sum of irreducible unitary representations of H:

π|_{H} ≃ _{\begin{matrix}∑ & ⊕ \\ μ∈Ĥ \end{matrix}}m_{π}(μ)μ (Hilbert direct sum),

with the multiplicity satisfying

(2.1.2)  m_{π}(μ) ≤ C(π) for all μ ∈ Ĥ.

2) C(π) = 1 if π is of scalar minimal K-type. Namely, the restriction π|_{H} is decomposed discretely into irreducible unitary representations of H with multiplicity free, for any unitary highest weight module π of G having scalar minimal K-type.

The infinitesimal classification of irreducible symmetric pairs was achieved by M. Berger [B]. For the reader’s convenience, we give a list of the infinitesimal classification of irreducible symmetric pair (G, H) satisfying the condition (1.3.1) (see Theorem B).

(𝔤, 𝔤^{τ}) satisfying (1.3.1) τZ = Z
𝔤  𝔤^{τ}
𝔰𝔲(p, q)  𝔰(𝔲(i, j) + 𝔲(p − i, q − j))
𝔰𝔲(n, n)  𝔰𝔬^{∗}(2n)
𝔰𝔲(n, n)  𝔰𝔭(n, ℝ)
𝔰𝔬^{∗}(2n)  𝔰𝔬^{∗}(2p) + 𝔰𝔬^{∗}(2n − 2p)
𝔰𝔬^{∗}(2n)  𝔲(p, n − p)
𝔰𝔬(2, n)  𝔰𝔬(2, p) + 𝔰𝔬(n − p)
𝔰𝔬(2, 2n)  𝔲(1, n)
𝔰𝔭(n, ℝ)  𝔲(p, n − p)
𝔰𝔭(n, ℝ)  𝔰𝔭(p, ℝ) + 𝔰𝔭(n − p, ℝ)
𝔢_{6(−14)}  𝔰𝔬(10) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔬^{∗}(10) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔬(8, 2) + 𝔰𝔬(2)
𝔢_{6(−14)}  𝔰𝔲(5, 1) + 𝔰𝔩(2, ℝ)
𝔢_{6(−14)}  𝔰𝔲(4, 2) + 𝔰𝔲(2)
𝔢_{7(−25)}  𝔢_{6} + 𝔰𝔬(2)
𝔢_{7(−25)}  𝔢_{6(−14)} + 𝔰𝔬(2)
𝔢_{7(−25)}  𝔰𝔬(10, 2) + 𝔰𝔩(2, ℝ)
𝔢_{7(−25)}  𝔰𝔬^{∗}(12) + 𝔰𝔲(2)
𝔢_{7(−25)}  𝔰𝔲(6, 2)

Table 2.1.3

\underline{2.2.} Here are simplest examples of Theorem A and Theorem B, respectively:

Example 2.2. We denote by π_{n} the holomorphic discrete series representation of SL(2, ℝ) with minimal K-type χ_{n} (n ≥ 2), where χ_{n} (n ∈ ℤ) stands for a character of SO(2). Then, the following branching formulae are well-known:

π_{m} ˆ⊗π_{n} ≃ \begin{matrix}∑ & ⊕ \\ k∈ℕ \end{matrix}π_{m+n+2k},
π_{n}|_{SO(2)} ≃ \begin{matrix}∑ & ⊕ \\ k∈ℕ \end{matrix}χ_{n+2k}.

Here, ℕ = {0, 1, 2, … }. We note that any holomorphic discrete series representation of SL(2, ℝ) is of scalar minimal K-type.

\underline{2.3.} The conditions “highest weight modules”, “discrete branching”, “scalar minimal K-type” are crucial in the multiplicity free, uniformly bounded, or bounded theorems in Theorem A and Theorem B. Here are related remarks:

Remark 2.3.

1) The discrete decomposability in Theorems A and B was previously known (see [Mr] and [Li], Theorem 4.2; [JV], Corollary 2.3 for a holomorphic discrete series π; see also [K2], Corollary 4.4; [K6], Theorem 7.4 for a general case). The novelty of Theorems A and B is the estimate of multiplicities (2.1.1) and (2.1.2).

2) The Cartan involution θ automatically satisfies (1.3.1). In this case, we have H = K and the multiplicity free result in Theorem B is known by B. Kostant, W. Schmid and K. Johnson ([Sc], [Jo]) by explicit branching laws in the case where π is a holomorphic discrete series representation of scalar type. Their formula will be generalized to a non-compact H also in §4.

3) If π = A_{𝔮}(λ) in the sense of Vogan-Zuckerman (e. g. a discrete series representation) and if (G, H) is a semisimple symmetric pair such that π|_{H} is discrete decomposable, then the multiplicity always satisfies

m_{π}(τ) < ∞ for any τ ∈ Ĥ

(Wallach conjecture; see [K5], Corollary 4.3). However, there is an example with

sup_{τ∈Ĥ} m_{π}(τ) = ∞

in this setting (e. g. [K7], Example 6.2). Namely, the multiplicity is always finite but not necessarily uniformly bounded in the discrete branching laws of non-highest weight modules with respect to a reductive symmetric pair.

4) The multiplicity can be infinite in the continuous spectrum if π = A_{𝔮}(λ) is not a highest weight module and if (G, H) is a symmetric pair (see [K2], Introduction).

5) It follows from R. Howe [H] and J. Repka [Re] that the irreducible decomposition of the tensor product π_{1} ˆ⊗π_{2} always involves a continuous spectrum, if π_{1} is a holomorphic discrete series representation and and π_{2} is an anti-holomorphic discrete series representation. This is regarded as an opposite extremal case to Theorem A. Likewise, if π is a highest weight module of scalar minimal K-type and if τ satisfies (1.3.2) instead of (1.3.1), then Ólafsson and B. Ørsted proved that π|_{H} is decomposed into only continuous spectrum with multiplicity free [OO]. This is an opposite extremal case to Theorem B (2).

6) If we drop the assumption of the scalar minimal K-type in Theorem A or Theorem B, then there is a counter example for multiplicity free (e. g. [K7], Example 6.2). Namely, C(π_{1}, π_{2}) in Theorem A (also C(π) in Theorem B) cannot be always taken to be 1.

7) Finally, we mention the case where dim π < ∞. Our method here also gives a sufficient condition for the multiplicity free branching laws for finite dimensional representations of compact groups, which is analogous to the second part of Theorems A and B. A complete list of the multiplicity free cases that can be obtained by our method is given in [K7], Theorem 7.3 and Theorem 7.4. Some of them could be also proved by using so called the Littlewood-Richardson rule and the algorithm of K. Koike and I. Terada in [KT2]. S. Okada recently obtained a number of multiplicity free branching laws by combinatorial arguments of character formulae for classical compact Lie groups [Ok]. It might be interesting from combinatorial view point to obtain explicit branching laws for the remaining cases (many of them are exceptional cases) for which the multiplicity is proved to be free by our method.

§3 Sketch of proof

\underline{3.1.} Let L → D be a holomorphic line bundle over a complex manifold D. We denote by O(L) the space of holomorphic sections of L → D. Then O(L) carries a Fréchet topology by the uniform convergence on compact sets. If a Lie group H acts holomorphically and equivariantly on the holomorphic line bundle L → D, then H defines a (continuous) representation on O(L) by the pull-back of sections.

Let {U_{α}} be a trivializing neighbourhood of D, and g_{αβ} ∈ O^{×}(U_{α} ∩U_{β}) the transition functions of the holomorphic line bundle L → D. Then an anti-holomorphic line bundle \overline{L} → D is a complex line bundle with the transition functions \overline{g_{αβ}}. We denote by \overline{O}(\overline{L}) the space of anti-holomorphic sections of \overline{L} → D.

Suppose σ is an anti-holomorphic diffeomorphism of D. Then the pull-back σ^{∗}L → D is an anti-holomorphic line bundle over D. In turn, \overline{σ^{∗}L} → D is a holomorphic line bundle over D.

\underline{3.2.} A main machinery for the proof of Theorem A and Theorem B is the commutativity of the commutant algebra

End_{H}(H) := {T ∈ End(H) : T is continuous, T π(h) = π(h)T for any h ∈ H},

if a unitary representation (π, H) of the group H is realized on holomorphic functions (or holomorphic sections) on a complex manifold D.

Faraut and Thomas [FT], in the case of trivial twisting parameter, gives a sufficient condition for the commutativity of End_{H}(H) by using the theory of reproducing kernels, which we extend to the general, twisted case below.

Lemma 3.2. Let (π, H) be a unitary representation of a Lie group H. Assume that there exist an H-equivariant holomorphic line bundle L → D and an antiholomorphic involutive diffeomorphism σ of D with the following three conditions: (3.2.1) There is an injective (continuous) H-intertwining map H → O(L). (3.2.2) There exists an isomorphism of H-equivariant holomorphic line bundles

Ψ: L ⟶^{∼} \overline{σ^{∗}L}.

(3.2.3) Given x ∈ D, there exists g ∈ H such that σx = g · x.

Then, End_{H}(H) is a commutative algebra.

\underline{3.3.} The idea of Lemma 3.2 parallels to [FT], which goes back to a lemma due to I. M. Gelfand:

Lemma 3.3 ([G], see also [La], IV, Theorem 1). Let G be a locally compact unimodular group, and K a compact subgroup. Assume that there exists an antiinvolutive automorphism σ of G such that given x ∈ g there exist k_{1}, k_{2} ∈ K satisfying σx = k_{1}xk_{2}. Then, the Hecke algebra L^{1}(K\G/K) is a commutative ring.

\underline{3.4.} The following is a key lemma to apply Lemma 3.2 by supplying a sufficient condition for (3.2.3) in the setting where D = G/K is a Riemannian symmetric space.

Lemma 3.4. Let G be a non-compact semisimple Lie group of finite center, K a maximal compact subgroup of G corresponding to a Cartan involution θ. Let σ and τ are involutive automorphisms of G. We assume the following two conditions: (3.4.1) σ, τ and θ commute with one another. (3.4.2) ℝ- rank 𝔤/𝔤^{τ} = ℝ- rank 𝔤^{σ}/𝔤^{σ,τ}.

Then for any x ∈ G/K, there exists g ∈ G_{0}^{τ} such that σ(x) = g · x.

The proof of Theorem B (similar, but easier for Theorem A) completes by showing the existence of σ ∈ Aut(G) satisfying (1.3.2), (3.4.1) and (3.4.2), for each τ ∈ Aut(G) satisfying (1.3.1).

§4 Explicit branching laws

— a generalization of the Kostant-Schmid formula

\underline{4.1.} Once we obtain (abstract) results on free multiplicities, then we with to obtain explicit formulae of such branching problems as a second stage. Theorem B asserts the multiplicity freeness of the branching law π|_{H}, especially in the case where

π ∈ Ĝ : holomorphic discrete series of scalar minimal K-type
H := G_{0}^{τ} : τ satisfies the condition (1.3.1).

This section presents an explicit branching law of π|_{H} in this setting. In particular, we generalize the Kostant-Schmid formula ([Sc], [Jo]) which corresponds to the case τ = θ (Cartan involution), namely H = K.

\underline{4.2.} Let us fix notation. Suppose that G is a simple non-compact connected Lie group of Hermitian type, and that τ ∈ Aut(G) satisfies (1.3.1). We take a Cartan subalgebra 𝔱 of 𝔨 such that 𝔱^{τ} := {X ∈ 𝔱 : τX = X} is also a Cartan subalgebra of 𝔨^{τ} := {X ∈ 𝔨 : τX = X}. We fix positive systems Δ^{+}(𝔨^{τ} , 𝔱^{τ}) and Δ^{+}(𝔨, 𝔱). Because τ satisfies (1.3.1), the direct sum decomposition

𝔤_{ℂ} = 𝔨_{ℂ} ⊕ 𝔭^{+} ⊕ 𝔭^{−}

is stable under τ (complex linear extension). Then we have a direct sum decomposition 𝔭^{+} = (𝔭^{+})^{τ} ⊕ (𝔭^{+})^{−τ}. Let Δ((𝔭^{+})^{−τ} , 𝔱^{τ}) (⊂ \sqrt{−1}(𝔱^{τ})^{∗}) be the set of weights of (𝔭^{+})^{−τ} with respect to 𝔱^{τ}.

The roots α and β are called strongly orthogonal if neither α + β nor α − β is a root. We take a maximal set of strongly orthogonal roots, say {ν_{1}, ν_{2}, …, ν_{k}}, such that

i) ν_{1} is the highest root among Δ((𝔭^{+})^{−τ} , 𝔱^{τ}),

ii) ν_{j+1} is the highest root in Δ((𝔭^{+})^{−τ} , 𝔱^{τ}) strongly orthogonal to ν_{1}, …, ν_{j}. We note that

k = ℝ- rank G/G^{τ} .

\underline{4.3.} We denote by V^{G}(μ) the irreducible highest weight module of G if (V^{G}(μ))^{𝔭^{+}} is an irreducible representation of K with highest weight μ ∈ \sqrt{−1}𝔱^{∗} with respect to Δ^{+}(𝔨, 𝔱) (see §1.2). Likewise, V^{H}(ν) denotes the irreducible highest weight module of H = G_{0}^{τ} if (V^{H}(ν))^{(𝔭^{+})^{τ}} is an irreducible representation of K_{0}^{τ} with highest weight μ ∈ \sqrt{−1}(𝔱^{τ})^{∗} with respect to Δ^{+}(𝔨^{τ} , 𝔱^{τ}).

Clearly, V^{G}(μ) is of scalar minimal K-type if and only if μ vanishes on the maximal semisimple ideal of 𝔨.

\underline{4.4.} Now we are ready to state an explicitly branching formula:

Theorem C. Let G be a connected non-compact simple Lie group of Hermitian type, and H := G_{0}^{τ} the connected component of the fixed point group G^{τ} of an involution τ ∈ Aut(G) satisfying (1.3.1). If V^{G}(μ) ∈ Ĝ is a holomorphic discrete series representation of scalar minimal K-type, then

(4.4.1)  V^{G}(μ)|_{H} ≃ _{^{∑⊕} ; \begin{matrix}a_{1}≥···≥a_{k}≥0 \\ a_{j}∈ℕ \end{matrix}}V^{H}(μ|_{𝔱^{τ}} + ∑_{j=1}^{k} a_{j}ν_{j}).

If τ = θ, then H = K and dim V^{H}(μ|_{𝔱^{τ}} + ∑_{j=1}^{k} a_{j}ν_{j}) < ∞. In this case, (4.4.1) coincides with the formula in [Sc] or [Jo].

References

[B] M. Berger, Les espaces symétriques non compacts, Ann. Sci. Ecole Norm. Sup. (3) 74 (1957), 85–177.

[FT] J. Faraut and E. Thomas, Invariant Hilbert spaces of holomorphic functions, (in preparation).

[G] I. M. Gelfand, Spherical functions on symmetric spaces, Dokl. Akad. Nauk. SSSR 70 (1950), 5–8.

[H] R. Howe, Reciprocity laws in the theory of dual pairs, Representation Theory of Reductive Groups (P. C. Trombi, ed.), vol. 40, Progress in Mathematics, Birkhäuser, Boston, 1983, pp. 159–175.

[JV] H. P. Jakobsen and M. Vergne, Restrictions and expansions of holomorphic representations, J. Funct. Anal. 34 (1979), 29–53.

[Jo] K. Johnson, On a ring of invariant polynomials on a Hermitian symmetric space, J. Algebra 67 (1980), 72–81.

[K1] T. Kobayashi, The restriction of A_{𝔮}(λ) to reductive subgroups, Proc. Japan Acad. 69 (1993), 262-267.

[K2] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181-205.

[K3] T. Kobayashi, The restriction of A_{𝔮}(λ) to reductive subgroups II, Proc. Japan Acad. 71 (1995), 24–26.

[K4] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups II — micro-local analysis and asymptotic K-support, Annals of Math. (to appear).

[K5] T. Kobayashi, Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups III — restriction of Harish-Chandra modules and associated varieties, Invent. Math. (to appear).

[K6] T. Kobayashi, Discrete series representations for the orbit spaces arising from two involutions of real reductive Lie groups, J. Funct. Anal. (to appear).

[K7] T. Kobayashi, Multiplicity free branching laws for unitary highest weight modules, preprint (1997).

[KT1] K. Koike and I. Terada, Young diagrammatic methods for the representation theory of the classical groups of type B_{n}, C_{n}, D_{n}, J. Algebra 107 (1987), 466–511.

[KT2] K. Koike and I. Terada, Young diagrammatic methods for the restriction of representations of complex classical Lie groups to reductive subgroups of maximal rank, Adv. in Math. 79 (1990), 104–135.

[La] S. Lang, SL_{2}(ℝ), Addison-Wesley, MA, 1975.

[Li] R. Lipsman, Restrictions of principal series to a real form, Pacific J. of Math. 89 (1980), 367-390.

[Mr] S. Martens, The characters of the holomorphic discrete series, Proc. Nat. Acad. Sci. USA 72 (1975), 3275-3276.

[Ok] S. Okada, Applications of minor summation formulas to rectangular-shaped representations of classical groups, preprint.

[OO] G. Ólafsson and B. Ørsted, Generalizations of the Bargmann transform, Proceedings of Workshop on Lie Theory and its application in physics, Clausthal, 1995 (to appear).

[Re] J. Repka, Tensor products of holomorphic discrete series representations, Can. J. Math. 31 (1979), 836-844.

[Sc] W. Schmid, Die Randwerte holomorphe Funktionen auf hermetisch symmetrischen Raumen, Invent. Math. 9 (1969-70), 61–80.

Department of Mathematical Sciences University of Tokyo Komaba, Meguro, Tokyo 153 Japan E-mail address: toshi@ms.u-tokyo.ac.jp