CONFORMAL GEOMETRY AND BRANCHING LAWS FOR UNITARY

REPRESENTATIONS ATTACHED TO MINIMAL NILPOTENT ORBITS

Toshiyuki KOBAYASHI and Bent Ørsted

University of Tokyo and Odense University

Abstract For the unitary representation of O(p, q) attached to the minimal nilpotent coadjoint orbit, we explicitly calculate its restriction to certain natural dual pairs in O(p, q). We furthermore show how the results are compatible with the orbit method, in particular when viewing the minimal nilpotent orbit as belonging to the limit set of semisimple orbits.

GÉOMÉTRIE CONFORME ET RESTRICTIONS DES REPRÉSENTATIONS UNITAIRES ASSOCIÉES AUX ORBITES NILPOTENTES MINIMALES

Résumé Nous donnons une description explicite de la restriction à certaines paires duales naturelles de la représentation unitaire de O(p, q) associée à l’orbite coadjointe nilpotente minimale. De plus, nous montrons que les résultats obtenus sont compatibles avec la méthode des orbites, particulièrement lorsqu’on considère l’orbite comme limite d’une famille bien déterminée d’orbites semi-simples.

Version française abrégée. Si G est un groupe de Lie, on désigne par Ĝ son dual unitaire, i.e. l’ensemble des classes d’équivalence de ses représentations unitaires irréductibles. Soit G un groupe réductif et G′ l’un de ses sous-groupes réductifs. Si π ∈ Ĝ, la restriction π|_{G′} à G′ n’est en général pas irréductible. Nous nous proposons donc d’obtenir, dans certains cas, une formule explicite pour la décomposition en irréductibles de cette restriction:

π|_{G′} ≃ ∫ ⊕ ; Ĝ′ m_{π}(τ)τdμ(τ) (intégrale hilbertienne),

où m_{π}(τ) ∈ ℕ ∪ {∞} et dμ est une mesure borélienne sur Ĝ′.

Dans la suite on suppose que G = O(p, q) et G′ := G′_{1} × G′_{2} = O(p′, q′) × O(p′′, q′′), avec p′ + p′′ = p, q′ + q′′ = q. Remarquons que G′_{1} et G′_{2} forment une paire duale réductive de G, i.e. ce sont deux sous-groupes réductifs, commutant l’un de l’autre, de G. Nous construisons les représentations irréductibles du groupe O(p, q) (p ≥ 2, q ≥ 2), ϖ^{p,q} et π_{+,λ}^{p,q}, π_{−,λ}^{p,q}, associées respectivement à l’orbite nilpotente minimale et aux orbites elliptiques minimales.

Typeset by AMS-TEX

En fait, si 𝔤_{0} désigne l’algèbre de Lie de O(p, q), les représentations π_{+,λ}^{p,q} et π_{−,λ}^{p,q} sont associées respectivement aux orbites elliptiques Ad^{∗}(G)λf_{i} (i = 1, 2) avec λ ∈ ℤ + \frac{1}{2}(p + q) et λ ≥ 0, f_{1}, f_{2} ∈ \sqrt{−1}𝔤_{0}^{∗} étant les formes linéaires définies par

⟨f_{1}, X⟩ := −\sqrt{−1}X_{12}, ⟨f_{2}, X⟩ := −\sqrt{−1}X_{p+1,p+2}, X = (X_{ij})_{1≤i,j≤p+q} ∈ 𝔤_{0}.

De plus, il est commode de définir π_{+,− \frac{1}{2}}^{1,0} = 1 (resp. π_{+, \frac{1}{2}}^{1,0} = sgn) comme étant la représentation triviale (resp. ”signe”) de O(1, 0) = O(1). Alors, si on pose

A(p, q) := \{ \begin{matrix}\{λ ∈ ℤ + \frac{p+q}{2} : λ > 0\} & (p > 1, q ≠ 0), \\ \{λ ∈ ℤ + \frac{p+q}{2} : λ ≥ \frac{p}{2} − 1\} & (p > 1, q = 0), \\ ∅ & (p = 1, q ≠ 0) or (p = 0), \\ \{−\frac{1}{2}, \frac{1}{2}\} & (p = 1, q = 0). \end{matrix}

les π_{+,λ}^{p,q}, λ ∈ A(p, q), constituent l’ensemble des éléments de la série discrète pour l’hyperboloïde réel G-homogène X(p, q) := {(x, y) ∈ ℝ^{p,q} : |x|^{2} − |y|^{2} = 1}.

Maintenant, nous nous intéressons au spectre discret de la restriction de la représentation unipotente minimale ϖ^{p,q} de G (définie par exemple dans [1]) à la paire réductive duale G′ = G′_{1} × G′_{2}, lorsque p + q ∈ 2ℕ, p ≥ 2, q ≥ 2 et (p, q) ≠ (2, 2). On pose A′(p, q) := A(p, q) ∩ {λ ∈ ℝ : λ > 1}.

Théorème A. La restriction ϖ^{p,q}|_{G′} contient

\begin{matrix}∑ & ⊕ \\ λ∈A′(p′,q′)∩A′(q′′,p′′) \end{matrix}π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} ⊕ \begin{matrix}∑ & ⊕ \\ λ∈A′(q′,p′)∩A′(p′′,q′′) \end{matrix}π_{−,λ}^{p′,q′} ⊠ π_{+,λ}^{p′′,q′′}

dans son spectre discret.

Si l’un des nombres p′, q′, p′′ or q′′ est nul, alors la restriction ϖ^{p,q}|_{G′} se décompose discrètement en représentations unitaires irréducibles de G′ comme il résulte d’un critère général se trouvant dans [4], Part II. Par suite on a

Théorème B. Supposons que p + q ∈ 2ℕ et que q′′ ≥ 2. Alors on a

ϖ^{p,q}|_{O(p,q′)×O(q′′)} ≃ ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix}π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ π_{−,l+ \frac{q′′}{2} −1}^{0,q′′}.

Supposons p + q ∈ 2ℕ, p, q ≥ 2 et p + q ≥ 8. Dans ces conditions, l’annulateur infinitésimal de la répresentation ϖ^{p,q} est l’idéal de Joseph ([1], [5]), si bien que la représentation est associée à l’orbite nilpotente minimale O_{min} de G dans \sqrt{−1}𝔤_{0}^{∗}, laquelle est de dimension 2(p + q − 3).

Définition. Soit M_{ν} une famille, paramétrée par ν ∈ ℝ_{+}, de sous-ensembles d’un espace topologique. Si M est une partie de cet espace topologique, on désigne par \overline{M} son adhérence. Alors, on définit l’ensemble limite de la famille M_{ν}, ν ∈ ℝ_{+}, en posant

lim_{ν↓0} M_{ν} := ⋂_{ε>0} \overline{⋃_{ε>ν>0} M_{ν}}.

Le résultat suivant donne la description de l’ensemble limite des orbites elliptiques O_{ν} = Ad^{∗}(G)νf_{1}, lorsqu’on s’approche du cône nilpotent:

Théorème C. Si p ≥ 2 et q ≥ 2, on a la décomposition en Ad^{∗}(G)-orbites:

lim_{ν↓0} O_{ν} = O_{0} ∪ O_{min} ∪ {0},

ou O_{0} est une orbite nilpotente de dimension 2(p + q − 2) de G dans \sqrt{−1}𝔤_{0}^{∗}.

§1. Introduction

\underline{1.1.} Let G be a reductive Lie group, and G′ a reductive subgroup of G. We denote by Ĝ the unitary dual of G, the equivalence classes of irreducible unitary representations of G. Likewise Ĝ′ for G′. If π ∈ Ĝ, then the restriction π|_{G′} is not necessarily irreducible. By a branching law, we mean an explicit irreducible decomposition formula:

(1.1.1)  π|_{G′} ≃ ∫ ⊕ ; Ĝ′ m_{π}(τ)τdμ(τ)  (direct integral),

where m_{π}(τ) ∈ ℕ ∪ {∞} and dμ is a Borel measure on Ĝ′.

\underline{1.2.} We denote by 𝔤_{0} the Lie algebra of G. The orbit method due to Kirillov-Kostant in the unitary representation theory of Lie groups indicates that the coadjoint representation Ad^{∗}: G → GL(𝔤_{0}^{∗}) often has a surprising intimate relation with the unitary dual Ĝ. It works perfectly for simply connected nilpotent Lie groups. For real reductive Lie groups G, known examples suggest that the set of coadjoint orbits \sqrt{−1}𝔤_{0}^{∗}/G (with certain integral conditions) still gives a fairly good approximation of the unitary dual Ĝ.

\underline{1.3.} There have been a number of attempts to construct representations attached to nilpotent orbits. Among all, the Segal-Shale-Weil representation (or the oscillator representation) of S̃p(n, ℝ), denoted by ϖ′, has been best studied, which is supposed to be attached to the minimal nilpotent orbit of 𝔰𝔭(n, ℝ). The restriction of ϖ′ to a reductive dual pair G′ = G′_{1}G′_{2} gives Howe’s correspondence ([2]).

The group S̃p(n, ℝ) is a split group of type C_{n}, and analogous to ϖ′, Kostant constructed a minimal representation of SO(n, n), a split group of type D_{n}, and then Binegar-Zierau generalized it for SO(p, q) with p + q ∈ 2ℕ. This representation (precisely, of O(p, q), see §2) will be denoted by ϖ^{p,q}.

\underline{1.4.} Let G′ := G′_{1}G′_{2} = O(p′, q′) × O(p′′, q′′), (p′ + p′′ = p, q′ + q′′ = q), be a subgroup of G = O(p, q). Our object of study is the branching law ϖ^{p,q}_{|G′}. We note that G′_{1} and G′_{2} form a mutually centralizing pair of subgroups in G.

A detailed proof will appear elsewhere. The first author expresses his sincere gratitude to Odense University for the warm hospitality.

§2. Construction of representations of O(p, q).

\underline{2.1.} In this section, we construct irreducible representations of the indefinite orthogonal group O(p, q) (p, q ≥ 2), denoted by ϖ^{p,q} and π_{+,λ}^{p,q}, π_{−,λ}^{p,q}, which are supposed to be attached to the minimal nilpotent orbit, and minimal elliptic orbits, respectively.

\underline{2.2.} Let ℝ^{p,q} be a manifold ℝ^{p+q} equipped with pseudo-Riemannian metric

ds^{2} = dx_{1}^{2} + ⋯ + dx_{p}^{2} − dy_{1}^{2} − ⋯ − dy_{q}^{2}.

We define submanifolds of ℝ^{p,q} by

X(p, q) := {(x, y) ∈ ℝ^{p,q} : |x|^{2} − |y|^{2} = 1},
Ξ  := {(x, y) ∈ ℝ^{p,q} : |x| = |y|} \ {0},
M  := {(x, y) ∈ ℝ^{p,q} : |x| = |y| = 1} ≃ S^{p−1} × S^{q−1}.

The indefinite orthogonal group G := O(p, q) acts naturally on ℝ^{p,q}, X(p, q), and Ξ. The action is denoted by z ↦ g · z (g ∈ G, z ∈ ℝ^{p,q}). G also acts on M. In fact, the dilation action of ℝ_{+}^{×} := {r ∈ ℝ : r > 0} on Ξ commutes with that of G. The induced action of G on M ≃ Ξ/ℝ_{+}^{×} will be denoted by x ↦ L_{h}x (x ∈ M, h ∈ G).

\underline{2.3.} We start with the conformal construction of the “minimal unipotent” representation ϖ^{p,q} for p + q ∈ 2ℕ by applying ideas from conformal geometry to M = S^{p−1} × S^{q−1} and G = O(p, q).

We equip M with the pseudo-Riemannian metric induced from ℝ^{p,q}. Then G acts conformally on M, and let Δ̃_{M} = Δ_{S^{p−1}} − Δ_{S^{q−1}} − (\frac{p−2}{2})^{2} + (\frac{q−2}{2})^{2}, be the so-called Yamabe operator. We set ν: Ξ → ℝ, (x, y) ↦ |x|, and define

(ϖ^{p,q}(h^{−1})f)(z) := ν(h · z)^{− \frac{p+q−4}{2}} f(L_{h}z),

for h ∈ G = O(p, q), z ∈ M = S^{p−1} × S^{q−1}, f ∈ V := {f ∈ C^{∞}(S^{p−1} × S^{q−1}) : Δ̃_{M}f = 0}. Then (ϖ^{p,q}, V) is a representation of G = O(p, q) by conformal geometry. Moreover, (ϖ^{p,q}, V) is a non-zero irreducible representation of G if p + q ∈ 2ℕ. By comparing the construction of [1], the underlying (𝔤, K)-module V_{K} is unitarizable with the inner product

(f_{1}, f_{2}) := ∫_{M}(Df_{1})\overline{f_{2}} dω, f_{1}, f_{2} ∈ V_{K}

where D := \sqrt{−Δ_{S^{p−1}} + \frac{(p−2)^{2}}{4}} , and dω is the standard measure on M. We use the same notation ϖ^{p,q} to denote the irreducible unitary representation of G.

\underline{2.4.} Next, we consider minimal elliptic coadjoint orbits of G = O(p, q). According to the notation in §1.3, we write π_{+,λ}^{p,q} for π_{λf_{1}}, and π_{−,λ}^{p,q} for π_{λf_{2}} attached to elliptic orbits Ad^{∗}(G)λf_{i} (i = 1, 2) for λ ∈ ℤ + \frac{1}{2}(p + q) with λ ≥ 0, where f_{1}, f_{2} ∈ \sqrt{−1}𝔤_{0}^{∗} is defined by

⟨f_{1}, X⟩ := −\sqrt{−1}X_{12}, ⟨f_{2}, X⟩ := −\sqrt{−1}X_{p+1,p+2}, for X = (X_{ij})_{1≤i,j≤p+q} ∈ 𝔤_{0},

where 𝔤_{0} = 𝔬(p, q). It is convenient to put π_{+,− \frac{1}{2}}^{1,0} = 1, the trivial representation of

O(1, 0) = O(1); and π_{+, \frac{1}{2}}^{1,0} = sgn, the signature representation of O(1). Then, \{π_{+,λ}^{p,q} :

λ ∈ A(p, q)\} (⊂ { \atop O(p, q)}) is the totality of discrete series representations for X(p, q), where we put

A(p, q) := \{ \begin{matrix}\{λ ∈ ℤ + \frac{p+q}{2} : λ > 0\} & (p > 1, q ≠ 0), \\ \{λ ∈ ℤ + \frac{p+q}{2} : λ ≥ \frac{p}{2} − 1\} & (p > 1, q = 0), \\ ∅ & (p = 1, q ≠ 0) or (p = 0), \\ \{−\frac{1}{2}, \frac{1}{2}\} & (p = 1, q = 0). \end{matrix}

§3. Branching laws

In this section, we consider the discrete spectrum in the branching law of the minimal unipotent representation ϖ^{p,q} ∈ Ĝ with respect to the reductive dual pair (G, G′) = (O(p, q), O(p′, q′) × O(p′′, q′′)), where p′ + p′′ = p (≥ 2), q′ + q′′ = q (≥ 2), p + q ∈ 2ℕ, and (p, q) ≠ (2, 2). We set A′(p, q) := A(p, q) ∩ {λ ∈ ℝ : λ > 1}.

Theorem 3.1. The restriction ϖ^{p,q}|_{G′} contains

\begin{matrix}∑ & ⊕ \\ λ∈A′(p′,q′)∩A′(q′′,p′′) \end{matrix}π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} ⊕ \begin{matrix}∑ & ⊕ \\ λ∈A′(q′,p′)∩A′(p′′,q′′) \end{matrix}π_{−,λ}^{p′,q′} ⊠ π_{+,λ}^{p′′,q′′}

as a discrete spectrum.

The idea of the proof is based on the conformal embedding

X(p′, q′) × X(q′′, p′′) ↪ M = S^{p−1} × S^{q−1}

together with the conformal construction of ϖ^{p,q} in §2.

\underline{3.2.} If one of p′, q′, p′′ or q′′ is zero, then the restriction ϖ^{p,q}|_{G′} is decomposed discretely into irreducible representations of G′ = O(p′, q′) × O(p′′, q′′) by a general criterion in [4], Part II. Then, we can determine the branching laws of ϖ^{p,q}|_{G′} as follows:

Theorem 3.2. Let p + q ∈ 2ℕ. If q′′ ≥ 2 and q′ + q′′ = q, then

ϖ^{p,q}|_{O(p,q′)×O(q′′)} ≃ ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix}π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ π_{−,l+ \frac{q′′}{2} −1}^{0,q′′}.

Remark 3.3. 1) π_{−,l+ \frac{q′′}{2} −1}^{0,q′′} is isomorphic to the irreducible representation of O(q′′) on the

spherical harmonics of degree l, that is, the representation space is H^{l} = {f ∈ C^{∞}(S^{q′′−1}) : Δ_{S^{q′′−1}}f = −l(l + q′′ − 2)f}.

2) The formula in Theorem 3.2 is nothing but a K-type formula when q′ = 0.

§4. Orbit methods

\underline{4.1.} Suppose p + q ∈ 2ℕ, p, q ≥ 2. The annihilator of the representation ϖ^{p,q} is the Joseph ideal for p + q ≥ 8 ([1], [5]). In this sense, ϖ^{p,q} is supposed to be attached to the unique minimal nilpotent coadjoint orbit, denoted by O_{min} (⊂ \sqrt{−1}𝔤_{0}^{∗}), whose dimension is 2(p + q − 3).

Definition 4.2. Let M_{ν} be a family of subsets, parametrized by ν ∈ ℝ_{+}, of a topological space. We denote by \overline{M} the closure of a subset M. Then, the “limit set” is defined by

lim_{ν↓0} M_{ν} := ⋂_{ε>0} \overline{⋃_{ε>ν>0} M_{ν}}.

\underline{4.3.} Here is the limit behavior as the elliptic orbits tend to the nilpotent orbits, setting O_{ν} = Ad^{∗}(G)νf_{1}:

Theorem 4.3. If p ≥ 2 and q ≥ 2, then we have the Ad^{∗}(G)-orbit decomposition:

lim_{ν↓0} O_{ν} = O_{0} ∪ O_{min} ∪ {0},

where O_{0} is a nilpotent orbit of dimension 2(p + q − 2).

\underline{4.4.} The construction of the representations π_{+,ν}^{p,q}, attached to elliptic orbits Ad^{∗}(G)νf_{1} (see §1.3, §2.4) is built on the polarization (or, equivalently, the θ-stable parabolic subalgebra if we employ Zuckerman-Vogan’s construction) depending on the signature of ν.

Let us consider the Dolbeault cohomology group with ν = −1 where the polarization is kept the same as that of a positive ν. It turns out that the Dolbeault cohomology group in the same degree (i.e. p − 2) is still non-zero in this special setting. We write lim_{ν→−1} π_{+,ν}^{p,q} for the underlying (𝔤, K)-module. Then, the following theorem may be regarded as a quantization of Theorem 4.3.

Theorem 4.4. Suppose p + q ∈ 2ℕ, and p, q ≥ 2. Then we have

lim_{ν→−1} π_{+,ν}^{p,q} = π_{+,1}^{p,q} + ϖ^{p,q}

in the Grothendieck group of (𝔤, K)-modules.

\underline{4.5.} Let G′ = O(p, q′) × O(q′′) (q′ + q′′ = q, p + q ∈ 2ℕ).

Theorem 4.5. Let pr_{𝔤→𝔤′}: 𝔤^{∗} → 𝔤′^{∗} be the projection dual to 𝔤′ ↪ 𝔤. Then we have

pr_{𝔤→𝔤′}(O_{min}) = Ad^{∗}(G′){λf_{1} + λf_{2} : λ > 0}.

The branching law in Theorem 3.2 may be regarded as a quantization of Theorem 4.5.

References

[1] B. Binegar and R. Zierau, Unitarization of a singular representation of SO(p, q), Comm. Math. Phys. 138 (1991), 245–258.

[2] R. Howe, Transcending classical invariant theory, Jour. A. M. S. 2 (1989), 535–552.

[3] T. Kobayashi, The restriction of A_{𝔮}(λ) to reductive subgroups, Proc. Acad. Japan 69 (1993), 262–267, Part II, 71, (1995), 24–26.

[4] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205; Part II (preprint); Part III (to appear in Invent. Math.).

[5] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4), Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory (Connes et al, eds.), vol. 92, Birkhäuser, Boston, 1990, pp. 85–124.

[6] W. Schmid, On a conjecture of Langlands, Ann. Math. 93 (1971), 1-42.

[7] D. Vogan Jr., Unitarizability of certain series of representations, Ann. of Math. (1984), 141–187.

[8] B. Ørsted, A note on the conformal quasi-invariance of the Laplacian on a pseudo-Riemannian manifold, Lett. Math. Phys. 1 (1977), 183.

[9] , Conformally invariant differential equations and projective geometry, J. Funct. Anal. 44 (1981), 1–23.

[10] H. Wong, Dolbeault cohomologies and Zuckerman modules associated with finite rank representations, ph. D. dissertation, Harvard University (1992).

Graduate School of Mathematical Sciences, University of Tokyo, Meguro, Komaba, 153, Tokyo, Japan

Department of Mathematics and Computer Science, Odense University, Campusvej 55, DK-5230, Odense M, Denmark

E-mail address: toshi@ms.u-tokyo.ac.jp; orsted@imada.ou.dk