ANALYSIS ON THE MINIMAL REPRESENTATION OF O(p, q)

– I. REALIZATION VIA CONFORMAL GEOMETRY

Toshiyuki KOBAYASHI and Bent ØRSTED

RIMS Kyoto and SDU-Odense University

Abstract. This is the first in a series of papers devoted to an analogue of the metaplectic representation, namely the minimal unitary representation of an indefinite orthogonal group; this representation corresponds to the minimal nilpotent coadjoint orbit in the philosophy of Kirillov-Kostant. We begin by applying methods from conformal geometry of pseudo-Riemannian manifolds to a general construction of an infinite-dimensional representation of the conformal group on the solution space of the Yamabe equation. By functoriality of the constructions, we obtain different models of the unitary representation, as well as giving new proofs of unitarity and irreducibility. The results in this paper play a basic role in the subsequent papers, where we give explicit branching formulae, and prove unitarization in the various models.

Contents

§1. Introduction §2. Conformal geometry §3. Minimal unipotent representations of O(p, q)

§1. Introduction

\underline{1.0.} This is the first in a series of papers devoted to a study of the so-called minimal representation of the semisimple Lie group G = O(p, q). We have taken the point of view that a rather complete treatment of this representation and its various realizations can be done in a self-contained way; also, such a study involves many different tools from other parts of mathematics, such as differential geometry (conformal geometry and pseudo-Riemannian geometry), analysis of solution spaces of ultrahyperbolic differential equations, Sobolev spaces, special functions such as hypergeometric functions of two variables, Bessel functions, analysis on semisimple symmetric spaces, and Dolbeault cohomology groups. Furthermore, the representation theory yields new results back to these areas, so we feel it is worthwhile to illustrate such an interaction in as elementary a way as possible. The sequel (Part II) to the present paper contains sections 4-9, and we shall also refer to these here. Part III is of more independent nature.

Working on a single unitary representation we essentially want to analyze it by understanding its restrictions to natural subgroups, and to calculate intertwining operators between the various models - all done very explicitly. We are in a sense

Typeset by AMS-TEX

studying the symmetries of the representation space by breaking the large symmetry present originally with the group G by passing to a subgroup. Geometrically the restriction is from the conformal group G to the subgroup of isometries H, where different geometries (all locally conformally equivalent) correspond to different choices of H. Changing H will give rise to radically different models of the representation, and at the same time allow calculating the spectrum of H.

Thus the overall aim is to elucidate as many aspects as possible of a distinguished unitary irreducible representation of O(p, q), including its explicit branching laws to natural subgroups and its explicit inner product on each geometric model. Our approach is also useful in understanding the relation between the representation and a certain coadjoint orbit, namely the minimal one, in the dual of the Lie algebra. In order to give a good view of the perspective in our papers, we are giving below a rather careful introduction to all these aspects.

For a semisimple Lie group G a particularly interesting unitary irreducible representation, sometimes called the minimal representation, is the one corresponding via “geometric quantization” to the minimal nilpotent coadjoint orbit. It is still a little mysterious in the present status of the classification problem of the unitary dual of semisimple Lie groups. In recent years several authors have considered the minimal representation, and provided many new results, in particular, Kostant, Torasso, Brylinski, Li, Binegar, Zierau, and Sahi, mostly by algebraic methods. For the double cover of the symplectic group, this is the metaplectic representation, introduced many years ago by Segal, Shale, and Weil. The explicit treatment of the metaplectic representation requires various methods from analysis and geometry, in addition to the algebraic methods; and it is our aim in a series of papers to present for the case of G = O(p, q) the aspects pertaining to branching laws. From an algebraic view point of representation theory, our representations ϖ^{p,q} are:

i) minimal representations if p + q ≥ 8 (i.e. the annihilator is the Joseph ideal). ii) not spherical if p ≠ q (i.e. no non-zero K-fixed vector). iii) not highest weight modules of SO_{0}(p, q) if p, q ≥ 3. Apparently our case provides examples of new phenomena in representation the-

ory, and we think that several aspects of our study can be applied to other cases as well. The metaplectic representation has had many applications in representation theory and in number theory. A particularly useful concept has been Howe’s idea of dual pairs, where one considers a mutually centralizing pair of subgroups in the metaplectic group and the corresponding restriction of the metaplectic representation. In Part II of our papers, we shall initiate a similar study of explicit branching laws for other groups and representations analogous to the classical case of Howe. Several such new examples of dual pairs have been studied in recent years, mainly by algebraic techniques. Our case of the real orthogonal group presents a combination of abstract representation theory and concrete analysis using methods from conformal differential geometry. Thus we can relate the branching law to a study of the Yamabe operator and its spectrum in locally conformally equivalent manifolds; furthermore, we can prove the existence of and construct explicitly an infinite discrete spectrum in the case where both factors in the dual pair are non-compact.

The methods we use are further motivated by the theory of spherical harmonics, extending analysis on the sphere to analysis on hyperboloids, and at the same time using elliptic methods in the sense of analysis on complex quadrics and the theory of Zuckerman-Vogan’s derived functor modules and their Dolbeault cohomological realizations. Also important are general results on discrete decomposability of representations and explicit knowledge of branching laws.

It is noteworthy, that as we have indicated, this representation and its theory of generalized Howe correspondence, illustrates several interesting aspects of modern representation theory. Thus we have tried to be rather complete in our treatment of the various models of the representations occurring in the branching law. See for example Fact 5.4, where we give three realizations: derived functor modules or Dolbeault cohomology groups, eigenspaces on semisimple symmetric spaces, and quotients of generalized principal series, of the representations attached to minimal elliptic orbits.

Most of the results of Part I and Part II were announced in [22], and the branching law in the discretely decomposable case (Theorem 7.1) was obtained in 1991, from which our study grew out. We have here given the proofs of the branching laws for the minimal unipotent representation and postpone the detailed treatment of the corresponding classical orbit picture as announced in [22] to another paper. Also, the branching laws for the representations associated to minimal elliptic orbits will appear in another paper by one of the authors.

It is possible that a part of our results could be obtained by using sophisticated results from the theory of dual pairs in the symplectic group, for example the see-saw rule (for which one may let our representation correspond to the trivial representation of one SL(2, ℝ) member of the dual pair ([11], [12]). We emphasize however, that our approach is quite explicit and has the following advantages:

(a) It is not only an abstract representation theory but also attempts new interaction of the minimal representation with analysis on manifolds. For example, in Part II we use in an elementary way conformal differential geometry and the functorial properties of the Yamabe operator to construct the minimal representation and the branching law in a way which seems promising for other cases as well; each irreducible constituent is explicitly constructed by using explicit intertwining operators via local conformal diffeomorphisms between spheres and hyperboloids.

(b) For the explicit intertwining operators we obtain Parseval-Plancherel type theorems, i.e. explicit L^{2} versions of the branching law and the generalized Howe correspondence. This also gives a good perspective on the continuous spectrum, in particular yielding a natural conjecture for the complete Plancherel formula.

A special case of our branching law illustrates the physical situation of the conformal group of space-time O(2, q); here the minimal representation may be interpreted either as the mass-zero spin-zero wave equation, or as the bound states of the Hydrogen atom (in q − 1 space dimensions). Studying the branching law means breaking the symmetry by for example restricting to the isometry group of De Sitter space O(2, q − 1) or anti De Sitter space O(1, q). In this way the original system (particle) breaks up into constituents with less symmetry.

In Part III, we shall realize the same representation on a space of solutions of the ultrahyperbolic equation □_{ℝ^{p−1,q−1}}f = 0 on ℝ^{p−1,q−1}, and give an intrinsic inner product as an integration over a non-characteristic hypersurface.

Completing our discussion of different models of the minimal representation, we find yet another explicit intertwining operator, this time to an L^{2} - space of functions on a hypersurface (a cone) in the nilradical of a maximal parabolic P in G. We find the K-finite functions in the case of p + q even in terms of modified Bessel functions. We remark that Vogan pointed out a long time ago that there is no minimal representation of O(p, q) if p + q > 8 is odd [34]. On the other hand, we have found a new interesting phenomenon that in the case p + q is odd there still exists a geometric model of a “minimal representation” of 𝔬(p, q) with a natural inner product (see Part III). Of course, such a representation does not have non-zero K-finite vectors for p + q odd, but have K′-finite vectors for smaller K′. What we construct in this case is an element of the category of (𝔤, P) modules in the sense that it globalizes to P (but not K); we feel this concept perhaps plays a role for other cases of the orbit method as well.

In summary, we give a geometric and intrinsic model of the minimal representation ϖ^{p,q} (not coming from the construction of ϖ^{p,q} by the θ-correspondence) on S^{p−1} × S^{q−1} and on various pseudo-Riemannian manifolds which are conformally equivalent, using the functorial properties of the Yamabe operator, a key element in conformal differential geometry. The branching law for ϖ^{p,q} gives at the same time new perspectives in conformal geometry, and we give an independent proof of the unitarity of a subspace of the kernel of the Yamabe operator. The main interest in this special case of a small unitary representation is not only to obtain the formulae, but also to investigate the geometric and analytic methods, which provide new ideas in representation theory.

Leaving the general remarks, let us now for the rest of this introduction be a little more specific about the contents of the present paper.

\underline{1.1.} Let G be a reductive Lie group, and G′ a reductive subgroup of G. We denote by Ĝ the unitary dual of G, the equivalence classes of irreducible unitary representations of G. Likewise Ĝ′ for G′. If π ∈ Ĝ, then the restriction π|_{G′} is not necessarily irreducible. By a branching law, we mean an explicit irreducible decomposition formula:

(1.1.1)  π|_{G′} ≃ ∫ ⊕ ; Ĝ′ m_{π}(τ)τdμ(τ)  (direct integral),

where m_{π}(τ) ∈ ℕ ∪ {∞} and dμ is a Borel measure on Ĝ′.

\underline{1.2.} We denote by 𝔤_{0} the Lie algebra of G. The orbit method due to Kirillov-Kostant in the unitary representation theory of Lie groups indicates that the coadjoint representation Ad^{∗}: G → GL(𝔤_{0}^{∗}) often has a surprising intimate relation with the unitary dual Ĝ. It works perfectly for simply connected nilpotent Lie groups. For real reductive Lie groups G, known examples suggest that the set of coadjoint orbits \sqrt{−1}𝔤_{0}^{∗}/G (with certain integral conditions) still gives a fairly good approximation of the unitary dual Ĝ.

\underline{1.3.} Here is a rough sketch of a unitary representation π_{λ} of G, attached to an elliptic element λ ∈ \sqrt{−1}𝔤_{0}^{∗}: The elliptic coadjoint orbit O_{λ} = Ad^{∗}(G)λ carries a G-invariant complex structure, and one can define a G-equivariant holomorphic line bundle L̃_{λ} := L_{λ} ⊗ (∧^{top}T^{∗}O_{λ})^{\frac{1}{2}} over O_{λ}, if λ satisfies some integral condition. Then, we have a Fréchet representation of G on the Dolbeault cohomology group HS ; ¯∂(O_{λ}, L̃_{λ}), where S := dim_{ℂ} Ad^{∗}(K)λ (see [36] for details), and of which a unique dense subspace we can define a unitary representation π_{λ} of G ([32]) if λ satisfies certain positivity. The unitary representation π_{λ} is irreducible and non-zero if λ is sufficiently regular. The underlying (𝔤, K)-module is so called “A_{𝔮}(λ)” in the sense of Zuckerman-Vogan after certain ρ-shift.

In general, the decomposition (1.1.1) contains both discrete and continuous spectrum. The condition for the discrete decomposition (without continuous spectrum) has been studied in [15], [16], [17], and [20], especially for π_{λ} attached to elliptic orbits O_{λ}. It is likely that if π ∈ Ĝ is “attached to” a nilpotent orbit, which is contained in the limit set of O_{λ}, then the discrete decomposability of π|_{G′} should be inherited from that of the elliptic case π_{λ}|_{G′}. We shall see in Theorem 4.2 that this is the case in our situation.

\underline{1.4.} There have been a number of attempts to construct representations attached to nilpotent orbits. Among all, the Segal-Shale-Weil representation (or the oscillator representation) of S̃p(n, ℝ), for which we write ϖ̃, has been best studied, which is supposed to be attached to the minimal nilpotent orbit of 𝔰𝔭(n, ℝ). The restriction of ϖ̃ to a reductive dual pair G′ = G′_{1}G′_{2} gives Howe’s correspondence ([10]).

The group S̃p(n, ℝ) is a split group of type C_{n}, and analogous to ϖ̃, Kostant constructed a minimal representation of SO(n, n), a split group of type D_{n}. Then Binegar-Zierau generalized it for SO(p, q) with p + q ∈ 2ℕ. This representation (precisely, of O(p, q), see Section 3) will be denoted by ϖ^{p,q}.

\underline{1.5.} Let G′ := G′_{1}G′_{2} = O(p′, q′) × O(p′′, q′′), (p′ + p′′ = p, q′ + q′′ = q), be a subgroup of G = O(p, q). Our object of study in Part II will be the branching law ϖ^{p,q}_{|G′}. We note that G′_{1} and G′_{2} form a mutually centralizing pair of subgroups in G.

It is interesting to compare the feature of the following two cases:

(i) the restriction ϖ̃|_{G′_{1}G′_{2}} (the Segal-Shale-Weil representation for type C_{n}), (ii) the restriction ϖ^{p,q}|_{G′_{1}G′_{2}} (the Kostant-Binegar-Zierau representation for type D_{n}).

The reductive dual pair (G, G′) = (G, G′_{1}G′_{2}) is of the ⊗-type in (i), that is, induced from GL(V) × GL(W) → GL(V ⊗ W); is of the ⊕-type in (ii), that is, induced from GL(V) × GL(W) → GL(V ⊕ W). On the other hand, both of the restrictions in (i) and (ii) are discretely decomposable if one factor G′_{2} is compact. On the other hand, ϖ̃ is (essentially) a highest weight module in (i), while ϖ^{p,q} is not if p, q > 2 in (ii).

\underline{1.6.} Let p + q ∈ 2ℕ, p, q ≥ 2, and (p, q) ≠ (2, 2). In this section we state the main results of the present paper and the sequels (mainly Part II; an introduction of Part III will be given separately in [24]). The first Theorem A below (Theorem 2.5) says that there is a general way of constructing representations of a conformal group by twisted pull-backs (see Section 2 for notation). It is the main tool to give different models of our representation.

Theorem A. Suppose that a group G acts conformally on a pseudo-Riemannian manifold M of dimension n.

1) Then, the Yamabe operator (see (2.2.1) for the definition)

Δ̃_{M}: C^{∞}(M) → C^{∞}(M)

is an intertwining operator from ϖ_{\frac{n−2}{2}} to ϖ_{\frac{n+2}{2}} (see (2.5.1) for the definition of ϖ_{λ}).

2) The kernel Ker Δ̃_{M} is a subrepresentation of G through ϖ_{\frac{n−2}{2}} .

Theorem B. 1) The minimal representation ϖ^{p,q} of O(p, q) is realized as the kernel of the Yamabe operator on S^{p−1} × S^{q−1}.

2) ϖ^{p,q} is also realized as a subspace (roughly, half) of the kernel of the Yamabe operator on the hyperboloid {(x, y) ∈ ℝ^{p,q} : |x|^{2} − |y|^{2} = 1}.

3) ϖ^{p,q} is also realized in a space of solutions to the Yamabe equation on ℝ^{p−1,q−1} which is a standard ultrahyperbolic constant coefficient differential equation.

4) ϖ^{p,q} is also realized as the unique non-trivial subspace of the Dolbeault cohomol-

ogy group Hp−2 ; ¯∂(G/L, L_{\frac{p+q−4}{2}} )

In Theorem B (1) is contained in Part I, Theorem 3.6.1; (2) in Part II, Corollary 7.2.1; and (3) in Part III, Theorem 4.7. In each of these models, an explicit model is given explicitly. In the models (2) and (3), the situation is subtle because the “action” of O(p, q) is no more smooth but only meromorphic. Then Theorem A does not hold in its original form, and we need to carry out a careful analysis for it (see Part II and Part III). The proof of the statement (4) will appear in another paper. Here G/L is an elliptic coadjoint orbit as in §1.3, and L = SO(2)×O(p−2, q).

The branching laws in Theorem C and Theorem D are the main themes in Part II; for notation see section 7 and section 9.

Theorem C. If q′′ ≥ 1 and q′ + q′′ = q, then the twisted pull-back Φ̃_{1}^{∗} of the local conformal map Φ_{1} between spheres and hyperboloids gives an explicit irreducible decomposition of the unitary representation ϖ^{p,q} when restricted to O(p, q′)×O(q′′):

ϖ^{p,q}|_{O(p,q′)×O(q′′)} ≃ ∞ ; \begin{matrix}∑ & ⊕ \\ l=0 \end{matrix}π_{+,l+ \frac{q′′}{2} −1}^{p,q′} ⊠ H^{l}(ℝ^{q′′}).

In addition, we give in §8, Theorem 8.5 the Parseval-Plancherel theorem for the situation in Theorem C on the “hyperbolic space model”. This may be also regarded as the unitarization of the minimal representation ϖ^{p,q}.

The twisted pull-back for a locally conformal diffeomorphism is defined for an arbitrary pseudo-Riemannian manifold (see Definition 2.3).

Theorem D. The twisted pull-back of the locally conformal diffeomorphism also constructs

\begin{matrix}∑ & ⊕ \\ λ∈A′(p′,q′)∩A′(q′′,p′′) \end{matrix}π_{+,λ}^{p′,q′} ⊠ π_{−,λ}^{p′′,q′′} ⊕ \begin{matrix}∑ & ⊕ \\ λ∈A′(q′,p′)∩A′(p′′,q′′) \end{matrix}π_{−,λ}^{p′,q′} ⊠ π_{+,λ}^{p′′,q′′}

as a discrete spectra in the branching law.

\underline{1.7.} The papers (Part I and Part II) are organized as follows: Section 2 provides a conformal construction of a representation on the kernel of a shifted Laplace-Beltrami operator. In section 3, we construct an irreducible unitary representations, ϖ^{p,q} of O(p, q) (p + q ∈ 2ℕ, p, q ≥ 2) “attached to” the minimal nilpotent orbit applying Theorem 2.5. This representation coincides with the minimal representation studied by Kostant, Binegar-Zierau ([2], [25]). In section 3 we give a new intrinsic characterization of the Hilbert space for the minimal representation in this model, namely as a certain Sobolev space of solutions, see Theorem 3.9 and Lemma 3.10. Such Sobolev estimate will be used in the construction of discrete spectrum of the branching law in section 9. Section 4 contains some general results on discrete decomposable restrictions ([16], [17]), specialized in detail to the present case. Theorem 4.2 characterizes which dual pairs in our situation provide discrete decomposable branching laws of the restriction of the minimal representation ϖ^{p,q}. In section 5, we introduce unitary representations, π_{±,λ}^{p,q} of O(p, q) “attached to” minimal elliptic coadjoint orbits. In sections 7 and 9, we give a discrete spectrum

of the branching law ϖ^{p,q}|_{G′} in terms of π_{±,λ}^{p′,q′} ∈ { \atop O(p′, q′)} and π_{±,λ}^{p′′,q′′} ∈ { \atop O(p′′, q′′)}.

In particular, if one factor G′_{2} = O(p′′, q′′) is compact (i.e. p′′ = 0 or q′′ = 0), the branching law is completely determined together with a Parseval-Plancherel theorem in section 8.

Notation: ℕ = {0, 1, 2, … }.

The first author expresses his sincere gratitude to SDU - Odense University for the warm hospitality.

§2. Conformal geometry

\underline{2.1.} The aim of this section is to associate a distinguished representation ϖ_{M} of the conformal group Conf(M) to a general pseudo-Riemannian manifold M (see Theorem 2.5).

\underline{2.2.} Let M be an n dimensional manifold with pseudo-Riemannian metric g_{M} (n ≥ 2). Let ∇ be the Levi-Civita connection for the pseudo-Riemannian metric g_{M}. The curvature tensor field R is defined by

R(X, Y)Z := ∇_{X}∇_{Y} Z − ∇_{Y} ∇_{X}Z − ∇_{[X,Y]}Z, X, Y, Z ∈ 𝔛(M).

We take an orthonormal basis {X_{1}, ⋯, X_{n}} of T_{x}M for a fixed x ∈ M. Then the scalar curvature K_{M} is defined by

K_{M}(x) := ∑_{i=1}^{n} ∑_{j=1}^{n} g_{M} (R(X_{i}, X_{j})X_{i}, X_{j}) .

The right side is independent of the choice of the basis {X_{i}} and so K_{M} is a welldefined function on M. We denote by Δ_{M} the Laplace-Beltrami operator on M. The Yamabe operator is defined to be

(2.2.1)  Δ̃_{M} := Δ_{M} − \frac{n − 2}{4(n − 1)}K_{M}.

See for example [26] for a good discussion of the geometric meaning and applications of this operator. Our choice of the signature of K_{M} and Δ_{M} is illustrated as follows:

Example 2.2. We equip ℝ^{n} and S^{n} with standard Riemannian metric. Then

For ℝ^{n};  K_{ℝ^{n}} ≡ 0,  Δ̃_{ℝ^{n}} = Δ_{ℝ^{n}} = ∑_{i=1}^{n} \frac{∂^{2}}{∂x_{i}^{2}} .

For S^{n};  K_{S^{n}} ≡ (n − 1)n,  Δ̃_{S^{n}} = Δ_{S^{n}} − \frac{1}{4}n(n − 2).

\underline{2.3.} Suppose (M, g_{M}) and (N, g_{N}) are pseudo-Riemannian manifolds of dimension n. A local diffeomorphism Φ: M → N is called a conformal map if there exists a positive valued function Ω on M such that

Φ^{∗}g_{N} = Ω^{2}g_{M}.

Φ is isometry if and only if Ω ≡ 1 by definition.

We denote the group of conformal transformation (respectively, isometry) of a pseudo-Riemannian manifold (M, g_{M}) by

Conf(M) := {Φ ∈ Diffeo(M) : Φ: M → M is conformal},
Isom(M) := {Φ ∈ Diffeo(M) : Φ: M → M is isometry}.

Clearly, Isom(M) ⊂ Conf(M).

If Φ is conformal, then we have (e.g. [37]; [9], Chapter II, Excer. A.5)

(2.3.1)  Ω^{\frac{n+2}{2}} (Φ^{∗}Δ̃_{N}f) = Δ̃_{M}(Ω^{\frac{n−2}{2}} Φ^{∗}f)

for any f ∈ C^{∞}(N). We define a twisted pull-back

(2.3.2)  Φ_{λ}^{∗}: C^{∞}(N) → C^{∞}(M), f ↦ Ω^{λ}(Φ^{∗}f),

for each fixed λ ∈ ℂ. Then the formula (2.3.1) is rewritten as

(2.3.1′)  Φ∗_{\frac{n+2}{2}} Δ̃_{N}f = Δ̃_{M}Φ∗_{\frac{n−2}{2}} f.

The case when λ = \frac{n−2}{2} is particularly important. Thus, we write the twisted pull-back for λ = \frac{n−2}{2} as follows:

Definition 2.3. Φ̃^{∗} = Φ∗_{\frac{n−2}{2}} : C^{∞}(N) → C^{∞}(M), f ↦ Ω^{\frac{n−2}{2}} (Φ^{∗}f).

Then the formula (2.3.1) implies that

(2.3.3)  Δ̃_{N}f = 0 on Φ(M) if and only if Δ̃_{M}(Φ̃^{∗}f) = 0 on M

because Ω is nowhere vanishing.

If n = 2, then Δ̃_{M} = Δ_{M}, Δ̃_{N} = Δ_{N}, and Φ̃^{∗} = Φ^{∗}. Hence, (2.3.3) implies a well-known fact in the two dimensional case that a conformal map Φ preserves harmonic functions, namely,

f is harmonic ⇔ Φ^{∗}f is harmonic.

\underline{2.4.} Let G be a Lie group acting conformally on a pseudo-Riemannian manifold (M, g_{M}). We write the action of h ∈ G on M as L_{h}: M → M, x ↦ L_{h}x. By the definition of conformal transformations, there exists a positive valued function Ω(h, x) (h ∈ G, x ∈ M) such that

L_{h}^{∗}g_{M} = Ω(h, ·)^{2}g_{M} (h ∈ G).

Then we have Lemma 2.4. For h_{1}, h_{2} ∈ G and x ∈ M, we have

Ω(h_{1}h_{2}, x) = Ω(h_{1}, L_{h_{2}}x) Ω(h_{2}, x).

Proof. It follow from L_{h_{1}h_{2}} = L_{h_{1}}L_{h_{2}} that

L_{h_{1}h_{2}}^{∗}g_{M} = L_{h_{2}}^{∗}L_{h_{1}}^{∗}g_{M}.

Therefore we have Ω(h_{1}h_{2}, ·)^{2} g_{M} = L_{h_{1}h_{2}}^{∗}g_{M} = L_{h_{2}}^{∗} (L_{h_{1}}^{∗}g_{M}) = L_{h_{2}}^{∗} (Ω(h_{1}, ·)^{2} g_{M}) = Ω(h_{1}, L_{h_{2}}·)^{2} Ω(h_{2}, ·)^{2} g_{M}. Since Ω is a positive valued function, we conclude that Ω(h_{1}h_{2}, x) = Ω(h_{1}, L_{h_{2}}x) Ω(h_{2}, x). □

\underline{2.5.} For each λ ∈ ℂ, we form a representation ϖ_{λ} ≡ ϖ_{M,λ} of the conformal group G on C^{∞}(M) as follows:

(2.5.1) (ϖ_{λ}(h^{−1})f) (x) := Ω(h, x)^{λ}f(L_{h}x), (h ∈ G, f ∈ C^{∞}(M), x ∈ M).

Then Lemma 2.4 assures that ϖ_{λ}(h_{1}) ϖ_{λ}(h_{2}) = ϖ_{λ}(h_{1}h_{2}), namely, ϖ_{λ} is a representation of G.

Denote by dx the volume element on M defined by the pseudo-Riemannian structure g_{M}. Then we have

L_{h}^{∗}(dx) = Ω(h, x)^{n}dx for h ∈ G.

Therefore, the map f ↦ f dx gives a G-intertwining operator from (ϖ_{n}, C^{∞}(M)) into the space of distributions D′(M) on M.

Here is a construction of a representation of the group of conformal diffeomorphisms of M.

Theorem 2.5. Suppose that a group G acts conformally on a pseudo-Riemannian manifold M of dimension n. Retain the notation before.

1) Then, the Yamabe operator

Δ̃_{M}: C^{∞}(M) → C^{∞}(M)

is an intertwining operator from ϖ_{\frac{n−2}{2}} to ϖ_{\frac{n+2}{2}} .

2) The kernel Ker Δ̃_{M} is a subrepresentation of G through ϖ_{\frac{n−2}{2}} .

Proof. (1) is a restatement of the formula (2.3.1). (2) follows immediately from (1). □

The representation of G on Ker Δ̃_{M} given in Theorem 2.5(2) will be denoted by ϖ ≡ ϖ_{M}.

\underline{2.6.} Here is a naturality of the representation of the conformal group Conf(M) in Theorem 2.5:

Proposition 2.6. Let M and N be pseudo-Riemannian manifolds of dimension n, and a local diffeomorphism Φ : M → N be a conformal map. Suppose that Lie groups G′ and G act conformally on M and N, respectively. The actions will be denoted by L_{M} and L_{N}, respectively. We assume that there is a homomorphism i: G′ → G such that

L_{N,i(h)} ∘ Φ = Φ ∘ L_{M,h} ( for any h ∈ G′).

We write conformal factors Ω_{M}, Ω_{N} and Ω as follows:

L_{M,h}^{∗}g_{M} = Ω_{M}(h, ·)^{2}g_{M}  (h ∈ G′),
L_{N,h}^{∗}g_{N} = Ω_{N}(h, ·)^{2}g_{N}  (h ∈ G),
Φ^{∗}g_{N} = Ω^{2}g_{M}.

1) For x ∈ M and h ∈ G′, we have

(2.6.1)  Ω(L_{M,h}x) Ω_{M}(h, x) = Ω(x) Ω_{N}(i(h), Φ(x)).

2) Let λ ∈ ℂ and Φ_{λ}^{∗} : C^{∞}(N) → C^{∞}(M) be the twisted pull-back defined in (2.3.2). Then Φ_{λ}^{∗} respects the G-representation (ϖ_{N,λ}, C^{∞}(N)) and the G′-representation (ϖ_{M,λ}, C^{∞}(M)) through i: G′ → G.

3) Φ̃^{∗} = Φ∗_{\frac{n−2}{2}} : C^{∞}(N) → C^{∞}(M) sends Ker Δ̃_{N} into Ker Δ̃_{M}. In particular, we have a commutative diagram:

(2.6.2)

Ker Δ̃_{N} \underrightarrow{Φ̃^{∗}} Ker Δ̃_{M}
ϖ_{N} (i(h))↓  ↓^{ϖ_{M} (h)}
Ker Δ̃_{N} \overrightarrow{Φ̃^{∗}} Ker Δ̃_{M}

for each h ∈ G′.

4) If Φ is a diffeomorphism onto N, then (Φ^{−1})_{λ}^{∗} is the inverse of Φ_{λ}^{∗} for each λ ∈ ℂ.

In particular, Φ̃^{∗} is a bijection between Ker Δ̃_{N} and Ker Δ̃_{M} with inverse { \atop (Φ^{−1})^{∗}}.

Proof. 1) Because L_{N,i(h)} ∘ Φ = Φ ∘ L_{M,h} for h ∈ G′, we have

(Φ^{∗}L_{N,i(h)}^{∗}g_{N})(x) = (L_{M,h}^{∗}Φ^{∗}g_{N})(x), for x ∈ M.

Hence,

Ω_{N}(i(h), Φ(x))^{2} Ω(x)^{2}g_{M}(x) = Ω(L_{M,h}x)^{2} Ω_{M}(h, x)^{2}g_{M}(x).

Because all conformal factors are positive-valued functions, we have proved (2.6.1).

2) We want to prove

(2.6.3)  (ϖ_{M,λ}(h^{−1})Φ_{λ}^{∗}f)(x) = (Φ_{λ}^{∗}ϖ_{N,λ}(i(h^{−1}))f)(x)

for any x ∈ M, h ∈ G′ and λ ∈ ℂ. In view of the definition, we have

the left side of (2.6.3) =(ϖ_{M,λ}(h^{−1})(Ω^{λ} Φ^{∗}f))(x)
=Ω_{M}(h, x)^{λ} Ω(L_{M,h}x)^{λ}(Φ^{∗}f)(L_{M,h}x)
=Ω(x)^{λ}Ω_{N}(i(h), Φ(x))^{λ}f(Φ ∘ L_{M,h}x).

Here the last equality follows from (2.6.1).

The right side of (2.6.3) =(Φ_{λ}^{∗}Ω_{N}(i(h), ·)^{λ} f(L_{N,h}·))(x)
=Ω(x)^{λ} Ω_{N}(i(h), Φ(x))^{λ} f(L_{N,i(h)} ∘ Φ(x)).

Therefore, we have (2.6.3), because L_{N,i(h)} ∘ Φ = Φ ∘ L_{M,h}.

3) If f ∈ C^{∞}(N) satisfies Δ̃_{N}f = 0, then Δ̃_{M}(Φ̃^{∗}f) = Ω^{\frac{n+2}{2}} (Φ^{∗}Δ̃_{N}f) = 0 by (2.3.1). Hence Φ̃^{∗}(Ker Δ̃_{N}) ⊂ Ker Δ̃_{M}. The commutativity of the diagram (2.6.2) follows from (2) and Theorem 2.5 (2), if we put λ = \frac{n−2}{2}.

4) Because (Φ^{−1})^{∗}g_{M} = (Ω ∘ Φ^{−1})^{−2}g_{N}, the twisted pull-back (Φ^{−1})_{λ}^{∗}F is given by the following formula from definition (2.3.2):

(Φ^{−1})_{λ}^{∗}: C^{∞}(M) → C^{∞}(N), F ↦ (Φ^{−1})_{λ}^{∗}F = (Ω ∘ Φ^{−1})^{−λ}(F ∘ Φ^{−1}).

Now the statement (4) follows immediately. □

§3. Minimal unipotent representations of O(p, q)

\underline{3.1.} In this section, we apply Theorem 2.5 to the specific setting where M = S^{p−1} × S^{q−1} is equipped with an indefinite Riemannian metric, and where the indefinite orthogonal group G = O(p, q) acts conformally on M. The resulting representation, denoted by ϖ^{p,q}, is non-zero, irreducible and unitary if p + q ∈ 2ℕ, p, q ≥ 2 and if (p, q) ≠ (2, 2). This representation coincides with the one constructed by Kostant, Binegar-Zierau ([2], [25]), which has the Gelfand-Kirillov dimension p + q − 3 (see Part II, Lemma 4.4). This representation is supposed to be attached to the unique minimal nilpotent coadjoint orbit, in the sense that its annihilator in the enveloping algebra U(𝔤) is the Joseph ideal if p + q ≥ 8, which is the unique completely prime primitive ideal of minimum nonzero Gelfand-Kirillov dimension.

Our approach based on conformal geometry gives a geometric realization of the minimal representation ϖ^{p,q} for O(p, q). One of the advantages using conformal geometry is the naturality of the construction (see Proposition 2.6), which allows us naturally different realizations of ϖ^{p,q}, not only on the K-picture (a compact picture in §3), but also on the N-picture (a flat picture) (see Part III), and on the hyperboloid picture (see Part II, §7, Corollary 7.2.1), together with the Yamabe operator in each realization. In later sections, we shall reduce the branching problems of ϖ^{p,q} to the analysis on different models on which the minimal representation ϖ^{p,q} is realized.

The case of SO(3, 4) was treated by [27]; his method was generalized in [30] to cover all simple groups with admissible minimal orbit, as well as the case of a local field of characteristic zero.

\underline{3.2.} We write a standard coordinate of ℝ^{p+q} as (x, y) = (x_{1}, …, x_{p}, y_{1}, …, y_{q}). Let ℝ^{p,q} be the pseudo-Riemannian manifold ℝ^{p+q} equipped with the pseudo-Riemannian metric:

(3.2.1)  ds^{2} = dx_{1}^{2} + ⋯ + dx_{p}^{2} − dy_{1}^{2} − ⋯ − dy_{q}^{2}.

We assume p, q ≥ 1 and define submanifolds of ℝ^{p,q} by

(3.2.2)  Ξ := {(x, y) ∈ ℝ^{p,q} : |x| = |y|} \ {0},
(3.2.3)  M := {(x, y) ∈ ℝ^{p,q} : |x| = |y| = 1} ≃ S^{p−1} × S^{q−1}.

We define a diagonal matrix by I_{p,q} := diag(1, …, 1, −1, …, −1). The indefinite orthogonal group

G = O(p, q) := {g ∈ GL(p + q, ℝ) : ^{t}gI_{p,q}g = I_{p,q}},

acts isometrically on ℝ^{p,q} by the natural representation, denoted by z ↦ g · z (g ∈ G, z ∈ ℝ^{p,q}). This action stabilizes the light cone Ξ. The multiplicative group ℝ_{+}^{×} := {r ∈ ℝ : r > 0} acts on Ξ as a dilation and the quotient space Ξ/ℝ_{+}^{×} is identified with M. Because the action of G commutes with that of ℝ_{+}^{×}, we can define the action of G on the quotient space Ξ/ℝ_{+}^{×}, and also on M through the diffeomorphism M ≃ Ξ/ℝ_{+}^{×}. This action will be denoted by

L_{h}: M → M, x ↦ L_{h}x (x ∈ M, h ∈ G).

In summary, we have a G-equivariant principal ℝ_{+}^{×}-bundle:

(3.2.4)  Φ: Ξ → M, (x, y) ↦ (\frac{x}{|x|}, \frac{y}{|y|}) = \frac{1}{ν(x, y)}(x, y),

where ν: Ξ → ℝ_{+} is defined by

(3.2.5)  ν(x, y) = |x| = |y|.

\underline{3.3.} Suppose N is a p+q−2-dimensional submanifold of Ξ. We say N is transversal to rays if Φ|_{N}: N → M is locally diffeomorphic. Then, the standard pseudo-Riemannian metric on ℝ^{p,q} induces a pseudo-Riemannian metric on N which has the codimension 2 in ℝ^{p,q}. The resulting pseudo-Riemannian metric is denoted by g_{N}, which has the signature (p − 1, q − 1). In particular, M ≃ S^{p−1} × S^{q−1} itself is transversal to rays, and the induced metric g_{S^{p−1}×S^{q−1}} = g_{S^{p−1}} ⊕ (−g_{S^{q−1}}), where g_{S^{n−1}} denotes the standard Riemannian metric on the unit sphere S^{n−1}.

Lemma 3.3. Assume that N is transversal to rays. Then Φ|_{N}: N → M is a conformal map. Precisely, we have

(3.3.1)  (Φ^{∗}g_{M})_{z} = ν(z)^{−2}(g_{N})_{z}, for z = (x, y) ∈ N.

Proof. Write the coordinates as (u_{1}, ⋯, u_{p}, v_{1}, ⋯, v_{q}) = Φ(x, y) ∈ S^{p−1} × S^{q−1}. Then

Φ^{∗} (du_{j}) = \frac{dx_{j}}{|x|} − \frac{x_{j}}{|x|^{3}} ∑_{i=1}^{p} x_{i}dx_{i}.

Therefore, we have

Φ^{∗} (∑_{j=1}^{p}(du_{j})^{2}) = |x|^{−2}∑_{j=1}^{p}(dx_{j})^{2} − 2|x|^{−4}(∑_{j=1}^{p} x_{j}dx_{j})^{2} + |x|^{−6}(∑_{j=1}^{p} x_{j}^{2})(∑_{i=1}^{p} x_{i}dx_{i})^{2}

= |x|^{−2}∑_{j=1}^{p}(dx_{j})^{2} − |x|^{−4}(∑_{j=1}^{p} x_{j}dx_{j})^{2}.

Similarly, we have

Φ^{∗} (∑_{j=1}^{q}(dv_{j})^{2}) = |y|^{−2}∑_{j=1}^{q}(dy_{j})^{2} − |y|^{−4}(∑_{j=1}^{q} y_{j}dy_{j})^{2}.

Because |x|^{2} = |y|^{2} and ∑_{j=1}^{p} x_{j}dx_{j} = ∑_{k=1}^{q} y_{k}dy_{k}, we have

Φ^{∗}(∑_{j=1}^{p}(du_{j})^{2} − ∑_{j=1}^{q}(dv_{j})^{2}) = \frac{1}{|x|^{2}} (∑_{j=1}^{p}(dx_{j})^{2} − ∑_{k=1}^{q}(dy_{k})^{2})

Hence, we have proved (3.3.1) from our definition of g_{M} and g_{N}. □

\underline{3.4.} If we apply Lemma 3.3 to the transformation on the pseudo-Riemannian manifold M = S^{p−1} × S^{q−1}, we have:

Lemma 3.4.1. G acts conformally on M. That is, for h ∈ G, z ∈ M, we have

L_{h}^{∗}g_{M} = \frac{1}{ν(h · z)^{2}} g_{M} at T_{z}M.

Proof. The transformation L_{h}: M → M is the composition of the isometry M → h · M, z ↦ h · z, and the conformal map Φ|_{h·M}: h · M → M, ξ ↦ \frac{ξ}{ν(ξ)}. Hence Lemma 3.4.1 follows. □

Several works in differential geometry treat the connection between the geometry of a manifold and the structure of its conformal group. For the identity

Conf(S^{p−1} × S^{q−1}) = O(p, q), (p > 2, q > 2),

see for example [13], Chapter IV.

As in Example 2.2, the Yamabe operator on M = S^{p−1} × S^{q−1} is given by the formula:

Δ̃_{M} = Δ_{S^{p−1}} − Δ_{S^{q−1}} − \frac{p + q − 4}{4(p + q − 3)} ((p − 1)(p − 2) − (q − 1)(q − 2))

= (Δ_{S^{p−1}} − \frac{1}{4}(p − 2)^{2}) − (Δ_{S^{q−1}} − \frac{1}{4}(q − 2)^{2}) (3.4.1)

= (Δ̃_{S^{p−1}} − \frac{1}{4}) − (Δ̃_{S^{q−1}} − \frac{1}{4}) .

We define a subspace of C^{∞}(S^{p−1} × S^{q−1}) by

(3.4.2)  V^{p,q} := {f ∈ C^{∞}(S^{p−1} × S^{q−1}) : Δ̃_{M}f = 0}.

By applying Theorem 2.5, we have

Theorem 3.4.2. Let p, q ≥ 1. For h ∈ O(p, q), z ∈ M = S^{p−1} × S^{q−1}, and f ∈ V^{p,q}, we define

(3.4.3)  (ϖ^{p,q}(h^{−1})f)(z) := ν(h · z)^{− \frac{p+q−4}{2}} f(L_{h}z).

Then (ϖ^{p,q}, V^{p,q}) is a representation of O(p, q).

\underline{3.5.} In order to describe the K-type formula of ϖ^{p,q}, we recall the basic fact of spherical harmonics. Let p ≥ 2. The space of spherical harmonics of degree k ∈ ℕ is defined to be

H^{k}(ℝ^{p}) = {f ∈ C^{∞}(S^{p−1}) : Δ_{S^{p−1}}f = −k(k + p − 2)f},

which is rewritten in terms of Δ̃_{S^{p−1}} = Δ_{S^{p−1}} − \frac{1}{4}(p − 1)(p − 3) (see Example 2.2) as

= \{f ∈ C^{∞}(S^{p−1}) : Δ̃_{S^{p−1}}f = (\frac{1}{4} − (k + \frac{p − 2}{2})^{2}) f\}. (3.5.1)

The orthogonal group O(p) acts on H^{k}(ℝ^{p}) irreducibly and we have the dimension formula:

(3.5.2)  dim_{ℂ} H^{k}(ℝ^{p}) = ( \begin{matrix}p + k − 2 \\ k \end{matrix} ) + ( \begin{matrix}p + k − 3 \\ k − 1 \end{matrix} )

For p = 1, it is convenient to define representations of O(1) by

H^{k}(ℝ^{1}) := \{ \begin{matrix}ℂ (trivial representation) & (k = 0) \\ ℂ (signature representation) & (k = 1) \\ 0 & (k ≥ 2). \end{matrix}

Then we have irreducible decompositions as O(p)-modules for p ≥ 1:

L^{2}(S^{p−1}) ≃ ∞ ; \begin{matrix}∑ & ⊕ \\ k=0 \end{matrix} H^{k}(ℝ^{p}) (Hilbert direct sum).

\underline{3.6.} Here is a basic property of the representation (ϖ^{p,q}, V^{p,q}).

Theorem 3.6.1. Suppose that p, q are integers with p ≥ 2 and q ≥ 2.

1) The underlying (𝔤, K)-module (ϖ^{p,q})_{K} of ϖ^{p,q} has the following K-type formula:

(3.6.1)  (ϖ^{p,q})_{K} ≃ ⨁_{\begin{matrix}a,b∈ℕ \\ a+ \frac{p}{2} =b+ \frac{q}{2} \end{matrix}} H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q}).

2) In the Harish-Chandra parametrization, the Z(𝔤)-infinitesimal character of ϖ^{p,q} is given by (1, \frac{p+q}{2} − 2, \frac{p+q}{2} − 3, …, 1, 0).

3) V^{p,q} is non-zero if and only if p + q ∈ 2ℕ.

4) If p + q ∈ 2ℕ and if (p, q) ≠ (2, 2), then (ϖ^{p,q}, V^{p,q}) is an irreducible representation of G = O(p, q) and the underlying (𝔤, K)-module (ϖ_{K}^{p,q}, V_{K}^{p,q}) is unitarizable.

Although Theorem 3.6.1 overlaps with the results of Kostant, Binegar-Zierau, Howe-Tan, Huang-Zhu obtained by algebraic methods ([2], [11], [12], [25]), we shall give a self-contained and new proof from our viewpoint: conformal geometry and discrete decomposability of the restriction with respect to non-compact subgroups. The method of finding K-types will be generalized to the branching law for noncompact subgroups (§7, §9). The idea of proving irreducibility seems interesting by its simplicity, because we do not need rather complicated computations (cf. [2], [11]) but just use the discretely decomposable branching law with respect to O(p, q′) × O(q′′). We shall give two different proofs of the unitarizability of ϖ^{p,q} because of the importance of “small” representations in the current status of unitary representation theory.

Proof. Let F ∈ V^{p,q} ⊂ C^{∞}(M). Then F is developed as

F = ∑_{a,b∈ℕ} F_{a,b} (F_{a,b} ∈ H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q})),

where the right side converges in the topology of C^{∞}(M). Applying the Yamabe operator, we have

Δ̃_{M}F = ∑_{a,b∈ℕ} (− (a + \frac{p − 2}{2})^{2} + (b + \frac{q − 2}{2})^{2}) F_{a,b}.

Since Δ̃_{M}F = 0, F_{a,b} can be non-zero if and only if

(3.6.2)  |a + \frac{p − 2}{2}| = |b + \frac{q − 2}{2}|,

whence (1) and (3). The statement (2) follows from Lemma 3.7.2 and (3.7.4). An explicit (unitarizable) inner product for ϖ^{p,q} will be given in §3.9 (see also Remark 3.9, §8).

We shall give a simple proof of the irreducibility of ϖ^{p,q} in Theorem 7.6 by using discretely decomposable branching laws to non-compact subgroups (Theorem 4.2 and Theorem 7.1). □

Remark 3.6.2. 1) ϖ^{2,2} contains the trivial one dimensional representation as a subrepresentation. The quotient ϖ^{2,2}/ℂ is irreducible as an O(2, 2)-module and splits into a direct sum of four irreducible SO_{0}(2, 2)-modules. The short exact sequence of O(2, 2)-modules 0 → ℂ → ϖ^{2,2} → ϖ^{2,2}/ℂ → 0 does not split, and ϖ^{2,2} is not unitarizable as an O(2, 2)-module.

This case is the only exception that ϖ^{p,q} is not unitarizable as a Conf(S^{p−1} × S^{q−1})-module.

2) The K-type formula for the case p = 1 or q = 1 is obtained by the same method as in Theorem 3.6.1. Then we have that

V^{p,q} ≃ \{ \begin{matrix}ℂ^{4} & if (p, q) = (1, 1), \\ ℂ^{2} & if (p, q) = (1, 3), (3, 1), \\ \{0\} & if p = 1 or q = 1 with p + q > 4 or if p + q ∉ 2ℕ. \end{matrix}

V^{p,q} consists of locally constant functions on S^{p−1} × S^{q−1} if (p, q) = (1, 1), (1, 3) and (3, 1).

3) In the case of the Kepler problem, i.e. the case of G = O(4, 2), the above Ktype formula has a nice physical interpretation, namely: the connected component of G acts irreducibly on the space with positive Fourier components for the action of the circle SO(2), the so-called positive energy subspace; the Fourier parameter n = 1, 2, 3, ... corresponds to the energy level in the usual labeling of the bound states of the Hydrogen atom, and the dimension (also called the degeneracy of the energy level) for the spherical harmonics is n^{2}, as it is in the labeling using angular momentum and its third component of the wave functions ψ_{nlm}. Here n corresponds to our b.

\underline{3.7.} Let us understand ϖ^{p,q} as a subrepresentation of a degenerate principal series.

For ν ∈ ℂ, we denote by the space

(3.7.1)  S^{ν}(Ξ) := {f ∈ C^{∞}(Ξ) : f(tξ) = t^{ν}f(ξ), ξ ∈ Ξ, t > 0}

of smooth functions on Ξ of homogeneous degree ν. Furthermore, for ϵ = ±1, we put

S^{ν,ϵ}(Ξ) := {f ∈ S^{ν}(Ξ) : f(−ξ) = ϵf(ξ), ξ ∈ Ξ}.

Then we have a direct sum decomposition

S^{ν}(Ξ) = S^{ν,1}(Ξ) + S^{ν,−1}(Ξ),

on which G acts by left translations, respectively.

Lemma 3.7.1. The restriction C^{∞}(Ξ) → C^{∞}(M), f ↦ f|_{M} induces the isomorphism of G-modules between S^{−λ}(Ξ) and (ϖ_{λ}, C^{∞}(M)) (given in (2.5.1)) for any λ ∈ ℂ.

Proof. If f ∈ S^{−λ}(Ξ), h ∈ G and z ∈ M, then

f(h · z) = f (ν(h · z)\frac{h · z}{ν(h · z)}) = ν(h · z)^{−λ}f(L_{h}z) = (ϖ_{λ}(h^{−1})f|_{M}) (z),

where the last formula follows from the definition (2.5.1) and Lemma 3.4.1. □

Let us also identify S^{ν,ϵ}(Ξ) with degenerate principal series representations in a standard notation. The indefinite orthogonal group G = O(p, q) acts on the light cone Ξ transitively. We put

(3.7.2)  ξ^{o} := ^{t}(1, 0, …, 0, 0, …, 0, 1) ∈ Ξ.

Then the isotropy subgroup at ξ^{o} is of the form M_{+}^{max}N^{max}, where M_{+}^{max} ≃ O(p − 1, q − 1) and N^{max} ≃ ℝ^{p+q−2} (abelian Lie group). We set

E := E_{1,p+q} + E_{p+q,1} ∈ 𝔤_{0},

where E_{ij} denotes the matrix unit. We define an abelian Lie group by A^{max} := exp ℝE (⊂ G), and put

(3.7.3)  m_{0} := −I_{p+q} ∈ G.

We define M^{max} to be the subgroup generated by M_{+}^{max} and m_{0}, then

P^{max} := M^{max}A^{max}N^{max}

is a Langlands decomposition of a maximal parabolic subgroup P^{max} of G. If a = exp(tE) (t ∈ ℝ), we put a^{λ} := exp(tλE) for λ ∈ ℂ. We put

ρ := \frac{p + q − 2}{2}.

For ϵ = ±1, we define a character χ_{ϵ} of M^{max} by the composition

χ_{ϵ}: M^{max} → M^{max}/M_{+}^{max} ≃ {1, m_{0}} → ℂ^{×},

such that χ_{ϵ}(m_{0}) := ϵ. We also write sgn for χ_{−1} and 1 for χ_{1}. We define F to be the A, B, C^{∞} or D′ valued degenerate principal series by

F- Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}) := {f ∈ F(G) : f(gman) = χ_{ϵ}(m^{−1})a^{−(λ+ρ)}f(g)},

which has Z(𝔤)-infinitesimal character

(3.7.4)  (λ, \frac{p + q}{2} − 2, \frac{p + q}{2} − 3, …, \frac{p + q}{2} − [\frac{p + q}{2} ])

in the Harish-Chandra parametrization. The underlying (𝔤, K)-module will be denoted by Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}). We note that Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}) is unitarizable if λ ∈ \sqrt{−1}ℝ.

In view of the commutative diagram of G-spaces:

G/M_{+}^{max}N^{max}  →^{∼} Ξ,  gM_{+}^{max}N^{max} ↦ g · ξ^{o}.
↓  ↓ Φ
G/P^{max} ←^{ℤ_{2}} G/M_{+}^{max}A^{max}N^{max} →^{∼} M ≃ Ξ/ℝ_{+}^{×}

we have an isomorphism of G-modules:

(3.7.5)  C^{∞}- Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}) ≃ S^{−λ− \frac{p+q−2}{2} ,ϵ}(Ξ).

It follows from Theorem 2.5 and Lemma 3.7.1 that (ϖ^{p,q}, V^{p,q}) is a subrepresentation of S^{− \frac{p+q−4}{2}} (Ξ). Furthermore, ϖ^{p,q}(m_{0}) acts on each K-type H^{a}(ℝ^{p})⊠H^{b}(ℝ^{q}) (a + \frac{p}{2} = b + \frac{q}{2}) as a scalar

(−1)^{a+b} = (−1)^{2a+ \frac{p−q}{2}} = (−1)^{\frac{p−q}{2}} .

Hence, we have the following:

Lemma 3.7.2. ϖ^{p,q} is a subrepresentation of S^{a,ϵ}(Ξ) with a = −\frac{p+q−4}{2} and ϵ = (−1)^{\frac{p−q}{2}} , or equivalently, of C^{∞}- Ind_{P^{max}}^{G}((−1)^{\frac{p−q}{2}} ⊗ ℂ_{−1}).

The quotient will be described in (5.5.5).

Remark 3.7.3. 1) ϖ^{p,q} splits into two irreducible components as SO(p, q)-modules, say ϖ_{±}^{p,q}, if p = 2 and q ≥ 4. Then, ϖ^{p,q} (or ϖ_{±}^{p,q} if p = 2 and q ≥ 4) coincides with the “minimal representations” constructed in [2], [25], [30].

2) In [2], it was claimed that the minimal representations of SO(p, q) are realized in the subspace of \{ψ ∈ C^{∞}(S^{p−1} × S^{q−1}) : ψ(−y) = (−1)^{d}ψ(y)\} for d = 2 − \frac{p+q}{2} . But this parity is not correct when both p and q are odd.

3) Our parametrization of S^{a,ϵ}(Ξ) is the same with S^{a,ϵ}(X^{0}) in the notation of [11].

\underline{3.8.} Let p ≥ 2. The differential operator −Δ_{S^{p−1}} + \frac{(p−2)^{2}}{4} acts on the space H^{a}(ℝ^{p}) of spherical harmonics as a scalar a(a + p − 2) + \frac{1}{4}(p − 2)^{2} = (a + \frac{p−2}{2})^{2}. Therefore, we can define a non-negative self-adjoint operator

(3.8.1)  D_{p}: L^{2}(S^{p−1}) → L^{2}(S^{p−1})

by

D_{p} := (−Δ_{S^{p−1}} + \frac{(p − 2)^{2}}{4} )^{\frac{1}{4}}

with the domain of definition given by

Dom(D_{p}) := \{F = ∑_{a=0}^{∞} F_{a} ∈ L^{2}(S^{p−1}) : ∑_{a=0}^{∞}(a + \frac{p − 2}{2})∥F_{a}∥_{L^{2}(S^{p−1})}^{2} < ∞\}.

Here is a convenient criterion which assures a given function to be in Dom(D_{p}):

Lemma 3.8.1. If F ∈ L^{2}(S^{p−1}) satisfies Y F ∈ L^{2− \frac{2}{p}} (S^{p−1}) for any smooth vector field Y on S^{p−1} then F ∈ Dom(D_{p}). Namely, D_{p}F is well-defined and D_{p}F ∈ L^{2}(S^{p−1}).

In order to prove Lemma 3.8.1, we recall an inequality due to W. Beckner:

Fact 3.8.2 ([1], Theorem 2). Let 1 ≤ δ ≤ 2 and F ∈ L^{δ}(S^{n}). Let F = ∑_{k=0}^{∞} F_{k} be the expansion in terms of spherical harmonics F_{k} ∈ H^{k}(ℝ^{n+1}), which converges in the distribution sense. Then

(3.8.2)  ∑_{k=0}^{∞} γ_{k}∥F_{k}∥_{L^{2}(S^{n})}^{2} ≤ ∥F ∥_{L^{δ}(S^{n})}^{2}, γ_{k} := \frac{Γ(\frac{n}{δ})Γ(k + n − \frac{n}{δ})}{Γ(n − \frac{n}{δ})Γ(k + \frac{n}{δ})}.

For our purpose, we need to give a lower estimate of γ_{k} in Fact 3.8.2. By Stirling’s formula for the Gamma function, we have

k^{b−a} \frac{Γ(k + a)}{Γ(k + b)} ∼ 1 + \frac{(a − b)(a + b − 1)}{2k} + …

as k → ∞. Hence, there exists a positive constant C depending only on n and δ so that

(3.8.3)  Ck^{n(1− \frac{2}{δ})} ≤ γ_{k}

for any k ≥ 1. Combining (3.8.2) and (3.8.3), we have:

(3.8.4)  C ∑_{k=1}^{∞} k^{n(1− \frac{2}{δ})}∥F_{k}∥_{L^{2}(S^{n})}^{2} ≤ ∥F ∥_{L^{δ}(S^{n})}^{2}.

Now we are ready to prove Lemma 3.8.1.

Proof of Lemma 3.8.1. Let {X_{i}} be an orthonormal basis of 𝔬(p) with respect to (−1)× the Killing form. The action of O(p) on S^{p−1} induces a Lie algebra homomorphism L: 𝔬(p) → 𝔛(S^{p−1}). Then we have Δ_{S^{p−1}} = ∑_{i} L(X_{i})^{2}. We write F = ∑_{k=0}^{∞} F_{k} where F_{k} ∈ H^{k}(ℝ^{p}). We note that L(X)F_{k} ∈ H^{k}(ℝ^{p}) for any k and for any X ∈ 𝔬(p), because Δ_{S^{p−1}} commutes with L(X). If we apply (3.8.4) with δ = 2 − \frac{2}{p} and n = p − 1, then we have

C ∑_{k=1}^{∞} k^{−1}∥L(X_{i})F_{k}∥_{L^{2}(S^{p−1})}^{2} ≤ ∥L(X_{i})F ∥_{L^{2− \frac{2}{p}} (S^{p−1})}^{2}.

Because L(X_{i}) is a skew-symmetric operator, we have

∑_{i} ∥L(X_{i})F_{k}∥_{L^{2}(S^{p−1})}^{2} = − ∑_{i}(Δ_{S^{p−1}}F_{k}, F_{k})_{L^{2}(S^{p−1})} = k(k + p − 2)∥F_{k}∥_{L^{2}(S^{p−1})}^{2},

and therefore

C ∑_{k=1}^{∞}(k + p − 2)∥F_{k}∥_{L^{2}(S^{p−1})}^{2} ≤ ∑_{i} ∥L(X_{i})F ∥_{L^{2− \frac{2}{p}} (S^{p−1})}^{2} < ∞.

Hence we have proved that D_{p}F is well-defined and

∥D_{p}F ∥_{L^{2}(S^{p−1})}^{2} = ∑_{k=0}^{∞}(k + \frac{p − 2}{2})∥F_{k}∥_{L^{2}(S^{p−1})}^{2} < ∞.

This completes the proof of Lemma 3.8.1 □

\underline{3.9.} Let p ≥ 2 and q ≥ 2. We extend D_{p} to a self-adjoint operator (with the same notation) on L^{2}(M). Then D_{p} is a pseudo-differential operator acting on

H^{a}(ℝ^{p}) ⊠ L^{2}(S^{q−1}) as a scalar \sqrt{a + \frac{p−2}{2}}. Likewise, we define D_{q} as a self-adjoint operator on L^{2}(S^{q−1}) and also on L^{2}(M). It follows from (3.6.2) that

(3.9.1)  D_{p} = D_{q} on V_{K}^{p,q}.

Let us consider an explicit unitary inner product for ϖ^{p,q}. We define the Knapp-Stein intertwining operator

A_{λ,ϵ}: Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{λ}) → Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{−λ})

by

(A_{λ,ϵ}f)(x) := ∫_{M} ψ_{λ−ρ,ϵ}([x, b]) f(b) db (x ∈ M).

Here, db is the Riemannian measure on M ≃ S^{p−1} × S^{q−1}, ρ = \frac{p+q−2}{2}, and

ψ_{ν,ϵ}(t) := \frac{1}{Γ(\frac{2ν+3−ϵ}{4})}|t|^{ν}χ_{ϵ}(sgn t).

Then, A_{λ,ϵ} has an eigenvalue on H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q}) (ϵ = (−1)^{a−b}) as follows ([21]):

(3.9.2)  \frac{2^{\frac{p+q+2}{2} −λ} π^{\frac{p+q−1}{2}} (−1)^{[ \frac{a−b}{2} ]} Γ(λ) Γ(−B_{λ}^{++})}{Γ(\frac{1}{2}) Γ(\frac{−2λ+p+q−1−ϵ}{4}) Γ(1 + B_{λ}^{−−}) Γ(1 + B_{λ}^{+−}) Γ(1 + B_{λ}^{−+})}

where for ϵ_{1}, ϵ_{2} = ±, we have put

(3.9.3)  B_{λ}^{ϵ_{1},ϵ_{2}} ≡ B_{λ}^{ϵ_{1},ϵ_{2}}(a, b) := λ − 1 − ϵ_{1}(a + \frac{p}{2} − 1) − ϵ_{2}(b + \frac{q}{2} − 1).

In particular, if p + q ∈ 2ℕ, p ≥ 2, q ≥ 2, (p, q) ≠ (2, 2) and ϵ = (−1)^{\frac{p−q}{2}} , then A_{1,ϵ}: Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{1}) → Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{−1}) is non-zero only on the submodule ϖ_{K}^{p,q}, and has an eigenvalue on H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q}) (a + \frac{p}{2} = b + \frac{q}{2}) given by

\frac{2^{\frac{p+q}{2}} π^{\frac{p+q−1}{2}} (−1)^{[ \frac{q−p}{4} ]} Γ(1) Γ(2a + p − 2)}{Γ(^{\frac{p+q−3−(−1)^{\frac{p−q}{2}}}{4}} ) Γ(2a + p − 1) Γ(1) Γ(1)} = \frac{2^{\frac{p+q−2}{2}} π^{\frac{p+q−1}{2}} (−1)^{[ \frac{q−p}{4} ]}}{Γ(^{\frac{p+q−3−(−1)^{\frac{p−q}{2}}}{4}} ) (a + \frac{p}{2} − 1)}

namely,

\frac{1}{a + \frac{p}{2} − 1} = \frac{1}{b + \frac{q}{2} − 1}

up to a non-zero constant scalar (−1)^{[ \frac{q−p}{4} ]}c_{1}, where

(3.9.4)  c_{1} = \frac{Γ(^{\frac{p+q−3−(−1)^{\frac{p−q}{2}}}{4}} )}{2^{\frac{p+q−2}{2}} π^{\frac{p+q−1}{2}}}.

Because the integration on G/P^{max} ≃ M/ ∼ ℤ_{2} gives a G-invariant non-degenerate sesquilinear form

Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{1}) × Ind_{P^{max}}^{G}(ϵ ⊗ ℂ_{−1}) → ℂ,

the underlying (𝔤, K)-module V_{K}^{p,q} (p, q ≥ 2, (p, q) ≠ (2, 2), p + q ∈ 2ℕ) is unitarizable with the inner product

(f_{1}, f_{2}) := ∫_{M}(a + \frac{p}{2} − 1)f_{1}\overline{f_{2}} dω, f_{1}, f_{2} ∈ H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q}),

namely,

(3.9.5)  (f_{1}, f_{2}) = ∫_{M}(D_{p}f_{1})\overline{D_{p}f_{2}} dω = ∫_{M}(D_{q}f_{1})\overline{D_{q}f_{2}} dω, f_{1}, f_{2} ∈ V_{K}^{p,q}

where dω is the standard measure on M.

We denote by \overline{V^{p,q}} the Hilbert completion of V^{p,q}, on which G acts as an (irreducible) unitary representation of G. We shall use the same notation ϖ^{p,q} to denote this unitary representation.

In view of §3.8, we can describe \overline{V^{p,q}} as follows: Let V be the Hilbert space of the completion of C^{∞}(M) by the norm defined by

∥F ∥_{L^{2}(M)}^{2} + ∥(D_{p} + D_{q})F ∥_{L^{2}(M)}^{2} for F ∈ C^{∞}(M).

Then, V is a dense subspace of L^{2}(M) and V = Dom(D_{p}) ∩ Dom(D_{q}). With this notation, the closure \overline{V^{p,q}} is characterized directly by the following:

Theorem 3.9.

\overline{V^{p,q}} = \{f ∈ V : D_{p}f = D_{q}f\} = \{f ∈ V : Δ̃_{M}f = 0\},

where Δ̃_{M}f = 0 is in the distribution sense. The norm for the unitary representation \overline{V^{p,q}} equals \frac{1}{4}∥(D_{p} + D_{q})F ∥_{L^{2}(M)}^{2}. Namely, if F = ∑_{a} F_{a,b} ∈ \overline{V^{p,q}} with

F_{a,b} ∈ H^{a}(ℝ^{p}) ⊠ H^{b}(ℝ^{q}) and b = a + \frac{p−q}{2} , then D_{p}F = ∑_{a} \sqrt{a + \frac{p−2}{2}}F_{a,b} and

∥F ∥^{2} = ∑_{a}(a + \frac{p − 2}{2})∥F_{a,b}∥_{L^{2}(M)}^{2}.

Here the sum is taken over integers a with a ≥ max(0, \frac{p−q}{2} ).

Remark. By comparing the construction of [2], (3.9.4) coincides with the formula obtained by Binegar-Zierau by a different method (see Remark 3.7.3 (1)). These authors defined a similar operator D_{n} of [2], page 249; we remark that there is a typographical error in the definition of D_{n}; n − 2 should read (n − 2)^{2}. Then, the square D_{p}^{2} of our operator corresponds to D_{p} if p ≠ 2; |D_{p}| if p = 2, with the notation in [2].

\underline{3.10.} The following lemma is rather weak, but is clear from the Sobolev estimate.

Lemma 3.10. Suppose W is an open set of M such that the measure of M \ W is zero. Suppose F is a C^{∞} function on W satisfying Δ̃_{M}F = 0 on W. If F ∈ L^{2}(M) and if Y Y′F ∈ L^{1}(M) for any Y, Y′ ∈ 𝔛(M) (differentiation in the sense of the Schwartz distributions), then F ∈ \overline{V^{p,q}}.

References

[1] W. Beckner, Geometric inequalities in Fourier analysis, Essays on Fourier analysis in honor of Elias M. Stein (C. Fefferman, R. Fefferman and S. Wainger, ed.), Princeton Univ. Press, 1995, pp. 36–68.

[2] B. Binegar and R. Zierau, Unitarization of a singular representation of SO(p, q), Comm. Math. Phys. 138 (1991), 245–258.

[3] R. Brylinski and B. Kostant, Minimal representations of E_{6}, E_{7} and E_{8} and the generalized Capelli identity, Proc. Nat. Acad. Sci. U.S.A. 91 (1994), 2469–2472.

[4] , Differential operators on conical Lagrangian manifolds, Lie theory and geometry, vol. 123, Progress in Math., 1994, pp. 65–96.

[5] , Lagrangian models of minimal representations of E_{6}, E_{7} and E_{8}, Functional analysis on the eve of the 21st century, 1, vol. 131, Progress in Math., 1995, pp. 13–63.

[6] A. Erdélyi, Higher Transcendental Functions, vol. I, McGraw-Hill, New York, 1953.

[7] A. Erdélyi, Tables of Integral Transforms, vol. II, McGraw-Hill, New York, 1954.

[8] I. M. Gelfand and G. E. Shilov, Generalized Functions, I, Academic Press, 1964.

[9] S. Helgason, Differential Geometry, Lie Groups and Symmetric Spaces, Pure and Appl. Math., vol. 80, Academic Press, 1978.

[10] R. Howe, Transcending classical invariant theory, Jour. A. M. S. 2 (1989), 535–552.

[11] R. Howe and E. Tan, Homogeneous functions on light cones, Bull. A. M. S. 28 (1993), 1–74.

[12] J-S. Huang and C-B. Zhu, On certain small representations of indefinite orthogonal groups, Representation Theory 1, 190–206.

[13] S. Kobayashi, Transformation Groups in Differential Geometry, Classics in Mathematics, vol. 70, Springer, 1995.

[14] T. Kobayashi, Singular Unitary Representations and Discrete Series for Indefinite Stiefel Manifolds U(p, q; 𝔽)/U(p − m, q; 𝔽), vol. 462, Memoirs A. M. S., 1992.

[15] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205.

[16] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups II — micro-local analysis and asymptotic K-support, Ann. of Math. 147 (1998), 709–729.

[17] , Discrete decomposability of the restriction of A_{𝔮}(λ) with respect to reductive subgroups III — restriction of Harish-Chandra modules and associated varieties, Invent. Math. 131 (1998), 229–256.

[18] , Multiplicity free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (eds. K. Mimachi) (1997), 9–17.

[19] , Multiplicity-free theorem in branching problems of unitary highest weight modules, preprint.

[20] , Discretely decomposable restrictions of unitary representations of reductive Lie groups, Analysis on Homogeneous Spaces and Representation Theory of Lie Groups, Okayama-Kyoto (T. Kobayashi et al, eds.), vol. 26, 2000, pp. 98–126.

[21] , Branching laws of O(p, q) associated to minimal elliptic orbits, (in preparation).

[22] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits, C. R. Acad. Sci. Paris 326 (1998), 925–930.

[23] , Analysis on the minimal representation of O(p, q) - II. Branching laws, Preprint.

[24] , Analysis on the minimal representation of O(p, q) - III. Ultrahyperbolic equations on ℝ^{p−1,q−1}, Preprint.

[25] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4), Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory (Connes et al, eds.), vol. 92, Birkhäuser, Boston, 1990, pp. 85–124.

[26] J. M. Lee and T. H. Parker, The Yamabe problem, Bull. A. M. S. 17 (1987), 37–91.

[27] H. Sabourin, Une représentation unipotente associée à l’orbite minimale: Le cas de SO(4,3), J. Funct. Anal. 137 (1996), 394–465.

[28] H. Schlichtkrull, Eigenspaces of the Laplacian on hyperbolic spaces: composition series and integral transforms, J. Funct. Anal. 70 (1987), 194–219.

[29] W. Schmid, Boundary value problems for group invariant differential equations (1985), Asterisque, hors s’erie, 311–321, Élie Cartan et les mathématiques d’aujourd’hui.

[30] P. Torasso, Méthode des orbites de Kirillov-Dufulo et representations minimales des groupes simples sur un corps local de caractéristique nulle, preprint (1997).

[31] D. Vogan Jr., Representations of Real Reductive Lie Groups, Progress in Math., vol. 15, Birkhäuser, 1981.

[32] , Unitarizability of certain series of representations, Ann. of Math. (1984), 141–187.

[33] , Unitary Representations of Reductive Lie Groups, Ann. Math. Stud. 118, Princeton U. P., Princeton, New Jersey, 1987.

[35] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 191–221.

[34] , Singular unitary representations, Springer Lecture Notes in Mathematics 880 (1980), 506–535.

[36] H. Wong, Dolbeault cohomologies and Zuckerman modules associated with finite rank representations, ph. D. dissertation, Harvard University (1992).

[37] B. Ørsted, A note on the conformal quasi-invariance of the Laplacian on a pseudo-Riemannian manifold, Lett. Math. Phys. 1 (1977), 183.

[38] , Conformally invariant differential equations and projective geometry, J. Funct. Anal. 44 (1981), 1–23.

RIMS, Kyoto University, Sakyo-ku, Kyoto, 606-8502, Japan

Department of Mathematics and Computer Science, SDU - Odense University, Campusvej 55, DK-5230, Odense M, Denmark

E-mail address: toshi@kurims.kyoto-u.ac.jp; orsted@imada.sdu.dk