Integral formula of the unitary inversion operator for the minimal representation of O(p, q)

By Toshiyuki Kobayashi ^{1)} and Gen Mano

RIMS, Kyoto University, Sakyo-ku, Kyoto, 606-8502, Japan

E-mail addresses: toshi@kurims.kyoto-u.ac.jp (T. Kobayashi), gmano@kurims.kyoto-u.ac.jp (G. Mano)

Abstract: The indefinite orthogonal group G = O(p, q) has a distinguished infinite dimensional unitary representation π, called the minimal representation for p + q even and greater than 6. The Schrödinger model realizes π on a very simple Hilbert space, namely, L^{2}(C) consisting of square integrable functions on a Lagrangean submanifold C of the minimal nilpotent coadjoint orbit, whereas the G-action on L^{2}(C) has not been well-understood. This paper gives an explicit formula of the unitary operator π(w_{0}) on L^{2}(C) for the ‘conformal inversion’ w_{0} as an integrodifferential operator, whose kernel function is given by a Bessel distribution. Our main theorem generalizes the classic Schrödinger model on L^{2}(ℝ^{n}) of the Weil representation, and leads us to an explicit formula of the action of the whole group O(p, q) on L^{2}(C). As its corollaries, we also find a representation theoretic proof of the inversion formula and the Plancherel formula for Meijer’s G-transforms.

Key words: minimal unitary representation; Schrödinger model; Weil representation; indefinite orthogonal group.

In this paper, we provide an explicit formula for the unitary inversion operator on the ‘Schrödinger model’ for the minimal representation π of the indefinite orthogonal group G = O(p, q) of type D.

For a reductive Lie group a particularly interesting irreducible unitary representation, sometimes called the minimal representation, is the one corresponding via ‘geometric quantization’ to the minimal nilpotent coadjoint orbit 𝒪. Minimal representations are one of the most fundamental irreducible unitary representations in the sense that they cannot be built up from any smaller groups by existing methods of induced representations.

The classic example of minimal representations is the oscillator representation, or sometimes referred to as the (Segal–Shale–)Weil representation of the metaplectic group Mp(n, ℝ). For the indefinite orthogonal group O(p, q), there is no minimal representation if p + q is odd and p, q > 3 by a result due to Howe and Vogan [18, Theorem 2.3]. On the other hand, if p + q is even and p, q ≥ 2, then O(p, q) has a distinguished unitary representation. This representation, denoted by π, is our main concern in this paper, and has the following properties:

(i) π is a minimal representation if p + q ≥ 8. (ii) π is not spherical if p ≠ q. (iii) dπ is not a highest weight module of the Lie

algebra 𝔰𝔬(p, q) if p, q ≥ 3.

In the special case q = 2, the differential representation dπ splits into the sum of highest and lowest weight modules of 𝔰𝔬(p, q), and these have been studied by many authors, in particular in the physics literature, interpreted as the mass-zero spin-zero wave equation, or as the bound states of the Hydrogen atom in p − 1 space dimensions.

Since 1990s, various models have been proposed to construct the unitary representation π of O(p, q) for p, q ≥ 3 by Kostant in [14] for p = q = 4, and by Binegar–Zierau [1], Huang–Zhu [7], and Kobayashi– Ørsted [12, 13] for general p, q ≥ 2 such that p + q is an even integer (≥ 6). Yet another construction is studied in Brylinski–Kostant [2] and Torasso [17].

2000 Mathematics Subject Classification. Primary 22E30; Secondary 22E46, 43A80.

1) Partially supported by Grant-in-Aid for Scientific Research (B) (18340037), Japan Society for the Promotion of Science.

From now on, suppose G = O(p, q) where

p ≥ q ≥ 2, p + q is even, ≥ 6.

One of the models of the minimal representation π is the realization as a subrepresentation of the most degenerate principal series representation [1, 6, 7]. Geometrically, the representation space can be characterized as the solution space of the Yamabe–Laplace operator in conformal geometry [8, 12, 13]. An advantage of this model (conformal model) is that the G-action on function spaces is easy to describe, whereas the inner product on the solution space is rather complicated.

By taking the Fourier transform of the conformal model on the flat pseudo-Eulidean space ℝ^{p−1,q−1}, we get in [13, III] another model (Schrödinger model) which has an advantage that the inner product on the representation space is very simple, whereas the group action is not. This model generalizes the classic Schrödinger model on L^{2}(ℝ^{n}) of the oscillator representation (e.g. [3, 5]), and we shall call it the Schrödinger model of the minimal representation π.

To explain the Schrödinger model of π, let C be the conical subvariety given by

C := {(x_{1}, ⋯, x_{p+q−2}) ∈ ℝ^{p+q−2} \ {0} :

x_{1}^{2} + ⋯ + x_{p−1}^{2} − x_{p}^{2} − ⋯ − x_{p+q−2}^{2} = 0},

and consider the Hilbert space L^{2}(C) ≡ L^{2}(C, dμ) of square integrable functions on C against the measure

dμ := \frac{1}{2}r^{p+q−5}drdωdη

in the polar coordinate

ℝ_{+} × S^{p−2} × S^{q−2} ≃ C, (r, ω, η) ↦ (rω, rη).

This variety C is so small that the whole group G cannot act on C. In fact, any non-trivial homogeneous space of G has a higher dimension than dim C. However, a maximal parabolic subgroup \overline{P^{max}} acts on L^{2}(C) as a unitary representation as follows (see [13, III, §3.3.7]):

Let e_{1}, ⋯, e_{p+q} be the standard basis of ℝ^{p+q}, and \overline{P^{max}} the stabilizer of ℝ(e_{1} − e_{p+q}) in the real projective space ℙ^{p+q−1}ℝ. The geometric meaning of the group \overline{P^{max}} is that it is (essentially) the conformal group on the flat pseudo-Riemannian Euclidean space ℝ^{p−1,q−1}. In a group language, \overline{P^{max}} is the maximal parabolic subgroup of G corresponding to non-positive weight vectors for the adjoint action of

E := E_{1,p+q} + E_{p+q,1}.

Then \overline{P^{max}} has a Langlands decomposition

\overline{P^{max}} = M^{max}A^{max}\overline{N^{max}},

where

M^{max} ≃ {±I_{p+q}} × O(p − 1, q − 1),
A^{max} = {e^{sE} : s ∈ ℝ}.

We give a coordinate of the unipotent radical \overline{N^{max}} by

\overline{n}_{b}(e_{1} − e_{p+q}) = ( \begin{matrix}1 − Q(b) \\ 2b \\ 1 + Q(b) \end{matrix}) for b ∈ ℝ^{p+q−2}.

Here, Q(b) := ∑_{j=1}^{p−1} b_{j}^{2} − ∑_{j=p}^{p+q−2} b_{j}^{2}. The correspondence b ↦ \overline{n}_{b} gives an isomorphism of abelian Lie groups, ℝ^{p+q−2} ≃ \overline{N^{max}}.

With this notation, any element g of \overline{P^{max}} is written as

g = δme^{sE}\overline{n}_{b}

for some δ = ±1, m ∈ O(p − 1, q − 1), s ∈ ℝ and b ∈ ℝ^{p+q−2}. Then, the \overline{P^{max}} action on L^{2}(C) is defined by

(π(g)u)(x) = δ^{\frac{p−q}{2}} e^{− \frac{p+q−4}{2} s}e^{2\sqrt{−1}⟨b,x⟩}u(e^{−s} ^{t}mx).

Here, ⟨ , ⟩ denotes the standard (positive definite) inner product on ℝ^{p+q−2}, and^{t}m denotes the transpose of the matrix m.

One of the main results in [13] asserts that this action of \overline{P^{max}} on L^{2}(C) extends to an irreducible unitary representation of G, giving rise to the minimal representation of G.

The missing point of [13] is an explicit formula for the action of the whole group G on L^{2}(C) other than the action of \overline{P^{max}}.

We set

w_{0} = (\begin{matrix}I_{p} & 0 \\ 0 & −I_{q} \end{matrix}) .

In light of the Bruhat decomposition

G = \overline{P^{max}} ∐ \overline{P^{max}}w_{0}\overline{P^{max}},

the whole group action will be understood if we find an explicit formula for the unitary inversion operator π(w_{0}). For the oscillator representation, the corresponding unitary inversion operator is nothing but the Fourier transform (e.g. [3]). For our minimal representation π, we proved in [10] that π(w_{0}) is given by the Hankel transform in the special case q = 2 (see also [9]). The general case is the main theme of this paper. We shall find the integro-differential kernel for the unitary inversion operator π(w_{0}) on L^{2}(C) for general p, q as follows.

We begin with the tempered distributions on ℝ given by

Ψ_{m}^{0}(t) :=(2t)_{+}^{− \frac{m}{2}} J_{m}(2\sqrt{2t_{+}}),
Ψ_{m}^{+}(t) :=(2t)_{+}^{− \frac{m}{2}} J_{m}(2\sqrt{2t_{+}}) − ∑_{l=0}^{m−1} \frac{(−\frac{1}{2})^{l}}{Γ(m − l)}δ^{(l)}(t),
Ψ_{m}(t) :=(2t)_{+}^{− \frac{m}{2}} Y_{m}(2\sqrt{2t_{+}})

+ \frac{2(−1)^{m+1}}{π} (2t)_{−}^{− \frac{m}{2}} K_{m}(2\sqrt{2t_{−}}).

Here, J_{ν}(x), Y_{ν}(x) and K_{ν}(z) are the (modified) Bessel functions, δ^{(l)}(t) denotes the l-th differential of the Dirac delta function δ(t), and t_{+}^{λ} and t_{−}^{λ} are the distributions with meromorphic parameter λ ∈ ℂ such that they are locally integrable functions for Re λ > −1:

t_{+}^{λ} := \begin{cases}t^{λ} & if t > 0 \\ 0 & if t < 0, \end{cases} t_{−}^{λ} := \begin{cases}0 & if t > 0 \\ |t|^{λ} & if t < 0. \end{cases}

We define a generalized function K(x, x′) on the direct product manifold C × C by

(1) K(x, x′) ≡ K(p, q; x, x′)
:= 2(−1)^{\frac{(p−1)(p+2)}{2}} π^{− \frac{p+q−4}{2}} Φ_{p,q}(⟨x, x′⟩),

where the distribution Φ_{p,q}(t) is defined as follows:

Φ_{p,q}(t) := \begin{cases}Ψ0_{\frac{p+q−6}{2}} (t)  if q = 2, \\ Ψ+_{\frac{p+q−6}{2}} (t)  if q > 2 is even, \\ Ψ_{\frac{p+q−6}{2}} (t)  if q > 2 is odd. \end{cases}

Then, here is our main result. Theorem 1. The unitary inversion operator

π(w_{0}) is given by the following integro-differential operator:

(2)  π(w_{0})u(x) = ∫_{C} K(x, x′)u(x′)dμ(x′),

for u ∈ L^{2}(C).

The following new phenomenon is noteworthy: the kernel distribution K(x, x′) for the unitary operator π(w_{0}) is not locally integrable if p, q ≥ 3 and p+ q > 6, equivalently, if π is a minimal representation which is a non-highest weight module.

The following corollaries concern with the functional equation of K(x, x′).

Corollary 2 (Plancherel formula). Let S : L^{2}(C) → L^{2}(C) be an integral transform whose kernel function is given by K(x, x′). Then S is unitary.

Since the group law w_{0}^{2} = 1 in O(p, q) implies π(w_{0})^{2} = id on L^{2}(C), we immediately obtain the inversion formula: S^{−1} = S. We pin down:

Corollary 3 (Reciprocal formula). The unitary operator S is of order two in L^{2}(C). Namely, we have the following reciprocal formula for u ∈ L^{2}(C):

u(x) = ∫_{C} K(x, x′′)(∫_{C} K(x′′, x′)u(x′)dμ(x′))dμ(x′′).

Corollaries 2 and 3 are regarded as a generalization of the Plancherel and inversion formulas for the Fourier–Bessel transforms (see [16, Chapter 8] for traditional approaches, and [3, 9, 10] for represenation theoretic approaches using Mp(n, ℝ) or O(p, 2)).

The proof of Theorem 1 is based on the following steps: Step 1) Analysis on the Radon transform ℛ for func-

tions supported on C (see [15]). Step 2) Decomposition formula of π(w_{0}) into the ‘ra-

dial’ part T_{l,k}. For Step 1), we identify a compactly supported

smooth function f on C with a tempered distribution fdμ on ℝ^{p+q−2} (p + q > 4). We define the Radon transform of fdμ by

ℛf(ξ, t) := ∫_{C} f(x)δ(⟨x, ξ⟩ − t)dμ(x).

Then, ℛf(ξ, t) satisfies the ultra-hyperbolic differential equation:

(∑_{j=1}^{p−1} \frac{∂^{2}}{∂ξ_{j}^{2}} − ∑_{j=p}^{p+q−2} \frac{∂^{2}}{∂ξ_{j}^{2}} )(ℛf)(ξ, t) = 0.

As for the differentiability with respect to t, we note that ℛf(ξ, t) is not of C^{∞} class at t = 0. The regularity at t = 0 is the main issue of [15], where we prove that ℛf(ξ, t) is [\frac{p+q−7}{2}] times continuously differentiable at t = 0. This regularity is sufficient to show that the singular integral (2) makes sense for u ∈ C_{0}^{∞}(C). Conversely, Theorem 1 leads us to:

Corollary 4. f can be recovered only from the restriction of the Radon transform ℛf(ξ, t) to C ×ℝ.

For Step 2), we use the polar coordinate to decompose the Hilbert space L^{2}(C) into the discrete direct sum as Hilbert spaces:

\begin{matrix}∞ \\ ∑ & ⊕ \\ l,k=0 \end{matrix} L^{2}(ℝ_{+}, r^{p+q−5}dr) ⊗ ℋ^{l}(ℝ^{p−1}) ⊗ ℋ^{k}(ℝ^{q−1}).

Here, ℋ^{l}(ℝ^{p−1}) is the space of spherical harmonics on S^{p−2} of degree l. Then, π(w_{0}) preserves each (l, k) summand, on which π(w_{0}) is of the form T_{l,k}⊗id ⊗ id for some unitary operator T_{l,k} on L^{2}(ℝ_{+}, r^{p+q−5}dr). Here is an explicit formula of T_{l,k}.

Theorem 5. For l, k ∈ ℕ, we set a := max(l+ \frac{p−q}{2} , k). Then, T_{l,k} is given by

(3) (T_{l,k}f)(r) = ∫_{0}^{∞} K_{l,k}(rr′)f(r′)r^{′p+q−5}dr′,

where the kernel function K_{l,k}(t) is defined by

K_{l,k}(t) := 2(−1)^{a}G_{04}^{20}(t^{2}|\frac{l + k}{2} , a + \frac{−p + 3 − l − k}{2} ,

\frac{−p − q + 6 − l − k}{2} , \frac{−q + 3 + l + k}{2} − a).

Here, G_{04}^{20}(x|b_{1}, b_{2}, b_{3}, b_{4}) denotes Meijer’s Gfunction.

The group law w_{0}^{2} = 1 in G implies π(w_{0})^{2} = id and consequently, T_{l,k}^{2} = id for every l, k ∈ ℕ. Hence, Theorem 5 gives a group theoretic proof for the Plancherel and reciprocal formulas on Meijer’s G-transforms which were first proved by C. Fox [4] by a completely different method.

Corollary 6 (Plancherel formula). Let b_{1}, b_{2}, γ be half-integers such that b_{1} ≥ 0, γ ≥ 1, \frac{1−γ}{2} ≤ b_{2} ≤ \frac{1}{2} + b_{1}. Then the integral transform

S_{b_{1},b_{2},γ} : f(x) ↦

\frac{1}{γ} ∫_{0}^{∞}G_{04}^{20}((xy)^{\frac{1}{γ}} |b_{1}, b_{2}, 1 − γ − b_{1}, 1 − γ − b_{2})f(y)dy

is a unitary operator on L^{2}(ℝ_{+}).

Corollary 7 (Reciprocal formula). The unitary operator S_{b_{1},b_{2},γ} is of order two in L^{2}(ℝ_{+}), that is, (S_{b_{1},b_{2},γ})^{−1} = S_{b_{1},b_{2},γ}.

The proof of Theorem 5 is based on an explicit construction of K-finite vectors in L^{2}(C), generalizing the computation of the minimal K-type vector in L^{2}(C) (see [13, III, Theorem 5.5]).

The results here accomplish the program of the L^{2}-model (Schrödinger model) of the mimimal representation of the indefinite orthogonal group O(p, q) of type D. Details of this paper will be given in another article [11].

References

[ 1 ] B. Binegar and R. Zierau, Unitarization of a singular representation of SO(p, q), Comm. Math. Phys. 138 (1991), 245–258.

[ 2 ] R. Brylinski and B. Kostant, Differential operators on conical Lagrangean manifolds, Lie Theory and Geometry, Progr. Math. 123, Birkhäuser, 1994, 65–96.

[ 3 ] B. Folland, Harmonic Analysis in Phase Space, Ann. of Math. Stud. 122, Princeton University Press, 1989.

[ 4 ] C. Fox, The G and H functions as symmetrical Fourier kernels, Trans. Amer. Math. Soc. 98 (1961), 395–429.

[ 5 ] R. Howe, The oscillator semigroup, in: Proc. Symp. Pure Math. 48, Amer. Math. Soc., 1988, 61–132.

[ 6 ] R. Howe and E.-C. Tan, Homogeneous functions on light cones, Bull. Amer. Math. Soc. 28 (1993), 1–74.

[ 7 ] J.-S. Huang and C.-B. Zhu, On certain small representations of indefinite orthogonal groups, Representation Theory 1 (1997), 190–206.

[ 8 ] T. Kobayashi, Conformal geometry and global solutions to the Yamabe equations on classical pseudo-Riemannian manifolds, Proceedings of the 22nd Winter School “Geometry and Physics” (Srni, 2002). Rend. Circ. Mat. Palermo (2) Suppl. 71 (2003), 15–40.

[ 9 ] T. Kobayashi and G. Mano, Integral formulas for the minimal representation of O(p, 2), Acta Appl. Math. 86 (2005), 103–113.

[ 10 ] T. Kobayashi and G. Mano, The inversion formula and holomorphic extension of the minimal representation of the conformal group, Special volume of R. Howe on the occasion of his sixtieth birthday (eds. E.-C. Tan and C.-B. Zhu), accepted for publication. math.RT/0607007.

[ 11 ] T. Kobayashi and G. Mano, The Schrödinger model for the minimal representation of the indefinite orthogonal group O(p, q), in preparation.

[ 12 ] T. Kobayashi and B. Ørsted, Conformal geometry and branching laws for unitary representations attached to minimal elliptic orbits, C. R. Acad. Sci. Paris 326 (1998), 925–930.

[ 13 ] T. Kobayashi and B. Ørsted, Analysis on the minimal representation of O(p, q), I, II, III. Adv. Math. 180 (2003), 486–512, 513–550, 551–595.

[ 14 ] B. Kostant, The vanishing scalar curvature and the minimal unitary representation of SO(4, 4), eds. Connes et al., Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory, Progr. Math. 92, Birkhäuser, 1990, 85– 124.

[ 15 ] G. Mano, Radon transform of functions supported on a homogeneous cone, Ph.D. thesis, RIMS, Kyoto University, 2007.

[ 16 ] E. Titchmarsh, Introduction to the Theory of Fourier Integrals, Oxford, 1937; third edition, Chelsea Publishing Co., New York, 1986.

[ 17 ] P. Torasso, Méthode des orbites de Kirillov–Duflo et représentations minimales des groupes simples sur un corps local de caractéristique nulle, Duke Math. J. 90 (1997), 261–377.

[ 18 ] D. Vogan, Singular unitary representations. Noncommutative harmonic analysis and Lie groups (Marseille, 1980), pp. 506–535, Lecture Notes in Math., 880, Springer, Berlin-New York, 1981.