Convex domains and the Fourier transform on

spaces of constant curvature

Toshiyuki Kobayashi* School of Mathematics

Institute for Advanced Study Princeton, NJ 08540, USA

toshi@math.ias.edu

Typeset by AMS-TEX

*Supported by DMS-9100383.

Preface

These notes represent a series of twelve expository lectures delivered by the author at the CIMPA (Centre international de mathematiques pures et appliquees) Spring School on “Invariant Differential Operators on Lie Groups and Homogeneous Spaces” at WuHan University in P. R. China, from April 21 to May 11, 1991.

The main subject here is the study of the analytic set 𝒩(Ω) (null variety) associated to a bounded domain Ω in ℝ^{n} and in the hyperbolic space SO_{0}(n, 1)/SO(n). In particular, we are interested in the injectivity of Ω ↦ 𝒩(Ω), in the description of 𝒩(Ω) in an asymptotic sense and in the Pompeiu problem where 𝒩(Ω) contains a hypersphere. We try to present these topics in an elementary fashion.

Here is a brief outline of the contents:

In §1 we discuss three different topics: the null variety 𝒩(Ω) = {ζ ∈ ℂ^{n}:

∫_{Ω} e^{\sqrt{−1}(x,ζ)}dx = 0\}, integral geometry and a free boundary problem. In a special case, called the Pompeiu problem or Schiffer’s conjecture, it is well known that these topics are closely related with one another (Theorem (1.4.1)). We also set forth various related problems and give a short survey of results known so far. However, many of the basic questions have not found a final answer.

In §2 we deal with the asymptotic behavior of the zeros of some holomorphic functions of one variable. It is a classical result that the zeros of some special functions (e.g. the Bessel function J_{ν}(ζ), the associated Legendre function P_{ζ− \frac{1}{2}}^{λ} (z) and so on) are regularly distributed, and that most of them are real if the parameter involved is ‘real’. We generalize this from the viewpoint of integral transforms. That is, after a definition of appropriate function spaces and their norms, we give a description of the asymptotic behavior of the zeros of their Fourier transforms and find error estimates in terms of their norms. Since the author does now know convenient references for these results, we have given essentially complete treatments of these materials.

In §3 we discuss an asymptotic behavior of the Fourier transform of a convex domain in ℝ^{n}, using a method based on the Radon transform and going back to F. John. Combining this with the results of §2, we give an asymptotic behavior of the null variety 𝒩(Ω) (precisely, the intersection with an n + 1 dimensional real manifold). This result yields partial solutions to Problems A and B posed in §1.

§4 deals with four further results concerning the null variety 𝒩(Ω) in the case n = 2. That is, a positive solution to the injectivity problem ‘if one knows 𝒩(Ω), can Ω be recovered?’ for strictly convex domains (§4.1), a characterization of convexity of Ω in terms of 𝒩(Ω) (§4.2), and partial positive solutions to Schiffer’s conjecture in the case where Ω is a thin convex domain (§4.3) and ∂Ω is algebraic (§4.4). §5 treats the Pompeiu problem under a perturbation of a ball in ℝ^{n} by using spherical harmonics. §4.2 and §5 are new results.

In §6 we present a rapid outline of some basic concepts of semisimple Lie groups, (Riemannian) symmetric spaces, invariant differential operators and Fourier transforms. We develop more materials than we need, hoping that the reader who is not very familiar with them could get some ‘feeling’ about how and why they work.

Introducing briefly (horospherical) convexity, a pair of ‘Gauss maps’ and a pair of ‘supporting functions’ with their basic properties in a Riemannian symmetric space of the non-compact type of rank one in §7, we generalize some of the Euclidean results in §3 to the hyperbolic space SO_{0}(n, 1)/SO(n) in §8. We collect some classical results on Bessel functions in §9 as an appendix.

I should like to thank Jean-Michel Lemaire, Guy Roos, Pierre Torasso who directed the school, Yu Jia-Rong, Deng Hong and the people of the China-French center of Mathematics and Information and of the Department of mathematics at WuHan University for their hospitality, and the other participants for stimulating feedback during the lectures. I have benefitted from various discussions with and comments by Sigurdur Helgason, Bent Ørsted, Toshio Oshima and Henrik Schlichtkrull. Hideko Sekiguchi provided a list of errors in earlier versions, which was very helpful. Charles Conley and Atsuko Yamamoto have examined parts of the manuscript, and provided helpful advice throughout its writing. Carlos Berenstein made detailed comments and suggestions on the entire manuscript. I take this opportunity to thank all of them now; I apologize for those errors which undoubtedly remain. The final manuscript was written while a member of the School of Mathematics of the Institute for Advanced Study, for whose hospitality I am grateful. In particular, I am grateful to Ms. Dottie Phares for her very skillful typing of the manuscript.

Table of Contents

§1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 §2 The zeros of a certain class of entire functions . . . . . . . . . . . . . . . 19 §3 Convex domains Ω in ℝ^{n} and asymptotics of the null variety 𝒩(Ω) . . . . . . 49 §4 Some more results on 𝒩(Ω) in the case Ω in ℝ^{2} . . . . . . . . . . . . . . 59 §5 Pompeiu problem and a perturbation of a ball in ℝ^{n} . . . . . . . . . . . . 63 §6 A quick review of semisimple Lie groups, symmetric spaces, invariant

differential operators and intgral transforms . . . . . . . . . . . . . 69 §7 Submanifolds and convexity in a rank one symmetric space . . . . . . . . . 87 §8 Asymptotics of the null variety 𝒩(Ω) in the hyperbolic space . . . . . . . . 97 §9 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 A scheme showing the dependence of the various sections is indicated as follows:

1  2  6
↙  ↘  ↙  ↓
5  3  ↘  7
↓  ↓
4  8

§1 Introduction

In §1 we discuss three independent topics, namely, the null variety 𝒩(Ω) (§1.1), integral geometry (§1.2) and a free boundary problem (§1.3). In a special case, these topics are closely related with one another (Pompeiu problem, Schiffer’s conjecture) (see §1.4).

1.1 Null variety 𝒩(Ω).

Given a measurable bounded set Ω in ℝ^{n}, we define the Fourier transform of the characteristic function χ_{Ω} of Ω by

(1.1.1)  ℱχ_{Ω}(ζ) : = ∫_{ℝ^{n}} χ_{Ω}(x)e^{\sqrt{−1} (x,ζ)}dx
= ∫_{Ω} e^{\sqrt{−1} (x_{1}ζ_{1}+···+x_{n}ζ_{n})}dx_{1} … dx_{n},

for ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n}. Then ℱχ_{Ω}(ζ) is a holomorphic function of the n-valuables ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n}. We note that

(1.1.1) (a)  ℱχ_{Ω}(0) = the volume of Ω.

We define an analytic set in ℂ^{n}, called the null variety of Ω, by

(1.1.2) (a)  𝒩(Ω) := {ζ ∈ ℂ^{n}: ℱχ_{Ω}(ζ) = 0}.

We also define the real points:

(1.1.2) (b)  𝒩(Ω)_{ℝ} := 𝒩(Ω) ∩ ℝ^{n} ≡ {ζ ∈ ℝ^{n}: ℱχ_{Ω}(ζ) = 0}.

The purpose of our study is to initiate a new line of investigation of the assignment

(1.1.3)

𝒩: {Bounded measurable sets in ℝ^{n}}  ⟶ {Analytic sets in ℂ^{n}},
∪  ∪
Ω  ∣ ⟶  𝒩(Ω)

through problems that we shall pose in a moment. First of all, we give typical examples of ℱχ_{Ω}, 𝒩(Ω) and 𝒩(Ω)_{ℝ}.

Example 1.1.4. Let Ω be a unit ball in ℝ^{n}. Then we have

(1.1.4)(a)  ℱχ_{Ω}(ζ) = (2π)^{\frac{n}{2}} \frac{J\frac{n}{2} (t)}{t^{\frac{n}{2}}},

where t = (ζ_{1}^{2} + ⋯ + ζ_{n}^{2})^{\frac{1}{2}} for ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n} and J_{ν}(t) denotes the ν-th Bessel function. Let j_{m} ≡ j (\frac{n}{2}, m) (m = 1, 2, …) be the enumeration of the positive zeros of J\frac{n}{2} (t) (see Theorem (9.3)(7)). Then we have a disjoint union

(1.1.4)(b)  𝒩(Ω) = ∐_{m=1}^{∞} {ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n} : ∑_{k=1}^{n} ζ_{k}^{2} = j_{m}^{2}} .

Consequently, 𝒩(Ω)_{ℝ} = 𝒩(Ω) ∩ ℝ^{n} consists of countably many concentric hyperspheres. See the figure (1.1.6)(a) which is a kind of ‘Newton ring’ appearing in the interface of electron wave. The derivation of the formula (1.1.4)(a) is easy, but we shall indicate four methods in Appendix §9 because they are highly illustrative for later generalizations. In fact, the method (9.4.1) of a Radon transform leads to Lemma (3.14) which is key to an asymptotic result on 𝒩(Ω) in Theorem (3.15); the method (9.4.2) of the polar coordinate leads to Theorem (5.12) which deals with a perturbation of a ball; the approach (9.4.3) of a differential equation can be seen in the relation of an overdetermined Neumann problem in Theorem (1.4.1).

Example 1.1.5. Let Ω be a cubic domain {x = (x_{1}, …, x_{n}) ∈ ℝ^{n} : |x_{j}| < 1, 1 ≤ j ≤ n}. Then by an argument of one variable, we have

ℱχ_{Ω}(ζ) = ∏_{k=1}^{n} \frac{2 sin ζ_{k}}{ζ_{k}} ,

where ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n}. Hence we have

𝒩(Ω) = ⋃_{k=1}^{n} ⋃_{m∈ℤ∖{0}} {ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n} : ζ_{k} = πm} .

See the figure (1.1.6)(b) for 𝒩(Ω)_{ℝ} in ℝ^{2}.

If Ω ⊂ ℝ^{n} is centrally symmetric, then ℱχ_{Ω}(ζ) takes real values for ζ ∈ ℝ^{n}. In particular, 𝒩(Ω)_{ℝ} is of codimension 1 in ℝ^{n} and we can view 𝒩(Ω)_{ℝ} in the plane ℝ^{2} as a hypersurface (i.e. a curve) when n = 2. Here are some examples:

(1.1.6) (a) Ω: unit disk in ℝ^{2}. Ω = {(x_{1}, x_{2}) ∈ ℝ^{2}: x_{1}^{2} + x_{2}^{2} < 1}.

(1.1.6) (b) Ω: square. Ω = {(x_{1}, x_{2}) ∈ ℝ^{2}: |x_{1}| < 1, |x_{2}| < 1}.

(1.1.6) (c) Ω: regular hexagon.

In the figures above, the black parts mean {(ζ_{1}, ζ_{2}) ∈ ℝ^{2}: ℱχ_{Ω}(ζ_{1}, ζ_{2}) > 0} and the white ones mean {(ζ_{1}, ζ_{2}) ∈ ℝ^{2}: ℱχ_{Ω}(ζ_{1}, ζ_{2}) < 0}. The null variety 𝒩(Ω)_{ℝ} is the boundary of the black parts and white ones. We remark that the ‘first’ zero point set of ℱχ_{Ω}(ζ_{1}, ζ_{2}) in (1.1.6) (c) looks like a circle, but is not actually a circle (cf. Example (1.2.12)(i)).

Observation 1.1.7. Let us give some elementary observations of the Figure (1.1.6), which lead us to suitable formulations for the study of the null variety 𝒩(Ω). a) All of the domains Ω in (1.1.6) are centrally symmetric.

b) All of the domains Ω in (1.1.6) are convex and only the domain in (1.1.6) (a) (a ball) is strictly convex.

c) The null variety 𝒩(Ω)_{ℝ} for a ball (1.1.6) (a) consists of infinitely many connected components, any of which is compact.

d) The null variety 𝒩(Ω)_{ℝ} for a cubic domain (1.1.6) (b) is noncompact and connected.

The feature (c) will be generalized for strictly convex domains – not merely convex – in ℝ^{n} and SO_{0}(n − 1)/SO(n) (see Theorem (3.15) and Theorem (8.5)).

Another interesting observation is that it looks much easier to distinguish the shapes of three null varieties 𝒩(Ω)_{ℝ} in (1.1.6) than to distinguish those of the original domains Ω. From the viewpoint of computer science, we might expect a new method of ‘recognition of shape’ (‘shape’ could involve some quantities in differential geometry) in some family of domains by using the null varieties 𝒩(Ω) via the Fourier transform. A natural question arises about the injectivity of 𝒩 which we shall discuss in Problem B in a moment.

Another observation is that 𝒩(Ω)_{ℝ} in the Figures (1.1.6) has a symmetry according to that of Ω. Conversely, we might expect that all the symmetry of 𝒩(Ω) comes from that of Ω. A precise formulation will be made in Problem (B.3) after we prepare a formal property of 𝒩 under the action of the affine transformation group.

In order to explain the invariance of an affine subgroup, we prepare some terminologies. We denote the affine transformation group of ℝ^{n} by a semi-direct product:

AF (ℝ^{n}) = GL(n, ℝ) ⋉ ℝ^{n}.

The group AF (ℝ^{n}) acts on ℝ^{n} and ℂ^{n} in the following way:

g · x = ax + b,

for g = ab ∈ AF (ℝ^{n}), a ∈ GL(n, ℝ), b ∈ ℝ^{n} and x ∈ ℝ^{n} (or ∈ ℂ^{n}). For a subset S ⊂ ℝ^{n} (or ⊂ ℂ^{n}), we set a subset of ℝ^{n} (or ℂ^{n}) by

g · S := {g · x: x ∈ S}.

Let

(1.1.8) (a)  π: AF (ℝ^{n}) ⟶ AF (ℝ^{n})/ℝ^{n} ≃ GL(n, ℝ)

be the natural quotient homomorphism. Suppose Ω is a bounded measurable set in ℝ^{n} and g = ab ∈ AF (ℝ^{n}), a = π(g) ∈ GL(n, ℝ). Then we have

ℱχ_{g·Ω}(ζ) = ∫_{ℝ^{n}} χ_{g·Ω}(x)e^{\sqrt{−1} (x,ζ)}dx
= ∫_{ℝ^{n}} χ_{Ω}(y)e^{\sqrt{−1} (ay+b,ζ)}(det a)dy

= (det a)e^{\sqrt{−1} (b,ζ)}ℱχ_{Ω}(^{t}aζ). (1.1.8) (b)

Since (det a)e^{\sqrt{−1} (b,ζ)} is a non-vanishing function of ζ ∈ ℂ^{n}, we have

(1.1.8) (c)  𝒩(g · Ω) = ^{t}π(g)^{−1}𝒩(Ω).

In particular, a parallel translation of Ω does not affect the null variety 𝒩(Ω). Retaining notation as above, we put

(1.1.8) (d)  AF (Ω) := {g ∈ AF (ℝ^{n}): g · Ω = Ω},
(1.1.8) (e)  GL(𝒩(Ω)) := {g ∈ GL(n, ℝ):^{t}g^{−1} · 𝒩(Ω) = 𝒩(Ω)}.

Then the formula (1.1.8) (c) implies an inclusive relation of groups:

(1.1.8) (f)  π(AF (Ω)) ⊂ GL(𝒩(Ω)).

Now we are ready to state our basic problems.

Problem A. “How does the null variety 𝒩(Ω) look?

— That is, we want to describe the null variety 𝒩(Ω), in particular an asymptotic behavior of 𝒩(Ω), in terms of some geometric quantities of Ω or of the boundary ∂Ω. We shall give an answer generalizing a typical example (1.1.6)(a) when Ω is strictly convex domain in ℝ^{n} or SO_{0}(n, 1)/SO(n). See Theorem (3.15) in §3 and Theorem (8.5) in §8. See also Conjecture (3.19). We note that there are two different kinds of definitions of convexity, that is, geodesical one and horospherical one, in SO_{0}(n, 1)/SO(n) because a horosphere is not totally geodesic. The horospherical convexity is the natural one in our study. (See Definition (7.1.7).)

Problem B (Injectivity). “Does the null variety 𝒩(Ω) determine Ω ?”

— That is, we want to know if the assignment 𝒩: Ω ⟼ 𝒩(Ω) is one to one. This naive question should require a finer formulation. First of all, as we saw in (1.1.8)(c), Ker(π: AF (ℝ^{n}) → GL(n, ℝ)) = the group of the parallel translations does not affect the null variety. Thus, we have to treat the injectivity problem up to parallel displacements. Also the following example shows that the injectivity of 𝒩 does not hold if we permit Ω to be disconnected.

Example 1.1.9 ([Kobayashi-a, 1989], Example 1.3). Fix two positive numbers A and B such that A ≥ 5B. Set

K := [\frac{A − B}{2B} ] ∈ ℕ (Gaussian integer).

For each integer j = 1, 2, …, K we put

a_{j} = \frac{2j + 2}{2j + 1}A + B,  b_{j} = \frac{2j + 2}{2j + 1}A − B,
c_{j} = \frac{2j}{2j + 1}A + B,  d_{j} = \frac{2j}{2j + 1}A − B.

Then 0 < d_{j} < c_{j} < b_{j} < a_{j} for any j. We set

(1.1.9) (a)  Ω_{j} := (−a_{j}, −b_{j}) ∪ (−c_{j}, −d_{j}) ∪ (d_{j}, c_{j}) ∪ (b_{j}, a_{j}) ⊂ ℝ.

Then we have

(1.1.9) (b)  ℱχ_{Ω_{j}}(ζ) = \frac{8}{ζ} sin(Bζ) cos(Aζ) cos (\frac{Aζ}{2j + 1})

and therefore 𝒩(Ω_{1}) = 𝒩(Ω_{2}) = … = 𝒩(Ω_{K}) ⊂ ℂ. A Cartesian product of such Ω’s yields finite number of domains in ℝ^{n} having the same null variety 𝒩(Ω) ⊂ ℂ^{n} for arbitrary n.

Thus we need to find a suitable class of bounded sets in ℝ^{n}, on which the restriction of 𝒩 is injective. So we start from

Problem B.1 Is the following one to one?

𝒩: {Bounded domains in ℝ^{n}}/parallel displacements ⟶ {Analytic sets in ℂ^{n}}.

Here, by a domain we mean a connected open set as usual. If 𝒩 is injective, a second stage should be:

Problem B.2. Find an explicit inversion formula of 𝒩.

With emphasis on convex domains, Problem (B.1) involves two sub-problems:

Problem B.1.a. Is the assignment

𝒩: {strictly convex domains in ℝ^{n}} ∋ Ω ⟼ 𝒩(Ω) ∈ {analytic sets in ℂ^{n}}

one to one up to parallel displacements?

Problem B.1.b. Suppose the null variety 𝒩(Ω) for a bounded domain is the same (asymptotically) as that of a strictly convex domain. Can we conclude that Ω is convex?

Problem (B.1.a) will be solved in special cases: n = 2 (Theorem (4.1.1)); Ω is centrally symmetric and n is general (Corollary (3.28)). The idea is based on the program (3.24) in §3. Namely, we shall see that Problem (B.1.a) leads us to a problem in differential geometry:

Problem B.1.c. Is a strictly convex domain determined up to parallel displacements by the breadth function (see (3.8)) and the ratio of Gaussian curvatures of the antipodal points on the boundary?

This problem is equivalent to Problem (3.27) in §3, the uniqueness of a differential equation of the supporting function on S^{n−1} up to linear functions. The general case (n ≥ 3) remains open.

We shall give a positive answer to Problem (B.1.b) in the case n = 2 in Theorem (4.2.1). Also Problem (B.1.b) is related to a problem in combinatorics (and Fourier analysis) as follows:

Problem B.1.d. Can we recover k ∈ ℕ_{+} and a_{j} ∈ ℝ, b_{j} > 0 (1 ≤ j ≤ k) from the following discrete set?

{ζ ∈ ℂ: cos ζ + ∑_{j=1}^{k} a_{j} cos(b_{j}ζ) = 0}

Because of the same phenomenon in Example (1.1.9), we should require {b_{j}} ‘generic’ in Problem (B.1.d). See [Dickson, 1957] for further references. See also Lemma (4.2.5) for an example of applications to our problems.

From the viewpoint of symmetry of an affine subgroup, Problem (B.1) involves the following:

Problem B.3. Recall notation (1.1.8). Does it hold

π(AF (Ω)) = GL(𝒩(Ω))?

That is, we want to tell the symmetry of the original domain Ω from that of the null variety 𝒩(Ω). A partial answer will be given in Corollary (3.29) when Ω is a strictly convex domain in ℝ^{n} and π(AF (Ω)) ∋ diag(−1, −1, …, −1).

So far we have tried to formulate weaker subproblems of Problem B because of its difficulty. On the other hand, it is also important to consider a stronger formulation in some special cases such as GL(𝒩(Ω)) = O(n). Here is a way to give stronger versions.

Problem B′. “Does a ‘component’ of the null variety 𝒩(Ω) determine Ω ?

This is still a vague formulation. For more precision, we state two interesting cases:

Problem B′.3 (cf. Problem B.3). If there exists a connected component 𝒩(Ω)_{α} of 𝒩(Ω) such that GL(𝒩(Ω)_{α}) = O(n), then is it true that π(AF (Ω)) = O(n) ?

Here GL(𝒩(Ω)_{α}) is the notation defined similarly to (1.1.8)(e) in an obvious sense. This problem is related to some classical problems in different fields of analysis, which we shall explain in §1.4. Problem (B′.3) is called a Schiffer conjecture (see Problem (1.3.2)) in its equivalent form in a free boundary problem, and is called a Pompeiu problem (see Problem (1.2.11)′) in its equivalent form in integral geometry. Another interesting ‘component’ of 𝒩(Ω) is the ‘first’ zero points of ℱχ_{Ω}(ζ), which should be the irreducible component nearest to the origin. Professor T. Oshima asked the following:

Problem B′.1.a (cf. Problem (B.1.a)). Do the ‘first’ zero points of ℱχ_{Ω}(ζ) determine a strictly convex domain in ℝ^{n} up to parallel displacements?

Although we have not given a precise definition of the set of ‘first’ zero points of ℱχ_{Ω}(ζ), we expect that it should be a component of 𝒩(Ω) whose intersection with S^{n−1} ×_{ℤ_{2}}

ℂ^{×} is 𝒩_{1} in the notation of Conjecture (3.19). Results in [Berenstein, 1980] Proposition 1 and [Aviles, 1986] (see Theorem (1.4.3)) may be interpreted as an analogue of Problem (B′.1.a) in eigenvalue problems. Theorem (5.12) and Corollary (5.13) can be regarded as results in this direction, which deals with a perturbation of a ball.

So far, we have given some formulations and posed some problems to study the assignment 𝒩. Some of them are vague at present but will be formulated more precisely in later sections.

1.2 Integral geometry and Pompeiu problem

In this section we discuss so-called ‘integral geometry’, by which we mean here the general problem of determining a function on a manifold by means of its integrals over certain family of submanifolds.

Let X be a manifold and ℳ_{λ} (λ ∈ Λ) be a family of submanifolds embedded in X. Suppose that each ℳ_{λ} is equipped with a measure dσ_{λ}. Consider the following integral transform

(1.2.1)  T: Γ(X) ⟶ Γ(Λ), f ⟼ T f,

where (T f)(λ) := ∫_{ℳ_{λ}} f(x)dσ_{λ}. Here Γ(X) stands for a certain class of function space on X. Γ(Λ) may be a different class of function spaces. Basic subjects in integral geometry are an understanding of the following problems:

Problem 1.2.2. Retain the above setting.

a) Find Ker T. a-1) If Ker T = 0, then find an explicit inversion formula of T. a-2) If Ker T = 0, then find minimal determining subsets of Λ.

b) Find Im T (Paley-Wiener type theorem).

c) Find functorial properties of T.

d) Find an L^{2}-correspondence (Plancherel type theorem).

The research which seeks explicit inversion formulae (see Problem (1.2.2)(a-1)) has been developed largely by I. M. Gel’fand’s school and by S. Helgason after [Funk, 1916], [Radon, 1917] and etc. (See also Notes of [Helgason, 1984], Chapter I for historical remarks). On the other hand, the search for minimal determining sets of integrals (see Problem (1.2.2)(a-2)) are emphasized in an expository paper [Zalcman, 1980] without seeking explicit inversions (see also the references therein). In [Gel’fand-Graev-Shapiro, 1967] the research is carried out in seeking both minimal determining sets of integrals and explicit inversions, in which there seems to be room for further developments. The L^{2}- correspondence (d) is sometimes a direct consequence of (a-1) and (c) of Problem (1.2.2). Another important branch of integral geometry is a representation of solutions of differential equations as the image of T (see Problem (1.2.2)(b),(c)) such as the Penrose transform. But we do not enter it here.

Our concern here is mainly with the injectivity of T without seeking explicit inversions. We call Λ (or, more precisely, (Λ, T)) good for Γ(X) if Ker T = {0}. As we shall see in Example (1.2.12), the class of function space Γ(X) is important in the injectivity problem.

With emphasis on a group action, we shall often consider the following setting of homogeneous spaces:

Setting 1.2.3. Let X = G/H be a homogeneous space of a Lie group G with an isotropy subgroup H. Fix a subset S ⊂ X and a measure dσ on S. We put

(1.2.3) (a)  G_{S} := {g ∈ G: g · S = S and g_{∗}dσ = dσ},

(1.2.3) (b)  Λ_{S} := G/G_{S},

(1.2.3) (c)  ℳ_{λ} := g^{−1} · S, dσ_{λ} := g_{∗}^{−1}dσ for λ := g^{−1} · S ∈ Λ_{S} = G/G_{S}.

In the above setting, the integral transform (1.2.1)

T: Γ(G/H) ⟶ Γ(G/G_{S}), (1.2.4)

T f(g) = ∫_{ℳ_{λ}} f(x)dσ_{λ}(x) = ∫_{g^{−1}·S} f(x)(g_{∗}^{−1}dσ)(x) = ∫_{S} f(gx)dσ(x),

is a G-homomorphism when G acts on both spaces by translation, if the integral over g^{−1} · S makes sense for Γ(G/H).

Here is a stupid example:

Example 1.2.5. In the setting (1.2.3), we let S := {o}, where o := eH ∈ G/H, and dσ = δ(o), the Dirac measure supported on o. Then (G/G_{S}, T) is good for continuous functions C(G/H). In fact, T is essentially the identity map. (For simplicity, if G/H has a G-invariant measure or equivalently if | det Ad(h)_{|𝔤/𝔥}| ≡ 1 for any h ∈ H, then G_{S} = H, Λ_{S} = G/H ≃ X and T is nothing but the identity map.)

The following two examples are classical (see also [Gindikin, 1992] for an exciting expository paper on them):

Example 1.2.6. Let 𝔽 = ℝ or ℂ and X := U(n, 𝔽) ⋉ 𝔽^{n}/U(n; 𝔽) (≃ 𝔽^{n}). We put S := 𝔽^{k} ⊂ X = 𝔽^{n} a k-dimensional subspace which is endowed with the Lebesgue measure dσ. Then Λ_{S} = G/G_{S} is a vector bundle over the Grassmann manifold G_{n,k}(𝔽) with a typical fiber 𝔽^{n−k}. Then (Λ_{S}, T) is good for C_{0}^{∞}(𝔽^{n}) and an explicit inversion formula is known ([Radon, 1917], [John, 1955], [Gel’fand-Graev-Shapiro, 1967], see also [Helgason, 1984], Chapter I §2). When k = n − 1, T is called a Radon transform. When k = 1, T is called an X-ray transform because of its applications to radiology.

Example 1.2.7 ([Funk, 1916]). Let X be the points on S^{2} and Λ be the set of great circles on S^{2}. That is, X = S^{2} ≃ O(3)/O(2), Λ = ℙ^{2}ℝ ≃ O(3)/O(1) × O(2). Put Γ(X) := C^{∞}(S^{2})^{ℤ_{2}} = {symmetric functions on S^{2}}. Funk proved that Λ = ℙ^{2}ℝ is good for C^{∞}(S^{2})^{ℤ_{2}}.

Furthermore we have the following examples:

Example 1.2.8 (two radius theorem). Let G = O(n) ⋉ ℝ^{n}, X = G/O(n) (≃ ℝ^{n}) and

S(r) ≡ S(0: r) := {x ∈ ℝ^{n}: ∑_{j=1}^{n} x_{j}^{2} = r^{2}} (r > 0)

endowed with the O(n) invariant measure dσ which is normalized to have total mass one. Then

Λ_{r} := G/G_{S(r)} = O(n) ⋉ ℝ^{n}/O(n) (≃ ℝ^{n})

and the integral transform T_{r}: C(X) = C(ℝ^{n}) → C(Λ_{r}) = C(ℝ^{n}) is given by

(T_{r}f)(a) = \frac{1}{Vol(S^{n−1})} ∫_{S^{n−1}} f(a + rω)dω.

Then Λ_{r} is not good for C(ℝ^{n}). However, if r_{1}, r_{2} > 0 and \frac{r_{1}}{r_{2}} is not a quotient of zeros of J_{\frac{n}{2} −1}(z), then Λ_{r_{1}} ∐ Λ_{r_{2}} (≃ ℝ^{n} ∐ ℝ^{n}) is good for C(X), that is, (T_{r_{1}}, T_{r_{2}}): C(X) → C(Λ_{r_{1}}) + C(Λ_{r_{2}}) is injective.

Next, we put S̃(r) := S(r) ∐{0} equippd with a measure dσ − δ(x), where δ(x) is the Dirac delta function in ℝ^{n}. Then G_{S̃(r)} ≃ G_{S(r)} = O(n) and the integral transform T̃_{r}: C(X) = C(ℝ^{n}) → C(Λ_{r}) = C(ℝ^{n}) is given by (T̃_{r}f)(a) = \frac{1}{Vol(S^{n−1})} ∫_{S^{n−1}} f(a + rω)dω − f(a). Thus, Ker T̃_{r} ⊃ ℋ(X) ≡ ℋ(ℝ^{n}) = {f ∈ C^{∞}(ℝ^{n}): Δf = 0}, the space of harmonic functions (Gauss’ mean-value theorem). It is known that Λ_{r_{1}} ∐ Λ_{r_{2}} is good for C(X)/ℋ(X), that is, Ker T̃_{r_{1}} ∩ Ker T̃_{r_{2}} = ℋ(X) if \frac{r_{1}}{r_{2}} is not a quotient of zeros of

\frac{2^{\frac{n}{2} −1}Γ(\frac{n}{2})J_{\frac{n}{2} −1}(z)}{z^{\frac{n}{2} −1}} − 1. ([Delsarte, 1958], see [Zalcman, 1980] for an extensive history. See also [Berenstein-Zalcman, 1976], [Berenstein-Taylor, 1976], [Thompson-Schonbek, 1980], [Zalcman, 1972]).

Example 1.2.9. Let X = O(n, 1)/O(n), the n-dimensional hyperbolic space (see §7.2). As a generalization of Example (1.2.6) we take S to be a k-dimensional totally geodesic submanifold of X which is endowed with the measure induced from a G = O(n, 1) invariant metric on X. Then Λ_{S} ≃ O(n, 1)/O(n − k) × O(k, 1) and (Λ_{S}, T) is good for C_{0}^{∞}(X). An explicit inversion formula is also known ([Helgason, 1959], [Helgason, 1990], [Berenstein-Tarabusi, 1991]. See also [Gel’fand-Graev-Vilenkin, 1962] Chapter 6 §4.6).

In a hyperbolic space a totally geodesic submanifold is one analogue of a subspace in an Euclidean space, while a holosphere is another analogue, which can be regarded as the limit of a sphere whose radius goes to infinity. A holosphere can be also defined for a more general homogeneous space as an orbit of an unipotent subgroup (see [Gel’gand-Graev, 1959]). Integral geometry for horospheres (see Problem (1.2.2)) has played an important (essential in some cases) role in harmonic analytic on homogeneous spaces. We explain its flavor in the following example. The readers who are not familiar with representation theory of a semisimple Lie group should postpone reading the following example until §6.5.

Example 1.2.10. Let G be a connected semisimple Lie group, σ an involutive automorphism of G, and H be an open subgroup of the fix points of σ. Then G/H is called a semisimple symmetric space. Take a Cartan involution θ of G which commutes with σ. Let 𝔞 be a maximal abelian subspace in {X ∈ 𝔤 = Lie (G) : θX = σX = −X}. Then Σ(𝔤, 𝔞) satisfies the axioms of root systems, and is called a restricted root system of (𝔤, 𝔥). Let P = MÃN ⊂ G be a Langlands decomposition of a parabolic subgroup of G corresponding to a positive system Σ^{+}(𝔤, 𝔞). Here Lie(Ã) ⊃ 𝔞. We put S := NH ⊂ G/H and we endow S with an N-invariant measure. S is called a horosphere (see also §7.1). In the case of σ = θ, G/H = G/K is a Riemannian symmetric space, P is a minimal parabolic subgroup and G_{NH} = MN. In the case of a complex group manifold, that is, G = G′ ×G′, σ(x, y) = (y, x) for x, y ∈ G′ where G′ is a complex semisimple Lie group, P is a Borel subgroup of G = G′ × G′. In either case, the inversion formula of the Fourier transform is explicitly known ([Gindikin-Karpelevič, 1962]; [Harish-Chandra, 1954], [Gel’fand-Graev, 1955], respectively) for C_{0}^{∞}(G/H). In particular, (G/G_{NH}, T) is good for C_{0}^{∞}(G/H). See also [Helgason, 1984] for an explicit inversion formula of T in the case of Riemannian symmetric spaces. On the other hand, in the case of complex semisimple group manifolds, the approach in [Gel’fand-Graev, 1955] is more direct to use an inversion formula of T. Recently, it has been announced that (G/G_{NH}, T) is good for C_{0}^{∞}(G/H) ([van den Ban-Schlichtkrull, in preparation]) for a general semisimple symmetric space G/H. In this general case, the assumption of compact supports for the function space C_{0}^{∞} should be important for the injectivity of T in view of the Plancherel formula announced by T. Oshima which may involve also less continuous series. From the above viewpoint of representation theory, it does not seem natural to try to find an inversion formula of T in a general case. On the other hand, it would be interesting to understand their (abstract) injectivity result of T for C_{0}^{∞}(G/H) by seeking a simpler and more direct proof.

So far we have treated some special cases of Setting (1.2.3) where S is of lower dimension. Now let us consider the following special case of Setting (1.2.3) where S is of positive volume with respect to a G-invariant measure on X = G/H.

Definition 1.2.11. Let G be a Lie group and H closed subgroup of G. Suppose that a homogeneous space X = G/H has a G-invariant measure dx. Let S = Ω be a measurable relatively compact set in X such that Vol(Ω) = ∫_{Ω} dx > 0. Then the transform (1.2.4)

T ≡ T_{Ω} : L_{loc}^{1}(G/H) ⟶ C(G/G_{Ω})

is given by (T_{Ω}f)(g) = ∫_{Ω} f(gx)dx. The injectivity of T_{Ω} depends on both Ω and a class of function spaces Γ(G/H). We call Ω a Pompeiu set if one of (and any of) the following equivalent conditions holds:

Ker(T_{Ω}: L_{loc}^{1}(G/H) → C(G/G_{Ω})) = {0},
Ker(T_{Ω}: C(G/H) → C(G/G_{Ω})) = {0},
Ker(T_{Ω}: C^{∞}(G/H) → C(G/G_{Ω})) = {0}.

The above equivalence follows from a standard smoothing procedure. We call Ω is an L^{p}- Pompeiu set if Ker(T_{Ω}: L^{p}(G/H) → C(G/G_{Ω})) = {0} (1 ≤ p ≤ ∞). Note that a Pompeiu set is an L^{p}-Pompeiu set for any 1 ≤ p ≤ ∞. We call G/H is an L^{p}-Pompeiu homogeneous space (1 ≤ p ≤ ∞) if any measurable relatively compact set Ω with a positive measure is an L^{p}-Pompeiu set. A group manifold G′ with both left and right actions can be regarded as a symmetric space G/H = G′ × G′/ diag G′. In this case, G′ is also called an L^{p}- Pompeiu group ([Carey-Kaniuth-Moran, 1991]) if the symmetric space G′ × G′/ diag G′ is an L^{p}-Pompeiu homogeneous space in our sense.

We begin with a typical example ℝ^{n} ≃ ℝ^{n}/{0} ≃ O(n) ⋉ ℝ^{n}/O(n).

Example 1.2.12 (ℝ^{n} case). I) First, we take G = ℝ^{n} and H = {0} so that G/H ≃ ℝ^{n}. The following sets Ω_{1} and Ω_{2} are not Pompeiu sets.

i) Ω_{1} ≡ Ω_{1}(r_{1}, …, r_{n}) := \{(x_{1}, …, x_{n}) ∈ ℝ^{n} : ∑_{j=1}^{n} \frac{x_{j}^{2}}{r_{j}^{2}} < 1\}, an ellipsoid

with principal radii r_{1}, …, r_{n} > 0. ii) Ω_{2} ≡ Ω_{2}(r_{1}, …, r_{n}) := {(x_{1}, …, x_{n}) ∈ ℝ^{n}: |x_{j}| < r_{j}(1 ≤ j ≤ n)}, for

r_{j} > 0.

In fact, Ω_{1} and Ω_{2} are not L^{∞}-Pompeiu sets for ℝ^{n}/{0}, because

u (λ(∑_{j=1}^{n} \frac{x_{j}^{2}}{r_{j}^{2}} )^{\frac{1}{2}} ) ∈ Ker T_{Ω_{1}} ∩ L^{∞}(ℝ^{n}).

∏_{j=1}^{n} sin \frac{k_{j}πx_{j}}{r_{j}} ∈ Ker T_{Ω_{2}} ∩ L^{∞}(ℝ^{n}).

Here, u(t) := t^{− \frac{n}{2} +1}J_{\frac{n}{2} −1}(t) is an entire function of t ∈ ℂ whose power series involves only even powers of t (Theorem (9.3)(1)), λ > 0 is any zero of J\frac{n}{2} and k_{j} ∈ ℤ (1 ≤ j ≤ n). It follows from (i) or (ii) that ℝ^{n}/{0} is not an L^{∞}-Pompeiu homogeneous space. This also means that ℝ^{n} is not an L^{∞}-Pompeiu group because ℝ^{n} is abelian. On the other hand, ℝ^{n} is an L^{1}-Pompeiu group. To see this, let us fix a bounded measurable set Ω in ℝ^{n}. Then the condition f ∈ Ker T_{Ω} implies a convolution equation

(1.2.12)(a)  χ_{Ω^{∨}} ∗ f ≡ 0,

where Ω^{∨} = {x ∈ ℝ^{n} : −x ∈ Ω}. If f ∈ Ker T_{Ω} ∩ L^{1}(ℝ^{n}), then we take the Fourier transform of (1.2.10)(a) to have

ℱχ_{Ω^{∨}}(ζ) ℱf(ζ) = 0.

Because ℱχ_{Ω^{∨}}(ζ) is a non-zero entire function of ζ ∈ ℂ^{n} and because ℱf(ζ) is continuous with respect to ζ ∈ ℝ^{n}, we conclude ℱf ≡ 0. Hence Ker T_{Ω} ∩ L^{1}(ℝ^{n}) = {0} for any Ω, that is, ℝ^{n} is an L^{1}-Pompeiu group.

II) Second, we take G = M(n) := O(n) ⋉ ℝ^{n} and H = O(n) so that G/H ≃ ℝ^{n}. Then the group G is larger than that of (I) by rotations. The following sets in (iii) and (iv) are not L^{∞}-Pompeiu sets for O(n) ⋉ ℝ^{n}/O(n).

iii) Ω_{1}(r, …, r) (ball),

iv) Ω_{1}(λ_{1}r, …, λ_{1}r)∖Ω(λ_{2}r, …, λ_{2}r) (annulus), where r > 0 and λ_{1} > λ_{2} >

0 are the zeros of J_{\frac{n}{2} (λ)}.

We should remark that there exist L^{∞}-Pompeiu sets for O(n) ⋉ ℝ^{n}/O(n) which are not L^{∞}-Pompeiu sets for ℝ^{n}/{0} if n ≥ 2. The following sets in (v) and (vi) are such examples:

v) Ω_{1}(r_{1}, …, r_{n}) (ellipsoid) unless r_{1} = … = r_{n},
vi) Ω_{2}(r_{1}, …, r_{n}) (box) for any r_{1}, …, r_{n} > 0.

Even though sets in ℝ^{n} are more likely to be Pompeiu sets for O(n) ⋉ ℝ^{n}/O(n) than for ℝ^{n}/{0} as we saw in (v) and (vi), O(n) ⋉ ℝ^{n}/O(n) is not an L^{∞}-Pompeiu homogeneous space but an L^{1}-Pompeiu homogeneous space, just like ℝ^{n}/{0}.

The above example suggests some directions to study. Here are such problems (Problem (1.2.13), (1.2.15)).

Problem 1.2.13. Classify Pompeiu sets for a homogeneous space G/H (see the setting in Definition (1.2.11)).

In particular, if G/H = O(n)⋉ℝ^{n}/O(n) (≃ ℝ^{n}), Problem (1.2.13) is called Pompeiu problem after a Rumanian mathematician Dimitrie Pompeiu ([Pompeiu, 1929-a], [Pompeiu, 1929-b]):

Problem 1.2.13′ (Pompeiu problem). Classify Pompeiu sets for O(n) ⋉ ℝ^{n}/O(n).

Example 1.2.14. Throughout this example, we assume G/H = O(n) ⋉ ℝ^{n}/O(n). A ball is not a Pompeiu set (see (1.2.12)(iii)). A combination of balls with a special ratio of radii can yield a non-Pompeiu set (e.g. (1.2.12)(iv)). See [Williams, 1976], page 184 for some more examples of this type. As far as the author knows, these are the only examples which have been proved not to be Pompeiu sets so far.

On the other hand, it is known that the following sets are Pompeiu sets (see also (1.2.12)(v)(vi)).

i) A polyhedron ([Brown-Schreiber-Taylor, 1973], Theorem (5.11); see also [Williams, 1976]).

ii) A ‘thin’ convex domain (n = 2). (see §4.3 for a precise formulation due

to [Brown-Kahane, 1982].) iii) A domain surrounded by an ‘algebraic curve’ (n = 2). (See §4.4 for a

precise statement due to [Garofalo-Segala, 1991].)

The above examples are evidence in support of the following old conjecture:

Conjecture 1.2.15. The ball is the only bounded domain with a connected boundary in ℝ^{n} which is not a Pompeiu set for O(n) ⋉ ℝ^{n}/O(n).

For another direction of the research, we recall that ℝ^{n} × ℝ^{n}/ diag(ℝ^{n}) and O(n) ⋉ ℝ^{n}/O(n) are L^{1}-Pompeiu homogeneous spaces. This feature can be generalized to a 1connected solvable Lie group and a Riemannian symmetric space of the noncompact type, respectively (see Theorem (1.2.17)(a), (e)). Thus we ask (see also [Berenstein, 1987] §4):

Problem 1.2.16. Which homogeneous space is an L^{p}-Pompeiu homogeneous space (in particular p = 1 or p = 2)?

This problem is closely related to the (abstract) Plancherel theorem for G/H. We collect partial solutions to Problem (1.2.16):

Theorem 1.2.17 (L^{p}-Pompeiu homogeneous space).

a) A (non-trivial) compact Lie group is not an L^{1}-Pompeiu group ([Carey-Kaniuth-Moran, 1991], Lemma 1.2).

b) A 1-connected solvable Lie group is an L^{1}-Pompeiu group ([Carey-Kaniuth-Moran, 1991], Corollary 3.4).

c) SL(2, ℝ) is not an L^{1}-Pompeiu group ([Sitaram, 1984]).

d) SL(2, ℂ) is not an L^{1}-Pompeiu group ([Scott-Sitaram, 1988]).

e) A Riemannian symmetric space G/K of the non-compact type is an L^{1}- Pompeiu homogeneous space.

f) A semisimple symmetric space G/H with rank G/H = rank K/H ∩ K is not an L^{p}-Pompeiu homogeneous space (1 ≤ p ≤ ∞).

f′) An indefinite Stiefel manifold U(r, q: 𝔽)/U(r − m, q: 𝔽) (𝔽 = ℝ, ℂ, ℍ, r ≥ 2m) is not an L^{p}-Pompeiu (1 ≤ p ≤ ∞).

The last two cases (f) and (f)′ are new results due to the author. A group manifold SU(p, q) and a homogeneous space GL(p + q, ℝ)/GL(p, ℝ) × GL(q, ℝ) are such examples.

We shall discuss the non-L^{p}-Pompeiu property (1 ≤ p ≤ ∞) for a general semisimple symmetric space which is not a Riemannian symmetric space (that covers (a), (c), (d), (f)) in another paper by an argument based on the study of the fundamental series for G/H (see [Flensted-Jensen, 1986], Chapter VI) and on the following simple observation (Lemma (1.2.18)).

Lemma 1.2.18. Let K be a compact subgroup of G. If there exists a non-trivial G-stable closed subspace V ⊂ L^{p}(G/K) such that V^{K} := {v ∈ V: k · v = v for any k ∈ K} = {0}, then any K-invariant measurable relatively compact set Ω with a positive measure is not an L^{p}-Pompeiu set (such sets exist).

Proof. Let us show V ⊂ Ker T_{Ω}. We write π for the regular representation of G on L^{p}(G/K). Take f ∈ V and g ∈ G. We put f_{1} := π(g^{−1})f ⊂ V. Let normalize a Haar measure dk on K so that ∫_{K} dk = 1. Then we have

(T_{Ω}f)(g) = ∫_{Ω} f(gx)dx = ∫_{K} ∫_{Ω} f(gkx)dk dx

= ∫_{K} ∫_{Ω} f_{1}(kx)dk dx = ∫_{Ω}(∫_{K}(π(k^{−1})f_{1})(x)dk)dx

= 0

Thus f ∈ Ker T_{Ω}. □

1.3 A “free boundary” problem.

The study of relations between the geometry of a given domain and the spectrum of the Laplace operator is very old. We treat a special case of this problem. Let us begin with the setting of a homogeneous space. Let H be a closed subgroup of a Lie group G. We denote by 𝔻(G/H) the totality of G-invariant differential operators on a homogeneous space G/H. Naturally, 𝔻(G/H) has a structure of ℂ-algebra. (For some basic facts on the structure of 𝔻(G/H), see §6.)

For example, we regard the Euclidean space ℝ^{n} as different homogeneous spaces

ℝ^{n}(= ℝ^{n}/{0}) ≃ M(n)/O(n) ≃ AF (ℝ^{n})/GL(n, ℝ),

where M(n) = O(n) ⋉ ℝ^{n} ⊂ AF (ℝ^{n}) = GL(n, ℝ) ⋉ ℝ^{n}. Correspondingly, we have

𝔻(ℝ^{n})  = ℂ [\frac{∂}{∂x_{1}} , …, \frac{∂}{∂x_{n}} ] ,
∪
𝔻(M(n)/O(n))  = ℂ[Δ_{ℝ^{n}}],
∪
𝔻(AF (ℝ^{n})/GL(n, ℝ)) = ℂ,

where Δ_{ℝ^{n}} = ∑_{j=1}^{m} \frac{∂^{2}}{∂x_{j}^{2}} is the Laplace-Beltrami operator on ℝ^{n}.

In this section, our interest is in the O(n)-invariance of a solution of differential equations which are defined by Δ_{ℝ^{n}} ∈ 𝔻(M(n)/O(n)). In the spherical coordinate x = rω (r ∈ ℝ^{×}, ω ∈ S^{n−1}) on ℝ^{n}, the Laplacian Δ_{ℝ^{n}} takes the form

Δ_{ℝ^{n}} = \frac{∂^{2}}{∂r^{2}} + \frac{n − 1}{r} \frac{∂}{∂r} + \frac{1}{r^{2}} Δ_{S^{n−1}},

where Δ_{S^{n−1}} is the Laplacian on the unit sphere with the standard metric. We put

u_{λ}(x) := \frac{J_{\frac{n}{2} −1}(λr)}{(λr)^{\frac{n}{2} −1}}, where r = (x_{1}^{2} + ⋯ + x_{n}^{2})^{\frac{1}{2}} (cf. Example (1.2.12)(i)). Then we have

(see Theorem (9.3)(2))

Δ_{ℝ^{n}}u_{λ} = −λ^{2}u_{λ}.

By virtue of a recurrence relation \frac{d}{dz} (z^{−ν}J_{ν}(z)) = −z^{−ν}J_{ν+1}(z) (see Theorem (9.3)(3)),

we have \frac{∂}{∂r} u_{λ}(rω) = \frac{−λJ_{\frac{n}{2}} (λr)}{(λr)^{\frac{n}{2} −1}}. Therefore, if λ ≠ 0 is a zero of J\frac{n}{2} (λ), then u_{λ} ∈ C^{∞}(ℝ^{n})

is a solution of the Neumann problem in Ω = B(0 : 1) := {x ∈ ℝ^{n}: ∑_{j=1}^{n} x_{j}^{2} < 1}:

(1.3.1)  \{ \begin{matrix}Δ_{ℝ^{n}}u + αu = 0 & in Ω, \\ \frac{∂u}{∂ν} = 0, u ≡ constant & on ∂Ω. \end{matrix}

Here \frac{∂}{∂ν} denotes the outer normal vector field on ∂Ω. In this case, α = λ^{2}. Since J\frac{n}{2} (λ) has countably many (real) zeros (Theorem (9.3)(7)), there are countably many eigenvalues −α with non-trivial solutions to (1.3.1).

Conversely, it is a problem to show for which domains Ω there exist non-trivial solutions to (1.3.1). The following is called Schiffer’s conjecture (see [Yau, 1982], problem 80):

Problem 1.3.2. Suppose that ∂Ω is isomorphic to S^{n−1}. If there exists a non-trivial solution to (1.3.1), is Ω a ball?

This problem remains open even in the simple case n = 2. On the other hand, some related problems are solved. For example, let transform the equation (1.3.1) by the linear substitution v = \frac{u}{αc} − \frac{1}{α} (α ≠ 0) to

\{ \begin{matrix}Δv + αv = −1 & in Ω, \\ v = 0, \frac{∂v}{∂ν} = 0 & on Ω. \end{matrix}

Here c is the constant in (1.3.1). In this connection, [Serrin, 1971] and [Weinberger, 1971] proved that the existence of a positive solution to an over determined Dirichlet problem of Poisson’s equation arising from fluid dynamics:

(1.3.3)  \{ \begin{matrix}Δv = −1 & in Ω, \\ v = 0, \frac{∂v}{∂ν} ≡ constant & on ∂Ω \end{matrix}

implies that Ω must be a ball of radius nc and v = ^{\frac{n^{2}c^{2}− ∑_{j=1}^{n} x_{j}^{2}}{2n}}. In particular, v is invariant under rotations.

See theorems (1.4.2), (1.4.3) and (1.4.4) for some contributions to Problem (1.3.2). See also Theorem (1.4.1) that allows different approaches to Problem (1.3.2).

1.4 Spectral Synthesis.

So far, we have explained three topics, the null variety 𝒩(Ω) (§1.1), integral geometry (§1.2) and a free boundary problem (§1.3). In the following special case with higher symmetry, these problems are related with one another:

Theorem 1.4.1 ([Brown-Schreiber-Taylor, 1973], [Williams, 1976]). Retain notations as in §1.1 – §1.3. Suppose that Ω is a bounded domain in ℝ^{n} with a connected Lipschitz boundary ∂Ω. Then the following three conditions on Ω are equivalent:

1) Λ_{Ω} = M(n)/M(n)_{Ω} (see Setting (1.2.3)) is not good for C(ℝ^{n}). (That is, Ω is not a Pompeiu set. See Definition (1.2.11).)

2) There exists λ ∈ ℂ^{×} and u(≠ 0) ∈ C^{2}(Ω) ∩ C^{1}(Ω̄) such that

(1.4.1)(a)  \{ \begin{matrix}Δu + λu = 0 & in Ω, \\ \frac{∂u}{∂ν} = 0, u ≡ constant & on Ω. \end{matrix}

3) There exists r > 0 such that

𝒩(Ω) ⊃ S_{ℂ}(0 : r) := {ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n} : ∑_{j=1}^{n} ζ_{j}^{2} = r^{2}} .

This result enables us to take a different approach, that is, a differential equation (1.4.1)(2) and the null variety (1.4.1)(3) in order to study the Pompeiu problem which is originally in integral geometry. From an argument of the null variety 𝒩(Ω) in Theorem (1.4.1)(3) we can obtain some contributions to the Pompeiu problem (e.g. Corollary (3.21), Theorem (3.23), Theorem (4.3.2) and Theorem (4.4.1)). Together with some related problems posed in §1.1, we shall discuss this approach in later sections (§3, 4, 5, 8).

On the other hand, from an argument of the differential equation (1.4.1)(a) in Theorem (1.4.1)(2) we can obtain some results on a generic property of Pompeiu sets (Theorem (1.4.2)), on small eigenvalues (Theorem (1.4.3)), and on the analyticity of the boundary (Theorem (1.4.4)). For the rest of this section we give a brief survey of these contributions to the Pompeiu problem, though a detailed treatment is beyond our scope.

Suppose that Ω is a domain in ℝ^{n} with a smooth connected boundary ∂Ω. A trivial remark is that ∂Ω determines the domain Ω. The totality of the boundaries ∂Ω, with an appropriate Sobolev norm, forms a separable Hilbert manifold Γ. A subset of Γ is called residual if it is a countable intersection of open dense subsets of Γ. The following result asserts that a ‘generic’ simply connected domain Ω in ℝ^{n} = M(n)/O(n) is not a Pompeiu set (see Definition (1.2.11)).

Theorem 1.4.2 (see [Kobayashi, 1986], Remark (3.3)). With the notation as above, there exists a residual subset B of Γ such that for any ∂Ω ∈ B there does not exist a non-trivial solution to the over-determined Neumann problem (1.4.1)(a).

This result is a direct consequence of [Fujiwara-Tanikawa-Yukita, 1978] Theorem 1 on a conjecture of Arnold ([Arnold, 1972]) (see also [Micheletti, 1972] Teorema C), and of a simple observation that if u is a solution to (1.4.1)(a) then v := \frac{∂u}{∂x_{j}} (1 ≤ j ≤ n) is a solution to the Dirichlet problem:

(1.4.2)(a)  \{ \begin{matrix}Δv + λv = 0 & in Ω, \\ v ≡ 0 & on ∂Ω. \end{matrix}

Let 0 = ν_{1} < ν_{2} ≤ … be the eigenvalues of the Neumann problem:

\{ \begin{matrix}Δu + νu = 0 & in Ω, \\ \frac{∂u}{∂ν} = 0 & on ∂Ω. \end{matrix}

If the eigenvalues of the over-determined Neumann problem (1.4.1)(a) are ‘small’ enough, then Ω must be a ball. These kinds of results were obtained in [Berenstein, 1980] Proposition 1 and generalized in:

Theorem 1.4.3 ([Aviles, 1986]). Suppose Ω is a bounded convex domain in ℝ^{n} with C^{1} boundary. If there exists a solution to (1.4.1)(a) with λ ≤ ν_{2n+2}, then Ω is a ball. If ∂Ω has non-negative mean curvature and if λ ≤ ν_{n+3}, then the same conclusion holds.

In the above result, the regularity of the boundary ∂Ω is not the issue. In fact Williams proved the analyticity of the boundary ∂Ω of a non-Pompeiu set Ω, following the approach in [Caffarelli, 1977]:

Theorem 1.4.4 ([Williams, 1981]). Suppose that Ω ⊂ ℝ^{n} is a bounded domain with a connected Lipschitz boundary. If there exists a non-trivial solution to (1.4.1)(a) (i.e. Ω is not a Pompeiu set), then ∂Ω is real analytic.

§2 The zeros of a certain class of entire functions.

The triangular function sin z has an infinite number of zeros, which are all real and distributed in a regular fashion. Similarly, for real ν, the Bessel function J_{ν}(z) has an infinite number of real zeros (Bessel, Lommel) with a special case going back to Daniel Bernoulli and Fourier (see [Watson, 1962], p.478 and the references therein). They are all real when ν > −1 (Fourier, Lommel, Hurwitz, see [Watson, 1962], pp. 482-483). They have an asymptotic behavior

j(ν, k) = kπ + \frac{π(2ν − 1)}{4} + O(k^{−1}), as k → ∞

where j(ν, k) denotes the k-th positive zero of J_{ν}(z) (Gauss, McMahon, see [Watson, 1962], pp. 505-507 and the references therein). Similar results for other special functions have been also investigated by many people. See [Tricomi, 1947] for a unified treatment in several special functions which is based on the results of Bessel functions. In this section we give a different method to prove a generalization.

The main result in this section is Corollary (2.20) (see also Theorem (2.16) and Theorem (2.18)) which gives an asymptotic behavior of the zeros of certain class of entire functions (the Fourier transform of (2.3)) and also gives an estimate of the upper bound of ‘exceptional’ zeros. The latter part implies the reality of most of the zeros under some symmetric condition on entire functions (see Corollary (2.22) for an estimate of the number of non-real zeros). We have given essentially complete treatments of the material in this section because the author does not know convenient references of these results (at least in the form used here). The reader may read only notations (2.1)∼(2.3), statement of results cited above and Examples, if he (or she) is more interested in its application to the study of the null variety 𝒩(Ω) in §3.

Notation 2.1. Let μ ∈ ℂ such that Re μ > −1. We put

x_{+}^{μ} := \{ \begin{matrix}x^{μ} & if x > 0, \\ 0 & if x ≤ 0, \end{matrix}

x_{−}^{μ} := \{ \begin{matrix}0 & if x ≥ 0, \\ |x|^{μ} & if x < 0. \end{matrix}

The mapping μ ↦ x_{+}^{μ} extends uniquely to a meromorphic mapping from ℂ to the space 𝒮′(ℝ) of tempered distributions. The poles are the points μ = −1, −2, … and they are all simple. The residue at μ = −k (k ∈ ℕ_{+}) is given by

(2.1.1)  Res_{μ=−k} x_{+}^{μ} = \frac{(−1)^{k−1}}{(k − 1)!} δ^{(k−1)}(x).

Usually we shall use the above notation in the case of Re μ > −1 in this section, however. If Re μ > 0, then denoting by [ ] the Gauss integer, we have

(2.1.2)  x_{+}^{μ}, x_{−}^{μ} ∈ C^{[μ−1],1}(ℝ).

Here, for N ∈ ℕ we put

(2.1.3)  C^{N,1}(ℝ) := \{f ∈ C^{N}(ℝ) : \frac{d^{N}f}{dx^{N}} is locally Lipschitz\} .

For δ > 0, we define a class of functions:

(2.2)  Ψ(δ) := \{ \begin{matrix}φ ∈ C^{∞}(ℝ) : & 0) φ is real valued, \\ & i) supp φ ⊂ [−2δ, 2δ], \\ & ii) φ(x) ≡ 1 if x ∈ [−δ, δ], \\ & iii) φ(x) = φ(−x). \end{matrix}\} .

For λ ∈ ℂ and N ∈ ℕ such that Re λ + N ≥ 0, we introduce a class of functions 𝒞^{N}(λ). Here a complex valued function f: ℝ → ℂ belongs to 𝒞^{N}(λ) if and only if: (2.3) there exist −∞ < α < β < ∞, a_{j}, b_{j} ∈ ℂ (j = 0, 1, …, N), 0 < δ < \frac{1}{4}(β − α), φ ∈ Ψ(δ) (Notation (2.2)) such that the following three conditions hold.

a_{0} ≠ 0, b_{0} ≠ 0, (2.3.1) f(x) = 0 if x < α or β < x, (2.3.2)

f(x) − ∑_{j=0}^{N} a_{j}(x − α)_{+}^{λ+j}φ(x − α) − ∑_{j=0}^{N} b_{j}(x − β)_{−}^{λ+j}φ(x − β) ∈ C^{[Re λ]+N,1}(ℝ). (2.3.3)

Then we have a natural inclusion

⋯ ⊃ 𝒞^{N}(λ) ⊃ 𝒞^{N+1}(λ) ⊃ · · · .

We put

𝒞^{∞}(λ) := ⋂_{\begin{matrix}N∈ℕ \\ N≥− Re λ \end{matrix}} 𝒞^{N}(λ).

The complex numbers a_{j}, b_{j} (0 ≤ j ≤ N) are obviously independent of the choice of φ ∈ Ψ(δ) and determined by f. So we write a_{j} = a_{j}(f), b_{j} = b_{j}(f) if we want to emphasize the dependence on f. Similarly we write α = α(f), β = β(f). Put

A ≡ A(f) := β(f) − α(f).

Then the maps

a_{j}, b_{j} : 𝒞^{N}(λ) ⟶ ℂ,  (0 ≤ j ≤ N),
α, β : 𝒞^{N}(λ) ⟶ ℝ,
A : 𝒞^{N}(λ) ⟶ ℝ_{+},

are clearly compatible with the inclusion map 𝒞^{N+1}(λ) ↪ 𝒞^{N}(λ). If f ∈ 𝒞^{N}(λ) and if we put f_{p,q}(x) := f(px + q) for p > 0 and q ∈ ℝ, then we have

(2.3.4)  f_{p,q} ∈ 𝒞^{N}(λ),
a_{j}(f_{p,q}) = a_{j}(f)p^{λ+j}, b_{j}(f_{p,q}) = b_{j}(f)p^{λ+j}, (0 ≤ j ≤ N),
α(f_{p,q}) = \frac{α(f) − q}{p} , β(f_{p,q}) = \frac{β(f) − q}{p} ,
A(f_{p,q}) = \frac{A(f)}{p}.

Let us fix a ψ ∈ Ψ(1) (Notation (2.2)). We introduce a norm ∥·∥_{𝒞^{N} (λ)} on 𝒞^{N}(λ) as follows: For f ∈ 𝒞^{N}(λ),

(2.3.5) ∥f∥_{𝒞^{N} (λ)} := ∑_{j=0}^{N} A(f)^{Re λ+j}(|a_{j}(f)| + |b_{j}(f)|)

+ A(f)^{[Re λ]+N+1} sup_{α≤x≤β}|(\frac{d}{dx})^{[Re λ]+N+1} (f(x) − ∑_{j=0}^{N} a_{j}(f)(x − α)_{+}^{λ+j}φ(x − α)

− ∑_{j=0}^{N} b_{j}(f)(x − β)_{−}^{λ+j}φ(x − β))|

where we put φ(x) := ψ (\frac{5x}{A(f)}). The point here is that the definition (2.3.5) is invariant under the affine transform of ℝ. That is,

(2.3.6)  ∥f∥_{𝒞^{N} (λ)} = ∥f_{p,q}∥_{𝒞^{N} (λ)}

for any p > 0, q ∈ ℝ. Recall that f_{p,q}(x) = f(px + q). Let −∞ < α < β < ∞ and L ∈ ℕ. For a complex valued function g ∈ C^{L}([α, β]), we define a norm by

(2.3.7)  ∥g∥_{C^{L}([α,β])} := ∑_{j=0}^{L}(β − α)^{j} sup_{α≤x≤β} |(\frac{d}{dx})^{j} g(x)| .

If we write g_{p,q}(x) := g(px + q) ∈ C^{L} ([\frac{α−q}{p} , \frac{β−q}{p} ]), then we have

(2.3.8)  ∥g_{p,q}∥_{C^{L}([\frac{α−q}{p} , \frac{β−q}{p}])} = ∥g∥_{C^{L}([α,β])},

for any p > 0 and q ∈ ℝ.

Suppose λ ∈ ℂ and N ∈ ℕ satisfies Re λ + N ≥ 0. We put L := [Re λ] + N + 1. Suppose f ∈ 𝒞^{N}(λ), g ∈ C^{L}([α, β]) such that α = α(f), β = β(f) (Notation (2.3.4)) and g(α)g(β) ≠ 0. Then we have g(x)f(x) ∈ 𝒞^{N}(λ), (2.3.9)(a) α(gf) = α(f), β(gf) = β(f), A(gf) = A(f), (2.3.9)(b)

a_{j}(gf) = ∑_{k=0}^{j} \frac{1}{k!}a_{j−k}(f)g^{(k)}(α), (2.3.9)(c)

b_{j}(gf) = ∑_{k=0}^{j} \frac{(−1)^{k}}{k!} b_{j−k}(f)g^{(k)}(β), (2.3.9)(d)

∥gf∥_{𝒞^{N} (λ)} ≤ C_{0}(λ, N)∥g∥_{C^{L}([α,β])}∥f∥_{𝒞^{N} (λ)}, (2.3.9)(e) where C_{0}(λ, N) ≥ 1 is a constant depending only on λ, N and ψ ∈ Ψ(1). The estimate (2.3.9)(e) is deduced from the following formula:

g(x)f(x) − ∑_{j=0}^{N} a_{j}(gf)(x − α)_{+}^{λ+j}φ(x − α) − ∑_{j=0}^{N} b_{j}(gf)(x − β)_{−}^{λ+j}φ(x − β)

= g(x){f(x) − ∑_{j=0}^{N} a_{j}(f)(x − α)_{+}^{λ+j}φ(x − α) − ∑_{j=0}^{N} b_{j}(f)(x − β)_{−}^{λ+j}φ(x − β)}

+ ∑_{j=0}^{N} a_{j}(f)(x − α)_{+}^{λ+j}φ(x − α)\{g(x) − ∑_{k=0}^{N−j} \frac{1}{k!}g^{(k)}(α)(x − α)_{+}^{k}\}

+ ∑_{j=0}^{N} b_{j}(f)(x − β)_{−}^{λ+j}φ(x − β)\{g(x) − ∑_{k=0}^{N−j} \frac{(−1)^{k}}{k!} g^{(k)}(β)(x − β)_{−}^{k}\}.

It follows from (2.3.6) and (2.3.8) that the constant C(N, λ) does not depend on α and β.

We call f ∈ 𝒞^{N}(λ) normalized if

(2.3.10)  α(f) = 0, β(f) = 1, A(f) = 1, a_{0}(f) = b_{0}(f) = 1.

Let

(2.3.11)  f_{p,q;r,s}(x) := e^{rx+s}f_{p,q}(x) ≡ e^{rx+s}f(px + q),

for p > 0, q ∈ ℝ, r ∈ ℂ and s ∈ ℂ. f_{p,q;r,s} ∈ 𝒞^{N}(λ) if and only if f ∈ 𝒞^{N}(λ). If we choose p, q, r and s so that

(2.3.11)(a)  p = A(f), q = α(f), e^{r} = \frac{a_{0}(f)}{b_{0}(f)}, e^{s} = \frac{1}{a_{0}(f)A(f)^{λ}} ,

then f_{p,q;r,s} is normalized (see (2.3.10)). We also have a formula

(2.3.11)(b)  ℱ(f_{p,q;r,s})(ζ) = \frac{e^{s}}{p} e^{− \frac{q}{p} (r+\sqrt{−1} ζ)}ℱf (\frac{ζ − \sqrt{−1} r}{p} ) .

For r ∈ ℂ and L ∈ ℕ, we put

e(L, r) : = ∥e^{rx}∥_{C^{L}([0,1])}

= ∑_{j=0}^{L} |r|^{j} sup_{0≤x≤1} |e^{rx}|

≤ L(1 + |r|^{L}) max(1, e^{Re r}).

In the setting of (2.3.11) and (2.3.11)(a), we have from (2.3.9)(e) and (2.3.6) ∥f_{p,q;r,s}∥_{𝒞^{N} (λ)} (2.3.11)(c)

= ∥e^{rx+s}f_{p,q}(x)∥_{𝒞^{N} (λ)}

≤ C_{0}(λ, N)|e^{s}| ∥e^{rx}∥_{C^{[Re λ]+N+1}([0,1])}∥f_{p,q}∥_{𝒞^{N} (λ)}

= C_{0}(λ, N)e([Re λ] + N + 1, r)\frac{∥f∥_{𝒞^{N} (λ)}}{|a_{0}(f)|A(f)^{Re λ}}

≤ C_{0}(λ, N)([Re λ] + N + 1) (1 + | log \frac{a_{0}(f)}{b_{0}(f)}|^{[Re λ]+N+1})

× \frac{∥f∥_{𝒞^{N} (λ)}}{min(|a_{0}(f)|, |b_{0}(f)|)A(f)^{Re λ}} .

Here we take a branch of log \frac{a_{0}(f)}{b_{0}(f)}, of which the imaginary part is contained in [−π, π). The rest of §2 is devoted to the study of the Fourier transform ℱf(ζ) for f ∈ 𝒞^{N}(λ). Example 2.4. For Re λ > −1, we set

(2.4.1)(a)  f_{1}(x) := \{ \begin{matrix}(1 − x^{2})^{λ} & if |x| < 1, \\ 0 & if |x| ≥ 1. \end{matrix}
(2.4.1)(b)  f_{2}(x) := \{ \begin{matrix}(cos x)^{λ} & if |x| < \frac{π}{2} \\ 0 & if |x| ≥ \frac{π}{2} . \end{matrix}

Then we have f_{i} ∈ 𝒞^{∞}(λ) (i = 1, 2) and α(f_{1}) = −1, β(f_{1}) = 1, A(f_{1}) = 2, a_{0}(f_{1}) = b_{0}(f_{1}) = 2^{λ}, (2.4.2)(a)

α(f_{2}) = −\frac{π}{2} , β(f_{2}) = \frac{π}{2} , A(f_{2}) = π, a_{0}(f_{2}) = b_{0}(f_{2}) = 1. (2.4.2)(b)

We also have (see [Gradshteyn-Ryzhik, 1980](3.631)(9) or (3.893)(2) for (2.4.3)(b).)

(2.4.3)(a)  ℱf_{1}(ζ) = \sqrt{π} Γ(λ + 1)(\frac{ζ}{2})^{−λ− \frac{1}{2}} J_{λ+ \frac{1}{2}} (ζ),
(2.4.3)(b)  ℱf_{2}(ζ) = \frac{πΓ(λ + 1)2^{−λ}}{Γ (\frac{λ+ζ+2}{2}) Γ (\frac{λ−ζ+2}{2})}.

Example 2.5. We generalize (2.4.1)(b) as follows: For Re λ > −1 and 0 < φ < π, we set

(2.5.1)  f(x) ≡ f(λ, φ)(x) := \{ \begin{matrix}(cos x − cos φ)^{λ} & if |x| < φ, \\ 0 & if |x| ≥ φ. \end{matrix}

Then f ≡ f(λ, φ) ∈ 𝒞^{∞}(λ) and

(2.5.2)  α(f) = −φ, β(f) = φ, A(f) = 2φ, a_{0}(f) = b_{0}(f) = (sin φ)^{λ}.

Notice that f (λ, \frac{π}{2} ) = f_{2} with notation in (2.4.1)(b). We write P_{ν}^{μ}(z) for the associated Legendre function of the first kind. That is,

(2.5.3)  P_{ν}^{μ}(z) = \frac{1}{Γ(1 − μ)} (\frac{z + 1}{z − 1})^{\frac{μ}{2}} F (−ν, ν + 1; 1 − μ, \frac{1 − z}{2} )

is a solution of the differential equation

(2.5.4)  (1 − z^{2})\frac{d^{2}u}{dz^{2}} − 2z \frac{du}{dz} + (\frac{μ^{2}}{1 − z^{2}} ) u = 0.

From the formula (8.714)(1) in [Gradshteyn-Ryzhik, 1980], we have

(2.5.5)  ℱf(λ, φ)(ζ) = \sqrt{2π} Γ(λ + 1)(sin φ)^{λ+ \frac{1}{2}} P_{ζ− \frac{1}{2}}^{−λ− \frac{1}{2}} (cos φ).

We note that

(2.5.6)(a)  ℱf (λ, \frac{π}{2} ) (ζ) = \frac{πΓ(λ + 1)^{2^{−λ}}}{Γ (\frac{λ+ζ+2}{2}) Γ (\frac{λ−ζ+2}{2})},

(2.5.6)(b)  ℱf(0, φ)(ζ) = \frac{2 sin(ζφ)}{ζ},

from direct calculations or from the following particular values (see [Gradshteyn-Ryzhik, 1980](8.756)(1) and (8.754)(3)):

(2.5.7)(a)  P_{ν}^{μ}(0) = \frac{2^{μ}\sqrt{π}}{Γ (\frac{ν−μ+2}{2}) Γ (\frac{−ν−μ+1}{2})},

(2.5.7)(b)  P_{ν− \frac{1}{2}}^{− \frac{1}{2}} (cos φ) = \sqrt{\frac{2}{π sin φ}} \frac{sin(νφ)}{ν}.

It is known that P_{ζ− \frac{1}{2}}^{−λ− \frac{1}{2}} (cos φ), considered as a function ζ, has infinitely many zeros for

λ ≥ −\frac{1}{2}. These are all simple and real. They are symmetric with respect to the origin (see [Gradshteyn-Ryzhik, 1980], (8.781)).

In order to investigate the Fourier transform ℱf(ζ) of f ∈ 𝒞^{N}(λ), we begin with the following formula:

Formula 2.6 (See [Gel’fand-Shilov, 1959], p.171). Suppose that λ, ζ ∈ ℂ satisfies Re λ > −1 and Im ζ > 0. Then we have

ℱ(x_{+}^{λ})(ζ) ≡ ∫_{−∞}^{∞} x_{+}^{λ}e^{\sqrt{−1} xζ}dx = p(λ)ζ^{−λ−1},

where we put

(2.6.1)  p(λ) := Γ(λ + 1)e^{\frac{1}{2} \sqrt{−1} π(λ+1)}.

Proof. Since Re λ > −1 and Im ζ > 0, we have x_{+}^{λ}e^{\sqrt{−1} xζ} ∈ L^{1}(ℝ). We put

I := ∫_{−∞}^{∞} x_{+}^{λ}e^{\sqrt{−1} xζ}dx = ∫_{0}^{∞} x^{λ}e^{\sqrt{−1} xζ}dx.

By making a change of a valuable y := −\sqrt{−1} xζ, we have

I = ∫_{L} (\frac{e^{\frac{1}{2} \sqrt{−1} π}}{ζ} )^{λ+1} y^{λ}e^{−y}dy,

where L is a ray from the origin to infinity whose angle with respect to the real axis is arg y = arg ζ − \frac{π}{2} ∈ (−\frac{π}{2} , \frac{π}{2} ). By Cauchy’s theorem

I = (\frac{e^{\frac{1}{2} \sqrt{−1} x}}{ζ} )^{λ+1} ∫_{0}^{∞} y^{λ}e^{−y}dy
= Γ(λ + 1)ζ^{−λ−1}e^{\frac{1}{2} \sqrt{−1} π(λ+1)}.

□

Next, we recall the following two elementary facts:

Sub-Lemma 2.7. Suppose that −∞ < α < β < ∞, k ∈ ℕ_{+} and g ∈ C^{k−1,1}(ℝ) (recall (2.1.3) for notation) such that supp g ⊂ [α, β]. Then we have

|ℱg(ζ)| ≤ (β − α)e^{−αη}|ζ|^{−k} sup_{x∈ℝ} |(\frac{d}{dx})^{k} g(x)| ,

for any ζ ∈ ℂ such that η = Im ζ > 0.

Proof. It follows from

|ζ|^{k}|ℱg(ζ)| = |ℱ ((\frac{d}{dx})^{k} g(x)) (ζ)|

≤ sup_{x∈ℝ} |(\frac{d}{dx})^{k} g(x)| ∫_{α}^{β} |e^{\sqrt{−1} ζx}|dx.

□ Sub-Lemma 2.8. Let F_{k}(x) := 1 + ∑_{n=1}^{∞} \frac{n^{k}}{n!} x^{n}, k ∈ ℤ. If k ∈ ℕ_{+}, then we have F_{k}(x) =

1 + f_{k}(x)e^{x}, where f_{k}(x) is a polynomial of x of degree k. In particular, for any k ∈ ℤ, there exists a constant C ≡ C(k) > 0 such that

|F_{k}(x)| ≤ C(1 + x^{k})e^{x},

for any x ∈ ℝ.

Proof. The first statement follows from the formula

f_{k}(x)e^{x} = (x\frac{d}{dx})^{k} e^{x}, (k ∈ ℕ).

Here we put f_{0}(x) = 1. Thus we have f_{k+1}(x) = x (f_{k}(x) + \frac{d}{dx}f_{k}(x)) by the induction on k. (f_{0}(x) = 1, f_{1}(x) = x, f_{2}(x) = x + x^{2}, …) The rest is easy. □

In the following two lemmas we shall give an estimate of ℱ(x_{ε}^{λ}φ(x))(ζ) (ε = ±1) in the upper half plane. The case ε = −1 is not trivial (Lemma (2.10)).

Lemma 2.9. Suppose λ ∈ ℂ, δ > 0 and k ∈ ℕ satisfies k ≥ Re λ > −1. If φ ∈ Ψ(δ) (see (2.2) for notation), then there exists a constant C > 0 such that

|ℱ(x_{+}^{λ}φ(x))(ζ) − p(λ)ζ^{−λ−1}| ≤ C|ζ|^{−k},

for any ζ ∈ ℂ such that Im ζ ≥ 1.

Proof. We put

h(x) := x_{+}^{λ}φ(x) − x_{+}^{λ} ∈ C^{∞}(ℝ) ∩ 𝒮′(ℝ),

and

C := sup_{x∈ℝ} |\frac{d^{k}}{dx^{k}} h(x)| (< ∞).

Then supp h ⊂ [δ, ∞) and ℱh(ζ) = ℱ(x_{+}^{λ}φ(x))(ζ) − p(λ)ζ^{−λ−1} (see Formula (2.6)) is a holomorphic function of ζ in the upper half plane Im ζ > 0 and we have

|ℱ (\frac{d^{k}}{dx^{k}} h) (ζ)| ≤ C ∫_{δ}^{∞} e^{−x Im ζ}dx ≤ \frac{C}{Im ζ} .

Therefore, if Im ζ ≥ 1, then we have |ℱh(ζ)| ≤ \frac{C}{|ζ|^{k}}. □

Lemma 2.10. Suppose λ ∈ ℂ, N ∈ ℕ and A, δ ∈ ℝ satisfy Re λ > −1 and A > 4δ > 0. Put L := [Re λ] + N + 1 ∈ ℕ. If φ ∈ Ψ(δ), then there exists a constant C > 0 such that

|ℱ(x_{−}^{λ}φ(x))(ζ) − ∑_{n=0}^{N} \frac{(2η)^{n}}{n!} p(λ + n)(−ζ̄)^{−λ−n−1}| ≤ C|ζ|^{−L}e^{Aη},

for any ζ ∈ ℂ such that η := Im ζ ≥ 1.

Proof. We put

h(x) := x_{+}^{λ}φ(x)e^{2ηx} − ∑_{n=0}^{N} \frac{(2η)^{n}}{n!} x_{+}^{λ+n}φ(x).

Then we have h(x) = ∑_{n=N+1}^{∞} \frac{(2η)^{n}}{n!} x_{+}^{λ+n}φ(x) ∈ C^{L−1,1}(ℝ) from (2.1.2) and supp h ⊂ [0, 2δ].

We put

∥φ∥_{L} := sup_{\begin{matrix}0≤k≤L \\ x∈ℝ \end{matrix}} |(\frac{d}{dx})^{k} φ(x)| .

Let x ∈ [0, 2δ]. Let us estimate |(\frac{d}{dx})^{L} h(x)| by means of a function of η > 0.

(2.10.1) |(\frac{d}{dx})^{L}h(x)| ≤ ∑_{n=0}^{∞} \frac{(2η)^{n+N+1}}{(n + N + 1)!}|(\frac{d}{dx})^{L}\{x_{+}^{λ+n+N+1}φ(x)\}|

≤ 2^{L}∥φ∥_{L} ∑_{n=0}^{∞} \frac{(2η)^{n+N+1}}{(n+N +1)!} max_{\begin{matrix}0≤k≤L \\ 0≤x≤2δ \end{matrix}}(∏_{j=0}^{k−1}(Re λ+n+N +1−j))x^{Re λ+n+N+1−k}

≤ 2^{L}∥φ∥_{L}( max_{\begin{matrix}0≤k≤L \\ 0≤x≤2δ \end{matrix}} x^{Re λ+N+1−k})(1 + ∑_{n=1}^{∞} \frac{(2η)^{n+N+1}}{n!n^{N+1}} (n + L + 1)^{L}x^{n})

≤ C_{1}η^{N+1}(1 + ∑_{n=1}^{∞} \frac{n^{[Re λ]}(2ηx)^{n}}{n!} )

≤ C_{2}η^{N+1}(1 + η^{[Re λ]})e^{2ηx},

for some constants C_{1}, C_{2} > 0 independent of η > 0. The last inequality follows from Sub-Lemma (2.8). On the other hand, if ζ = Im η > 0, we have

|ℱ(x_{−}^{λ}φ(x))(ζ) − ∑_{n=0}^{N} \frac{(2η)^{n}}{n!} p(λ + n)(−ζ̄)^{−λ−n−1}|

= |ℱ(x_{+}^{λ}φ(x))(−ζ) − ∑_{n=0}^{N} \frac{(2η)^{n}}{n!} p(λ + n)(−ζ̄)^{−λ−n−1}|

= |ℱ(x_{+}^{λ}φ(x)e^{2xη})(−ζ̄) − ℱ(∑_{n=0}^{N} \frac{(2η)^{n}}{n!} x_{+}^{λ+n})(−ζ̄)|

≤ |ℱh(−ζ̄)| + ∑_{n=0}^{N} \frac{(2η)^{n}}{n!} |ℱ(x_{+}^{λ+n} − x_{+}^{λ+n}φ(x))(−ζ̄)|

From (2.10.1), Sub-Lemma (2.7) and Lemma (2.9), there exist constants C′, C > 0 such that

≤ C′η^{N+1}(1 + η^{[Re λ]})e^{4δη}|η|^{−L} + C′(∑_{n=0}^{N} \frac{(2η)^{n}}{n!} )|ζ|^{−L}

≤ C|ζ|^{−L}e^{Aη},

for any ζ ∈ ℂ such that η = Im ζ ≥ 1. □

Recall that we have fixed a compactly supported C^{∞} function ψ ∈ Ψ(1) in the definition of the norm ∥ · ∥_{𝒞^{N} (λ)} in (2.3.5). Here is an approximation of ℱf(ζ) for the half plane Im ζ ≥ 1. Note that Proposition (2.11) essentially gives an estimate of ℱf(ζ) for ζ ∈ ℂ. In fact, the case Im ζ ≤ −1 is quite similar, and the case of a bounded imaginary part is given in Lemma (2.13) as a consequence of Proposition (2.11).

Proposition 2.11. Suppose λ ∈ ℂ and N ∈ ℕ satisfy Re λ > −1 and Re λ + N ≥ 0. Then there exists a constant C ≡ C(λ, N) > 0 (depending only on λ, N and ψ(x)) such that

(2.11.1) |ℱf(ζ) − e^{\sqrt{−1} α(f)ζ} ∑_{j=0}^{N−1} a_{j}(f)p(λ + j)ζ^{−λ−j−1}

− e^{\sqrt{−1} β(f)ζ} ∑_{\begin{matrix}j,k∈ℕ \\ j+k<N \end{matrix}} b_{j}(f)\frac{(2η)^{k}}{k!} p(λ + j + k)(−ζ̄)^{−λ−j−k−1}|

≤ C(λ, N)A(f)^{−[Re λ]−N}∥f∥_{𝒞^{N} (λ)}e^{−α(f)η}|ζ|^{−[Re λ]−N−1},

for any f ∈ 𝒞^{N}(λ) and for any ζ ∈ ℂ such that η = Im ζ ≥ 1. (Recall Notation (2.3).)

In particular, there exists a constant C′ ≡ C′(λ) > 0 (depending only on λ and ψ(x)) such that

(2.11.2) |ℱf(ζ) − a_{0}(f)e^{\sqrt{−1} α(f)ζ}p(λ)ζ^{−λ−1} − b_{0}(f)e^{\sqrt{−1} β(f)ζ}p(λ)(−ζ̄)^{−λ−1}|
≤ C′(λ)A(f)^{− Re λ−1}∥f∥_{𝒞^{2}(λ)}e^{−α(f)η}|ζ|^{− Re λ−2},

for any f ∈ 𝒞^{2}(λ) and for any ζ ∈ ℂ such that η = Im ζ ≥ 1.

Proof. It is sufficient to prove Proposition when A(f) ≡ β(f) − α(f) = 1 because of the formulae (2.3.4), (2.3.6) and

(2.11.3)  (ℱf_{p,q})(ζ) = \frac{1}{p}e^{− \frac{\sqrt{−1} qζ}{p}} (ℱf) (\frac{ζ}{p}) ,

where f_{p,q}(x) = f(px + q) for p > 0, q ∈ ℝ.

Let us put φ(x) := ψ(5x), where ψ(x) ∈ Ψ(1) is the function in the definition of the norm ∥ · ∥_{𝒞^{N} (λ)} in (2.3.5).

Let f ∈ 𝒞^{N}(λ) such that A(f) = 1. From the definition of 𝒞^{N}(λ) in (2.3), we have

f(x) − ∑_{j=0}^{N} a_{j}(f)(x − α(f))_{+}^{λ+j}φ(x − α(f))

− ∑_{j=0}^{N} b_{j}(f)(x − β(f))_{−}^{λ+j}φ(x − β(f)) ∈ C^{[Re λ]+N,1}(ℝ).

Taking the Fourier transform, we have from Sub-Lemma (2.7),

(2.11.4)  |ℱf(ζ) − e^{\sqrt{−1} α(f)ζ} ∑_{j=0}^{N} a_{j}(f)ℱ(x_{+}^{λ+j}φ(x))(ζ)

− e^{\sqrt{−1} β(f)ζ} ∑_{j=0}^{N} b_{j}(f)ℱ(x_{−}^{λ+j}φ(x))(ζ)|

≤ ∥f∥_{𝒞^{N} (λ)}e^{−α(f)η}|ζ|^{−[Re λ]−N−1}

for any ζ ∈ ℂ such that η = Im ζ > 0. From Lemma (2.9) and Lemma (2.10), there exists a constant C_{1} > 0 depending only on λ, N and ψ(x), so that

(2.11.5) |ℱ(x_{+}^{λ+j}φ(x))(ζ) − p(λ + j)ζ^{−λ−j−1}| ≤ C_{1}|ζ|^{−[Re λ]−N−1},

|ℱ(x_{−}^{λ+j}φ(x))(ζ) − ∑_{k=0}^{N} \frac{(2η)^{k}}{k!} p(λ + j + k)(−ζ̄)^{−λ−j−k−1}| ≤ C_{1}e^{η}|ζ|^{−[Re λ]−N−1},

for any ζ ∈ ℂ such that η = Im ζ ≥ 1 and for any j (0 ≤ j ≤ N). Here we have used |ζ|^{j} ≥ 1. Also there exists a constant C_{2} > 0 depending only on N such that

|η^{k}| ≤ C_{2}e^{η}

for any η ≥ 1 and for any k ∈ ℕ (0 ≤ k ≤ N). Because

(2.11.6)  |z^{α}| = |z|^{Re α}e^{−(Im α)(arg z)}, (z ∈ ℂ^{×}, α ∈ ℂ),

we have

(2.11.7)  |η^{k}(−ζ̄)^{−λ−j−k−1}| ≤ C_{2} max{1, e^{π Im λ}}e^{η}|ζ|^{−[Re λ]−N−1},

if j, k ∈ ℕ, j + k ≥ N, 0 ≤ k ≤ N and if η = Im ζ ≥ 1.

Combining (2.11.4), (2.11.5) and (2.11.7), we have (2.11.1) when A(f) ≡ β(f) − α(f) = 1. The estimate (2.11.2) is an easy consequence of (2.11.1). □

Now we are ready to prove a key step in proving main results of this section.

Proposition 2.12. Suppose Re λ > −1. If f ∈ 𝒞^{1}(λ), then we have

(2.12.1)  sup{| Im ζ|: ℱf(ζ) = 0} < ∞.

To be more precise, we put

C_{1}(λ) := \frac{2C(λ, 1)}{|p(λ)|} max\{1, e^{−π Im λ}\},
C_{2}(λ) := \frac{2C′(λ)}{|p(λ)|} max\{1, e^{−π Im λ}\},

where C(λ, N) and C′(λ) are the constants in Proposition (2.11) and p(λ) = Γ(λ + 1)e^{\frac{1}{2} \sqrt{−1}π(λ+1)} (see (2.6.1)). With this notation, if f ∈ 𝒞^{1}(λ) is normalized (see (2.3.10) for definition), then we have

(2.12.2) sup \{| Im ζ| : ℱf(ζ) = 0, |ζ| ≥ (C_{1}(λ)∥f∥_{𝒞^{1}(λ)})^{\frac{1}{1+[Re λ]−Re λ}} \} ≤ 1 + π| Im λ|.

Furthermore, if f ∈ 𝒞^{2}(λ) is normalized, then we have

(2.12.3)  sup{| Im ζ|: ℱf(ζ) = 0, |ζ| ≥ C_{2}(λ)∥f∥_{𝒞^{2}(λ)}} ≤ 1 + π| Im λ|.

Proof. We write f^{∨}(x) := f(β(f) − x). Then it is clear that f ∈ 𝒞^{N}(λ) if and only if f^{∨} ∈ 𝒞^{N}(λ), that f is normalized if and only if f^{∨} is normalized, and that a_{j}(f) = b_{j}(f^{∨}), b_{j}(f) = a_{j}(f^{∨}), A(f) = A(f^{∨}) and ∥f∥_{𝒞^{N} (λ)} = ∥f^{∨}∥_{𝒞^{N} (λ)}. Since ℱf^{∨}(ζ) = e^{\sqrt{−1}ζ}ℱf(−ζ) for a normalized f, it is sufficient to estimate {Im ζ: ℱf(ζ) = 0} only from the above in order to prove the proposition.

First, we suppose f ∈ 𝒞^{1}(λ) is normalized. Then from (2.11.1) we have

|ℱf(ζ) − p(λ)ζ^{−λ−1} − e^{\sqrt{−1}ζ}p(λ)(−ζ̄)^{−λ−1}| ≤ C(λ, 1)∥f∥_{𝒞^{1}(λ)}|ζ|^{−[Re λ]−2}

for any ζ ∈ ℂ such that η = Im ζ ≥ 1. Thus, if η = Im ζ ≥ 1, then we have

e^{−η}e^{π| Im λ|} ≥ |e^{\sqrt{−1}ζ} (\frac{−ζ̄}{ζ})^{−λ−1}|

≥ |\frac{ζ^{λ+1}ℱf(ζ)}{p(λ)} − 1| − \frac{C(λ, 1)}{|p(λ)|} max (1, e^{−π Im λ}) ∥f∥_{𝒞^{1}(λ)}|ζ|^{Re λ−[Re λ]−1}

= |\frac{ζ^{λ+1}ℱf(ζ)}{p(λ)} − 1| − \frac{1}{2}C_{1}(λ)∥f∥_{𝒞^{1}(λ)}|ζ|^{Re λ−[Re λ]−1}.

Therefore, if ℱf(ζ) = 0 and if |ζ| ≥ (C_{1}(λ)∥f∥_{𝒞^{1}(λ)})^{\frac{1}{− Re λ+[Re λ]+1}} and η = Im ζ ≥ 1, then we have

e^{−η}e^{π| Im λ|} ≥ 1 − \frac{1}{2} = \frac{1}{2} > e^{−1}

that is,

η ≤ π| Im λ| + 1,

whence we conclude (2.12.2).

Next, we suppose that f ∈ 𝒞^{2}(λ) satisfies α(f) = 0, β(f) = 1 and so A(f) = 1. (For later use, we start from a little more general setting than a normalized one.) Then from (2.11.2) we have

|ℱf(ζ) − a_{0}(f)p(λ)ζ^{−λ−1} − b_{0}(f)e^{\sqrt{−1}ζ}p(λ)(−ζ̄)^{−λ−1}| ≤ C′(λ)∥f∥_{𝒞^{2}(λ)}|ζ|^{− Re λ−2},

for any ζ ∈ ℂ such that η = Im ζ ≥ 1. Thus we have

|\frac{b_{0}(f)}{a_{0}(f)}| e^{−η}e^{π| Im λ|} ≥ |\frac{b_{0}(f)}{a_{0}(f)}e^{\sqrt{−1}ζ} (\frac{−ζ̄}{ζ})^{−λ−1}|

≥ |\frac{ζ^{λ+1}ℱf(ζ)}{a_{0}(f)p(λ)} − 1| − \frac{C′(λ) max(1, e^{−π Im λ})}{|a_{0}(f)||p(λ)|}∥f∥_{𝒞^{2}(λ)}|ζ|^{−1}

= |\frac{ζ^{λ+1}ℱf(ζ)}{a_{0}(f)p(λ)} − 1| − \frac{1}{2|a_{0}(f)|}C_{2}(λ)∥f∥_{𝒞^{2}(λ)}|ζ|^{−1}, (2.12.4)

for any ζ ∈ ℂ such that η = Im ζ ≥ 1. Therefore if ℱf(ζ) = 0 and if |ζ| ≥ \frac{C_{2}(λ)}{|a_{0}(f)|}∥f∥_{𝒞^{2}(λ)} and η = Im ζ ≥ 1, then we have

|\frac{b_{0}(f)}{a_{0}(f)}| e^{−η}e^{π| Im λ|} ≥ 1 − \frac{1}{2} = \frac{1}{2} > e^{−1},

that is,

η < 1 + π| Im λ| + log |\frac{a_{0}(f)}{b_{0}(f)}| .

Therefore, if f ∈ 𝒞^{2}(λ) is normalized, then we have

sup{Im ζ: ℱf(ζ) = 0, |ζ| ≥ C_{2}(λ)∥f∥_{𝒞^{2}(λ)}} ≤ 1 + π| Im λ|.

□

Thanks to Proposition (2.12), it is sufficient to study the asymptotic behavior of ℱf(ζ) with η = Im ζ bounded in order to study the zeros of ℱf(ζ). The following lemma is a consequence of Proposition (2.11).

Lemma 2.13. Suppose that λ ∈ ℂ such that Re λ > −1. Then there exists a constant C ≡ C(λ) > 0 (depending only on λ and ψ(x) (see (2.3.5))) such that

(2.13.1) |ℱf(ζ) − a_{0}(f)e^{\sqrt{−1}α(f)ζ}p(λ)ζ^{−λ−1} − b_{0}(f)e^{\sqrt{−1}β(f)ζ}p(λ)(−ζ)^{−λ−1}|

≤ C(λ)e^{1.01E}A(f)^{− Re λ−1}∥f∥_{𝒞^{2}(λ)}e^{−α(f)η}|ζ|^{− Re λ−2}, for any f ∈ 𝒞^{2}(λ), for any E ≥ 1 and for any ζ ∈ ℂ such that |η| = | Im ζ| ≤ \frac{E}{A(f)} and |ζ| ≥ \frac{5E}{A(f)}. Here the branch of (±ζ)^{−λ−1} is taken according as

arg(∓ζ) = \{ \begin{matrix}π & if ±ξ > 0 and η = 0, \\ 0 & if ±ξ < 0 and η = 0 , \end{matrix}

where ζ = ξ + \sqrt{−1} η.

Proof. As in the proof of Proposition (2.11), by using the formulae (2.3.4), (2.3.6) and (2.11.3), it can be easily checked that f satisfies (2.13.1) for |5η| ≤ \frac{5E}{A(f)} ≤ |ζ| if and only if f_{p,q} satisfies (2.13.1) with the same constant C(λ) for |5η| ≤ \frac{5E}{A(f_{p,q})} ≤ |ζ|. Therefore, we may assume that α(f) = 0, β(f) = 1 in order to prove (2.13.1). That is, we want to show that there exists a constant C(λ) > 0 such that

(2.13.1)′  |ℱf(ζ) − a_{0}(f)p(λ)ζ^{−λ−1} − b_{0}(f)e^{\sqrt{−1}ζ}p(λ)(−ζ)^{−λ−1}|
≤ C(λ)e^{1.01E}∥f∥_{𝒞^{2}(λ)}|ζ|^{− Re λ−2},

for any f ∈ 𝒞^{2}(λ) such that α(f) = 0 and β(f) = 1 and for any E ≥ 1 and any ζ ∈ ℂ such that |η| = | Im ζ| ≤ E and |ζ| ≥ 5E. We put

f_{E}(x) := f(x)e^{(E+1)x}.

From (2.3.9) we have

f_{E} ∈ 𝒞^{2}(λ) (2.13.3)(a) α(f_{E}) = α(f) = 0, β(f_{E}) = β(f) = 1, A(f_{E}) = A(f) = 1, (2.13.3)(b) a_{0}(f_{E}) = a_{0}(f)e^{α(f)(E+1)} = a_{0}(f), (2.13.3)(c) b_{0}(f_{E}) = b_{0}(f)e^{β(f)(E+1)} = b_{0}(f)e^{E+1}, (2.13.3)(d) ∥f_{E}∥_{𝒞^{2}(λ)} ≤ C′′(λ)e^{1.01E}∥f∥_{𝒞^{2}(λ)}. (2.13.3)(e)

Here C′′(λ) ≥ 1 is a constant depending only on λ and ψ(x). The inequality (2.13.3)(e) follows from (2.3.9)(e) and from

∥e^{(1+E)x}∥_{C^{[Re λ]+3}([0,1])} = ∑_{j=0}^{[Re λ]+3}(E + 1)^{j}e^{E+1}.

From (2.11.2), we have

|ℱf_{E}(ζ) − a_{0}(f_{E})p(λ)ζ^{−λ−1} − b_{0}(f_{E})e^{\sqrt{−1}ζ}p(λ)(−ζ̄)^{−λ−1}| ≤ C′(λ)∥f_{E}∥_{𝒞^{2}(λ)}|ζ|^{− Re λ−2}

for η = Im ζ ≥ 1.

Replacing ζ by ζ + \sqrt{−1} (E + 1), we have
|ℱf_{E}(ζ + \sqrt{−1} (E + 1)) − a_{0}(f_{E})p(λ)(ζ + \sqrt{−1} (E + 1))^{−λ−1}

− b_{0}(f_{E})e^{\sqrt{−1}(ζ+\sqrt{−1}(E+1))}p(λ)(−ζ̄ + \sqrt{−1} (E + 1))^{−λ−1}|

≤ C′(λ)∥f_{E}∥_{𝒞^{2}(λ)} |ζ + \sqrt{−1}(E + 1)|^{− Re λ−2}

for η = Im ζ ≥ 1 − (1 + E) = −E.

Since ℱf_{E}(ζ) = ℱf (ζ − \sqrt{−1} (E + 1)), we have

(2.13.4) |ℱf(ζ) − a_{0}(f)p(λ)ζ^{−λ−1} − b_{0}(f)e^{\sqrt{−1}ζ}p(λ)(−ζ)^{−λ−1}|
≤ |a_{0}(f)p(λ)| |(ζ + \sqrt{−1} (E + 1))^{−λ−1} − ζ^{−λ−1}|

+ |b_{0}(f)e^{\sqrt{−1}ζ}p(λ)| |(−ζ̄ + \sqrt{−1} (E + 1))^{−λ−1} − (−ζ)^{−λ−1}| + C′(λ)C′′(λ)e^{1.01E}∥f∥_{𝒞^{2}(λ)} |ζ + \sqrt{−1}(E + 1)|^{− Re λ−2} ,

for η = Im ζ ≥ −E.

Now, let z ∈ ℂ satisfy |z| ≤ \frac{4}{5}. Then we have

|(1 + z)^{−λ−1} − 1| ≤ \frac{C_{1}(λ)}{e^{2π Im λ}} |z|,

where

C_{1}(λ) := e^{2π Im λ} sup_{|t|≤ \frac{4}{5}} |(\frac{d}{dt}) (1 + t)^{−λ−1}| .

Therefore, if ζ ∈ ℂ^{×} satisfies |ζ| ≥ \frac{5}{4}|R| and if we take a branch of ζ such that | arg ζ| ≤ 2π, then we have

|(ζ + R)^{−λ−1} − ζ^{−λ−1}| = |ζ^{−λ−1}| |(1 + \frac{R}{ζ})^{−λ−1} − 1|
≤ C_{1}(λ)|ζ|^{− Re λ−2}|R|.

In particular, if |ζ| ≥ 5E, |η| = | Im ζ| ≤ E and if E ≥ 1, then we have

\frac{5}{4}|E + 1| ≤ 5E ≤ |ζ| and \frac{5}{4}|2η + E + 1| ≤ 5E ≤ |ζ|

and so

|(ζ + \sqrt{−1} (E + 1))^{−λ−1} − ζ^{−λ−1}| ≤ 5C_{1}(λ)E|ζ|^{− Re λ−2}, (2.13.5)
|(−ζ̄ + \sqrt{−1} (E + 1))^{−λ−1} − (−ζ)^{−λ−1}| = |(−ζ + \sqrt{−1} (2η + E + 1))^{−λ−1} − (−ζ)^{−λ−1}|

≤ 5C_{1}(λ)E|ζ|^{− Re λ−2}.

Hence, if we put

C(λ) := 5C_{1}(λ)|p(λ)| + 2^{Re λ+2}C′(λ)C′′(λ) (≥ 1),

then from (2.13.4) and from (2.13.5), we have

|ℱf(ζ) − a_{0}(f)p(λ)ζ^{−λ−1} − b_{0}(f)e^{\sqrt{−1}ζ}p(λ)(−ζ)^{−λ−1}| ≤ C(λ)e^{1.01E}∥f∥_{𝒞^{2}(λ)}|ζ|^{− Re λ−2},

for ζ ∈ ℂ such that |η| = | Im ζ| ≤ E, |ζ| ≥ 5E ≥ 5, which we wanted to prove. □

Now we prepare an elementary lemma on the distribution of the zeros of a holomorphic function approximated by e^{aζ} − b.

Lemma 2.14. Let A > 0 and λ_{0} = e^{\sqrt{−1} Aμ_{0}} ∈ ℂ^{×} such that −π ≤ Re Aμ_{0} < π. We put

ϕ_{1}(ζ) := e^{\sqrt{−1} Aζ} − λ_{0}

Suppose that R > 0, D > 0 and C > 0 and that a holomorphic function ϕ_{2}(ζ) defined in

W ≡ W (R, D) := {ζ ∈ ℂ: Re ζ ≥ R, | Im ζ| ≤ D}

satisfies

(2.14.1)  |ϕ_{2}(ζ)| ≤ C|ζ|^{−1}

for any ζ ∈ W. Assume that

(2.14.2)  R ≥ \frac{6C}{|λ_{0}|} and AD ≥ 1 + |log |λ_{0}||.

For m ∈ ℕ_{+}, we put

(2.14.3)  B(m) := B (\frac{2(m + 1)π}{A} + μ_{0}, \frac{C}{mπ|λ_{0}|}) ,

where B(z, r) := {ζ ∈ ℂ: |ζ − z| ≤ r} for z ∈ ℂ and r > 0.

Fix m_{0} ∈ ℕ_{+} such that m_{0} ≥ \frac{AR}{2π}. Put R′ := \frac{2(m_{0}+1)π}{A} + Re μ_{0} and ϕ(ζ) := ϕ_{1}(ζ) − ϕ_{2}(ζ). Then we have R′ ≥ R and

(2.14.4)(a)  {ζ ∈ W (R′, D) : ϕ(ζ) = 0} = ∐_{m=m_{0}}^{∞} {ζ ∈ B(m) : ϕ(ζ) = 0} ,

(2.14.4)(b)  # {ζ ∈ B(m) : ϕ(ζ) = 0} = 1.

Here (2.14.4)(b) is counted with multiplicities. In particular, any zero of ϕ(ζ) in W (R′, D) is simple.

Proof. Suppose that ζ_{0} ∈ ℂ is a zero of ϕ_{1}(ζ). We shall estimate |ϕ_{1}(ζ)| from the below in the following cases:

i) ζ = z + ζ_{0}, where |z| ≤ \frac{1}{3A},

ii) ζ = ζ_{0} ± \frac{π}{A} + \sqrt{−1} y, where y ∈ ℝ,
iii) ζ = ζ_{0} ± \frac{\sqrt{−1}}{A} ± x, where x ∈ ℝ.

Case i) First we recall a formula for a holomorphic function g(ζ) near ζ_{0} ∈ ℂ: g(z + ζ_{0}) − g(ζ_{0}) − zg′(ζ_{0}) = ∫_{0}^{z}(z − t)g′′(t + ζ_{0})dt. Therefore we have

|g(z + ζ_{0}) − g(ζ_{0})| ≥ |z| |g′(ζ_{0})| − |z|^{2} sup_{|t|≤|z|} |g′′(t + ζ_{0})|

Applying this inequality to the case g(ζ) = ϕ_{1}(ζ) and ϕ_{1}(ζ_{0}) = 0, we have

|ϕ_{1}(z + ζ_{0})| ≥ A|λ_{0}| |z|(1 − A|z|e^{A|z|}). Since re^{r} < \frac{1}{2} if 0 ≤ r ≤ \frac{1}{3}, we have

(2.14.5)  |ϕ_{1}(z + ζ_{0})| > \frac{1}{2}A|λ_{0}| |z|

for any z ∈ ℂ such that 0 < |z| ≤ \frac{1}{3A}.

Case ii) Since ϕ_{1} (ζ_{0} ± \frac{π}{A} + \sqrt{−1} y) = −λ_{0}(e^{−Ay} + 1), we have

(2.14.6) |ϕ_{1} (ζ_{0} ± \frac{π}{A} + \sqrt{−1} y)| ≥ |λ_{0}|,

for any y ∈ ℝ.

Case iii) Since ϕ_{1} (ζ_{0} ± \frac{\sqrt{−1}}{A} + x) = λ_{0} (e^{∓1+\sqrt{−1}Ax} − 1) and since |e^{∓1} − 1| > \frac{1}{2},

we have

(2.14.7) |ϕ_{1} (ζ_{0} ± \frac{\sqrt{−1}}{A} + x)| ≥ \frac{1}{2}|λ_{0}|,

for any x ∈ ℝ.

Let m ∈ ℕ_{+}. We define

V (m) := {ζ ∈ ℂ : (2m + 1)π ≤ A Re(ζ − μ_{0}) ≤ (2m + 3)π, | Im ζ| ≤ D} ,
V′(m) := {ζ ∈ ℂ : (2m + 1)π ≤ A Re(ζ − μ_{0}) ≤ (2m + 3)π, A| Im(ζ − μ_{0})| ≤ 1} .

Since AD ≥ 1 + |log |λ_{0}|| = 1 + A| Im μ_{0}| ((2.14.2)), we have

(2.14.8)(a)  B(m) ⊂ B (\frac{2(m + 1)π}{A} + μ_{0}, \frac{1}{3A}) ⋐ V′(m) ⊂ V (m) ⊂ W (R, D)

if and only if

(2.14.8)(b)  \frac{C}{mπ|λ_{0}|} ≤ \frac{1}{3A} and \frac{(2m + 1)π}{A} + Re μ_{0} ≥ R.

Here B ⋐ V means that the closure of B is contained in the interior of V. The condition (2.14.8)(b) is satisfied if

(2.14.8)(c)  m ≥ \frac{AR}{2π},

since R ≥ \frac{6C}{|λ_{0}|} ((2.14.2)) and since π + A Re μ_{0} ≥ 0.

Now let us prove |ϕ_{1}(ζ)| > |ϕ_{2}(ζ)| on ∂V (m) ∪ ∂B(m) if m ∈ ℕ_{+} satisfies m ≥ \frac{AR}{2π} ((2.14.8)(c)).

If ζ ∈ ∂V (m) ∪ ∂B(m) ⊂ V (m), then we have

|ζ| ≥ | Re ζ| ≥ \frac{(2m + 1)π}{A} + Re μ_{0} ≥ \frac{2mπ}{A} (≥ R),

and so

(2.14.9)(a)  |ϕ_{2}(ζ)| ≤ C|ζ|^{−1} ≤ \frac{AC}{2mπ} (≤ \frac{C}{R} ) .

If ζ ∈ ∂V (m), then from (2.14.6) and from (2.14.7) we have

(2.14.9)(b)  |ϕ_{1}(ζ)| ≥ \frac{1}{2}|λ_{0}|.

If ζ ∈ ∂B(m), then from (2.14.5) and from (2.14.8)(a), we have

(2.14.9)(c)  |ϕ_{1}(ζ)| > \frac{1}{2}A|λ_{0}|\frac{C}{mπ|λ_{0}|} = \frac{AC}{2mπ} .

Since R ≥ \frac{6C}{|λ_{0}|} ((2.14.2)), we have proved

|ϕ_{1}(ζ)| > |ϕ_{2}(ζ)|,

for any ζ ∈ ∂V (m) ∪ ∂B(m). Now the lemma follows from Rouché’s theorem and from

W (R′, D) = ∪_{m=m_{0}}^{∞} V (m). □

By using a similar argument, Lemma (2.14) can be extended to a more general setting:

Proposition 2.15. Given ℓ ∈ ℕ_{+}, A > 0, λ_{i} = e^{\sqrt{−1} Aμ_{i}} ∈ ℂ^{×} (λ_{i} ≠ λ_{j} if i ≠ j, −π ≤ Re μ_{0} < π) and n_{i} ∈ ℕ_{+} (1 ≤ i ≤ ℓ), we define an entire function of ζ ∈ ℂ:

ϕ_{1}(ζ) := ∏_{i=1}^{ℓ} (e^{\sqrt{−1} Aζ} − λ_{i})^{n_{i}} .

Let D > 1 + max_{1≤i≤ℓ} | Im μ_{i}| and define

W := {ζ ∈ ℂ : Re ζ ≥ 1, | Im ζ| ≤ D} .

Suppose that there exist constants C > 0 and ε > 0 and that a holomorphic function ϕ_{2}(ζ) in W satisfies

|ϕ_{2}(ζ)| ≤ C|ζ|^{−ε}

for any ζ ∈ W. We put ϕ(ζ) := ϕ_{1}(ζ) − ϕ_{2}(ζ). Then there exists a constant L > 0 such that

\{ζ ∈ W : ϕ(ζ) = 0\} ⊂ ⋃_{i=1}^{ℓ} ⋃_{m∈ℕ_{+}} B (\frac{2mπ}{A} + μ_{i}, Lm^{− \frac{ε}{n_{i}}} ) ,

where we recall B(z, r) = {ζ ∈ ℂ : |ζ − z| ≤ r}. Furthermore, there exists m_{0} ∈ ℕ_{+} such that

# \{ζ ∈ B (\frac{2mπ}{A} + μ_{i}, Lm^{− \frac{ε}{n_{i}}} ) : ϕ(ζ) = 0\} = n_{i}

for any m ∈ ℕ_{+} such that m ≥ m_{0} and for any i (1 ≤ i ≤ ℓ). Here we take multiplicities of the zeros into account.

We do not go into details such as an estimate of L which we did in Lemma (2.14). Also we omit a proof of Proposition (2.15).

Now we are ready to prove the asymptotic behavior of the zeros of ℱf(ζ) for f ∈ 𝒞^{2}(λ):

Theorem 2.16. Suppose Re λ > −1. There exists a constant C_{0}(λ) (> 0) (see (2.16.2)(b)) depending only on λ and on the choice of ψ(x) ∈ Ψ(1) (see (2.3.5)) such that the following holds: Suppose f ∈ 𝒞^{2}(λ) is normalized (see (2.3.10) for definition). Put

(2.16.1)  n_{0} ≡ n_{0}(λ, f) := [(\frac{C_{0}(λ)}{2π} − \frac{5}{4}) ∥f∥_{𝒞^{2}(λ)}] − [\frac{Re λ − 1}{2}] + 1 ∈ ℕ_{+}.

Then we have

\{ζ ∈ ℂ : ℱf(ζ) = 0\} ⊂ \{ζ = ξ + \sqrt{−1} η ∈ ℂ : max\{|ξ|, |η|\} ≤ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}\}

∪ ∐_{ε=±1} ∐_{n=n_{0}}^{∞} B (ε(2n + λ)π, \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)}) .

Moreover, if n ≥ n_{0} and if ε = ±1, then each of B (ε(2n + λ)π, \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)}) contains a unique zero of ℱf(ζ). These zeros are all simple.

Here B(z, r) := {ζ ∈ ℂ : |ζ − z| ≤ r} for z ∈ ℂ and r > 0. The symbol ∐ denotes a disjoint union. The constant C_{0}(λ) is given in (2.16.2)(b) below.

Proof. Put

(2.16.2)(a)  E := 1 + π| Im λ|,

and

(2.16.2)(b)  C_{0}(λ) := max \{\frac{5}{2}(3π + π|λ|), \frac{6C(λ)e^{1.01E}e^{\frac{π}{2} | Im λ|}}{|p(λ)|} + \frac{5}{2}π, C_{2}(λ)\} ,

where C(λ) is the constant in Lemma (2.13) and C_{2}(λ) is the one in Proposition (2.12). Suppose that f ∈ 𝒞^{2}(λ) is normalized (see (2.3.10) for definition). Then from Proposition (2.12) and (2.12.3), we have

(2.16.3)  sup{| Im ζ| : ℱf(ζ) = 0, |ζ| ≥ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}}
≤ 1 + π| Im λ| = E.

Since C_{0}(λ)∥f∥_{𝒞^{2}(λ)} ≥ 2C_{0}(λ) ≥ 5E > E, we have

(2.16.4) {ζ ∈ ℂ : ℱf(ζ) = 0} ⊂ {ζ ∈ ℂ : max{| Re ζ|, | Im ζ|} ≤ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}}

∪ {ζ ∈ ℂ : | Re ζ| ≥ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}, | Im ζ| ≤ E} .

If Re ζ ≥ (C_{0}(λ) − \frac{5}{2}π) ∥f∥_{𝒞^{2}(λ)} (≥ 5E) and if | Im ζ| ≤ E, then the inequality (2.13.1) gives

|\frac{e^{\sqrt{−1}(λ+1)π}ζ^{λ+1}ℱf(ζ)}{p(λ)} − (e^{\sqrt{−1}ζ} − e^{\sqrt{−1}λπ})|

≤ \frac{C(λ)e^{1.01E}e^{\frac{π}{2} | Im λ|}}{|p(λ)|} e^{−π Im λ}|ζ|^{−1}∥f∥_{𝒞^{2}(λ)}

≤ \frac{1}{6} (C_{0}(λ) − \frac{5}{2}π) |e^{\sqrt{−1}πλ}| |ζ|^{−1}∥f∥_{𝒞^{2}(λ)}.

We put

(2.16.5)  ϕ_{1}(ζ) := e^{\sqrt{−1}ζ} − e^{\sqrt{−1}λπ},
ϕ(ζ) ≡ ϕ_{1}(ζ) − ϕ_{2}(ζ) := \frac{e^{\sqrt{−1}(λ+1)π}ζ^{λ+1}ℱf(ζ)}{p(λ)},
μ_{0} := λπ − 2π [\frac{Re λ + 1}{2}] ,
A := 1,
λ_{0} := e^{\sqrt{−1}λπ} (= e^{\sqrt{−1}μ_{0}}) ,
C := \frac{1}{6} (C_{0}(λ) − \frac{5}{2}π) e^{−π Im λ}∥f∥_{𝒞^{2}(λ)},
R := (C_{0}(λ) − \frac{5}{2}π) ∥f∥_{𝒞^{2}(λ)} (= \frac{6C}{|λ_{0}|}) ,
D := 1 + π| Im λ| (≡ E),
m_{0} := [\frac{R}{2π} ] + 1,
R′ := 2(m_{0} + 1)π + Re μ_{0}.

We note that −π ≤ Re μ_{0} < π. Applying Lemma (2.14) with the above data, we have

(2.16.6)  {ζ ∈ W (R′, D) : ϕ(ζ) = 0} = ∐_{m=m_{0}}^{∞} {ζ ∈ B(m) : ϕ(ζ) = 0} .

Here we recall that W (R′, D) = {ζ ∈ ℂ : Re ζ ≥ R′, | Im ζ| ≤ D}, that B(m) = B (2(m + 1)π + μ_{0}, \frac{C}{mπ|λ_{0}|}) ⊂ B (π (2m + 2 + λ − 2 [\frac{Re λ+1}{2}]) , \frac{C_{0}(λ)}{6πm}∥f∥_{𝒞^{2}(λ)}) and that B(z, r) = {ζ ∈ ℂ : |ζ − z| ≤ r}.

Since R′ < R + 5π ≤ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}, we have

{ζ ∈ ℂ: ℱf(ζ) = 0, Re ζ ≥ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}}

= {ζ ∈ ℂ : ℱf(ζ) = 0, Re ζ ≥ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}, | Im ζ| ≤ 1 + π| Im λ|} = {ζ ∈ W (C_{0}(λ)∥f∥_{𝒞^{2}(λ)}, 1 + π| Im λ|) : ℱf(ζ) = 0} ⊂ {ζ ∈ W (R′, 1 + π| Im λ|) : ℱf(ζ) = 0}

⊂ ∐_{m=m_{0}}^{∞} B (π (2m + 2 + λ − 2 [\frac{Re λ + 1}{2}]) , \frac{C_{0}(λ)}{6πm}∥f∥_{𝒞^{2}(λ)})

⊂ ∐_{n=m_{0}−[\frac{Re λ−1}{2}]}^{∞} B (π(2n + λ), \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)})

Here the first equality follows from (2.16.3), the second one is by definition, the third inclusion is because R′ < R + 5π ≤ C_{0}(λ)∥f∥_{𝒞^{2}(λ)}, the fourth one is (2.16.6), and the last one is because

|\frac{n}{m}| ≤ \frac{π}{2} ,

if n = m + 1 − [\frac{Re λ+1}{2}] and if m ≥ m_{0}. (In fact, m_{0} ≥ \frac{R}{2π} = \frac{C_{0}(λ)− \frac{5}{2} π}{2π} ∥f∥_{𝒞^{2}(λ)} ≥ \frac{5(|λ|+1)}{2}∥f∥_{𝒞^{2}(λ)} ≥ \frac{5}{2}(|λ| + 2) > (π − 2)(|λ| + 3) and |\frac{n}{m} | = |1 + \frac{1−[\frac{Re λ+1}{2}]}{m} | ≤ 1 + \frac{|λ|+3}{2m_{0}}.)

The union ∐_{n=n_{0}}^{∞} B (π(2n + λ), \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)}) is still disjoint because of (2.14.8)(a)

and \frac{1}{3A} · \frac{π}{2} < \frac{1}{A}. We note that

m_{0} − [\frac{Re λ − 1}{2}] = [(\frac{C_{0}(λ)}{2π} − \frac{5}{4}) ∥f∥_{𝒞^{2}(λ)}] − [\frac{Re λ − 1}{2}] + 1 ≡ n_{0}

by definition (2.16.5) and (2.16.1).

If we put f^{∨}(x) = f(1 − x), then we have ℱf^{∨}(ζ) = e^{\sqrt{−1}ζ}ℱf(−ζ). Thus we have similarly

{ζ ∈ ℂ : ℱf(ζ) = 0, Re ζ ≤ −C_{0}(λ)∥f∥_{𝒞^{2}(λ)}}

⊂ ∐_{n=n_{0}}^{∞} B (−π(2n + λ), \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)}) .

From (2.16.4) we have proved the theorem. □

Remark 2.17. In Theorem (2.16) we assume f ∈ 𝒞^{2}(λ). We could obtain similar estimate for f ∈ 𝒞^{1}(λ). To do this we need to use (2.12.2) instead of (2.12.3) and to use Proposition (2.15) with ℓ = n_{i} = 1, 0 < ε ≤ 1 instead of Lemma (2.14) and to modify Lemma (2.13) for f ∈ 𝒞^{1}(λ). The price to pay is that the radius of the ball containing the zero of ℱf(ζ) in the theorem tends to zero, possibly estimated by a slower order than n^{−1} as n goes to infinity.

The following theorem gives an upper bound of the number of the zeros of ℱf(ζ) that are contained in a fixed bounded domain in ℂ:

Theorem 2.18. Suppose that Re λ > −1 and that f ∈ 𝒞^{2}(λ) is normalized (see (2.3.10) for definition). We set

(2.18.1)  V (L, E) := \{ζ = ξ + \sqrt{−1} η ∈ ℂ : |ξ| ≤ L, |η| ≤ E\} ,

for

(2.18.2)(a) E ≥ max {3 + π| Im λ|, 3C_{0}(λ, 2)C_{2}(λ)∥f∥_{𝒞^{2}(λ)}} ,
(2.18.2)(b) L ≥ max\{15C_{0}(λ, 2)C(λ)e^{1.01E}e^{\frac{π}{2} | Im λ|}∥f∥_{𝒞^{2}(λ)}|p(λ)|^{−1}, 30\}.

Here C_{0}(λ, 2) (≥ 1), C_{2}(λ) > 0, and C(λ) ≥ 1 are the constants in (2.3.9)(e), in Proposition (2.12) and in Lemma (2.13), respectively. Then we have

# {ζ ∈ V (L, E) : ℱf(ζ) = 0} ≤ 2L + E.

Proof. Suppose that f ∈ 𝒞^{2}(λ) is normalized. We put

(2.18.3)  f_{1}(x) := (2x − 1)f(x).

Then a_{0}(f_{1}) = −1, b_{0}(f_{1}) = 1 because a_{0}(f) = b_{0}(f) = 1 (see (2.3.9)). Thus we have

\frac{ℱ(xf(x))(ζ)}{ℱf(ζ)} = \frac{1}{2} (1 + \frac{ℱf_{1}(ζ)}{ℱf(ζ)}) ,

and so

(2.18.4)  | \frac{\frac{d}{dζ} (ℱf(ζ))}{ℱf(ζ)}| ≤ \frac{1}{2} (1 + |\frac{ℱf_{1}(ζ)}{ℱf(ζ)}|) .

From now on, we estimate |\frac{ℱf_{1}(ζ)}{ℱf(ζ)} | from the above when ζ lies on the boundary of the rectangle

V (L, E) := \{ζ = ξ + \sqrt{−1} η ∈ ℂ : |ξ| ≤ L, |η| ≤ E\} ,

for a suitably large L > 0 and E > 0.

Case I) η = E and ξ ∈ ℝ. Let h = f or f_{1}. Then from (2.12.4) we have

|\frac{ζ^{λ+1}ℱh(ζ)}{a_{0}(h)p(λ)} − 1| ≤ e^{−η}e^{π| Im λ|} + \frac{1}{2}C_{2}(λ)∥h∥_{𝒞^{2}(λ)}|ζ|^{−1},

if ζ ∈ ℂ such that η = Im ζ > 1. Thus, if η ≥ 3 + π| Im λ| and if |ζ| ≥ C_{2}(λ) max(∥f∥_{𝒞^{2}(λ)}, ∥f_{1}∥_{𝒞^{2}(λ)}), then we have

|\frac{ζ^{λ+1}ℱh(ζ)}{a_{0}(h)p(λ)} − 1| ≤ e^{−3} + \frac{1}{2} < \frac{3}{5},

and since a_{0}(h) = ±1, we have

\frac{2}{5} ≤ |\frac{ζ^{λ+1}ℱh(ζ)}{p(λ)}| ≤ \frac{8}{5}.

Therefore, if

(2.18.5)  E ≥ max {3 + π| Im λ|, C_{2}(λ)∥f∥_{𝒞^{2}(λ)}, C_{2}(λ)∥f_{1}∥_{𝒞^{2}(λ)}} ,

and if η = Im ζ = E, then we have

(2.18.6)  |\frac{ℱf_{1}(ζ)}{ℱf(ζ)}| ≤ \frac{8}{5}/\frac{2}{5} = 4.

Case II) ξ = L and |η| ≤ E. From (2.13.1), if ζ ∈ ℂ satisfies |η| ≤ E, |ζ| ≥ 5E (E ≥ 1) and Re ζ > 0, then we have

|\frac{ℱh(ζ)}{a_{0}(h)p(λ)ζ^{λ+1}} − 1 − \frac{b_{0}(h)}{a_{0}(h)}e^{\sqrt{−1}(ζ−(λ+1)π)}| ≤ C(λ, E)|ζ|^{−1}∥h∥_{𝒞^{2}(λ)},

where C(λ, E) := C(λ)e^{\frac{π}{2} | Im λ|}e^{1.01E}|p(λ)|^{−1} and h ∈ 𝒞^{2}(λ) such that α(h) = 0, β(h) = 1 and |a_{0}(h)| = 1. Suppose E ≥ 1 and

(2.18.7)(a)  L ≥ 5 max {E, C(λ, E)∥f∥_{𝒞^{2}(λ)}, C(λ, E)∥f_{1}∥_{𝒞^{2}(λ)}} ,
(2.18.7)(b)  L − (Re λ + 1)π ∈ 2ℤπ.

If ξ = Re ζ = L and |η| = | Im ζ| ≤ E, then we have

|\frac{ℱf(ζ)}{p(λ)ζ^{λ+1}} − (1 + e^{−η+π Im λ})| ≤ \frac{1}{5},

|\frac{ℱf_{1}(ζ)}{p(λ)ζ^{λ+1}} − (1 − e^{−η+π Im λ})| ≤ \frac{1}{5}.

Thus

|\frac{ℱf(ζ)}{p(λ)ζ^{λ+1}} | ≥ \frac{4}{5} + e^{−η+π Im λ},
|\frac{ℱf_{1}(ζ)}{p(λ)ζ^{λ+1}} | ≤ \frac{6}{5} + e^{−η+π Im λ},

and so we have

(2.18.8)  |\frac{ℱf_{1}(ζ)}{ℱf(ζ)}| ≤ \frac{3}{2}.

We can reduce Case I)′ η = −E and ξ ∈ ℝ, Case II)′ ξ = −L and |η| ≤ E, to Case (I), Case (II), respectively, by putting f^{∨}(x) := f(1−x). Hence, if E and L satisfy (2.18.5) and (2.18.7), then we have

(2.18.9)  |∫_{∂V (L,E)} \frac{\frac{d}{dζ} ℱf(ζ)}{ℱf(ζ)}dζ| ≤ \frac{1}{2}(1 + 4) × 4L + \frac{1}{2} (1 + \frac{3}{2}) × 4E
= 10L + 5E

from (2.18.4), (2.18.6) and (2.18.8). Thus

(2.18.10)  # \{ζ ∈ V (L, E) : ℱf(ζ) = 0\} ≤ \frac{1}{2π} (10L + 5E),

counted with multiplicities.

Now let us estimate ∥f_{1}∥_{𝒞^{2}(λ)} by ∥f∥_{𝒞^{2}(λ)} from the above, so that the conditions on E and L in (2.18.5) and (2.18.7) become simpler.

From the definition in (2.3.7), we have

∥2x − 1∥_{C^{L}([0,1])} = 1 + 1 · 2 = 3,

if L ≥ 1. Hence from (2.3.9)(e), we have

∥f_{1}(x)∥_{𝒞^{2}(λ)} ≤ 3C_{0}(λ, 2)∥f∥_{𝒞^{2}(λ)},

where C_{0}(λ, 2) ≥ 1 is the constant in (2.3.9)(e). Now, let E and L satisfy (2.18.2)(a),(b). We put L′ ∈ ℝ such that

L ≤ L′ < L + 2π, L′ − (Re λ + 1)π ∈ 2ℤπ.

Then E and L′ satisfy (2.18.5) and (2.18.7). Since L ≥ 30, we have \frac{L′}{L} ≤ \frac{30+2π}{30} < \frac{4π}{10} . Thus, from (2.18.10), we have

# {ζ ∈ V (L, E) : ℱf(ζ) = 0}

≤ # {ζ ∈ V (L′, E) : ℱf(ζ) = 0}

≤ \frac{1}{2π} (10L′ + 5E)

< 2L + E.

□ For later use, we also need the following modification of Theorem (2.16):

Lemma 2.19. Suppose we are in the setting of Theorem (2.16). Let K be any positive number satisfying

(2.19.1)  K ≥ (\frac{C_{0}(λ)}{2π} − \frac{5}{4}) ∥f∥_{𝒞^{2}(λ)} + 2.

We put

(2.19.2)  n_{1} := [K] − [\frac{Re λ}{2}] ∈ ℕ_{+}.

Then we have

(2.19.3)  n_{0} ≤ n_{1} ≤ K − \frac{Re λ}{2} + 1.

For L > 0, we set

(2.19.4)  V (L) ≡ V (L, L) := {ζ ∈ ℂ : | Re ζ| ≤ L, | Im ζ| ≤ L} .

Then we have

(2.19.5) {ζ ∈ ℂ : ℱf(ζ) = 0}

⊂ V (max \{C_{0}(λ)∥f∥_{𝒞^{2}(λ)}, 2Kπ\}) ∪ ∐_{n=n_{1}}^{∞} ∐_{ε=±1} B (ε(2n + λ)π, \frac{πK}{6n} ) ,

if f ∈ 𝒞^{2}(λ) is normalized.

Proof. From (2.19.3) we have

⋃_{n=n_{0}}^{n_{1}−1} ⋃_{ε=±1}B (ε(2n + λ)π, \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)})

⊂ ⋃_{n=n_{0}}^{n_{1}−1} ⋃_{ε=±1} B(ε(2n + λ)π, π)

⊂ V ((2(n_{1} − 1) + Re λ + 1)π) ⊂ V ((2K − 1)π) ⊂ V (2Kπ).

By virtue of Theorem (2.16), we have (2.19.6)

\{ζ ∈ ℂ : ℱf(ζ) = 0\} ⊂ V (C_{0}(λ)∥f∥_{𝒞^{2}(λ)}) ∪ ⋃_{n=n_{0}}^{n_{1}−1} ⋃_{ε=±1} B (ε(2n + λ)π, \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)})

∪ ∐_{n=n_{1}}^{∞} ∐_{ε=±1} B (ε(2n + λ)π, \frac{C_{0}(λ)}{12n}∥f∥_{𝒞^{2}(λ)})

⊂ V (max \{C_{0}(λ)∥f∥_{𝒞^{2}(λ)}, 2Kπ\}) ∪ ⋃_{n=n_{1}}^{∞} ⋃_{ε=±1} B (ε(2n + λ)π, \frac{πK}{6n} ) .

On the other hand,

K ≥ (\frac{C_{0}(λ)}{2π} − \frac{5}{4}) ∥f∥_{𝒞^{2}(λ)} + 2
≥ \frac{1}{2π} (\frac{5π(|λ| + 1)}{2} − \frac{5}{4}) × 2 + 2  (2.16.2)(b)

≥ \frac{5}{2}|λ| + 4

Therefore we have

n_{1} ≥ K − \frac{Re λ}{2} − 1

≥ \frac{3}{4}K + \frac{1}{4} (\frac{5}{2}|λ| + 4) − \frac{Re λ}{2} − 1

≥ \frac{3}{4}K.

Hence

\frac{πK}{6n} ≤ \frac{πK}{6n_{1}} ≤ \frac{4πK}{18K} < π,

if n ≥ n_{1}. This implies that the union ∪_{n=n_{1}}^{∞} ∪_{ε=±1} B (ε(2n + λ)π, \frac{πK}{6n} ) in (2.19.6) is still

disjoint. Now the proof of the lemma is complete. □

Corollary 2.20. Suppose λ ∈ ℂ satisfies Re λ > −1. Then there exist constants B(λ) > 0 (see (2.20.10)) and D(λ) > 0 (see (2.20.5)(c)) with the following properties: For f ∈ 𝒞^{2}(λ), we fix r(f) ∈ ℂ such that

(2.20.1)  e^{r(f)} = \frac{a_{0}(f)}{b_{0}(f)},

and we put

(2.20.2)  ⟨f⟩ := \frac{(1 + |r(f)|^{[Re λ]+3}) ∥f∥_{𝒞^{2}(λ)}}{A(f)^{Re λ} min \{|a_{0}(f)|, |b_{0}(f)|\}} (≥ 2).

(We recall notations in (2.3). If f is normalized in the sense of (2.3.10), then we can choose r(f) = 0 and ⟨f⟩ = ∥f∥_{𝒞^{2}(λ)}.)

We put

(2.20.3)  n_{1} ≡ n_{1}(λ, f) := [\frac{D(λ)⟨f⟩}{2π}] − [\frac{Re λ}{2}] .

Then there exists a finite set S(f) ⊂ ℂ such that

(2.20.4)(a) {ζ ∈ ℂ : ℱf(ζ) = 0}

= S(f) ∪ ∐_{n=n_{1}}^{∞} ∐_{ε=±1} B (\frac{ε(2n + λ)π − \sqrt{−1} r(f)}{A(f)}, \frac{D(λ)⟨f⟩}{12A(f)n}) ,

(2.20.4)(b)

S(f) ⊂ \{ζ ∈ ℂ : max \{|Re (ζ + \frac{\sqrt{−1} r(f)}{A(f)})| , |Im (ζ + \frac{\sqrt{−1} r(f)}{A(f)})|\} ≤ \frac{D(λ)⟨f⟩}{A(f)}\} ,

(2.20.4)(c)  #S(f) ≤ exp(B(λ)⟨f⟩).

Furthermore, each of B (^{\frac{ε(2n+λ)π−\sqrt{−1} r(f)}{A(f)}}, \frac{D(λ)⟨f⟩}{12A(f)n}) (n ≥ n_{1}, ε = ±1) contains a unique zero of ℱf(ζ), which is simple.

Proof. As in (2.3.11), we put

f_{p,q;r,s}(x) := e^{rx+s}f(px + q)

with p = A(f) > 0, q = α(f) ∈ ℝ and r ≡ r(f), s ∈ ℂ such that

e^{r(f)} = \frac{a_{0}(f)}{b_{0}(f)} and e^{s} = \frac{1}{a_{0}(f)A(f)^{λ}} .

Then f_{p,q;r,s} ∈ 𝒞^{2}(λ) is normalized in the sense of (2.3.10). From (2.3.11)(c) we have

(2.20.5)(a)  ∥f_{p,q;r,s}∥_{𝒞^{2}(λ)} ≤ C_{0}(λ, 2)([Re λ] + 3)⟨f⟩.

Thus

(2.20.5)(b)  C_{0}(λ)∥f_{p,q;r,s}∥_{𝒞^{2}(λ)} ≤ D(λ)⟨f⟩,

where we put

(2.20.5)(c)  D(λ) := C_{0}(λ)C_{0}(λ, 2)([Re λ] + 3).

Applying Lemma (2.19) for f_{p,q;r,s} ∈ 𝒞^{2}(λ) with K := \frac{1}{2π} D(λ)⟨f⟩ and n_{1} := [\frac{D(λ)⟨f⟩}{2π}]

− [\frac{Re λ}{2}] ∈ ℕ_{+}, we have

(2.20.6) {ζ ∈ ℂ : ℱf_{p,q;r,s}(ζ) = 0}

⊂ V (D(λ)⟨f⟩) ∪ ∐_{n=n_{1}}^{∞} ∐_{ε=±1} B (ε(2n + λ)π, \frac{D(λ)⟨f⟩}{12n}) ,

where we have used

K = \frac{1}{2π} D(λ)⟨f⟩ ≥ \frac{1}{2π} C_{0}(λ)∥f_{p,q;r,s}∥_{𝒞^{2}(λ)} ≥ (\frac{C_{0}(λ)}{2π} − \frac{5}{4}) ∥f_{p,q;r,s}∥_{𝒞^{2}(λ)} + 2.

From the formula (2.3.11)(b) and (2.20.6) we have

(2.20.7) \begin{aligned}\{ζ ∈ ℂ : ℱf(ζ) = 0\} & = \{\frac{ζ − \sqrt{−1} r(f)}{A(f)} : ℱf_{p,q;r,s}(ζ) = 0\} \\ & ⊂ \{ζ ∈ ℂ : ζ + \frac{\sqrt{−1} r(f)}{A(f)} ∈ V (\frac{D(λ)⟨f⟩}{A(f)})\} \\ & ∪ ∐_{n=n_{1}}^{∞} ∐_{ε=±1} B (\frac{ε(2n + λ)π − \sqrt{−1} r(f)}{A(f)}, \frac{D(λ)⟨f⟩}{12nA(f)}) . \end{aligned}

We put a finite set by

(2.20.8)  \begin{aligned}S(f) : & = \{ζ ∈ ℂ : ℱf(ζ) = 0, ζ + \frac{\sqrt{−1} r(f)}{A(f)} ∈ V (\frac{D(λ)⟨f⟩}{A(f)})\} \\ & = \{\frac{ζ − \sqrt{−1} r(f)}{A(f)} : ℱf_{p,q;r,s}(ζ) = 0, ζ ∈ V (D(λ)⟨f⟩)\} . \end{aligned}

Then (2.20.7) shows (2.20.4)(a) and (b). Now, it is sufficient to estimate the number of S(f) in order to complete the proof of the corollary.

We recall that positive constants C_{0}(λ) (> 15), C_{0}(λ, 2) (≥ 1), C(λ) (≥ 1) and C_{2}(λ) are the ones in (2.16.2)(b), (2.3.9)(e), Lemma (2.13) and Proposition (2.12), respectively. We put

(2.20.9)(a)  C_{3}(λ) := max {C_{0}(λ), 3C_{0}(λ, 2)C_{2}(λ)} ,
(2.20.9)(b)  C_{4}(λ) := C_{3}(λ)C_{0}(λ, 2)([Re λ] + 3),
(2.20.9)(c)  E := C_{4}(λ)⟨f⟩.

Then C_{3}(λ) ≥ C_{0}(λ) ≥ \frac{1}{2}(3 + | Im λ|), E ≥ C_{3}(λ)∥f_{p,q;r,s}∥_{𝒞^{2}(λ)} and E satisfies (2.18.2)(a) for f_{p,q;r,s}. We also put

(2.20.9)(d)  L := 5C_{0}(λ, 2)e^{2C_{4}(λ)⟨f⟩}

Then L satisfies (2.18.2)(b) for f_{p,q;r,s} and also we have L ≥ E ≥ D(λ)⟨f⟩ (≥ 90). Therefore, from Theorem (2.18), we have

#S(f) = # {ζ ∈ V (D(λ)⟨f⟩) : ℱf_{p,q;r,s}(ζ) = 0}
≤ # {ζ ∈ V (L, E) : ℱf_{p,q;r,s}(ζ) = 0}
≤ 2L + E
≤ 3L.

Finally, we put

(2.20.10)  B(λ) := 2C_{4}(λ) + \frac{15}{2}C_{0}(λ, 2).

Then using ⟨f⟩ ≥ 2 we have

e^{B(λ)⟨f⟩} ≥ e^{2C_{4}(λ)⟨f⟩}e^{15C_{0}(λ,2)}
≥ 3L.

Thus we conclude

#S(f) ≤ e^{B(λ)⟨f⟩},

which we wanted to prove. □

Let f ∈ 𝒞^{N}(λ) and we put
(2.21.1)(a)  f^{∨}(x) := f(−x + α(f) + β(f)),
(2.21.1)(b)  f̄(x) := \overline{f(x)}.

Then it is clear from the definition (2.3) that f^{∨} ∈ 𝒞^{N}(λ) and f̄ ∈ 𝒞^{N}(λ̄). We say f is symmetric if

(2.21.2)  f^{∨} = f.

We say f is real if

(2.21.3)  f̄ = f.

It follows from the definition that if f is symmetric then a_{j}(f) = b_{j}(f) for 0 ≤ j ≤ N with the notation (2.3) and that if f is real then λ is also real.

Corollary 2.22. Suppose λ ∈ ℂ satisfies Re λ > −1. If f ∈ 𝒞^{2}(λ) is symmetric and real in the sense of (2.21), then there are countably many real zeros of ℱf(ζ) and there are at most a finite number of non-real zeros. More precisely, we have

(2.22.1)  # {ζ ∈ ℂ ∖ ℝ : ℱf(ζ) = 0} ≤ exp(B(λ)⟨f⟩),

where B(λ) is the constant in Corollary (2.20) and ⟨f⟩ is defined in (2.20.2).

Proof. First we note that λ ∈ ℝ because f is real. We put

f^{∗}(x) := \overline{f^{∨}}(x) ≡ \overline{f(−x + α(f) + β(f))}.

Then we have

ℱf^{∗}(ζ̄) = e^{\sqrt{−1}(α(f)+β(f))ζ̄}\overline{ℱf(ζ)}.

Since f^{∗} = f, we have

(2.22.2)  {ζ ∈ ℂ : ℱf(ζ) = 0} = {ζ̄ ∈ ℂ : ℱf(ζ) = 0} .

Now let us apply Corollary (2.20.4). We may choose r(f) = 0 since a_{0}(f) = b_{0}(f). We denote by ζ_{n,ε} (n ≥ n_{1}, ε = ±1) the unique zero contained in the ball

B (\frac{ε(2n + λ)π − \sqrt{−1} r(f)}{A(f)}, \frac{D(λ)⟨f⟩}{12A(f)n}) .

This ball is symmetric with respect to the real axis since λ ∈ ℝ and r(f) = 0, and thus we conclude

\overline{ζ_{n,ε}} = ζ_{n,ε}.

Hence, ℱf(ζ) has countably many zeros and we have

{ζ ∈ ℂ ∖ ℝ : ℱf(ζ) = 0} ⊂ S(f),

where S(f) is a finite set satisfying the property (2.20.4). In particular, it follows from (2.20.4)(c) that

# {ζ ∈ ℂ ∖ ℝ : ℱf(ζ) = 0} ≤ exp(B(λ)⟨f⟩).

□

Remark 2.23. The previous corollary is a kind of generalization of classical results that assert some ‘special functions’ with real parameter (e.g. Bessel function J_{λ}(ζ) with λ ∈ ℝ, λ > −1) have countably many real zeros, and have no non-real zeros. However, there may exist finite number of non-real zeros in our general setting. In fact, for any λ ∈ ℂ such that Re λ > −1, we can find a sequence of functions f_{K} ∈ 𝒞^{2}(λ) (K = 1, 2, …) such that

lim_{K→∞} # {ζ ∈ ℂ ∖ ℝ : ℱf_{K}(ζ) = 0} = ∞.

The following example was pointed out by Hiroyuki Ochiai. We fix f ∈ 𝒞^{2}(λ) which is real and symmetric in the sense of (2.21) and choose r_{j} > 0 (j = 1, 2, …). We define

(2.23.1)  f_{K}(x) := ∏_{j=1}^{K} (\frac{d^{2}}{dx^{2}} + r_{j}) (f(x)(x − α(f))_{+}^{K}(x − β(f))_{−}^{K}) .

Then f_{K} ∈ 𝒞^{2}(λ) is real and symmetric and

(2.23.2)  # {ζ ∈ ℂ ∖ ℝ : ℱf_{K}(ζ) = 0} ≥ 2K.
Notice that lim_{K→∞}⟨f_{K}⟩ = ∞ in this case.

§3 Convex domains Ω in ℝ^{n} and asymptotics of the null variety 𝒩(Ω).

First of all, we review some standard facts about convex domains. Unless otherwise stated, we suppose the following setting throughout this section:

(3.1) Ω is a bounded domain with a connected smooth boundary ∂Ω of dimension n − 1.

We equip ℝ^{n} with the standard inner product ( , ) and denote the unit sphere by S^{n−1}. We define the Gauss map

(3.2)  ν ≡ ν_{Ω} : ∂Ω → S^{n−1},

by its outer normal vector field, and define the Gauss-Kronecker curvature

(3.3)  K ≡ K_{Ω} : ∂Ω → ℝ

by the Jacobian of dν with respect to the induced metric from ℝ^{n}. Here we choose an orientation of ∂Ω so that ν preserves an orientation. In particular, K is everywhere positive if Ω is a ball. The following characterization of (strict) convexity is classical:

Fact 3.4 (Hadamard, see [Chern-Lashof, 1958]). In the setting of (3.1), the following two conditions on Ω are equivalent:

1) K_{Ω}(∂Ω) ⊂ ℝ_{+},
2) ν_{Ω} : ∂Ω ⟶^{∼} S^{n−1} is a (surjective) diffeomorphism.

Furthermore, one of (therefore all of) the above two conditions implies the following four equivalent conditions:

3) Ω is geodesically convex, that is, tx + (1 − t)y ∈ Ω for any x, y ∈ Ω and for any t ∈ [0, 1].

4) Ω lies in one of the halfspaces determined by any hyperplane tangent to ∂Ω.

5) K_{Ω}(∂Ω) ⊂ ℝ_{+} ∪ {0} and the mapping degree of ν_{Ω} is one.

6) {0} ⫋ K_{Ω}(∂Ω) ⊂ ℝ_{+} ∪ {0}.

Definition 3.5. With notation as in Fact (3.4), Ω is called strictly convex if and only if one of (any of) (1) and (2) is satisfied. Ω is called convex if and only if one of (any of) (3), (4), (5) and (6) is satisfied. The notion of convexity will be generalized to two different ways in the case of Riemannian symmetric spaces of rank one in §7 (see Remark (7.1.8)).

Definition 3.6. Suppose Ω is a convex domain. The supporting function of Ω is a map

h ≡ h_{Ω} : S^{n−1} ⟶ ℝ,

given by

(3.6.1)  h_{Ω}(ω) : = (ω, ν^{−1}(ω))

= sup_{x∈Ω}(x, ω) (ω ∈ S^{n−1}).

We denote the linear extension of h by

h̃ ≡ h̃_{Ω} : ℝ^{n} ⟶ ℝ.

That is,

(3.6.2)  h̃(x) := \{ \begin{matrix}|x|h (\frac{x}{|x|}) , & x ∈ ℝ^{n} ∖ \{0\}, \\ 0, & x = 0. \end{matrix}

A convex domain can be recovered by its supporting function: Lemma 3.7. Suppose that Ω is a convex domain in ℝ^{n}. Then we have

(3.7.1)  Ω = ⋂_{ω∈S^{n−1}}{x ∈ ℝ^{n} : (x, ω) < h(ω)}.

Furthermore, if Ω is strictly convex, then we have more directly,

(3.7.2)  ν_{Ω}^{−1}(ω) = (\frac{∂h̃}{∂x_{1}} (ω), …, \frac{∂h̃}{∂x_{n}} (ω)) .

Proof. The first formula (3.7.1) is clear. To see the second one, we extend a diffeomorphism

ν_{Ω}^{−1} : S^{n−1} ⟶^{∼} ∂Ω to

(3.7.3) ν̃_{Ω}^{−1} : ℝ^{n} ∖ {0} ⟶ ∂Ω

as a homogeneous function of degree 0. Then we have

(3.7.4)  h̃_{Ω}(x) = (x, ν̃_{Ω}^{−1}(x)) ,  x ∈ ℝ^{n},
(3.7.5)  (\frac{∂ν̃_{Ω}^{−1}}{∂x_{j}} (x), x) = 0,  (j = 1, 2, …, n).

Thus, we have (3.7.2). □

The breadth function of a convex domain Ω in ℝ^{n} is given by

H ≡ H_{Ω} : S^{n−1} ⟶ ℝ,
∪  ∪
ω ∣ ⟶ h(ω) + h(−ω)

From the definition, we have

(3.8.1)  H(ω) > 0, H(ω) = H(−ω) for any ω ∈ S^{n−1}.

If Ω is strictly convex, we put

(3.9.1)  κ ≡ κ_{Ω} : S^{n−1} → ℝ_{+}, ω ⟼ K_{Ω} ∘ ν_{Ω}^{−1}(ω),
(3.9.2)  d ≡ d_{Ω} : S^{n−1} → ℝ, ω ⟼ \frac{log κ_{Ω}(−ω) − log κ_{Ω}(ω)}{2H_{Ω}(ω)}.

From the definition, we have

d(ω) + d(−ω) = 0, for any ω ∈ S^{n−1}.

Notice that the functions H_{Ω}, κ_{Ω} and d_{Ω} are invariant under parallel displacements of Ω, while h_{Ω} is not. The following fact is a uniqueness part of Minkowski’s problem:

Fact 3.10 (Alexandroff-Fenchel-Jessen, see [Chern, 1959]). With the definition (3.9.1) κ determines a strictly convex domain up to parallel displacements.

We do not give a proof here. It reduces to the uniqueness of a solution h up to a linear function (restricted to S^{n−1}) of the differential equation (see (3.25) for definition):

(3.10)(a)  det(D^{2}h + h)(ω) = \frac{1}{κ(ω)},

where κ(ω) is a given function. It is a standard method to use the maximum principle to show the uniqueness of solutions of elliptic partial differential equations. A proof in [Chern, 1959] is based on this idea. One of the points here is how to deal with linear functions on S^{n−1} coming from parallel translations in ℝ^{n}, up to which we have to prove the uniqueness. See also Problem (3.27) for a related unsolved problem (n ≥ 3).

As a direct consequence of Fact (3.10), we have

Lemma 3.11. Let Ω be a strictly convex domain. Then the following two conditions on Ω are equivalent.

1) d_{Ω}(ω) ≡ 0 for any ω ∈ S^{n−1},

2) Ω is centrally symmetric with respect to an inner point of Ω.

Proof. 1) ⇒ 2) Put Ω^{∨} := {x ∈ ℝ^{n} : −x ∈ Ω}. Then the assumption (1) implies that κ_{Ω} ≡ κ_{Ω^{∨}}. From Fact (3.10), Ω and Ω^{∨} differ from each other by a parallel displacement. Hence (2).

2) ⇒ 1) Clear. □

Let δ denote the Dirac’s delta function of a single variable. The Fourier-Laplace transform on ℝ^{n} is decomposed into a Radon transform and a Fourier transform of a single variable:

ℱf(ζ · ω) = ∫_{ℝ^{n}} f(x)e^{\sqrt{−1}(x,ζ·ω)}dx
= ∫_{−∞}^{∞}(∫_{ℝ^{n}} f(x)δ(p − (x, ω))dx)e^{\sqrt{−1} pζ}dp,

for ζ ∈ ℂ and ω ∈ S^{n−1}.

We put

S ≡ S_{Ω} : S^{n−1} × ℝ ⟶ ℝ
∪  ∪
(ω, p)  ∣ ⟶ ∫_{ℝ^{n}} χ_{Ω}(x)δ(p − (x, ω))dx

for a bounded measurable set Ω in ℝ^{n}. Then we have

(3.13)  ℱχ_{Ω}(ζ · ω) = ∫_{−∞}^{∞} S_{Ω}(ω, p)e^{\sqrt{−1} pζ}dp

= ∫_{−h_{Ω}(−ω)}^{h_{Ω}(ω)} S_{Ω}(ω, p)e^{\sqrt{−1} pζ}dp,

for ζ ∈ ℂ and ω ∈ S^{n−1}. A key lemma to an application of the previous section is the following:

Lemma 3.14. Suppose that Ω is a strictly convex domain in ℝ^{n}. We define an open subset V in S^{n−1} × ℝ by

(3.14.1)  V := {(ω, p) ∈ S^{n−1} × ℝ : −h(−ω) < p < h(ω)}.

Retain notation in (2.3) in §2. Then we have

i) S_{Ω} ∈ C(S^{n−1} × ℝ), supp S_{Ω} = V̄ and S_{Ω}|_{V} ∈ C^{∞}(V).
ii) S_{Ω}(ω, · ) ∈ 𝒞^{∞} (\frac{n−1}{2}).

Furthermore, we have

(3.14.2)  α(S_{Ω}(ω, ·)) = −h(−ω), β(S_{Ω}(ω, ·)) = h(ω), A(S_{Ω}(ω, ·)) = H(ω),
(3.14.3)(a) a_{0}(S_{Ω}(ω, ·)) = \frac{(2π)^{\frac{n−1}{2}} κ_{Ω}^{− \frac{1}{2}} (−ω)}{Γ (\frac{n+1}{2})},

(3.14.3)(b) b_{0}(S_{Ω}(ω, ·)) = \frac{(2π)^{\frac{n−1}{2}} κ_{Ω}^{− \frac{1}{2}} (ω)}{Γ (\frac{n+1}{2})}.

Sketch of Proof. The first statement is clear from the geometrical meaning of S_{Ω}. The second one is divided into three steps:

Step 1. Find the leading term of the asymptotic expansion of S(ω, p) as p → h(ω) (or −h(−ω)). That is to find the formula (3.14.3).

Step 2. S(ω, p) is smooth as a function of ω and (p − h(ω))−^{\frac{1}{2}} near p = h(ω). (Also similar statement near p = −h(−ω).)

Step 3. All coefficients of the power (p − h(ω))−^{\frac{n+k}{2}} (k = 0, 2, 4, …) vanish in the expansion of Step 2.

Step 1 goes back to [John, 1934]. That is, if we write κ_{i}(x) (1 ≤ i ≤ n − 1) for the principal curvature at x, then the first approximation of S(ω, p) is the volume of the

ellipsoid of dimension n − 1 with the radii of curvature (\frac{h(ω)−p}{κ_{i}∘ν^{−1}(ω)})^{\frac{1}{2}} (1 ≤ i ≤ n − 1),

which amounts to \frac{(2π)^{\frac{n−1}{2}} (h(ω)−p)^{\frac{n−1}{2}}}{Γ(\frac{n+1}{2})κ(ω)^{\frac{1}{2}}} .

We can prove Step 2 easily by the implicit function theorem. Step 3 is non-trivial. In fact, Step 3 gives better error terms in the level of the geometrical quantities of a convex domain. If we take its Fourier transform and apply Lemma (2.13) with E = 0 (i.e. ζ ∈ ℝ^{n}) and with f = S_{Ω}(ω, · ), then Lemma (3.14) gives an alternative proof of Theorem 3(ii) in [Herz, 1962], where the ‘most difficult part’ was to improve error terms (see page 83, line 1 there). See [Kobayashi-a, 1989] Lemma (2.2.8) for a detailed proof of Step 2 and Step 3. □

Before stating the main theorem in this section, let us review notation defined so far. Suppose that Ω is a strictly convex domain.

ν ≡ ν_{Ω} : ∂Ω → S^{n−1},  the Gauss map,
K ≡ K_{Ω} : ∂Ω → ℝ_{+},  the Gauss-Kronecker curvature,
κ ≡ κ_{Ω} : S^{n−1} → ℝ_{+},  ω ↦ K_{Ω} ∘ ν_{Ω}^{−1}(ω),
H ≡ H_{Ω} : S^{n−1} → ℝ_{+},  the breadth function,
d ≡ d_{Ω} : S^{n−1} → ℝ,  ω ↦ \frac{log κ(−ω) − log κ(ω)}{2H(ω)},
ℱχ_{Ω}(ζ) = ∫_{Ω} e^{\sqrt{−1}(x,ζ)}dx,
𝒩(Ω) = {ζ ∈ ℂ^{n} : ℱχ_{Ω}(ζ) = 0} .

Finally, we introduce an n + 1 dimensional real analytic manifold

S := S^{n−1} ×_{ℤ_{2}} ℂ^{×} = {ζ · ω : ζ ∈ ℂ^{×}, ω ∈ S^{n−1}} ⊂ ℂ^{n}.

Now we are in a position to describe the asymptotic behavior of the null variety 𝒩(Ω) for a strictly convex domain in ℝ^{n}.

Theorem 3.15. Let Ω be a strictly convex domain in ℝ^{n}. Retain notation as before. Then there exists an integer m_{0} ≡ m_{0}(Ω) depending only on Ω such that we have a disjoint union

(3.15.1)  𝒩(Ω) ∩ S = ( ∐_{m=m_{0}}^{∞} 𝒩_{m}) ∐ (compact set).

Here for each integer m ≥ m_{0}, 𝒩_{m} is a regular submanifold in S(⊂ ℂ^{n}), and is analytically diffeomorphic to S^{n−1}. More precisely, 𝒩_{m} has the following asymptotic behavior: There is a family of analytic maps

(3.15.2)  F_{m} : S^{n−1} ⟶ ℂ (m ∈ ℕ, m ≥ m_{0})

such that

(3.15.3)(a) F_{m}(ω) = \frac{2πm}{H(ω)} + (\frac{π(n − 1)}{2H(ω)} + \sqrt{−1}d(ω)) + O(m^{−1}), as m → ∞.
(3.15.3)(b) F_{m}(ω) = \overline{F_{m}(−ω)}.
(3.15.3)(c) 𝒩_{m} = {F_{m}(ω) · ω : ω ∈ S^{n−1}} (⊂ S ⊂ ℂ^{n}).

In (3.15.3) the estimate of the error terms is uniform with respect to ω ∈ S^{n−1}.

Sketch of Proof. The theorem is a direct consequence of Lemma (3.14) and Corollary (2.20), except for the analyticity of the maps F_{m}, which is derived from the following lemma.

Lemma 3.16. There exists a constant R ≡ R(Ω) > 0 depending only on Ω with the following property: If ℱχ_{Ω}(ζ · ω) = 0, ζ ∈ ℂ, ω ∈ S^{n−1} and if Re ζ > R, then we have

\frac{d}{dζ} ℱχ_{Ω}(ζ · ω) ≠ 0,

where \frac{d}{dζ} is a holomorphic derivative.

See [Kobayashi-a, 1989] Lemma (2.4.25) for the proof of Lemma (3.16).

Example 3.17. Let j(ν, m) (m ∈ ℕ_{+}) be the m-th positive zero of the Bessel function J_{ν}(z). Then the following asymptotic behavior of j(ν, m) is well-known:

j(ν, m) = \frac{π(4m + 2ν − 1)}{4} + O(m^{−1}) as m → ∞.

If Ω is the unit ball in ℝ^{n}, then from Example (1.1.4) we have a disjoint union 𝒩(Ω) =

∐_{m=1}^{∞} ℳ_{m}, where

ℳ_{m} ≡ S (0 : j (\frac{n}{2}, m)) := \{ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n} : ∑_{k=1}^{n} ζ_{k}^{2} = j (\frac{n}{2}, m)^{2}\} .

In this case, H(ω) ≡ 2 and d(ω) ≡ 0, so that F_{m}(ω) = πm + \frac{π(n−1)}{4} + O(m^{−1}) as m → ∞ in (3.15.3)(a). Hence 𝒩_{m} = S ∩ ℳ_{m} is precisely the m-th component in (3.15.1). This is a kind of so-called the phase principle.

Remark 3.18. By virtue of the estimates (2.20.4)(b) and (2.20.4)(c) in Corollary (2.20), we could estimate the ‘exceptional’ compact set of 𝒩(Ω) in (3.15.1) by a suitable Sobolev norm of the boundary ∂Ω. We leave this for an interested reader. However, we might expect that there would be no ‘exceptional compact set’ in Theorem (3.15). Some experiments of computer graphics lead us to the following conjecture:

Conjecture 3.19. In the setting of Theorem (3.15), we would have a disjoint union of countably many regular submanifolds:

(3.19.1)  𝒩(Ω) ∩ S ≃ ∐_{m=1}^{∞} 𝒩_{m}.

This conjecture involves:

(3.19.2)(a) the ‘compact set’ in (3.15.1) would be removed,

(3.19.2)(b) m_{0}(Ω) = 1 (the phase principle).

The following three Corollaries (3.20), (3.21) and (3.22) are due to [Kobayashi, 1986], which are easy applications of Theorem (3.15). See also [Kobayashi-a, 1989] for a detailed proof. We also emphasize that the first result in the direction of Corollary (3.21) was obtained in [Berenstein, 1976]. Another application of Theorem (3.15) will be discussed in §4.

Corollary 3.20. Suppose that Ω is strictly convex domain in ℝ^{n}. Then the following five conditions are equivalent:

(a) π(AF (Ω)) ∋ diag(−1, −1, …, −1). (See (1.1.8)(d) for notation.)

(b) Ω is centrally symmetric with respect to the center of gravity.

(c) GL(𝒩(Ω)) ∋ diag(−1, −1, …, −1). (See (1.1.8)(e) for notation.)

(d) 𝒩(Ω)_{ℝ} contains countably many hypersurfaces in ℝ^{n} as connected components.

(e) 𝒩(Ω)_{ℝ} is composed of a compact set and a disjoint union of countably many closed hypersurfaces in ℝ^{n} without boundaries.

Corollary 3.21 (see also [Berenstein, 1976], [Berenstein, 1980], [Berenstein-Yang, 1987]). Suppose that Ω is a strictly convex domain in ℝ^{n}. Then the following eleven conditions on Ω are equivalent:

(a) Ω is a ball in ℝ^{n}.

(a)′ π(AF (Ω)) = O(n).

(b) GL(𝒩(Ω)) = O(n).

(c) 𝒩(Ω) is composed of countably many concentric complex hyperspheres whose common center is the origin.

(c)′ 𝒩(Ω) contains countably many concentric complex hyperspheres whose common center is the origin.

(c)′′ 𝒩(Ω) contains countably many complex hyperspheres.

(d) 𝒩(Ω)_{ℝ} is composed of countably many concentric hyperspheres whose common center is the origin.

(d)′ 𝒩(Ω)_{ℝ} contains countably many concentric hyperspheres whose common center is the origin.

(d)′′ 𝒩(Ω)_{ℝ} contains countably many hyperspheres.

(d)′′′ 𝒩(Ω)_{ℝ} contains countably many hypersurfaces which approximate hyperspheres asymptotically.

(e) There exist countably many eigenvalues for the overdetermined Neumann problem:

Δ_{ℝ^{n}}u = λu  in Ω
\frac{∂u}{∂ν} |_{∂Ω} = 0,  u|_{∂Ω} ≡ constant,
u ∈ C^{2}(Ω) ∩ C^{1}(Ω̄), u ≢ 0.

Here a complex hypersphere (or a complex quadric) is a subset in ℂ^{n} defined by

S_{ℂ}(a : r) := {ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n} : ∑_{k=1}^{n}(ζ_{k} − a_{k})^{2} = r^{2}} ,

for some a = (a_{1}, …, a_{n}) ∈ ℂ^{n} and r ∈ ℂ. We write S(a : r)_{ℝ} := S_{ℂ}(a : r) ∩ ℝ^{n}. In (d)′′′, ‘asymptotically’ means that there exist a sequence of a(j) ∈ ℝ^{n} (j ∈ ℕ_{+}), an increasing sequence ℝ ∋ r(j) ↑ ∞ (as j → ∞), a constant C > 0, a constant 0 < ε ≤ 1 and a sequence of hypersurfaces X_{j} ⊂ 𝒩(Ω)_{ℝ} such that

dist(X_{j}, S(a(j) : r(j))_{ℝ}) ≤ Cr(j)^{−ε},

for any j ∈ ℕ_{+}. here for closed subsets S, T ⊂ ℝ^{n}, we define

dist(S, T) := max_{x∈S} min_{y∈T} |x − y| + max_{y∈T} min_{x∈S} |x − y|.

We remark that the centers of hyperspheres in (d)′′ and (d)′′′ are not necessarily the origin. The following implications in Corollary (3.21) are obvious:

(a) ⟷ (a)′

↓ (b) ↓

(c) ⟶ (c)′ ⟶ (c)′′
↓ ↓ ↓
(d) ⟶ (d)′ ⟶ (d)′′ ⟶ (d)′′′.

See Theorem (1.4.1) for the the equivalence (c)′ ↔(e).

Corollary 3.22. Suppose that Ω is strictly convex domain in ℝ^{n}. Then the following conditions on Ω are equivalent.

(a) ∂Ω is a hypersurface with constant breadth.

(b) pr(𝒩(Ω) ∩ S) (⊂ ℝ^{n}) contains a sequence of hypersurfaces which approximate hyperspheres asymptotically.

(c) pr(𝒩(Ω) ∩ S) (⊂ ℝ^{n}) is composed of hypersurfaces which approximates hyperspheres asymptotically. Here the projection pr : S = S^{n−1} ×_{ℤ_{2}} ℂ^{×} →

ℝ^{n} is defined by

pr : S ∋ (ω, ξ + \sqrt{−1} η) mod ℤ_{2} ⟼ (ξω_{1}, …, ξω_{n}) ∈ ℝ^{n}.

The equivalence between (a) and (c)′ in Corollary (3.21) can be generalized without convexity assumption:

Theorem 3.23 ([Berenstein-Yang, 1987]). Suppose that Ω is a bounded domain in ℝ^{n} with a connected Lipschitz boundary. Then (a) and (c)′ in Corollary (3.21) are equivalent.

Now that we have described the asymptotic behavior of the null variety 𝒩(Ω) in answer to Problem A in §1.1, we want to solve the injectivity problem (see Problem (B.1.a)) by using Theorem (3.15). Our strategy here is explained in the following diagram (recall the notations in Theorem (3.15)): Problem 3.24. Recover a convex domain Ω from the null variety 𝒩(Ω) in the following procedure:

(3.24)(a) Ω : a strictly convex domain in ℝ^{n}.

> ↓

(3.24)(b) 𝒩(Ω) : the null variety in ℂ^{n}.

> ↓

(3.24)(c) The asymptotic behavior of 𝒩(Ω) ∩ S.

> ↓

H_{Ω} : S^{n−1} → ℝ and d_{Ω} : S^{n−1} → ℝ. (3.24)(d)

> ↓

A differential equation of the supporting function h_{Ω} : S^{n−1} → ℂ :

(3.24)(e) det(D^{2}h_{Ω} + h_{Ω})(ω) = A(ω) det(D^{2}(B − h_{Ω}) + B − h_{Ω})(ω).

> ↓

h_{Ω} : S^{n−1} → ℝ. (3.24)(f)

Here (a)⇒(b) is the definition of the null variety (1.1.2)(a), and Theorem (3.15) gives a differential equation of the supporting function h_{Ω} by means of 𝒩(Ω). The problem is now (e)⇒(f) (see Problem (3.27)).

In order to explain the notation in the partial differential equation (3.24)(e), we recall that for a function f defined on a Riemannian manifold M with a Levi-Civita connection ∇ the Hessian D^{2}f is a (0,2) tensor given by

(3.25)(a)  D^{2}f(X, Y) := X̃Ỹ f − (∇_{X} Ỹ)f  (X, Y ∈ T_{x}M),

where X̃ (resp. Ỹ) is any local extension to a vector field of X (resp. Y). Note that

(3.25)(b)  Δf = Trace(D^{2}f).

Suppose that M = S^{n−1} is equipped with the standard Riemannian metric and that h_{Ω} : S^{n−1} → ℝ is the supporting function of a strictly convex domain Ω in ℝ^{n}. Then we have

(3.25)(c)  det(D^{2}h_{Ω} + h_{Ω})(ω) = \frac{1}{κ_{Ω}(ω)}

For a strictly convex domain Ω in ℝ^{n}, we put

(3.26)(a)  A(ω) := \frac{κ_{Ω}(−ω)}{κ_{Ω}(ω)} ≡ \frac{K_{Ω} · ν_{Ω}^{−1}(−ω)}{K_{Ω} · ν_{Ω}^{−1}(ω)},
(3.26)(b)  B(ω) := H_{Ω}(ω) (≡ h_{Ω}(ω) + h_{Ω}(−ω)).

Then the supporting function h_{Ω} : S^{n−1} → ℝ satisfies the partial differential equation

(3.24)(e). Also a function of the form h_{Ω}(ω) + ∑_{j=1}^{n} a_{j}ω_{j} (ω = (ω_{1}, …, ω_{n}) ∈ S^{n−1}) satisfies

(3.24)(e) for any a = (a_{1}, …, a_{n}) ∈ ℝ^{n}. Consider:

Problem 3.27. Is a solution of (3.24)(e) unique up to a linear function (restricted to S^{n−1})?

This problem is equivalent to Problem (B.1.c) in §1. The affirmative solution of Problem (3.27) leads to the injectivity of 𝒩 for strictly convex domains in ℝ^{n} (see Problem (B.1.a)). If we assume furthermore Ω is a centrally symmetric domain, we can tell h_{Ω}(ω) directly from H_{Ω}(ω) because H_{Ω}(ω) = 2h_{Ω}(ω) after a suitable parallel translation. From the above argument we have as a corollary of Theorem (3.15) the following:

Corollary 3.28. Suppose Ω_{1} and Ω_{2} are strictly convex domains in ℝ^{n} such that 𝒩(Ω_{1}) = 𝒩(Ω_{2}). Assume that Ω_{1} is centrally symmetric with respect to some inner point (i.e. π(GL(Ω)) ∋ diag(−1, −1, …, −1) in the notation (1.1.8)(a),(e)). Then Ω_{1} and Ω_{2} differ from each other only by a parallel translation.

This also gives a partial answer to Problem (B.3) in §1:

Corollary 3.29. Suppose that Ω is a strictly convex domain in ℝ^{n} such that π(GL(Ω)) ∋ diag(−1, −1, …, −1). Then we have

π(AF (Ω)) = GL(𝒩(Ω)).

In the case n = 2, we shall carry out the program (3.24) in §4.1 without the centrally symmetric condition.

§4 Some more results on 𝒩(Ω) in the case Ω in ℝ^{2}

In this section we discuss some more results concerning the null variety 𝒩(Ω) in the case n = 2. §4.1∼4.4 are independent from one another and the reader can enter any of them.

4.1 Injectivity of 𝒩. In this section we treat Problem (B.1.a) in the case of n = 2:

“Is the assignment

𝒩 :{Strictly convex domains in ℝ^{n}} ∋ Ω

↦ 𝒩(Ω) ∈ {analytic sets in ℂ^{n}}

one to one up to parallel displacements?”

To do this we carry out the program (3.24). That is, we prove an affirmative answer to the uniqueness problem of the differential equation (see Problem (3.27) and Problem (B.1.c)), which leads to the following

Theorem 4.1.1 ([Kobayashi-a, 1989], Corollary (2.3.10)). Let Ω_{1} and Ω_{2} be strictly convex domains in ℝ^{2}. If 𝒩(Ω_{1}) = 𝒩(Ω_{2}), then Ω_{1} and Ω_{2} differ from each other by a parallel translation.

Proof. Let Ω be a strictly convex domain in ℝ^{2}. We parametrize S^{1} by θ ∈ [0, 2π) through S^{1} ≃ ℝ/2πℤ. The radius of curvature is given by (see (3.9.1) for notation)

(4.1.2)  ρ ≡ ρ_{Ω} : S^{1} → ℝ_{+}, θ ↦ \frac{1}{κ_{Ω}(θ)} = \frac{1}{K_{Ω} · ν_{Ω}^{−1}(θ)}.

Then from the formula (3.25)(c), we have

ρ(θ) = h(θ) + h′′(θ).

By virtue of Theorem (3.15), we have two functions A ≡ A_{𝒩(Ω)} , B ≡ B_{𝒩(Ω)} : S^{1} → ℝ_{+} which are defined by the asymptotic data of 𝒩(Ω) such that

h(θ) + h′′(θ) = A(θ)(B(θ) − h(θ) + B′′(θ) − h′′(θ)).

(See (3.24)(e), (3.26).) Therefore, we obtain the following linear ordinary differential equation of h(θ) on S^{1}:

(4.1.3)  h_{Ω}(θ) + h′′_{Ω}(θ) = \frac{A_{𝒩(Ω)} (θ)}{1 + A_{𝒩(Ω)} (θ)} (B_{𝒩(Ω)} (θ) + B′′_{𝒩(Ω)} (θ)) .

(Here the right hand side is a well-defined C^{∞} function because A_{𝒩(Ω)} (θ) > 0.) Therefore, h_{Ω}(θ) is uniquely determined by 𝒩(Ω) up to the space

{h(θ) ∈ C^{∞}(S^{1}) : h(θ) + h′′(θ) = 0} = ℝ ⟨cos θ⟩ + ℝ ⟨sin θ⟩ ,

the ℝ-span of cos θ and sin θ. The right-hand side is nothing but a restriction of linear functions of ℝ^{2} to S^{1}, which corresponds to parallel translations. Hence we have proved the injectivity. □

4.2 Characterization of convexity by 𝒩(Ω).

In this section we treat Problem (B.1.b), that is, a characterization of convexity by the asymptotic behavior of 𝒩(Ω) in the case of n = 2. Recall that S = S^{1} ×_{ℤ_{2}} ℂ =

{ζ · ω : ζ ∈ ℂ, ω ∈ S^{1} ⊂ ℝ^{2}} ⊂ ℂ^{2}. Then we have

Theorem 4.2.1. Suppose that Ω is a bounded multiply-connected domain in ℝ^{2} with finitely many analytic boundaries. If 𝒩(Ω) has the following asymptotic behavior (4.2.2), then Ω is a strictly convex domain.

(4.2.2) There exist m_{0} ∈ ℕ, a continuous function H : S^{1} → ℝ_{+}, d : S^{1} → ℝ and F_{m} : S^{1} → ℂ (ℕ ∋ m ≥ m_{0}) such that

F_{m}(ω) = \frac{2mπ}{H(ω)} + \frac{π}{2H(ω)} + \sqrt{−1} d(ω) + o(1) as m → ∞, (4.2.2)(a)

(4.2.2)(b) 𝒩(Ω) ∩ S = ( ∐_{m≥m_{0}} 𝒩_{m}) ∐ (compact set) (disjoint union).

Here we put

𝒩_{m} := {F_{m}(ω) · ω : ω ∈ S^{1}} ⊂ ℂ^{2}.

Moreover, we have H(ω) = H_{Ω}(ω) (see (3.8)) and d(ω) = d_{Ω}(ω) (see (3.9.2)).

A detailed proof will appear in [Kobayashi, in preparation].

Sketch of Proof. The proof is divided into three steps:

Step 1) The number of connected components of ∂Ω is at most two.

Step 2) Ω is either convex or of the form Ω_{1} ∖ Ω_{2} where Ω_{2} ⊂ Ω_{1} are convex.

Step 3) ∂Ω is connected.

We put

(4.2.3)  S_{reg}^{1} := {ω ∈ S^{1} : K(p) ≠ 0 for any p ∈ ν^{−1}(ω) ∪ ν^{−1}(−ω)} .

Since ∂Ω is real analytic, we have S_{reg}^{1} = S^{1} ∖ {finite number of points}. The point of the proof in Steps 1 and 2 is to show the existence of a constant C > 0 such that

(4.2.4)  |(ω, q) − (ω, p)| ≥ C

for any ω ∈ S_{reg}^{1} and p, q ∈ ν^{−1}(ω) ∪ ν^{−1}(−ω) (p ≠ q).

In proving (4.2.4) we use the asymptotic behavior of ℱχ_{Ω}(ζ) and the following elementary result (cf. Problem (B.1.d) in §1):

Lemma 4.2.5. For a_{j} ∈ ℂ^{×} (j = 0, 1, …, n) and x_{j} ∈ ℝ (j = 0, 1, …, n) such that

0 = x_{0} < x_{1} < ⋯ < x_{n}, we put f(m) = ∑_{j=0}^{n} a_{j}e^{\sqrt{−1} mx_{j}}. If lim_{ℕ∋m→∞} f(m) = 0, then we

have x_{j} ∈ πℚ for any j and ∑_{j=0}^{n} a_{j} = 0.

4.3 A result of Brown and Kahane. If Ω is a ball in ℝ^{2}, then every connected component of 𝒩(Ω) is of the form

(4.3.1)  S_{ℂ}(α) := {(ζ_{1}, ζ_{2}) ∈ ℂ^{2} : ζ_{1}^{2} + ζ_{2}^{2} = α^{2}}  (α > 0).

If Ω is a convex domain in ℝ^{2} and 𝒩(Ω) contains S_{ℂ}(α) for some α ∈ ℂ, then Ω is close to a ball in the following sense:

Theorem 4.3.2 ([Brown-Kahane, 1982]). Suppose Ω is a convex domain in ℝ^{2}. If

(4.3.3)  2 min_{ω∈S^{1}} H_{Ω}(ω) ≤ max_{ω∈S^{1}} H_{Ω}(ω),

then 𝒩(Ω) ⊅ S_{ℂ}(α) for any α ∈ ℂ.

Since their proof is simple and elegant, we introduce it here:

Proof. If 𝒩(Ω) ⊃ S_{ℂ}(α), then a result due to Williams asserts that ∂Ω is real analytic (see Theorem (1.4.4)) and α > 0.

Put r := \sqrt{α} (> 0). We put F (ζ, ω) := e^{\sqrt{−1}ζh(−ω)}ℱχ_{Ω}(ζ · ω) for ζ ∈ ℂ and ω ∈ S^{1}. From the formula (3.13) of a Radon transform we have

(4.3.4)  F (ζ, ω) = ∫_{0}^{H(ω)} S(ω, p − h(−ω))e^{\sqrt{−1} pζ}dp,

where S(ω, p) = ∫_{ℝ^{n}} χ_{Ω}(x)δ(p − (x, ω))dx (see (3.12)). The condition 𝒩(Ω) ⊃ S_{ℂ}(α) = S_{ℂ}(r^{2}) implies F (r, ω) = 0 for any ω ∈ S^{1}. We shall show that the condition (4.3.3) would lead to a contradiction. Let us divide into two cases. Put

(4.3.5)  M := max_{ω∈S^{1}} H(ω), m = min_{ω∈S^{1}} H(ω).

Case 1. 0 < r < \frac{π}{m}.

In this case, we fix ω_{0} ∈ S^{1} such that H(ω_{0}) attains the minimum m of H(ω). Then we have

Im F (r, ω_{0}) = ∫_{0}^{H(ω_{0})} S(ω_{0}, p − h(−ω_{0})) sin(rp)dp > 0.

Case 2. \frac{π}{m} ≤ r.

In this case, we have

r ∈ [\frac{π}{m}, ∞) ⊂ [\frac{2π}{M} , ∞) = ⋃_{k∈ℕ_{+}} [\frac{2kπ}{M} , \frac{2(k + 1)π}{M} ] ⊂ ⋃_{k∈ℕ_{+}} [\frac{2kπ}{M} , \frac{2kπ}{m} ]

because \frac{m}{M} ≤ \frac{1}{2} ≤ \frac{k}{k+1} (k ∈ ℕ_{+}). Then

Re F (r, ω_{0}) = ∫_{0}^{H(ω_{0})} S(ω_{0}, p − h(−ω_{0})) cos(rp)dp

= − ∫0^{\frac{2πk}{r}} \frac{∂S}{∂p} (ω_{0}, p − h(−ω_{0}))\frac{sin(rp)}{r}dp.

As the convexity of Ω implies that \frac{∂S}{∂p} is decreasing, we have Re F (r, ω_{0}) < 0. Thus F (r, ω) ≢ 0 as a function of ω ∈ S^{1}.

Remark 4.3.6. The point of the above proof is that \frac{∂S_{Ω}}{∂p} (ω, p) is decreasing as a function of p if Ω is convex in ℝ^{2}. However, this is not true for higher dimensions. In fact for a unit ball Ω in ℝ^{n} we have

S_{Ω}(ω, p) = \frac{π^{\frac{n−1}{2}}}{Γ (\frac{n+1}{2})}(1 − p^{2})^{\frac{n−1}{2}} ,

for −1 < p < 1 (see (9.4.1)). Then we have

\frac{∂^{2}S_{Ω}(ω, p)}{∂p^{2}} = −\frac{2π^{\frac{n−1}{2}}}{Γ (\frac{n−1}{2})}(1 − p^{2})^{\frac{n−5}{2}} \{1 − (n − 2)p^{2}\} .

Thus if n ≥ 4, then \frac{∂S_{Ω}(ω,p)}{∂p} is not necessarily decreasing for −1 < p < 1.

4.4 A result of Garofalo and Segala.

To end §4, we mention another sufficient condition for Pompeiu sets in ℝ^{2} ≃ O(2) ⋉ ℝ^{2}/O(2) which was obtained recently in [Garofalo-Segala, 1991]. The proof also takes an approach of the null variety (cf. §1.4).

Theorem 4.4.1 ([Garofalo-Segala, 1991]). Let Ω ⊂ ℝ^{2} be a bounded simply-connected domain whose boundary ∂Ω is a closed, simple curve parametrized (x_{1}(θ), x_{2}(θ)), θ ∈ [−π, π]. If there exists a polynomial of degree N (≥ 2) with complex coefficients

p(ω) = ∑_{k=0}^{N} a_{k}ω^{k}, ω ∈ ℂ

such that x_{1}(θ) + \sqrt{−1} x_{2}(θ) = p (e^{\sqrt{−1} θ}), then Ω is a Pompeiu set.

§5 Pompeiu problem and a perturbation of a ball in ℝ^{n}.

We are particularly interested in the shape of a domain Ω in ℝ^{n} with the property

(5.1)  𝒩(Ω) ⊃ S_{ℂ}(0 : r),

because of its connection with the Pompeiu problem and Schiffer’s conjecture (see Theorem (1.4.1) and also §1.2 and §1.3). Here we write

S_{ℂ}(a : r) := {ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n}: ∑_{j=1}^{n}(ζ_{j} − a_{j})^{2} = r^{2}} ,

for a = (a_{1}, …, a_{n}) ∈ ℝ^{n} and r ∈ ℝ. We also write

S(a : r) = S_{ℂ}(a : r) ∩ ℝ^{n},
B(a : r) = {x ∈ ℝ^{n}: ∑_{j=1}^{n}(x_{j} − a_{j})^{2} < r^{2}} .

We recall that if Ω is the unit ball in ℝ^{n} then 𝒩(Ω) ⊃ S_{ℂ}(0 : r) whenever J\frac{n}{2} (r) = 0, r > 0.

Loosely speaking, the previous result (Theorem (4.3.2)) due to [Brown-Kahane, 1982] says that if Ω is a long thin convex domain in ℝ^{2} then 𝒩(Ω) ⊅ S_{ℂ}(0 : r) for any r > 0.

Conversely, we shall treat the case where Ω is ‘sufficiently near’ to a ball. Roughly speaking, our results asserts that for a fixed R > 0, if Ω ⊂ ℝ^{n} is sufficiently near to a ball and if there exists r ∈ ℝ such that 0 < r < R and that 𝒩(Ω) ⊃ S_{ℂ}(0 : r), then Ω must be a ball (see Theorem (5.12)). The results in this section are taken from [Kobayashi, 1986]. See [Kobayashi, 1987] Chapter 3, [Kobayashi, 1992] for a detailed proof. See also [Agranovsky, Preprint] for a related result.

For a precise formulation, we need to get rid of the effect of parallel translations and similarity transformations of Ω. In fact, let a ∈ ℝ^{n} and c > 0. We write

(5.2.1)(a)  Ω + a := {x + a ∈ ℝ^{n} : x ∈ Ω} ,
(5.2.1)(b)  c · Ω := {cx : x ∈ Ω}.

Then we have for ζ ∈ ℂ^{n},

(5.2.2)(a)  ℱχ_{Ω+a}(ζ) = e^{\sqrt{−1}(a,ζ)}ℱχ_{Ω}(ζ),
(5.2.2)(b)  ℱχ_{c·Ω}(ζ) = c^{n}ℱχ_{Ω}(cζ),

and correspondingly,

(5.2.3)(a)  𝒩(Ω + a) = 𝒩(Ω),
(5.2.3)(b)  𝒩(c · Ω) = \frac{1}{c}𝒩(Ω).

Therefore, if Ω has the property (5.1) for some r > 0, then so does c · Ω + a for any a ∈ ℝ^{n} and c > 0. Thus we need to define a perturbation of S^{n−1} ‘modulo’ parallel translations and similarity translations. For this purpose we recall spherical harmonics. Let us induce a Riemannian metric g on S^{n−1} from the standard metric on ℝ^{n}. Let Δ be the Laplace-Beltrami operator on S^{n−1} with respect to this Riemannian metric. The orthogonal group O(n) naturally acts on the unit sphere S^{n−1}. This action is transitive and we have

(5.3.1)  S^{n−1} ≃ O(n)/O(n − 1) ≃ SO(n)/SO(n − 1).

As O(n) acts on S^{n−1} isometrically, Δ belongs to 𝔻(O(n)/O(n − 1)), the algebra of O(n)- invariant differential operators on S^{n−1}. As we shall see in §6.6 (see also Example (6.6.5)), we have

(5.3.2)  𝔻(O(n)/O(n − 1)) ≃ 𝔻(SO(n)/SO(n − 1)) ≃ ℂ[Δ].

The Riemannian metric g also induces an O(n)-invariant measure on S^{n−1}. Denote by L^{2}(S^{n−1}) the Hilbert space of square integrable functions with respect to this measure. Then O(n) acts on L^{2}(S^{n−1}) = L^{2}(O(n)/O(n − 1)) unitarily (regular representation) by f(x) ↦ f(g^{−1}x) for f ∈ L^{2}(S^{n−1}), x ∈ S^{n−1} and g ∈ O(n). For k ∈ ℕ we put

(5.4)  E_{k} := {f ∈ C^{∞}(S^{n−1}) : Δf = −k(k + n − 2)f} .

Since Δ is an O(n)-invariant operator, each eigenspace E_{k} is an O(n)-submodule of L^{2}(S^{n−1}). Now we review a classical result on spherical harmonics from the viewpoint of (finite dimensional) representation theory.

Lemma 5.5 (see also [Helgason, 1984], Introduction, Theorem 3.1).

1) L^{2}(S^{n−1}) = ∑_{k=0}^{∞} ⊕E_{k}, (orthogonal Hilbert space decomposition).

2)

dim E_{k} = \{ \begin{matrix}\frac{(2k+n−2)(k+n−3)!}{(n−2)!k!}, & (n ≥ 3), \\ 1, & (n = 2, k = 0), \\ 2, & (n = 2, k ≠ 0). \end{matrix}

3) E_{k} is irreducible as an O(n)-module.

4) E_{k} is the ℂ-span of

{(a_{1}x_{1} + ⋯ + a_{n}x_{n})^{k} : a_{j} ∈ ℂ, ∑_{j=1}^{n} a_{j}^{2} = 0, (x_{1}, …, x_{n}) ∈ S^{n−1}} .

In particular,

E_{0} = ℂ · 1,
E_{1} = ℂx_{1} ⊕ … ⊕ ℂx_{n}.

Sketch of Proof. The space L^{2}(S^{n−1}) is spanned by matrix coefficients of irreducible finite dimensional representations of SO(n) which have non-zero SO(n − 1) fixed vectors. The highest weights of such representations are explicitly characterized by Cartan-Helgason’s theorem (see [Helgason, 1970], 2, III, §3). These representations are also stable under the action of O(n) if n ≥ 3. The eigenvalues of Δ can be calculated as the evaluation by the Casimir element (see Example (6.6.4)) and (2) follows from the Weyl’s dimension formula. We can prove (4) directly. □

We introduce a family of seminorms | · |′_{r} on L^{2}(S^{n−1}) parametrized by r > 0 by

(5.6.1)  |h|′_{r} := \{ ∑_{k=2}^{∞} ∥h_{k}∥_{L^{2}(S^{n−1})}^{2}J_{k+ \frac{n}{2} −1}(r)^{2}\}^{\frac{1}{2}}

for h = ∑_{k=0}^{∞} h_{k} ∈ L^{2}(S^{n−1}) = ∑_{k=0}^{∞} ⊕E_{k}. Since ∑_{k=0}^{∞} J_{k+ \frac{n}{2} −1}(r)^{2} < ∞ (see (9.3.5)(a)), we

have |h|′_{r} < ∞ for any r > 0 and any h ∈ L^{2}(S^{n−1}). Let j(ν, k) (k ∈ ℕ_{+}) be the positive zero of J_{ν}(z) in ascending order. For R > j (\frac{n}{2}, 1) let k_{R} ∈ ℕ_{+} be the integer such that

(5.6.2)  0 < j (\frac{n}{2}, 1) < j (\frac{n}{2}, 2) < ⋯ < j (\frac{n}{2}, k_{R}) ≤ R < j (\frac{n}{2}, k_{R} + 1) .

For h ∈ L^{2}(S^{n−1}), we define

(5.6.3)  |h|_{R} := min_{1≤k≤k_{R}} |h|′_{j(\frac{n}{2} ,k)}.

For a continuous function

f: S^{n−1} ⟶ ℝ_{+},

we define a star-shaped domain Ω(f) with respect to the origin by

(5.7)  Ω(f) := {ρ · η ∈ ℝ^{n} : η ∈ S^{n−1}, 0 ≤ ρ < f(η)} .

Note that for a constant function f ≡ 1 Ω(f) is the unit ball B(0 : 1).

Suppose that

(5.8.1)  g: (−1, 1) × S^{n−1} ⟶ ℝ_{+}

is a function of C^{2}-class such that

(5.8.2)  g(0, η) ≡ 1,  (η ∈ S^{n−1}).

Then Ω(g(t, ·)) is a one-parameter family of domains in ℝ^{n} with Ω(g(0, ·)) being the unit ball. As we have explained at the beginning of this section, the property that 𝒩(Ω) contains some S_{ℂ}(0 : r) (r > 0) is preserved under the affine transform Ω → aΩ+b (a > 0, b ∈ ℝ^{n}).

In order to avoid these unessential perturbations, we need a normalization of g(t, η) as follows: Put

(5.9.1)  h(η) := \frac{∂g}{∂t} (0, η),

and let h(η) = ∑_{k=0}^{∞} h_{k}(η) be the decomposition corresponding to L^{2}(S^{n−1}) = ∑_{k=0}^{∞} ⊕E_{k}

in Lemma (5.5). We call g is normalized if h(η) = \frac{∂g}{∂t} (0, η) satisfies the following three conditions.

(5.9.2)(a)  h_{0}(η) ≡ 0,  (η ∈ S^{n−1}),
(5.9.2)(b)  h_{1}(η) ≡ 0,  (η ∈ S^{n−1}),
(5.9.2)(c)  ∥h∥_{∞} = 1.

Here ∥h∥_{∞} = sup_{η∈S^{n−1}} |h(η)|. The condition (5.9.2)(a) (resp. (5.9.2)(b)) corresponds to

the normalization for similarity transforms (resp. parallel translations). We note that if h ∈ L^{2}(S^{n−1}) satisfies both (5.9.2)(a) and (5.9.2)(b) then

(5.10)  |h|_{R} = 0 ⟺ h = 0

from (9.3.8) in §9 Appendix.

With the notation as above, we put

(5.11)  [g]_{R} := \frac{|\frac{∂g}{∂t} (0, η)|_{R}}{1 + max_{\begin{matrix}|t|≤ \frac{1}{2} \\ η∈S^{n−1} \end{matrix}} |\frac{∂^{2}g}{∂t^{2}} (t, η)|}

Now we are ready to state the main result in this section.

Theorem 5.12 ([Kobayashi, 1986], Theorem 3). Retain notation as above. Fix R ≫ 0 (e.g. R > j (\frac{n}{2}, 1)). Then there exists a constant C(n, R) > 0 with the following property: Let g: (−1, 1) × S^{n−1} → ℝ be any function of C^{2}-class satisfying (5.8.2) and (5.9.2)(a), (b), (c). If there exists a ∈ ℝ^{n}, t ≥ 0 and r > 0 such that

(5.12.1)  0 ≤ t < C(n, R)[g]_{R}.

(5.12.2)  𝒩(Ω(g(t, · )))_{ℝ} ∩ B(0 : R) ⊃ S(a : r),

then a = t = 0 and r = j (\frac{n}{2}, k) for some k (1 ≤ k ≤ k_{R}). That is, Ω(g(t, · )) must be the unit ball.

Corollary 5.13. Let R ≫ 0 and retain the setting as in Theorem (5.12). Assume that there exist

0 < λ < R^{2}. (5.13.1)(a) 0 ≤ t < C(n, R)[g]_{R}. (5.13.1)(b)

u ∈ C^{2}(Ω(g(t, · ))) ∩ C^{1}(\overline{Ω(g(t, · ))}), u ≠ 0 (5.13.1)(c)

such that

Δu = −λu  in Ω(g(t, · )),
\frac{∂u}{∂ν} = 0, u ≡ constant on ∂Ω(g(t, · )),

then Ω(g(t, · )) must be the unit ball.

For the rest of this section, we give an outline of the proof. See [Kobayashi, 1992] for a detailed proof.

Define F: ℂ × ℂ → ℂ by

(5.14.1)  F (s, u) := \frac{(−1)^{n}(n − 1)!}{(\sqrt{−1} u)^{n−1}} \{1 − e^{\sqrt{−1} su} ∑_{k=0}^{n−1} \frac{(\sqrt{−1} su)^{k}}{k!} \} .

By a simple calculation we have

(5.14.2)  \frac{∂F}{∂s} (s, u) = s^{n−1}e^{\sqrt{−1} su},

and thus

(5.14.3)  ∫_{0}^{s} e^{\sqrt{−1} ρu}ρ^{n−1}dρ = F (s, u).

Let f: S^{n−1} → ℝ_{+} be a continuous function and Ω(f) be a star-shaped domain as in (5.7). From (5.14.3) we have

Lemma 5.15. With notation as above,

ℱχ_{Ω(f)}(ζ) = ∫_{S^{n−1}} F (f(η), (η, ζ))dη, ζ ∈ ℂ^{n}.

By using Lemma (5.15), we have a key formula in the proof of Theorem (5.12):

Lemma 5.16. In the setting of Theorem (5.12), we have

(5.16.1)  \frac{∂}{∂t}|_{t=0} ℱχ_{Ω(g(t, · ))}(r · ω) = \frac{(2π)^{\frac{n}{2}}}{r^{\frac{n}{2} −1}} ∑_{k=0}^{∞} \sqrt{−1}^{k}h_{k}(ω)J_{k+ \frac{n}{2} −1}(r),

for r ∈ ℝ and ω ∈ S^{n−1}. Moreover, if g(t, η) satisfies (5.9.2)(a) and (b), then

(5.16.2)  ∥ \frac{∂}{∂t}|_{t=0} ℱχ_{Ω(g(t, · ))}(r · ω)∥_{L^{2}(S^{n−1})} = \frac{(2π)^{\frac{n}{2}}}{r^{\frac{n}{2} −1}} |h|′_{r}.

Proof. By using Lemma (5.15) and the formula (5.14.2), we have

\frac{∂}{∂t}|_{t=0} ℱχ_{Ω(g(t, · ))}(r · ω) = \frac{∂}{∂t}|_{t=0} ∫_{S^{n−1}} F (g(t, η), (η, r · ω))dη

= ∫_{S^{n−1}} e^{\sqrt{−1} r(η,ω)}h(η)dη

= ∑_{k=0}^{∞} ∫_{S^{n−1}} e^{\sqrt{−1} r(η,ω)}h_{k}(η)dη

= \frac{(2π)^{\frac{n}{2}}}{r^{\frac{n}{2} −1}} ∑_{k=0}^{∞} \sqrt{−1}^{k}h_{k}(ω)J_{k+ \frac{n}{2} −1}(r).

We have used the formula (9.3.4)(a) for the fourth equality. The second formula (5.16.2) is a direct consequence of the definition (5.6.1), the first formula (5.16.1) and the assumption (5.9.2)(a),(b) of normalization. □

§6 A quick review of semisimple Lie groups, symmetric spaces, invariant differential operators and integral transforms. The object of this section is to give a rapid outline of some basic concepts of semisim-

ple Lie groups, (Riemannian) symmetric spaces, invariant differential operators and Fourier transforms. We actually develop more material than we will need in this paper, hoping it would make easier for the reader to overview these subjects and their applications in this paper. If a reader is not very familiar with them, then there is enough material in §6 to give him (her) the feeling of these subjects. On the other hand, if the reader is already familiar with them, then he (she) should skip this section, using it only as a dictionary of notation.

§6.5 is devoted to standard structural results on semisimple Lie groups. For more information the reader should consult [Warner, G., 1972] Chapter I, [Helgason, 1978] Chapter IX and [Wallach, 1973] Chapter 7. In §6.7 we will develop some basic facts on the Fourier transform and the Poisson transform, starting from a general framework of integral transforms. See also [Flensted-Jensen, 1986] Chapter IV for a nice survey. Other sections are devoted to a description of 𝔻(G/H), the ring of invariant differential operators on G/H. That is, we start from the relations among 𝔻(G), U(𝔤) and S(𝔤) in the case of a group manifold G/{e} = G in §6.1 and then treat the following cases:

general homogeneous spaces (§6.2)
reductive homogeneous spaces (§6.3)
spherical homogeneous spaces (§6.4)
Riemannian symmetric spaces (§6.6)

The more special the cases we treat are, the finer the structural results we obtain become.

This section §6 is independent of other sections in this paper. A scheme showing the dependence of the various sections is indicated as follows:

6.1

↓
6.2  6.4  6.5
↓  ↓
6.3  ⟶ 6.6 ⟶ 6.7

6.1 Enveloping algebras

Let G be a Lie group and 𝔤 be its Lie algebra. Denote by T (𝔤) the tensor algebra of 𝔤_{ℂ} = 𝔤 ⊗_{ℝ} ℂ, by U(𝔤) the enveloping algebra of 𝔤_{ℂ}, by S(𝔤) the symmetric algebra of 𝔤_{ℂ}.

That is, if we define the two-sided ideals of T (𝔤) by

(6.1.1)(a)  𝒥 := ⟨X ⊗ Y − Y ⊗ X : X, Y ∈ 𝔤_{ℂ}⟩ ,
(6.1.1)(b)  ℐ := ⟨X ⊗ Y − Y ⊗ X − [X, Y] : X, Y ∈ 𝔤_{ℂ}⟩ ,

then we have

(6.1.1)(c)  S(𝔤) ≃ T (𝔤)/𝒥,
(6.1.1)(d)  U(𝔤) ≃ T (𝔤)/ℐ.

We write 𝔻(G) for the ℂ-algebra of left G-invariant differential operators on G. For X ∈ 𝔤 we define a left G-invariant vector field X̃ over G by

(X̃f)(g) := \frac{d}{dt}|_{t=0} f(g exp tX), f ∈ C^{∞}(G), g ∈ G.

We extend a linearization 𝔤 → 𝔻(G), X ↦ X̃ to a ℂ-algebra homomorphism

(6.1.1)(e)  Φ : U(𝔤) → 𝔻(G),

which we can prove to be a surjective ℂ-algebra isomorphism. Take a basis X_{1}, …, X_{n} ∈ 𝔤 over ℝ. We regard P ∈ S(𝔤) as a polynomial over 𝔤^{∗}. Define λ: S(𝔤) → 𝔻(G) by

(6.1.2)  (λ(P)f)(g) := P (\frac{∂}{∂t_{1}} , …, \frac{∂}{∂t_{n}} )|_{t=0} f(g exp(t_{1}X_{1} + ⋯ + t_{n}X_{n})),

for f ∈ C^{∞}(G), g ∈ G. Then λ is a bijective ℂ-linear map which does not depend on the choice of a basis. The composition map λ̃ := Φ^{−1} ∘ λ: S(𝔤) → U(𝔤) is also a bijective ℂ-linear map given by

(6.1.3)  λ̃(X_{1} ⋯ X_{k}) = \frac{1}{k!} ∑_{σ∈𝔖_{k}} X_{σ(1)} … X_{σ(k)}.

Thus we have a diagram:

λ̃ .................................................................................................................................................................................................................. ....................

(6.1.4)  \begin{aligned}S(𝔤) & ⊃ & 𝔤 & ⊂ & U(𝔤) \\ _{λ} ↘ & & ↓ & & ↙ _{Φ} \\ & & 𝔻(G) \end{aligned}

These three maps λ, Φ and λ̃ respect the adjoint action of G.

6.2 Invariant differential operators (general case)

Let H be a closed subgroup of a Lie group G. We denote by 𝔻(G/H) the ℂ-algebra of G-invariant differential operators on G/H. As a generalization of the identification of 𝔻(G) = 𝔻(G/{e}) with the enveloping algebra U(𝔤) via Φ in §6.1, we want to describe 𝔻(G/H) in terms of U(𝔤) ≃ 𝔻(G) and its subalgebras. The results here are valid with a slight modification for G-invariant twisted differential operators which act on sections of an associated line bundle over G/H. The treatment in §6.2 follows [Koornwinder, 1981]. The advantage of our treatment is that the results here can be applied not only to reductive homogeneous spaces (Theorem 6.3.4) but also to non-reductive homogeneous spaces such as horospherical spaces G/MN (see §6.5 for notations). For a straightforward proof in the case of reductive homogeneous spaces, see also [Helgason, 1984], Theorem 4.6 which starts from 𝔻_{H}(G) (see (6.3.2) for definition), which is a subspace of 𝔻(G) (C^{∞}(G/H)) defined below. The proof in this section is formal (sometimes uses a standard argument of graded modules), and we shall give a sketch.

We begin with some elementary linear algebra. Suppose that R is a (possibly noncommutative) algebra and that a vector space U is an R-module. Let V be a subspace of U. V may not be stable under R. We put

(6.2.1)(a)  R(V) := {r ∈ R : rV ⊂ V } ,
(6.2.1)(b)  Ann(V) := {r ∈ R : rV = {0}} .

R(V) is a subalgebra of R and Ann(V) is an ideal of R(V). Then V is naturally an R(V)-module. More precisely, it is a R(V)/ Ann(V)-module as follows:

(6.2.2)

R ⟶ End(U),
∪
0 ⟶ Ann(V) ⟶ R(V) ⟶ End(V).

For example, if R = End(U), then we have a surjection R(V)/ Ann(V) ⟶^{∼} End(V). Now, let V := C^{∞}(G/H), U := C^{∞}(G) and R := 𝔻(G). We regard an element f ∈ C^{∞}(G/H) as a function on G which is right H-invariant, so that we have V ⊂ U. Then the following result can be proved in a local coordinate (the surjectivity is non-trivial):

Lemma 6.2.3. With notation as above, the following is an exact sequence of ℂ-algebras:

0 → Ann(C^{∞}(G/H)) → 𝔻(G)(C^{∞}(G/H)) → 𝔻(G/H) → 0.

By using a filtration of 𝔻(G) according to the degree of differential operators, we have

Theorem 6.2.4 (see [Koornwinder, 1981]). Retain the above notation.

Ann(C^{∞}(G/H)) = 𝔻(G)𝔥. 1)

2) 𝔻(G)(C^{∞}(G/H)) = {D ∈ 𝔻(G) : Ad(h)D − D ∈ 𝔻(G)𝔥 for any h ∈ H }

3) 𝔻(G) = λ(S(𝔪)) ⊕ 𝔻(G)𝔥 as a vector space. 𝔻(G)(C^{∞}(G/H)) ∩ λ(S(𝔪)) ≃ 𝔻(G)(C^{∞}(G/H))/𝔻(G)𝔥 ≃ 𝔻(G/H). 4)

Here the first isomorphism in (4) is as ℂ-vector spaces and the second one is as ℂ-algebras.

Fix a complementary subspace 𝔪 of 𝔤 so that

(6.2.5)  𝔤 = 𝔥 ⊕ 𝔪

is a direct sum of vector spaces. Then the tangent space T_{o}(G/H) at o = eH ∈ G/H is identified with 𝔤/𝔥 ≃ 𝔪. Let σ: S(𝔤) → S(𝔪) be the algebra homomorphism which is an extension of the projection 𝔤 → 𝔪. We put

(6.2.6)(a)  I_{mod}(𝔪) := {P ∈ S(𝔪) : σ ∘ Ad(h)P = P for any h ∈ H} .
(6.2.6)(b)  S(𝔤/𝔥)^{H} := {P ∈ S(𝔤/𝔥): h · P = P for any h ∈ H}.

Then both I_{mod}(𝔪) and S(𝔤/𝔥)^{H} equip with a natural algebra structure and clearly we have an isomorphism I_{mod}(𝔪) ≃ S(𝔤/𝔥)^{H} as ℂ-algebras. With a mild assumption we can conclude that λ: S(𝔤) → 𝔻(G) induces an isomorphism S(𝔤/𝔥)^{H} ≃ 𝔻(G/H) as ℂ-vector spaces. To be precise, we have:

Proposition 6.2.7. Assume λ(I_{mod}(𝔪)) ⊂ 𝔻(G)(C^{∞}(G/H)). Then we have

𝔻(G)(C^{∞}(G/H)) = λ(I_{mod}(𝔪)) ⊕ 𝔻(G)𝔥

In particular, we have isomorphisms as ℂ-vector spaces:

(S(𝔤/𝔥)^{H} ≃) λ(I_{mod}(𝔪)) = 𝔻(G)(C^{∞}(G/H)) ∩ λ(S(𝔪)) (≃ 𝔻(G/H)).

The proof also uses a filtration of S(𝔤) and 𝔻(G). The assumption in Proposition (6.2.7) is satisfied in the case where G/H is a horospherical space G/MN (see §6.5) or a reductive homogeneous space (see §6.3). Even though some of the isomorphisms so far are only as ℂ-vector spaces, they have enough information to determine the ℂ-algebra structure of 𝔻(G/H) in some special cases. For example, we have:

Example 6.2.8. Let G = GL(n, ℝ) and H be the group consisting of strictly upper triangular matrices with the diagonal entries 1. We write 𝔞 for the n-dimensional subspace {diag(a_{1}, …, a_{n}) ∈ 𝔤𝔩(n, ℝ) : a_{j} ∈ ℝ}. Then the assumption in Proposition (6.2.7) is satisfied and we have isomorphisms as ℂ-algebra:

S(𝔤/𝔥)^{H} ≃ S(𝔞) ≃ 𝔻(G/H).

Note that G/H is not reductive (see §6.3 for definition) in this example.

6.3 Invariant differential operators on reductive homogeneous spaces

The coset space G/H is called a reductive homogeneous space if there exists a subspace 𝔪 ⊂ 𝔤 such that

𝔤 = 𝔥 ⊕ 𝔪

is a direct sum decomposition as Ad_{G}(H)-modules (see [Kobayashi-Nomizu, 1969], Chapter X, §2).

Example 6.3.1. The coset space G/H is reductive in the following cases:

1) Ad_{G}(H) is compact.

2) A group manifold G/H = G′ × G′/ diag G′.

3) H ⊂ G ⊂ GL(n, ℝ) are closed subgroups which are stable under trans-

pose.

Suppose that G/H is a reductive homogeneous space. We put

(6.3.2)(a)  I(𝔪) := {P ∈ S(𝔪) : Ad(h)P = P for any h ∈ H} .
(6.3.2)(b)  𝔻_{H}(G) := {D ∈ 𝔻(G) : Ad(h)D = D for any h ∈ H} .

Then it is easy to see the following Lemma 6.3.3. λ(I_{mod}(𝔪)) = λ(I(𝔪)) ⊂ 𝔻_{H}(G) ⊂ 𝔻(G)(C^{∞}(G/H)).

By virtue of Theorem (6.2.4)(3) and Proposition (6.2.7), we have a direct sum of vector spaces:

(6.3.4)  \begin{aligned}𝔻(G) & = λ(S(𝔪)) & ⊕ 𝔻(G)𝔥 \\ ∪ & ∪ & ∥ \\ 𝔻(G)(C^{∞}(G/H)) & = λ(I(𝔪)) & ⊕ 𝔻(G)𝔥 \\ ∪ & ∥ & ∪ \\ 𝔻_{H}(G) & = λ(I(𝔪)) & ⊕ (𝔻_{H}(G) ∩ 𝔻(G)𝔥) \end{aligned}

From Theorem (6.2.4) we conclude: Theorem 6.3.5. Suppose that G/H is reductive. Then we have ℂ-linear bijections

S(𝔪)^{H} = I(𝔪) ⟶^{∼} λ(I(𝔪)) ≃ 𝔻_{H}(G)/𝔻_{H}(G) ∩ 𝔻(G)𝔥

and isomorphisms of ℂ-algebras:

𝔻_{H}(G)/𝔻_{H}(G) ∩ 𝔻(G)𝔥 ≃ 𝔻(G)(C^{∞}(G/H))/𝔻(G)𝔥 ≃ 𝔻(G/H).

6.4 Spherical homogeneous spaces

The ring of invariant differential operators 𝔻(G/H) depends only on its ‘complexification’ G_{ℂ}/H_{ℂ} by an argument of holomorphic continuation. Example 6.4.1. We have a natural ℂ-algebra isomorphism: 𝔻(GL(n, ℝ)/O(p, n − p)) ≃ 𝔻(U(n)/O(n)), through a holomorphic continuation in GL(n, ℝ)/O(p, n − p) ⊂ GL(n, ℂ)/ O(n, ℂ) ⊃ U(n)/O(n).

Suppose that H ⊂ G are both algebraic reductive groups over ℝ. Then G/H is a typical case of Example (6.3.1)(3) (see also [Kobayashi-Ono, 1990], Definition (3.3) and Example (3.6)) and therefore it is a reductive homogeneous space. In this case there exist compact groups H_{U} ⊂ G_{U} such that both G/H and G_{U}/H_{U} have the same complexification. In particular, we have a natural ℂ-algebra isomorphism 𝔻(G/H) ≃ 𝔻(G_{U}/H_{U}) as we saw in Example (6.4.1), and the study of invariant differential operators on G/H can be reduced to the compact case.

In this section §6.4 we give a brief survey on spherical homogeneous spaces, where G is compact and 𝔻(G/H) is commutative. We give various characterizations for spherical homogeneous spaces in Theorem (6.4.3). We try to indicate a sketch of proof as elementary as possible. The reader is supposed to be familiar with (finite) dimensional representation theory of compact groups. The reader may skip this section since we do not use the results here in later sections. In fact, the purpose of §6.4 is only to explain a beautiful and rather simple feature of 𝔻(G/H) for spherical homogeneous spaces that lies between a formal and general result for reductive homogeneous spaces (Theorem (6.3.5)) and a fine result (with some unavoidable notations) for Riemannian symmetric spaces (Theorem (6.6.2))

Suppose that H is a closed subgroup of a connected compact Lie group G. Take a complexified Lie group G_{ℂ} ⊃ G and define a complex subgroup H_{ℂ}(⊂ G_{ℂ}) as a subgroup generated by H and exp(𝔥_{ℂ}). In particular, H_{ℂ} is connected if H is connected. We fix a Borel subgroup B of G_{ℂ}. Let Ĝ be the equivalence class of irreducible (unitary) representations of G. We write a parametrization of Ĝ by highest weights as follows:

(6.4.2)  Ĝ ≃ Hom(B, ℂ^{×}), π_{χ} ⟷ χ.

We also regard π_{χ} as a holomorphic representation of G_{ℂ}. We denote by L^{2}(G/H, τ) for a unitary representation τ of H the L^{2}-sections of the associated vector bundle G ×_{H} τ. By

Frobenius reciprocity Hom_{H}(τ, π|_{H}) ≃ Hom_{G}(π, L^{2}(G/H, τ)) for any τ ∈ Ĥ and π ∈ Ĝ. We write L^{2}(G/H)(= L^{2}(G/H, 1) for the trivial representation 1 ∈ Ĥ as usual. We denote by Ind_{H}^{G} 1 the G-finite part of L^{2}(G/H). That is, Ind_{H}^{G} 1 is the induced representation from 1 ∈ Ĥ in the algebraic sense. Then the following result is due to [Vinberg-Kimel’fel’d, 1978], [Servedio, 1973], [Brion-Luna-Vust, 1986].

Theorem 6.4.3. Suppose G is a connected compact Lie group and H is a closed subgroup of G. Retain the notation as above. Then the following seven conditions are equivalent:

a) #(H_{ℂ}\G_{ℂ}/B) < ∞.

a)′ H_{ℂ} has an open orbit on G_{ℂ}/B.

b) dim Hom_{G}(π, L^{2}(G/H, τ)) ≤ dim τ for any τ ∈ Ĥ and any π ∈ Ĝ.

b)′ dim Hom_{G}(π, L^{2}(G/H)) ≤ 1 for any π ∈ Ĝ.

b)′′ sup dim Hom_{G}(π, L^{2}(G/H)) < ∞.

c) End_{G}(Ind_{H}^{G} 1) is a commutative ring.

c)′ 𝔻(G/H) is a commutative ring.

A homogeneous space with the above equivalent properties is called a spherical homogeneous space. A symmetric pair (G, H) is a typical example of spherical homogeneous space. The spherical homogeneous spaces are classified in [Krämer, 1979] for a simple Lie group G, and in [Brion, 1987] for a semisimple Lie group G. An important ingredient in classification is the dimensional criterion dim G ≤ 2 dim H + rank G (see also [Guillemin-Sternberg, 1984]).

Sketch of proof.

(a)⇒(a)′ Clear.

(b)⇒(b)′ ⇒ (b)′′ Clear.
(c)⇒ (c)′ Clear.
(a)′ ⇒ (b)′ ((a)′ ⇒ (b) is similar.) Let π_{χ} ∈ Ĝ for χ ∈ Hom(B, ℂ^{×}). We define
𝒪(π_{χ}, H) := {f ∈ 𝒪(G_{ℂ}) : f(bgh) = χ(b)f(g)} .

Then by taking highest weight vectors we have

Hom_{G}(π_{χ}, L^{2}(G/H)) ≃ Hom_{G_{ℂ}}(π_{χ}, 𝒪(G_{ℂ}/H_{ℂ}))
≃ 𝒪(π_{χ}, H).

Since a holomorphic function is determined by its restriction to an open set, the existence of an open H_{ℂ} orbit on G_{ℂ}/B implies dim_{ℂ} 𝒪(π_{χ}, H) ≤ 1.

(b)′ ⇒ (a)′ We omit it. See [Vinberg-Kimel’fel’d, 1978], Theorem 1.

(a)′ ⇒ (a) We explain a new proof due to [Matsuki, 1990] which is quite simple and elementary. See [Servedio, 1973] for a proof using algebraic geometry. First consider

G_{ℂ} = SL(2, ℂ). Let A_{ℂ} = \{( \begin{matrix}a & 0 \\ 0 & a^{−1} \end{matrix} ) : a ∈ ℂ^{×}\} and N_{ℂ} = \{( \begin{matrix}1 & z \\ 0 & 1 \end{matrix} ) : z ∈ ℂ\}. If H_{ℂ} has

an open orbit on G_{ℂ}/B, then dim_{ℂ} H_{ℂ} ≥ 1 and therefore H_{ℂ} contains A_{ℂ} or N_{ℂ} after a conjugation by Ad(G_{ℂ}). By an identification G_{ℂ}/B ≃ ℙ^{1}ℂ, the A_{ℂ}-orbits on G_{ℂ}/B are ℂ^{×}∪{0}∪{∞} and the N_{ℂ}-orbits on G_{ℂ}/B are ℂ∪{∞}, and so either of them is finite. By virtue of Matsuki’s rank one reduction (in which the point is that B is a minimal parabolic subgroup of G_{ℂ}, cf. Remark (6.4.5)), the general case follows from the 𝔰𝔩(2, ℂ) case which is the unique complex semisimple Lie algebra of real rank one.

(b)′′ ⇒ (b)′ (The author learned this proof from T. Oshima). It is enough to show that if dim_{ℂ} 𝒪(π_{χ}, H) ≥ 2 for some χ ∈ Hom(B, ℂ^{×}), then dim_{ℂ} 𝒪(π_{χ^{N}} , H) ≥ N +1 for any N ∈ ℕ. Take linearly independent elements f, g ∈ 𝒪(π_{χ}, H). Then {f^{N−k}g^{k} : 0 ≤ k ≤ N}

are linearly independent elements in 𝒪(π_{χ^{N}} , H) because ∑_{k=0}^{N} a_{k}f^{N−k}g^{k} (a_{k} ∈ ℂ) can be

written as a product of N linear functions of f and g and because f, g ∈ 𝒪(G_{ℂ}).

(b)′ ⇔ (c) This is essentially the following fact: GL(n, ℂ) is commutative if and only if n = 1.

(c)′ ⇒ (b)′ and (c) We need a result asserting that 𝔻(G/H) is dense in End_{G}(Ind_{H}^{G} 1) in some sense. We can show U(𝔤) is dense in ℬ(G) = 𝒜(G)′ as G is compact, and then 𝔻(G/H) is dense in End_{G,conti}(𝒜(G/H)). □

Remark 6.4.4. The conditions (a), (a)′, and (c) in Theorem (6.4.3) depend only on a complexification G_{ℂ}/H_{ℂ}.

Remark 6.4.5. The implication (a)′ ⇒ (a) does not always hold if we replace a Borel sub-

group by a general parabolic subgroup. For example, let G_{ℂ} = \overbrace{SL(n, ℂ) × … × SL(n, ℂ)}^{n+1},

H_{ℂ} = diag(SL(n, ℂ)) and P be a parabolic subgroup so that G_{ℂ}/P = \overbrace{ℙ^{1}ℂ × … × ℙ^{1}ℂ}^{n+1}. Then H_{ℂ} has an open orbit on G_{ℂ}/P. But #(H_{ℂ}\G_{ℂ}/P) = ∞ if n ≥ 3. In this case P is a Borel subgroup of G_{ℂ} if and only if n = 2. If n = 2, then #(H_{ℂ}\G_{ℂ}/P) < ∞ and the unique open sense orbit means a well-known fact that any three points in ℙ^{1}ℂ can be transformed into {0, 1, ∞} by a linear-fractional transform.

Remark 6.4.6. A Riemannian symmetric space of the compact type is a spherical homogeneous space. In this case, the property (a)′ in Theorem (6.4.3) is easy to see from the Iwasawa decomposition (6.5.16)(c) for a dual symmetric space of the non-compact type. In particular, we conclude that the ring of invariant differential operators 𝔻(G/H) for a semisimple symmetric space is commutative. It is known that 𝔻(G/H) depends only on the Lie algebras (𝔤, 𝔥) and does not depend on a disconnected part in this semisimple symmetric case. We also indicate a different approach to a finer structural result on 𝔻(G/H) based on spherical functions and on §6.6.

6.5 Semisimple Lie groups

In this section we review some standard materials on semisimple Lie groups. This section is written for the readers who have a basic knowledge of Lie groups (e.g. [Warner, F., 1981]) but who are not familiar with semisimple groups.

Let G be a Lie group, and 𝔤 be its Lie algebra. The Killing form B: 𝔤 × 𝔤 → ℝ is a ℝ-bilinear form given by

(6.5.1)  B(X, Y) := Trace(ad X ad Y), X, Y ∈ 𝔤.

Then B is G-invariant, that is, B(Ad(g)X, Ad(g)Y) = B(X, Y) for any g ∈ G, X, Y ∈ 𝔤.

A connected Lie group G is called semisimple if B is non-degenerate. Then, the center Z(G) ≃ Ker(Ad: G → GL(𝔤)) is discrete.

Example 6.5.2. G = SL(n, ℝ), SO(n, ℂ), Sp(n, ℝ), Sp(n, ℂ), SO_{0}(p, q) (p + q ≥ 3), SU(p, q), Sp(p, q), … are semisimple Lie groups.

Suppose that G is semisimple. Let (p, q) be the signature of B, that is, the numbers of positive (resp. negative) eigenvalues with multiplicities. Then p + q = dim 𝔤 and Ad(G) ⊂ SO(p, q). In particular, if B is negative definite, then Ad(G) is a subgroup of a compact group SO(0, q) = SO(dim 𝔤). This suggests G should be compact. This is the case. In fact we have Theorem 6.5.3. Suppose G is a connected Lie group. Then the following two conditions are equivalent:

a) B is negative definite,

b) G is compact semisimple.

In fact, the implication (a)⇒(b) follows from an observation that Ad(G) ⊂ SO(dim 𝔤) as before, that Ad(G) is closed in GL(𝔤) (Lemma (6.5.4)) and that Ker Ad = Z(G) is finite (Weyl).

If 𝔤 is a semisimple Lie algebra, then Ad(G) is a linear Lie group (i.e. Ad(G) is a subgroup of GL(n, ℝ) with n = dim 𝔤) whose Lie algebra is 𝔤. In general, a linear semisimple Lie group has the following properties:

Lemma 6.5.4. Suppose G is a connected semisimple Lie group which is a subgroup of GL(n, ℝ). Then we have

1) (See [Yosida, 1938]) G is closed in SL(n, ℝ).

2) There exists an element g of GL(n, ℝ) such that G^{g} = ^{t}(G^{g}), where G^{g} := gGg^{−1}. Here ^{t} denotes the transpose in GL(n, ℝ).

Definition 6.5.5. Suppose that G is a connected semisimple Lie group. If σ ∈ Aut(G) is an involutive automorphism (i.e. σ^{2} = 1) and if H is an open subgroup of G^{σ} := {g ∈ G : σg = g}, then G/H is called a semisimple symmetric space. We also write σ (i.e. σ ∈ Aut(𝔤)) for the derivation of σ. Let

(6.5.5)(a)  𝔤 = 𝔥 + 𝔮

be the decomposition into +1, −1 eigenspaces of σ ∈ Aut(𝔤). Then 𝔥 = Lie(H). σ is called a Cartan involution if B|_{𝔮×𝔮} is positive definite and if B|_{𝔥×𝔥} is negative definite. If σ is a Cartan involution, then we usually write θ for σ, 𝔤 = 𝔨 + 𝔭 for 𝔤 = 𝔥 + 𝔮 ((6.5.5)(a)). We write K := G^{θ}. Then the coset space G/K carries a G-invariant metric g (e.g. the one induced from B|_{𝔭×𝔭}). There exists a unique map s: G/K → Isom(G/K, g) (the group of the isometries) such that s(x) · x = x and ds(x)_{x} = −id|_{T_{x}G/K} for any x ∈ G/K. In view of this, (G/K, g) is called a Riemannian symmetric space of the non-compact type.

Example 6.5.6.

a) G is compact semisimple. θ ≡ 1 is a Cartan involution of G.

b) G = SL(n, ℝ). θ(g) = ^{t}g^{−1} gives a Cartan involution of G.

Suppose G is a linear Lie group. Retain notation as in Lemma (6.5.4). Then θ(x) := tx^{−1} (x ∈ SL(n, ℝ)) (see Example (6.5.6)) defines a Cartan involution of G^{g} = gGg^{−1}, and so induces that of G. In particular, this shows that there exists a Cartan involution of Ad(G). In general we have:

Lemma 6.5.7. Suppose that G is a connected semisimple Lie group.

1) There exists a Cartan involution θ of G.

2) If θ_{1}, θ_{2} are Cartan involutions, then there exists an element g of G such that θ_{1} = Ad(g) ∘ θ_{2} ∘ Ad(g)^{−1}.

3) If θ is a Cartan involution of G, then K := G^{θ} is connected and Ad_{G}(K) is compact (recall Ad_{G} ≡ Ad : G → GL(𝔤)).

Suppose that G is a noncompact connected semisimple Lie group. Retain notations as above. Fix a maximal abelian subspace 𝔞 of 𝔭. The dimension of 𝔞 over ℝ is called the real rank of G denoted by ℝ-rank, or the rank of the symmetric space G/K. This definition is independent of the choice of 𝔞. In fact we have

Lemma 6.5.8 (Cartan). If 𝔞_{1}, 𝔞_{2} are maximal abelian subspaces of 𝔭, then there exists an element k ∈ K such that Ad(k)𝔞_{1} = 𝔞_{2}. □

Sketch of proof (see [Hunt, 1956]). Fix generic elements H_{i} ∈ 𝔞_{i}. Here, by generic, we mean {X ∈ 𝔭: [X, H_{i}] = 0} = 𝔞_{i}. As K is compact, the function f: K → ℝ, f(u) := B(Ad(u)H_{1}, H_{2}) has a critical value, say, at k ∈ K. Then the vanishing of the first derivative of f tangent to K at k yields [Ad(k)H_{1}, H_{2}] = 0, which in turn implies Ad(k)𝔞_{1} = 𝔞_{2}.

In the above proof, it is easy to see that the number of the critical points are the order of the Weyl group W defined in (6.5.15)(c).

We write 𝔞_{ℂ} = 𝔞 ⊗_{ℝ} ℂ, 𝔞^{∗} := Hom_{ℝ}(𝔞, ℝ) and 𝔞_{ℂ}^{∗} := Hom_{ℝ}(𝔞, ℂ) = Hom_{ℂ}(𝔞_{ℂ}, ℂ). For

α ∈ 𝔞^{∗}, the root space of 𝔤 with respect to 𝔞 is given by

(6.5.9)(a)  𝔤(𝔞 : α) := {X ∈ 𝔤 : [H, X] = α(H)X, for all H ∈ 𝔞} .

The finite set

(6.5.9)(b)  Σ ≡ Σ(𝔤, 𝔞) := {α ∈ 𝔞^{∗} : 𝔤(𝔞 : α) ≠ 0} ∖ {0}

is called the restricted root system of 𝔤. Fix an element H_{0} ∈ 𝔞 such that α(H_{0}) ≠ 0 for any α ∈ Σ(𝔤, 𝔞). We put

(6.5.10)(a)  Σ^{+} ≡ Σ^{+}(𝔤, 𝔞) := {α ∈ Σ(𝔤, 𝔞) : α(H_{0}) > 0} ,
(6.5.10)(b)  𝔞_{+} := {H ∈ 𝔞 : α(H) > 0 for any α ∈ Σ^{+}(𝔤, 𝔞)} ,
(6.5.10)(c)  𝔫 := ∑_{α∈Σ^{+}} 𝔤(𝔞 : α),
(6.5.10)(d)  𝔫̄ := θ𝔫 = ∑_{α∈Σ^{+}} 𝔤(𝔞 : −α),
(6.5.10)(e)  𝔪 := 𝔨 ∩ 𝔤(𝔞 : 0) = {X ∈ 𝔨 : [H, X] = 0 for any H ∈ 𝔞} ,
(6.5.10)(f)  ρ := \frac{1}{2} ∑_{α∈Σ^{+}} dim_{ℝ} 𝔤(𝔞 : α)α = \frac{1}{2} trace(ad |_{𝔞}(𝔫 → 𝔫)).

Lemma 6.5.11. With notation as above, we have direct sum decompositions as ℝ-vector spaces.

(6.5.11)(a)  𝔤 = 𝔨 ⊕ 𝔭  (Cartan decomposition),
(6.5.11)(b)  = 𝔫̄ ⊕ 𝔞 ⊕ 𝔪 ⊕ 𝔫 (Gel’fand-Naimark decomposition),
(6.5.11)(c)  = 𝔨 ⊕ 𝔞 ⊕ 𝔫  (Iwasawa decomposition).

Sketch of Proof. The first two decompositions are clear from definition. Iwasawa decomposition follows from Gel’fand-Naimark decomposition and from the following bijection:

𝔫̄ + 𝔪 ⟶^{∼} 𝔨, (X, Y) ⟼ X + θX + Y.

Alternatively, it can be deduced from the result for 𝔤 = 𝔰𝔩(n, ℝ) case (see Example (6.5.12)) by using a compatible choice of a Cartan involution (Lemma (6.5.4)(2)).

Example 6.5.12. Let G = SL(n, ℝ) and θ ∈ Aut(G) be given by θ(g) = ^{t}g^{−1}. Then we have K =SO(n) and 𝔭={X ∈M(n, ℝ) : Trace X =0, X =^{t}X}. Let 𝔞 := {diag(a_{1}, …, a_{n})

∈ M(n, ℝ) : a_{j} ∈ ℝ, ∑_{j=1}^{n} a_{j} = 0\} and H_{0} := diag (\frac{n−1}{2}, \frac{n−3}{2}, …, −\frac{n−1}{2}) ∈ 𝔞. Then 𝔪 = \{0\}

and 𝔫 (resp. 𝔫̄) is the set of strictly upper (resp. lower) matrices.

Let N (resp. A) be the analytic subgroup of G having 𝔫 (resp.𝔞) as its Lie algebra. Then

Lemma 6.5.13. All of the following maps

(6.5.13)(a)  exp : 𝔭 ⟶ exp(𝔭)
(6.5.13)(b)  exp : 𝔞 ⟶ A
(6.5.13)(c)  exp : 𝔫 ⟶ N
(6.5.13)(d)  exp : 𝔞 + 𝔫 ⟶ AN

are surjective C^{ω}-diffeomorphisms.

Remark 6.5.14. In general, exp : 𝔤 → G is neither injective nor surjective.

Definition 6.5.15. We put

(6.5.15)(a)  M := Z_{K}(𝔞) = {g ∈ K : Ad(g)H = H for any H ∈ 𝔞} ,
(6.5.15)(b)  M′ := N_{K}(𝔞) = {g ∈ K : Ad(g)𝔞 = 𝔞} ,
(6.5.15)(c)  W := M′/M,
(6.5.15)(d)  P := MAN  (⊂ G).

W is a finite group which is naturally identified with the Weyl group of the root system Σ(𝔤, 𝔞). P is called a minimal parabolic subgroup of G and P = MAN is called a Langlands decomposition. If G is a complex semisimple Lie group, then MA ≃ (ℂ^{×})^{ℓ} where ℓ = ℝrank G and P is called a Borel subgroup. We should remark that 𝔭 = {X ∈ 𝔤: θX = −X} is merely a subspace of 𝔤 and is not a Lie subalgebra corresponding to P.

Now we have a global version of Lemma (6.5.11).

Theorem 6.5.16. Suppose that G is a connected semisimple Lie group. Retain notation as above. We have

(6.5.16)(a)  G = (exp 𝔭)K (Cartan decomposition),
(6.5.16)(b)  = K\overline{A_{+}}K  (Cartan decomposition),
(6.5.16)(c)  = NAK  (Iwasawa decomposition),
(6.5.16)(d)  = ∐_{w∈W} NwP (Bruhat decomposition).

Here A_{+} := exp 𝔞_{+} (see (6.5.10)(b)) and \overline{A_{+}} denotes the closure of A_{+}. Furthermore, both of the following maps

𝔭 × K ⟶ G, (X, k) ⟼ (exp X)k, (6.5.16)(e) N × A × K ⟶ G, (n, a, k) ⟼ nak, (6.5.16)(f)

are C^{ω}-diffeomorphisms.

Sketch of Proof. In the case where G is linear (i.e. G ⊂ GL(n, ℝ) for some n, the proof except for (6.5.16)(d) can be reduced from the result for the case G = SL(n, ℝ) (or GL(n, ℝ)) by using a compatible choice of a Cartan involution (Lemma (6.5.4)(2). The result for a non-linear case is also reduced to a linear case. In the case of G = GL(n, ℝ), (6.5.16)(e) is a polar decomposition and the inverse map is given by g ↦ (\frac{1}{2} log(g^{t}g), exp (\frac{1}{2} log(g^{t}g)) g) where log: exp(𝔭) → 𝔭 is the inverse of (6.5.13)(a). The inverse map of (6.5.16)(f) for the case G = GL(n, ℝ) can be written explicitly in the process of Gram-Schmidt’s orthogonalization. We omit (6.5.16)(d). □

One of the points in Theorem (6.5.16) is that (6.5.16)(b) and (c) give orthogonal coordinate systems, that is, a polar coordinate and a horospherical one.

Definition 6.5.17 (Iwasawa projection). We define a map

A: G → 𝔞

by the property g ∈ Ne^{A(g)}K. The map A is called the Iwasawa projection.

6.6 Invariant differential operators on Riemannian symmetric spaces

In a proof of the Cartan decomposition G = K\overline{A_{+}}K (see (6.5.16)(b)) we use the fact that 𝔞 ↪ 𝔭 induces an isomorphism of the quotient spaces

𝔞/W ⟶^{∼} 𝔭/K.

We might anticipate ℂ-algebra isomorphisms 𝔻(G/K) ? ≃ S(T_{o}(G/K))^{K} ≃ S(𝔭)^{K} ? ≃ S(𝔞)^{W}. However, the first isomorphism is only as ℂ-vector spaces (see Theorem (6.3.5)) and is not as ℂ-algebras. The third isomorphism S(𝔭)^{K} ≃ S(𝔞)^{W} is true (Chevalley’s restriction theorem, see also [Helgason, 1984], p.340). Instead of the Cartan decomposition (6.5.16)(b), if we use the Iwasawa decomposition (6.5.16)(c), we can establish a ℂ-algebra isomorphism 𝔻(G/K) ≃ S(𝔞)^{K} as follows: By virtue of the Iwasawa decomposition 𝔤 = 𝔨 + 𝔞 + 𝔫 (see Lemma (6.5.11)(c)) and the Poincaré-Birkhoff-Witt theorem, it follows that

(6.6.1)  U(𝔤) = 𝔨_{ℂ}U(𝔤) ⊕ U(𝔞) ⊕ U(𝔤)𝔫_{ℂ}.

Let δ be the projection of U(𝔤) onto U(𝔞) with respect to this decomposition. Let η: U(𝔞) → U(𝔞) be the ℂ-algebra homomorphism generated by η(X) := X −ρ(X) (X ∈ 𝔞) (recall (6.5.10)(f) for the definition of ρ). Put γ := η ∘ δ: U(𝔤) → U(𝔞). Then we have

Theorem 6.6.2. Retain notation as above. If we restrict a homomorphism γ: U(𝔤) → U(𝔞) to U(𝔤)^{K} ≃ 𝔻_{K}(G), then we have a ℂ-algebra isomorphism:

U(𝔤)^{K}/U(𝔤)^{K} ∩ U(𝔤)𝔨_{ℂ} ⟶_{γ}^{∼} U(𝔞)^{W} .

Combining with Theorem (6.3.5) for reductive homogeneous spaces, we have ℂ-algebra isomorphisms

(6.6.2)(a)  𝔻(G/K) ⟵_{Γ}^{∼} U(𝔤)^{K}/U(𝔤)^{K} ∩ U(𝔤)𝔨_{ℂ} ⟶_{γ}^{∼} U(𝔞)^{W} .

This theorem is a first non-trivial result in this section §6. See [Helgason, 1984], Chapter II, Theorem 5.17 for a proof, where the surjectivity of γ is based on a result on spherical functions. We also note that Trossian has recently obtained a structural theorem for 𝔻(G/K) for a general symmetric space.

Since 𝔞 is an abelian Lie algebra, we can identify U(𝔞) with the polynomials in 𝔞_{ℂ}^{∗}. By a theorem due to Chevalley, U(𝔞)^{W} is isomorphic to a polynomial ring generated by (dim 𝔞)-algebraically independent homogeneous elements and 1. For λ ∈ 𝔞_{ℂ}^{∗} we define an algebra homomorphism χ_{λ}: 𝔻(G/K) → ℂ by χ_{λ}(Γ(u)) := γ(u)(λ) for u ∈ U(𝔤)^{K}. Then we have a bijection

(6.6.3)  Hom_{ℂ−alg}(𝔻(G/K), ℂ) ≃ 𝔞_{ℂ}^{∗}/W, χ_{λ} ⟷ λ.

Example 6.6.4. Retain the setting as before. The Killing form B induces an identification 𝔤 ≃ 𝔤^{∗}. Let us recall the notation (6.1.1)(d) and consider the following composition map:

(6.6.4)(a)  φ: 𝔤^{∗} ⊗ 𝔤^{∗} ≃_{B} 𝔤 ⊗ 𝔤 ⊂ T (𝔤) → T (𝔤)/ℐ ≃ U(𝔤).

The image of the Killing form B ∈ 𝔤^{∗} ⊗ 𝔤^{∗} under the map φ is called the Casimir element denoted by c_{G} ∈ U(𝔤). Since φ is a G-module map under the adjoint representation of G, c_{G} is G-invariant, that is, c_{G} ∈ U(𝔤)^{G} ⊂ U(𝔤)^{K}. In the isomorphisms 𝔻(G/K) ⟵_{Γ}^{∼}

U(𝔤)^{K}/U(𝔤)^{K} ∩ U(𝔤)𝔨_{ℂ} ⟶_{γ}^{∼} U(𝔞)^{W} (see (6.6.2)(a)), we have by a simple calculation

(6.6.4)(b)  Γ(c_{G}) = Δ_{G/K},
(6.6.4)(c)  γ(c_{G}) = ∑_{i=1}^{ℓ} H_{i}^{2} + 2H_{ρ}.

Here Δ_{G/K} is the Laplacian on G/K with a Riemannian metric coming from the restriction of B to 𝔭 ≃ T_{o}(G/K), o = eK ∈ G/K. {H_{1}, …, H_{ℓ}} is an orthonormal basis of 𝔞 with respect to B|_{𝔞×𝔞}, ℓ = dim_{ℝ} 𝔞 and H_{ρ} ∈ 𝔞 is the corresponding element to ρ ∈ 𝔞^{∗} ((6.5.10)(f)) via the identification 𝔞 ≃ 𝔞^{∗} by B|_{𝔞×𝔞}.

Example 6.6.5. Suppose that G is a connected semisimple Lie group of ℝ-rank one. Then the Weyl group W ≃ ℤ_{2} acts on 𝔞 ≃ ℝ as a change of the sign. If we identify S(𝔞) with a polynomial ring ℂ[x], then S(𝔞)^{W} ≃ ℂ[x^{2}]. In this case we have an isomorphism of ℂ-algebras:

(6.6.5)(a)  𝔻(G/K) ≃ ℂ[Δ_{G/K}].

More generally, the ring of invariant differential operators on a two point Riemannian homogeneous space is generated by the Laplacian. A Riemannian symmetric space of rank one is such a typical example.

6.7 Fourier transform and Poisson transform

In this section we review the Fourier transform on a Riemannian symmetric space, which we shall use in §8. We shall treat it in a general framework so that it leads us naturally to the definition of the Fourier transform and its ‘adjoint’, the Poisson transform.

We begin with an observation in a general context. Let G be a Lie group and H_{1}, H_{2} closed subgroups of G. For simplicity we assume that G/H_{i}(i = 1, 2) has a G-invariant measure. Given K ∈ Γ(G/H_{2} × G/H_{1}) we define integral transforms

(6.7.1)(a)  T ≡ T_{K} : Γ(G/H_{1}) ⟶ Γ(G/H_{2}),
(6.7.1)(b)  S ≡ S_{K} : Γ(G/H_{2}) ⟶ Γ(G/H_{1}).

by

(T f)(x) := ∫_{G/H_{1}} K(x, y)f(y)dy,
(SF)(y) := ∫_{G/H_{2}} K(x, y)F (x)dx.

Here Γ denotes a suitable class of function spaces that we do not specify at this moment. (More precisely, (6.7.1)(a) should read T : Γ_{1}(G/H_{1}) → Γ_{2}(G/H_{2}) and (6.7.1)(b) should read S : Γ_{3}(G/H_{2}) → Γ_{4}(G/H_{1}), where Γ_{i} are suitable classes of function spaces.) We note that

(6.7.1)(c)  ∫_{G/H_{2}}(T f)(x)F (x)dx = ∫_{G/H_{1}} f(y)(SF)(y)dy

if the integral makes sense for f ∈ Γ(G/H_{1}), F ∈ Γ(G/H_{2}). Since the role of T and S is the same, we shall treat only T for a while. (Later T will be the Poisson transform and S will be the Fourier transform on G/H if we take H_{1} = P, a minimal parabolic subgroup of G, if we take H_{2} = H such that HP ⊂ G is open and if we also generalize Γ(G/H_{1}) to sections of line bundles.)

Here are some examples of a class of function spaces Γ.

Example 6.7.2. The following T_{K} is well-defined and continuous.

a)  K ∈ 𝒟′(G/H_{2} × G/H_{1}), T_{K} : C^{∞}(G/H_{1}) → C^{∞}(G/H_{2}).
b)  K ∈ L^{2}(G/H_{2} × G/H_{1}), T_{K} : L^{2}(G/H_{1}) → L^{2}(G/H_{2}).
c)  K ∈ L^{∞}(G/H_{2} × G/H_{1}), T_{K} : L^{1}(G/H_{1}) → L^{∞}(G/H_{2}).

It corresponds to the condition

(6.7.3)  K(gx, gy) = K(x, y) for any g ∈ G

that T_{K} (or S_{K}) is a G-map when G acts on Γ(G/H_{1}) and Γ(G/H_{2}) by translations. Suppose that K satisfies (6.7.3). We put

(6.7.4)  V (g) := K(1, g).

Then we have

K(x, y) = K(1, x^{−1}y) = V (x^{−1}y). (6.7.5)(a) V ∈ Γ(H_{2}\G/H_{1}). (6.7.5)(b)

In particular, we might expect

(6.7.6)  dim Hom_{G,continuous}(Γ(G/H_{1}), Γ(G/H_{2})) ≤ #(H_{2}\G/H_{1}),

for a suitable class of function spaces. In particular, it is interesting in the case where H_{2} has finite orbits on G/H_{1}. The above observation can be extended formally to sections of associated vector bundles over G/H_{i} (i = 1, 2) after a minor modification.

From now on, we suppose that G is a connected semisimple Lie group. Retain notation in §6.5. In particular, θ is a Cartan involution, K = G^{θ} and P = MAN is a Langlands decomposition of a minimal parabolic subgroup of G. From the Iwasawa decomposition (6.5.16)(c) we have #(K\G/P) = 1. Let us apply the above argument with H_{1} = P and H_{2} = K (in the case of sections of a line bundle). Let λ ∈ 𝔞_{ℂ}^{∗}. Let Γ be one of the sheaf of 𝒜 ⊂ C^{∞} ⊂ 𝒟′ ⊂ ℬ (hyperfunctions). If Y is a compact manifold, then ℬ(Y) is the same as the set of analytic functionals on Y. We define the Γ-valued spherical principal series by (6.7.7)

Γ(G/P, ℒ_{λ}) := {f ∈ Γ(G) : f(gman) = a^{λ−ρ}f(g), for any m ∈ M, a ∈ A, n ∈ N}.

Since #(K\G/P) = 1, a K-invariant element in Γ(G/P, ℒ_{λ}) is an analytic function on G and is determined by the value at e. We define

(6.7.8)(a)  1_{λ} ∈ 𝒜(G/P, ℒ_{λ})

by

(6.7.8)(b)  \{ \begin{matrix}1_{λ}(kg) = 1_{λ}(g) & for any k ∈ K, \\ 1_{λ}(e) = 1. \end{matrix}

Then we have

(6.7.8)(c)  1_{λ}(g) = e^{⟨−λ+ρ,A(g^{−1})⟩}

by using the Iwasawa projection A: G = NAK → 𝔞 (see Definition (6.5.17)). We put

(6.7.9)  p_{λ}(x, g) := 1_{λ}(x^{−1}g), for x, g ∈ G.

Note that for each x ∈ G (or x ∈ G/K) p_{λ}(x, g) ∈ 𝒜(G/P, ℒ_{λ}). If G = SL(2, ℝ), the kernel p_{λ}(x, g) is the classical kernel of the Poisson integral, raised to a power. As we saw in (6.7.1), the kernel p_{λ}(x, g) defines a pair of G-module maps which we call the Poisson transform and the Fourier transform. That is:

The Poisson transform

(6.7.10)  𝒫_{λ} : ℬ(G/P, ℒ_{λ}) → ℬ(G/K), f ↦ 𝒫_{λ}f

is defined by

(6.7.10)(a) (𝒫_{λ}f)(x) : = ∫_{K/M} f(k)1_{−λ}(x^{−1}k)dk = ∫_{K/M} f(k)p_{−λ}(x, k)dk
= ∫_{K/M} f(xk)dk

for x ∈ G, where dk is the K-invariant measure on K/M ≃ G/P such that ∫_{K/M} dk = 1. The Radon-Fourier transform introduced by [Helgason, 1965] is defined for F ∈ ℰ′(G/K) and for ζ ∈ 𝔞_{ℂ}^{∗} and g ∈ G as

(6.7.11)(a)  (ℱF)(ζ, g) : = ∫_{G/K} F (x)p_{−\sqrt{−1}ζ}(x, g)dx
= ∫_{G/K} F (x)e^{⟨\sqrt{−1}ζ+ρ,A(g^{−1}x)⟩}dx.

Then for each fixed ζ ∈ 𝔞_{ℂ}^{∗}, (ℱF)(ζ, g) ∈ 𝒜(G/P, ℒ_{−\sqrt{−1}ζ}). In a compact picture 𝒜(G/P, ℒ_{−\sqrt{−1}ζ}) ≃ 𝒜(K/M), the (Radon-)Fourier transform is a regarded as a map

(6.7.11)(b)  ℱ: ℰ′(G/K) ⟶ 𝒜(𝔞_{ℂ}^{∗} × K/M).

The price to pay in a compact picture (6.7.11)(b) is that the G-equivariance of ℱ is less obvious.

Now let us review some facts on these transforms. For λ ∈ 𝔞_{ℂ}^{∗}, we define

(6.7.12) 𝒜(G/K, ℳ_{λ}) : = {φ ∈ ℬ(G/K) : Dφ = χ_{λ}(D)φ for any D ∈ 𝔻(G/K)}
= {φ ∈ 𝒜(G/K) : Dφ = χ_{λ}(D)φ for any D ∈ 𝔻(G/K)} .

The second equality follows from the fact that 𝔻(G/K) contains the Laplacian Δ_{G/K} which is elliptic. We note that

(6.7.13)  𝒜(G/K, ℳ_{λ}) = 𝒜(G/K, ℳ_{wλ}) for any w ∈ W.

Then we have

Lemma 6.7.14. The Poisson transform 𝒫_{λ} maps ℬ(G/P, ℒ_{λ}) into 𝒜(G/K, ℳ_{λ}) for all λ ∈ 𝔞_{ℂ}^{∗}.

Proof. Let D := Γ(u) ∈ 𝔻(G/K), u ∈ U(𝔤)^{K} and f ∈ ℬ(G/P, ℒ_{λ}). Since u commutes with K, we have

D𝒫_{λ}f(xK) = ∫_{K/M}(uf)(xk)dk
= δ(u)(λ − ρ) ∫_{K/M} f(xk)dk
= χ_{λ}(D)(𝒫_{λ}f)(xK).

□ In fact, the Poisson transform is an isomorphism for most values of λ, in particular

for those λ such that Re⟨λ, α⟩ ≥ 0 (α ∈ Σ^{+}). That is, we give an integral representation of simultaneous eigenfunctions of 𝔻(G/K) by means of an integral of a Poisson kernel over the minimal boundary G/P of G/K.

Theorem 6.7.15 ([Helgason, 1976], Theorem 6.1; [Kashiwara et al., 1978], [Schmid, 1985]). The Poisson transform

(6.7.15)(a)  𝒫_{λ}: ℬ(G/P, ℒ_{λ}) ⟶ 𝒜(G/K, ℳ_{λ})

is bijective if and only if e_{λ} ≠ 0. Here we denote by Σ_{0} the indivisible roots and define

(6.7.15)(b) e_{λ}^{−1} := ∏_{α∈Σ_{0}^{+}} Γ ( \frac{\frac{1}{2}m_{α} + 1 + ⟨\sqrt{−1} λ, α⟩}{2}) Γ ( \frac{\frac{1}{2}m_{α} + m_{2α} + ⟨\sqrt{−1} λ, α⟩}{2}.)

Basic properties of the Fourier transform are as follows.

Lemma 6.7.16. Retain notations as above.

a) For each fixed ζ ∈ 𝔞_{ℂ}^{∗}, (ℱF)(ζ, g) ∈ 𝒜(G/P, ℒ_{−\sqrt{−1}ζ}).

b) ℱ defines a family of G-maps parametrized by ζ ∈ 𝔞_{ℂ}^{∗}:

ℰ′(G/K) ⟶ ℬ(G/P, ℒ_{−\sqrt{−1}ζ}), F ⟼ (ℱF)(ζ, g).

c) If D ∈ 𝔻(G/K), then

ℱ(DF)(ζ, g) = χ_{−\sqrt{−1}ζ}(^{t}D)(ℱF)(ζ, g).

Proof. (a) and (b) are clear because p_{λ}(x, g) ∈ 𝒜(G/P, ℒ_{λ}) and because p_{λ}(g_{1}x, g_{1}g) = p_{λ}(x, g) for any g_{1} ∈ G. (c) is deduced from the following observation. Let δ ∈ ℰ′(G/P, ℒ_{λ}) ≃ ℰ′(K/M) denote the Dirac measure supported at the origin. Then for each k ∈ K,

p_{−λ}(x, k) = (𝒫_{λ}δ(k^{−1}·))(x).

Thus, as a function of x ∈ G/K with k ∈ K fixed

p_{−λ}(x, k) ∈ 𝒜(G/K, ℳ_{λ}).

Thus we have

ℱ(DF)(ζ, g) = ∫_{G/K}(DF)(x)p_{−\sqrt{−1}ζ}(x, g)dx
= ∫_{G/K} F (x)^{t}D_{x}p_{−\sqrt{−1}ζ}(x, g)dx
= χ_{−\sqrt{−1}ζ}(^{t}D) ∫_{G/K} F (x)p_{−\sqrt{−1}ζ}(x, g)dx

□ §7 Submanifolds and convexity in a rank-one symmetric space

In this section we introduce a notion of horospherical convexity and a pair of generalized ‘Gauss maps’ in a non-compact Riemannian symmetric space of rank one. We give an outline of their basic properties after we introduce a pair of ‘Gauss-Kronecker’ curvatures as some symmetric polynomials of principal curvatures. By means of these terminologies we shall find the first and second leading terms of the asymptotic behavior of the null variety 𝒩(Ω) in the hyperbolic space in §8 as in a generalization of Euclidean results in §3. This section is taken from Chapter 3, §2 and §3 in [Kobayashi-a, 1989].

7.1 Submanifolds in a rank-one symmetric space.

Throughout this subsection we suppose that X = G/K is a non-compact Riemannian symmetric space of rank one. This means that G has an Iwasawa decomposition G = NAK with dim A = 1 (see §6.5).

SO_{0}(n, 1)/SO(n), (7.1.1)

SU(n, 1)/S(U(n) × U(1)),
Sp(n, 1)/Sp(n) × Sp(1),
F_{4(−20)}/SO(9).

Here, G_{0} denotes the identity component of a Lie group G. F_{4(−20)} denotes a connected real semisimple Lie group such that dim 𝔭 − dim 𝔨 = −20 and that 𝔤_{ℂ} = 𝔤 ⊗_{ℝ} ℂ is of the

exceptional type 𝔣_{4}.

If G/K is of rank one, then the algebra of invariant differential operators 𝔻(G/K) is isomorphic to a polynomial ring ℂ[Δ] generated by the Laplace-Beltrami operator Δ ≡ Δ_{G/K} on G/K as we saw in Example (6.6.5). The Weyl group W for Σ(𝔤, 𝔞) is isomorphic to ℤ_{2}. We fix an element m_{−} ∈ M′ = N_{K}(𝔞) and write m_{+} := e so that

W = {[m_{+}] = 1, [m_{−}]}.

Fix a positive Weyl chamber (a half line) 𝔞^{+}. Let a_{0} ∈ A be the element determined by the condition:

(7.1.2)  ∥ log a_{0}∥ = 1 and log a_{0} ∈ 𝔞^{+}.

Here ∥ · ∥ denotes the norm on 𝔭 ≃ T_{o}X induced from the Riemannian metric g at o = eK ∈ X = G/K, and log: A → 𝔞 is the inverse of the bijective map exp: 𝔞 → A. Set an element of A by

a_{0}^{t} := exp(t log a_{0}), t ∈ ℝ.

If we write

Exp: T_{p}X ⟶ X

for the exponential mapping at p in X with the Levi-Civita connection ∇, then we have

a_{0}^{t} · o = Exp_{o}(t log (a_{0})), t ∈ ℝ.

A horosphere in X is the image of the map

N → X, n ↦ g_{1}ng_{2}^{−1}o,

where g_{1} and g_{2} are fixed elements of G. It is known that this map is always diffeomorphic to a submanifold in X which is a closed subset. Denote by Ξ the totality of horospheres in X. Let S(T X) be the unit tangent sphere bundle of the Riemannian manifold (X, g), and L(X) be the totality of oriented complete geodesics in X.

Lemma 7.1.3. With notation as above, we define the following four maps.

(7.1.3)(a) K/M × A ⟶ G/MN,  (kM, a) ⟼ kaMN,
(7.1.3)(b) G/MN ⟶ Ξ,  gMN ⟼ gN · o,
(7.1.3)(c) G/M  ⟶ S(T X),  gM  ⟼ \frac{d}{dt}|_{t=0} ga_{0}^{t} · o,
(7.1.3)(d) G/MA ⟶ L(X),  gMA ⟼ {ga_{0}^{t} · o ∈ X : t ∈ ℝ} .

Then all of the above maps are surjective diffeomorphisms.

Remark 7.1.4. The isomorphism K/M × A ⟶^{∼} G/MN ⟶^{∼} Ξ holds without the rank one assumption ([Helgason, 1970], Proposition 1.1).

An important geometric property of horospheres are:

Lemma 7.1.5. Retain notation as in Lemma (7.1.3). For each element g of G, there exist precisely two horospheres orthogonal to the unit normal vector at g · o ∈ X corresponding to gM ∈ G/M ≃ S(T X). These two horospheres are given by gm_{ε}N · o (ε = ±1).

Let Ω be a bounded domain in X = G/K with a connected smooth boundary ∂Ω. We shall introduce a pair of maps ψ_{ε}: ∂Ω → G/P = K/M (ε = ±1), which are an analogue of the Gauss map in ℝ^{n} (3.2). We shall also introduce a pair of maps h_{ε}: ∂Ω → ℝ (ε = ±1) when Ω is strictly horospherical convex in the sense of Definition (7.7), which are an analogue of the supporting function (see Definition (3.6)) in ℝ^{n}.

Let V ⊂ X = G/K be a local hypersurface oriented by a normal vector field ν ∈ Γ(V, T X|_{V}). We define

(7.1.6)  ψ_{ε} ≡ ψ_{V,ε}: V ⟶ G/P = K/M

as a composition of the following three maps:

(7.1.6)(a)  V ⟶ S(T X),  p ⟼ (p, ν(p)),
(7.1.6)(b)  S(T X) ⟶^{∼} G/M,  (7.1.3)(c),
(7.1.6)(c)  π_{ε}: G/M ⟶ G/P ≃ K/M,  gM ⟼ gm_{ε}P.

If V = ∂Ω and ν is the outer normal vector field, then we also write ψ_{Ω,ε} for ψ_{ε} ≡ ψ_{∂Ω,ε}. We call ψ_{Ω,ε} (ε = ±1) the Gauss map for the embedding ∂Ω into a noncompact Riemannian symmetric space X of rank one. Since the map π_{ε}: G/M → G/P ≃ K/M does not depend on the choice of representative m_{ε} ∈ M′ of the Weyl group W ≃ M′/M, ψ_{ε} is well defined.

Definition 7.1.7. Suppose that Ω is a bounded domain in X = G/K with a connected smooth boundary. We call Ω a strictly horospherically convex (or strictly H-convex) domain if and only if both ψ_{1} and ψ_{−1} are surjective diffeomorphisms. We call Ω a horospherically convex (or H-convex) domain if and only if Ω lies in one of the half spaces determined by each of two horospheres tangent to ∂Ω.

Remark 7.1.8. If a complete Riemannian manifold has a non-negative curvature (e.g. X = G/K is a Riemannian symmetric space of the noncompact type), then there exists a unique geodesic which goes through given two points. Therefore, we can define a notion of geodesical convexity just as in Euclidean case. We should notice that geodesical convexity does not imply horospherical convexity in a non-compact Riemannian symmetric space of rank one, since a horosphere is not totally geodesic. On the other hand, the correspondence of a horosphere in ℝ^{n} is a hyperplane, which is obviously totally geodesic.

Suppose that V ⊂ X = G/K is a local hypersurface which is oriented by the (outer) normal vector field ν ∈ Γ(V, T X|_{V}). We define a map

(7.1.9)  h̃_{ε} ≡ h̃_{V,ε}: V ⟶ ℝ,  (ε = ±1),

as a composition of the following four maps:

(7.1.9)(a)  V ⟶ S(T X),  p ⟼ (p, ν(p)),
(7.1.9)(b)  S(T X) ⟶^{∼} G/M,  (7.1.3)(c),
(7.1.9)(c)  G/M ⟶ 𝔞,  gM ⟼ −A(m_{ε}^{−1}g^{−1}),
(7.1.9)(d)  𝔞 ⟶^{∼} ℝ,  c log a_{0} ⟼ c.

Here A: G = NAK → 𝔞 is the Iwasawa projection defined in (6.5.17).

Suppose that Ω ⊂ X = G/K is a strictly horospherically convex domain. We introduce a pair of the supporting functions for Ω:

(7.1.10)  h_{ε} ≡ h_{Ω,ε}: G/P ⟶ ℝ,  (ε = ±1),

as a composition of

ψ_{ε}^{−1}: G/P ⟶^{∼} ∂Ω  (7.1.6)
h̃_{∂Ω,ε}: ∂Ω ⟶ ℝ,  (7.1.9)

The definition of h_{ε} (ε = ±1) does not depend on the choice of m_{ε} ∈ M′. The breadth function of a strictly horospherically convex domain Ω is defined by

(7.1.12)  H ≡ H_{Ω} := h_{1} − h_{−1}: G/P ≃ K/M ⟶ ℝ.

Consider the action of G. From the commutativity of the diagram

(7.1.13)  \begin{aligned}∂Ω & ⟶ & S(T X) & ⟶^{∼} & G/M & ⟶^{π_{ε}} & G/P \\ ^{g·}↓ & & ^{g·}↓ & & ^{g·}↓ & & ^{g·}↓ \\ ∂(g · Ω) & ⟶ & S(T X) & ⟶^{∼} & G/M & ⟶^{π_{ε}} & G/P, \end{aligned}

we have the following:

Lemma 7.1.14. Retain notation as before. For g ∈ G, we set g ·Ω := {g · x ∈ X : x ∈ Ω}. Then we have

(7.1.14)(a)  gψ_{Ω,ε}(x) = ψ_{g·Ω,ε}(g · x), (x ∈ Ω),
(7.1.14)(b)  h_{b^{−1}·Ω,ε}(gP) = h_{Ω,ε}(bgP), (gP ∈ G/P, b ∈ K).

7.2 A space with a negative constant curvature SO_{0}(n, 1)/SO(n).

From now on, we suppose G = SO_{0}(n, 1), the identity component of the matrix group

O(n, 1) = {g ∈ GL(n + 1, ℝ) : gI_{n,1}^{t}g = I_{n,1}} ,

where I_{n,1} = diag(1, 1, …, 1, −1) ∈ GL(n + 1, ℝ). Then G/K = SO_{0}(n, 1)/SO(n) is a Riemannian symmetric space of rank one. The Lie algebra of G is given by

𝔤 = 𝔰𝔬 (n, 1) = {Y ∈ M(n + 1, ℝ) : Y I_{n,1} + I_{n,1}^{t}Y = 0} .

Let E_{ij} (1 ≤ i, j ≤ n + 1) be the matrix unit and put

(7.2.1)(a)  H := E_{n,n+1} + E_{n+1,n},
(7.2.1)(b)  N_{j} := E_{n,j} + E_{n+1,j} − E_{j,n} + E_{j,n+1} (1 ≤ j ≤ n − 1),
(7.2.1)(c)  𝔞 := ℝH (⊂ 𝔭),
(7.2.1)(d)  𝔞^{+} := ℝ_{+}H ⊂ 𝔞,
(7.2.1)(e)  𝔫 := ∑_{j=1}^{n−1} ℝN_{j}.

Then we have Σ(𝔫, 𝔞) = Σ^{+}(𝔤, 𝔞) = {α}, where α ∈ 𝔞^{∗} is a weight such that α(H) = 1. We identify 𝔞_{ℂ}^{∗} with ℂ by

(7.2.2)(a)  𝔞_{ℂ}^{∗} ≃ ℂ, ζα ⟼ ζ.

Via this identification, we look upon the Fourier transform ℱ as a map from ℰ′(X) to 𝒜(ℂ × K/M).

Fix a positive number k > 0 and fix a Riemannian metric g on X so that X has a constant sectional curvature ≡ −k^{2}. Then we have

(7.2.2)(b)  a_{0} = exp(kH) ∈ A,

following the notation (7.1.2).

Let us recall here briefly some standard models of the hyperbolic space X = G/K = SO_{0}(n, 1)/SO(n): For x = ^{t}(x_{1}, …, x_{n+1}) ∈ ℝ^{n+1}, we define a quadric form

[x, x] := − ∑_{j=1}^{n}(x_{j})^{2} + (x_{n+1})^{2}.

1) (Quadric model)

(7.2.3)(a) X_{1} : = {x = ^{t}(x_{1}, …, x_{n}, x_{n+1}) ∈ ℝ^{n+1} : [x, x] > 0, x_{n+1} > 0} /ℝ_{+}^{×}
= {x = ^{t}(x_{1}, …, x_{n}, x_{n+1}) ∈ ℝ^{n+1} : [x, x] = 1, x_{n+1} > 0} ,
(7.2.3)(b) o_{1} : = ^{t}(0, 0, …, 0, 1) ∈ X_{1}
(7.2.3)(c) ∂X_{1} : = {x=^{t}(x_{1}, …, x_{n}, x_{n+1})∈ℝ^{n+1} : [x, x]=0, x_{n+1} >0} /ℝ_{+}^{×},
(7.2.3)(d) v_{1} : = ^{t}(0, …, 0, 1, 1) ∈ ∂X_{1}

Here ℝ_{+}^{×} acts on ℝ^{n+1} diagonally and ∂X is the geometric boundary of X.

2) (Unit ball model)

(7.2.4)(a)  X_{2} := {u = ^{t}(u_{1}, …, u_{n}) ∈ ℝ^{n} : u_{1}^{2} + ⋯ + u_{n}^{2} < 1} ,
(7.2.4)(b)  o_{2} := ^{t}(0, …, 0) ∈ X_{2},
(7.2.4)(c)  ∂X_{2} := {u = ^{t}(u_{1}, …, u_{n}) ∈ ℝ^{n} : u_{1}^{2} + ⋯ + u_{n}^{2} = 1} ,
(7.2.4)(d)  v_{2} := ^{t}(0, 0, …, 0, 1) ∈ ∂X_{2}.

Riemannian metrics on X_{2} and ∂X_{2} are given by

(7.2.4)(e)  ds^{2} = \frac{4}{k^{2}} \frac{du_{1}^{2} + ⋯ + du_{n}^{2}}{(1 − (u_{1}^{2} + ⋯ + u_{n}^{2}))^{2}} on X_{2},
(7.2.4)(f)  ds^{2} = \frac{1}{k^{2}} (du_{1}^{2} + ⋯ + du_{n}^{2})  on ∂X_{2}.

3) (Upper half-space model)

(7.2.5)(a)  X_{3} := {y = ^{t}(y_{1}, …, y_{n}) ∈ ℝ^{n} : y_{n} > 0} ,
(7.2.5)(b)  o_{3} := ^{t}(0, …, 0, 1) ∈ X_{3},
(7.2.5)(c)  ∂X_{3} := {y = ^{t}(y_{1}, …, y_{n}) ∈ ℝ^{n} : y_{n} = 0} ∪ {∞},
(7.2.5)(d)  v_{3} := {∞} ∈ ∂X_{3}.

Riemannian metrics on X_{3} and ∂X_{3} are given by

(7.2.5)(e)  ds^{2} = \frac{1}{k^{2}} \frac{dy_{1}^{2} + ⋯ + dy_{n}^{2}}{y_{n}^{2}} ,
(7.2.5)(f)  ds^{2} = \frac{4}{k^{2}} \frac{dy_{1}^{2} + ⋯ + dy_{n−1}^{2}}{(1 + y_{1}^{2} + ⋯ + y_{n−1}^{2})^{2}} .

The natural action of G = SO_{0}(n, 1) on ℝ^{n+1} preserves the quadratic form [ , ], and so X_{1} and ∂X_{1} are stable under this action. The G-actions on X_{j} and ∂X_{j} (j = 2, 3) are defined via the following isometries

(7.2.6)  Φ_{ji}: \overline{X_{i}} = X_{i} ∪ ∂X_{i} → \overline{X_{j}} = X_{j} ∪ ∂X_{j}, (1 ≤ i, j ≤ 3),

which are defined by

Φ_{jj} = id|\overline{X_{j}}, Φ_{ij} ∘ Φ_{jk} = Φ_{ik}

and

t(u_{1}, …, u_{n}) = Φ_{21}(x_{1}, …, x_{n+1}) := \frac{1}{[x, x]^{\frac{1}{2}} + x_{n+1}}^{t}(x_{1}, …, x_{n}), (7.2.6)(a)

t(y_{1}, …, y_{n}) = Φ_{32}(u_{1}, …, u_{n}) (7.2.6)(b)

:= \frac{1}{u_{1}^{2} + ⋯ + u_{n−1}^{2} + (u_{n} − 1)^{2}}^{t}(2u_{1}, …, 2u_{n−1}, 1 − ∑_{j=1}^{n} u_{j}^{2}).

Then we have

t(u_{1}, …, u_{n}) = Φ_{23}(y_{1}, …, y_{n}) (7.2.6)(c)

= \frac{1}{y_{1}^{2} + ⋯ + y_{n−1}^{2} + (y_{n} + 1)^{2}}^{t}(2y_{1}, …, 2y_{n−1}, ∑_{j=1}^{n} y_{j}^{2} − 1),

t(y_{1}, …, y_{n}) = Φ_{31}(x_{1}, …, x_{n}) = \frac{1}{x_{n+1} − x_{n}}^{t}(x_{1}, …, x_{n−1}, 1), (7.2.6)(d)

t(x_{1}, …, x_{n+1}) = Φ_{13}(y_{1}, …, y_{n}) (7.2.6)(e)

= \frac{1}{2y_{n}}^{t}(2y_{1}, …, 2y_{n−1}, ∑_{j=1}^{n} y_{j}^{2} − 1, ∑_{j=1}^{n} y_{j}^{2} + 1).

Here x = ^{t}(x_{1}, …, x_{n+1}) is normalized such that [x, x] = 1. We have

(7.2.7)  Φ_{ij}(v_{j}) = v_{i}, Φ_{ij}(o_{j}) = o_{i}.

The isotropy subgroups at o_{j}, v_{j} are K, P, respectively. The metric on X_{j} is G-invariant, while that on ∂X_{j} is merely K-invariant. Moreover the metrics on X_{2} and X_{3} are conformal to the standard metric in ℝ^{n} in each model.

The following calculation will be used to calculate the Jacobian of the ‘Gauss maps’ in Lemma (7.2.11). Since the geodesic {a_{0}^{t} · o_{2} : t ∈ ℝ} ⊂ X_{2} is given by

a_{0}^{t} · o_{2} = ^{t} (0, …, 0, \frac{e^{kt} − 1}{e^{kt} + 1}) ,

we have

1 − ∑_{j=1}^{n} u_{j}^{2} = \frac{4e^{kt}}{(1 + e^{kt})^{2}} ,

\frac{4}{1 − ∑_{j=1}^{n} u_{j}^{2}} − 2 = \frac{1 + e^{2kt}}{e^{kt}} = e^{kt} + e^{−kt},

for u = ^{t}(u_{1}, …, u_{n}) = a_{0}^{t} · o_{2}. From the K = SO(n) invariance, we obtain

(7.2.8)  \frac{4}{1 − ∑_{j=1}^{n} u_{j}^{2}} − 2 = 2 cosh(kd(o_{2}, u))

for any u = ^{t}(u_{1}, …, u_{n}) ∈ X_{2}, where d(x, y) denotes the Riemannian distance between x and y (x, y ∈ X).

Suppose that Ω is a bounded domain in X with a smooth boundary ∂X. Let ν ∈ Γ(∂Ω, T X|_{∂Ω}) be the outer unit normal vector field on ∂X as usual. Fix an orthonormal frame {f_{1} …, f_{n−1}} in a neighborhood of a given point x ∈ ∂Ω. The principal curvatures at x for the embedding ∂Ω ↪ X are the eigenvalues

(7.2.9)  λ_{j}(x) ≡ λ_{∂Ω,j}(x)  (1 ≤ j ≤ n − 1)

of the second fundamental form (−g_{x}(∇_{f_{j}(x)}f_{k}, ν(x)))_{1≤j,k≤n−1}. We introduce a pair of functions on ∂Ω

(7.2.10)  K_{ε} ≡ K_{∂Ω,ε}: ∂Ω ⟶ ℝ, (ε = ±1),

by K_{ε}(x) := ∏_{j=1}^{n−1}(λ_{j}(x) + εk) for x ∈ ∂Ω as an analogue of the Gauss-Kronecker curvature

in Euclidean case. We also write K_{Ω,ε} for K_{∂Ω,ε}. The point here is that the Jacobian of the ‘Gauss maps’ ψ_{ε} (ε = ±1) is a multiple of the ‘Gauss-Kronecker curvatures’ K_{ε} (ε = ±1) in the following lemma, which is due to [Berenstein-Yang, 1982] in the case of n = 2 and due to [Berenstein-Yang, 1987] and [Kobayashi-a, 1989], Lemma (3.4.4) for general n. We try to restate the latter one in an intrinsic way that is independent of the choice of models (7.2.3)–(7.2.5), because it might give us some insight to study the ‘Gauss’ map in a more general setting such as a Riemannian symmetric space of rank one other than a hyperbolic space (see (7.1.6)). For the benefit of the reader we give a proof.

Lemma 7.2.11. Suppose V ⊂ X = G/K is a local hypersurface oriented by a normal vector field ν ∈ Γ(V, T X|_{V}). Recall that ∂X is equipped with a K-invariant Riemannian structure (see (7.2.4)(f) and (7.2.5)(f)). We put

(7.2.12)  P_{ε}: V → ℝ, p ↦ 2 cosh(kd(o, p)) − exp(kh̃_{V,−ε}(p)),

where d(o, p) is the Riemannian distance between o = eK ∈ G/K and p ∈ V. Recall that h̃_{V,−ε} is defined in (7.1.9) and that −k^{2} is a sectional curvature of G/K (see (7.2.2)). Then we have

P_{ε}(p) > 0, (7.2.13)

det(Jψ_{ε})(p) = (εP_{ε}(p))^{1−n}K_{ε}(p), (7.2.14) for any p ∈ V. Here J stands for the Jacobian matrix and det is defined as a map between finite dimensional vector spaces of the same rank equipped with inner products.

Proof. Let p ∈ V. Take b ∈ K such that

ψ_{V,−ε}(p) = bM ∈ K/M.

Then from (7.1.14)(a), we have

ψ_{b^{−1}·V,−ε}(b^{−1} · p) = b^{−1}ψ_{V,−ε}(p) = eM ∈ K/M.

That is, in an upper half-space model (7.2.5), we have

(7.2.15)  ψ_{b^{−1}·V,−ε}(b^{−1} · p) = v_{3} ≡ ∞.

From the K-invariance of the metrics on X and ∂X and from Lemma 3.4.4 in [Kobayashi-a, 1989], we have

(7.2.16) det(Jψ_{V,ε})(p) = det(Jψ_{b^{−1}·V,ε})(b^{−1} · p)

= ε^{n−1}k^{1−n} (\frac{1 + ∑_{j=1}^{n} y_{j}(b^{−1} · p)^{2}}{y_{n}(b^{−1} · p)} )^{1−n} K_{b^{−1}·V,ε}(b^{−1} · p)

= ε^{n−1}k^{1−n} (\frac{1 + ∑_{j=1}^{n−1} y_{j}^{2}}{y_{n}} )^{1−n} (b^{−1} · p) K_{V,ε}(p).

Here (y_{1}, …, y_{n}) is the coordinate in the upper half-space model (7.2.5). Now, let us

calculate ^{\frac{1+ ∑_{j=1}^{n−1} y_{j}(b^{−1}·p)^{2}}{y_{n}(b^{−1}·p)}} . Let f be a function which represents b^{−1} · V as a graph near b^{−1} · p ∈ b^{−1} · V. We may assume that y_{1}, …, y_{n−1} are in the principal directions at b^{−1} · p corresponding to the principal curvatures λ_{b^{−1}·V,ℓ} (1 ≤ ℓ ≤ n − 1) (see (7.2.9)) after replacing b by bm for some m ∈ M. Then we have a local expression

(7.2.17) f(y_{1}, …, y_{n−1}) = exp(kh̃_{V,−ε}(p))

+ \frac{ε}{2k} exp(−2kh̃_{V,−ε}(p) ∑_{ℓ=1}^{n−1}(λ_{b^{−1}·V,ℓ} − εk)(y_{ℓ} − y_{ℓ}(b^{−1} · p))^{2}

+ O(∑_{ℓ=1}^{n−1} |y_{ℓ} − y_{ℓ}(b^{−1} · p)|^{3})).

(See [Kobayashi-a, 1989], page 465.) In particular, we have

(7.2.17)(a)  y_{n}(b^{−1} · p) = exp(kh̃_{V,−ε}(p)).

We put u(b^{−1} · p) := Φ_{23}(y(b^{−1} · p)) ∈ X_{2}. From the formula y = Φ_{32}(u) in (7.2.6)(b), we have

\frac{1 + ∑_{j=1}^{n−1} y_{j}^{2}}{y_{n}} = \frac{4}{1 − ∑_{j=1}^{n−1} u_{j}^{2}} − 2 − y_{n}.

Thus, from (7.2.8) and (7.2.17)(a) we have

(7.2.18)  \frac{1 + ∑_{j=1}^{n−1} y_{j}(b^{−1} · p)^{2}}{y_{n}(b^{−1} · p)} = 2 cosh(kd(o_{2}, u(b^{−1} · p))) − exp(kh̃_{V,−ε}(p))
= 2 cosh(kd(o_{2}, u(p))) − exp(kh̃_{V,−ε}(p))
= P_{ε}(p).

From (7.2.16) and (7.2.18) we conclude the Lemma.

This Lemma (7.2.11) assures that our definitions of a pair of ‘Gauss maps’ ψ_{Ω,ε} (ε = ±1) in (7.1.6) and a pair of ‘Gauss-Kronecker curvatures’ K_{Ω,ε} (ε = ±1) in (7.2.10) are the right ones. Using Lemma (7.2.11) and a Morse theory (see [Milnor, 1963]), we obtain a characterization of horospherical convexity:

Proposition 7.2.19 ([Kobayashi-a, 1989], Proposition 3.4.2). Let Ω be a bounded domain in X = SO_{0}(n, 1)/SO(n) with a connected smooth boundary.

1) The following three conditions on Ω are equivalent:

a) Ω is strictly horospherically convex.

a)′ ψ_{−1}: ∂Ω → K/M is locally diffeomorphic.

b) K_{ε} ≡ K_{∂Ω,ε}: ∂Ω → ℝ are positive valued for any ε = ±1.

2) Strictly horospherical convexity implies horospherical convexity.

3) Horospherical convexity implies

(7.2.19)(a)  λ_{∂Ω,j}(x) ≥ k (1 ≤ j ≤ n − 1)

for any x ∈ ∂Ω (Recall (7.2.9) for the definition of λ_{∂Ω,j}(x).)

As a corollary, we can recover a strictly horospherically convex domain from one of its supporting functions:

Corollary 7.2.20 ([Kobayashi-a, 1989], Corollary 3.4.12). Suppose that Ω is a strictly horospherically convex domain. We denote by h_{Ω,ε} (ε = ±1) a pair of its supporting functions (see (7.1.10)). Then Ω is recovered from one of the supporting functions h_{Ω,1} and h_{Ω,−1} as follows.

(7.2.20)(a)  Ω = ⋂_{b∈K} {ba_{0}^{t}n · o ∈ X : −∞ < t < h_{Ω,1}(b), n ∈ N} .
(7.2.20)(b)  Ω = ⋂_{b∈K} {ba_{0}^{t}n · o ∈ X : h_{Ω,−1}(b) < t < ∞, n ∈ N} .

We do not enter a detailed a proof of Proposition (7.2.19) and Corollary (7.2.20) here. In a proof of Corollary (7.2.20) we also need a geometric understanding of the supporting functions h_{Ω,ε}:

“The horosphere a_{0}^{t}N · o is tangent to ∂Ω if and only if t = h_{Ω,ε}(e) (ε = ±1).

Moreover, the point of tangency is given by ψ_{Ω,ε}^{−1}(e).”

See the following figure (7.2.21).

Figure (7.2.21)

§8 Asymptotics of the null variety 𝒩(Ω) in the hyperbolic space

Let X = G/K be a Riemannian symmetric space of the noncompact type. Retain notation in §6.7. Given a measurable bounded domain Ω in X, we define the Fourier transform of the characteristic function χ_{Ω} of Ω by

ℱχ_{Ω}(ζ, bM) = ∫_{G/K} χ_{Ω}(x)e^{⟨\sqrt{−1}ζ+ρ,A(b^{−1}x)⟩}dx.

Then ℱχ_{Ω}(ζ, bM) is a holomorphic function of ζ ∈ 𝔞_{ℂ}^{∗} with bM ∈ K/M fixed. We note that

ℱχ_{Ω} (\sqrt{−1} ρ, bM) = the volume of Ω.

We define an analytic set in 𝔞_{ℂ}^{∗} × K/M, called the null variety of Ω, by

𝒩(Ω) := {(ζ, bM) ∈ 𝔞_{ℂ}^{∗} × K/M : ℱχ_{Ω}(ζ, bM) = 0} .

This is a generalization of the definition (1.1.1) in ℝ^{n} case. In this section we shall restrict ourselves to X = G/K = SO_{0}(n, 1)/SO(n), the hyperbolic space. Similarly to Theorem (3.15) in ℝ^{n} case, we shall describe the asymptotic behavior of the null variety 𝒩(Ω) in this section.

Let X = G/K = SO_{0}(n, 1)/SO(n), the hyperbolic space and Ω be measurable bounded in X. Define the Radon transform of the characteristic function χ_{Ω}

(8.1)  S ≡ S_{Ω}: K/M × ℝ ⟶ ℝ

by

S_{Ω}(bM, t) : = ∫_{N} χ_{Ω}(ba_{0}^{t}n · o)dn
= ∫_{N} χ_{b^{−1}·Ω}(a_{0}^{t}n · o)dn.

Lemma 8.2 (cf. Lemma 3.14 in Euclidean case). Suppose that Ω is a strictly horospherically convex domain in G/K = SO_{0}(n, 1)/SO(n) (see Definition (7.1.7)). We define an open subset V in K/M × ℝ by

(8.2.1)  V := {(bM, p) ∈ K/M × ℝ : bM ∈ K/M, h_{−1}(bM) < p < h_{1}(pM)} .

Retain notation in (2.3). Then we have

i) S_{Ω} ∈ C(K/M × ℝ), supp S_{Ω} = V̄, and S_{Ω}|_{V} ∈ C^{∞}(V).
ii) S_{Ω}(bM, · ) ∈ 𝒞^{∞} (\frac{n−1}{2}).

Furthermore, we have

(8.2.2)  α(S_{Ω}(bM, ·)) = h_{−1}(bM), β(S_{Ω}(bM, ·)) = h_{1}(bM),
A(S_{Ω}(bM, ·)) = H_{Ω}(bM),
(8.2.3)(a)  a_{0}(S_{Ω}(bM, ·)) = \frac{(2π)^{\frac{n−1}{2}} (κ_{−1}(bM))^{− \frac{1}{2}}}{Γ (\frac{n+1}{2})} ,

(8.2.3)(b)  b_{0}(S_{Ω}(bM, ·)) = \frac{(2π)^{\frac{n−1}{2}} (κ_{1}(bM))^{− \frac{1}{2}}}{Γ (\frac{n+1}{2})} .

Here we put

(8.2.4)  κ_{ε} ≡ κ_{Ω,ε}: K/M ⟶ ℝ_{+}

by κ_{ε}(bM) := K_{ε} ∘ ψ_{ε}^{−1}(bM).

A Fourier transform on a non-compact Riemannian symmetric space is a composition of a Radon transform and a Fourier transform on 𝔞. So, via the identification 𝔞_{ℂ}^{∗} ≃ ℂ (7.2.2)(a), we have

(8.3)  ℱχ_{Ω}(ζ, bM) = ∫_{X} χ_{Ω}(x)e^{⟨\sqrt{−1}ζ+ρ,A(b^{−1}x)⟩}dx
= ∫_{X} χ_{Ω}(x)e^{⟨\sqrt{−1}ζ+ρ,A(x)⟩}dx
= ∫_{A} ∫_{N} χ_{Ω}(ban · o)e^{⟨\sqrt{−1}ζ+ρ,A(an)⟩}dnda
= ∫_{−∞}^{∞} ∫_{N} χ_{Ω}(ba_{0}^{t}n · o)e^{⟨\sqrt{−1}ζ+ρ,A(a_{0}^{t}na_{0}^{−t}a_{0}^{t})⟩}dndt
= ∫_{−∞}^{∞} ∫_{N} χ_{Ω}(ba_{0}^{t}n · o)e^{⟨\sqrt{−1}ζ+ρ,A(a_{0}^{t})⟩}dndt
= ∫_{−∞}^{∞} S_{Ω}(bM, t)e^{kt(\sqrt{−1}ζ+ \frac{n−1}{2})}dt.

Now that we have established Lemma (8.2) and the formula (8.3), we can obtain the asymptotic behavior of the null variety 𝒩(Ω) for a strictly horospherically convex domain in SO_{0}(n, 1)/SO(n) by the same method as Euclidean case by using Corollary (2.20). We put

(8.4)  d ≡ d_{Ω}: G/P ≃ K/M ⟶ ℝ

by

d(bM) = \frac{− log κ_{1}(bM) + log κ_{−1}(bM)}{2H(bM)}

for bM ∈ K/M (see (8.2.4)). Then the following theorem gives an answer to Problem A for a horospherically convex domain in the hyperbolic space:

Before stating Theorem (8.5), let us review some notation defined so far. Suppose that Ω is a strictly horospherically convex domain in the hyperbolic space G/K = SO_{0}(n, 1)/SO(n). We notice that G/P ⟵^{∼} K/M is diffeomorphic to S^{n−1} and that 𝔞_{ℂ}^{∗} is identified with ℂ ((7.2.2)(a)).

ψ_{ε} ≡ ψ_{Ω,ε} : ∂Ω → G/P = K/M,  the ‘Gauss map’ (7.1.6),
K_{ε} ≡ K_{Ω,ε} : ∂Ω → ℝ_{+},  the ‘Gauss-Kronecker curvature’ (7.2.10),
κ ≡ κ_{Ω,ε} : K/M → ℝ_{+}, b ↦ K_{ε} ∘ ψ_{ε}^{−1}(b) (8.2.4),
H ≡ H_{Ω} : K/M → ℝ,  the breadth function (7.1.12),
d ≡ d_{Ω} : K/M → ℝ,  (8.4),
A  : G → 𝔞,  Iwasawa projection (6.5.17),

ℱχ_{Ω}(ζ, bM) = ∫_{G/K} χ_{Ω}(x)e^{⟨\sqrt{−1}ζ+ρ,A(b^{−1}x)⟩}dx,

𝒩(Ω) = {(ζ, bM) ∈ 𝔞_{ℂ}^{∗} × K/M : ℱχ_{Ω}(ζ, bM) = 0} .

Theorem 8.5 (cf. Theorem (3.15)). Let Ω be a strictly horospherically convex domain in SO_{0}(n, 1)/SO(n). Retain notation as before. Then there exists an integer m_{0} ≡ m_{0}(Ω) depending only on Ω such that we have a disjoint union

(8.5.1)  𝒩(Ω) = ∐_{m=m_{0}}^{∞} (𝒩_{m}^{+} ∐ 𝒩_{m}^{−}) ∐ (compact set).

Here for each integer m ≥ m_{0}, 𝒩_{m}^{±} is a regular submanifold in ℂ×K/M, and is analytically diffeomorphic to K/M ≃ S^{n−1}. More precisely, the diffeomorphism 𝒩_{m}^{±} ≃ S^{n−1} is given as follows. There is a family of analytic maps

(8.5.2)  F_{m}^{ε}: K/M ⟶ ℂ, (ε = ±, m ≥ m_{0})

such that

(8.5.3)(a) F_{m}^{+}(bM) = \frac{2πm}{kH(bM)} + \frac{π(n − 1)}{2kH(bM)} + \sqrt{−1} (\frac{d(bM)}{k} + \frac{n − 1}{2}) + O(m^{−1})
as m → ∞,
(8.5.3)(b) F_{m}^{+}(bM) = −\overline{F_{m}^{−}(bM)},
(8.5.3)(c)  𝒩_{m}^{ε} = {(F_{m}^{ε}(bM), bM) ∈ ℂ × K/M : bM ∈ K/M} .

Example 8.6. For 0 < r < 1 we put R := \frac{1}{k} log \frac{1+r}{1−r} and Ω = B(R) := \{u ∈ ℝ^{n}: ∑_{j=1}^{n} u_{j}^{2} <

r^{2}} ⊂ X_{2} in the unit ball model (7.2.4). Then B(R) is a geodesic ball with radius R. We

have

K_{B(R),ε} ≡ (\frac{k(1 + εr)^{2}}{2r} )^{n−1} = (\frac{2εk exp(2εkR)}{exp(2εkR) − 1})^{n−1} ,
h_{B(R),ε} ≡ εR, H_{B(R)} ≡ 2R,
d_{B(R)} ≡ −\frac{k(n − 1)}{2},
ℱχ_{B(R)}(ζ, bM) = \frac{(2π)^{\frac{n}{2}}}{k^{n}} (sinh kR)^{\frac{n}{2}} P_{−\sqrt{−1}ζ− \frac{1}{2}}^{− \frac{n}{2}} (cosh kR).

The zeros of ℱχ_{B(R)}(ζ, bM) have the asymptotic behavior:

ζ = ±\frac{4m + (n − 1)}{4kR}π + O(m^{−1}) (ℤ ∋ m ≫ 0).

Similar to the formula of the Gauss-Kronecker curvature by means of the supporting function in ℝ^{n} case (see (3.25)(c)), we have a formula for a strictly horospherical convex domain in SO_{0}(n, 1)/SO(n) when n = 2 (see [Kobayashi-a, 1989], Proposition 3.7.21):

(8.7.1)  \frac{1}{K_{Ω,ε} ∘ ψ_{Ω,ε}^{−1}(θ)} = \frac{2k \frac{d^{2}h_{Ω,ε}}{dθ^{2}} − k^{2} (\frac{dh_{Ω,ε}}{dθ} )^{2} + e^{2kh_{Ω,ε}(θ)} − 1}{2εke^{2kh_{Ω,ε}(θ)}}

The injectivity problem of 𝒩 remains open in the hyperbolic space even in the relatively simple case n = 2 (cf. §4.1 for ℝ^{2} case). Let us carry out the program (3.24) for the injectivity of 𝒩 in a very special case, that is 𝒩(Ω) = 𝒩(B(R)), where B(R) is a geodesic ball with radius R with its center at the origin. In contrast to the ℝ^{2} case where we got a linear ordinary differential equation (4.1.3), we have a Duffing equation

(8.7.2)  f′′(θ) = −\frac{1}{4}(f(θ) − f^{−3}(θ)),

for f(θ) := exp (\frac{−k(h_{Ω,−1}(θ)+R)}{2}) ∈ C^{∞}(S^{1}), which is nonlinear (see [Kobayashi-a, 1989], (3.7.35)). From this we can conclude

Proposition 8.8. Suppose Ω is a strictly horospherically convex domain in SO_{0}(2, 1)/ SO(2). Retain notation as above. If 𝒩(Ω) and 𝒩(B(R)) have the same asymptotic behavior, then there exists g ∈ SO_{0}(2, 1) such that

g · Ω = B(R).

If 𝒩(Ω) contains countably many hyperspheres (not in an asymptotic sense but exactly hyperspheres), then Berenstein and Yang gave a partial answer to Problem (B′.3) in §1.3 for the hyperbolic space of arbitrary dimension n by using a result of [Alexandrov, 1962]:

Theorem 8.9 ([Berenstein-Yang, 1987]. Suppose that Ω is a bounded domain in SO_{0}(n, 1)/SO(n) with a connected Lipschitz boundary. Let Δ be the Laplace-Bertrami operator on the Riemannian symmetric space SO_{0}(n, 1)/SO(n). If there exists a sequence of complex numbers ζ_{j} ∈ ℂ (j ∈ ℕ) and if 𝒩(Ω) contains

𝒩_{j} := {(ζ_{j}, bM) ∈ ℂ × K/M : bM ∈ K/M} ,

then Ω is a geodesic ball.

By virtue of a similar relation between the null variety 𝒩(Ω) and some overdetermined problem in SO_{0}(n, 1)/SO(n) as we saw in §1.4 for ℝ^{n} case, we conclude from Theorem (8.9) the following:

Corollary 8.10 ([Berenstein-Yang, 1987], Theorem 2). Suppose that Ω is a bounded domain in SO_{0}(n−1)/SO(n) with a connected Lipschitz boundary. Let Δ be the Laplace-Beltrami operator on the Riemannian symmetric space SO_{0}(n, 1)/SO(n). If there exists a sequence of over-determined eigenfunctions u_{j} to

(8.10.1)  \{ \begin{matrix}Δu + αu = 0 & in Ω, α > 0 \\ \frac{∂u}{∂ν} = 0 and u ≡ constant & on ∂Ω, \end{matrix}

then Ω is a geodesic ball.

§9 Appendix

In this section we collect various properties of the Bessel function J_{ν}(z) which are used in this paper (§2, §3, §5). Also we indicate different methods to find a formula (9.4).

For ν ∈ ℂ and z ∈ ℂ ∖ (−∞, 0], we set

(9.1.1)  J_{ν}(z) : = (\frac{z}{2})^{ν} ∑_{n=0}^{∞} \frac{(−1)^{n}(\frac{z}{2})^{2n}}{n!Γ(ν + n + 1)}

= \frac{(\frac{z}{2})ν}{\sqrt{π}Γ (ν + \frac{1}{2})} ∫_{−1}^{1} e^{\sqrt{−1} zt}(1 − t^{2})^{ν− \frac{1}{2}} dt. (9.1.2)

J_{ν}(z) is called the ν-th Bessel function. If we put θ = z \frac{d}{dz}, then J_{ν}(z) and J_{−ν}(z) are the solutions of the ordinary differential equation:

(9.2)  (θ^{2} + (z^{2} − ν^{2}))f(z) = 0.

Here is a collection of classical results on the Bessel function:

Theorem 9.3.

1) \frac{J_{ν}(z)}{z^{ν}} is an entire function of z ∈ ℂ.

(9.3.1)(a)  \frac{J_{ν}(z)}{z^{ν}} |_{z=0} = \frac{1}{2^{ν}Γ(ν + 1)}.

2) f(z) := \frac{J_{k+ \frac{n}{2} −1}(λz)}{(λz)^{\frac{n}{2} −1}} is a solution of the following ordinary differential equation:

(9.3.2)(a) (θ^{2} + (n − 2)θ − k(k + n − 2) + λ^{2}z^{2})f(z)

= (z^{2} \frac{d^{2}}{dz^{2}} + (n − 1)z \frac{d}{dz} − k(k + n − 2) + λ^{2}z^{2}) f(z)

= 0

3) (recurrence formulae, see [Erdélyi et al., 1953], (7.2.50), (7.2.51))

\frac{d}{dz} (z^{ν}J_{ν}(z)) = z^{ν}J_{ν−1}(z), (9.3.3)(a)

\frac{d}{dz} (z^{−ν}J_{ν}(z)) = −z^{−ν}J_{ν+1}(z). (9.3.3)(b)

4) (spherical harmonics, see [Erdélyi et al., 1953], p. 247, (11.4.22)* ; see also [Helgason, 1984], Introduction, Lemma 3.6)

*S_{p}(η) should read S_{n}(η).

If h ∈ E_{k} = {f ∈ L^{2}(S^{n−1}) : Δf = −k(k + n − 2)f} (k ∈ ℕ) (see (5.4)), then we have for ω ∈ S^{n−1} and r ∈ ℂ

(9.3.4)(a)  ∫_{S^{n−1}} e^{\sqrt{−1} r(η,ω)}h(η)dη = (2π)^{\frac{n}{2}} \sqrt{−1}^{k}h(ω)\frac{J_{k+ \frac{n}{2} −1}(r)}{r^{\frac{n}{2} −1}}.

5) Let ν > 0 and x ≥ 0. Then we have

(9.3.5)(a)  J_{ν}(x)^{2} + 2 ∑_{n=1}^{∞} J_{ν+n}(x)^{2} = 2ν ∫_{0}^{x} J_{ν}(t)^{2} \frac{dt}{t}.

In particular,

(9.3.5)(b)  |J_{ν}(x)| ≤ 1, |J_{ν+1}(x)| ≤ \frac{1}{\sqrt{2}}.

6) Let z ∈ ℂ.

∑_{n=−∞}^{∞} J_{n}(z)^{2} = 1. (9.3.6)(a)

∑_{n=0}^{∞}(2n + 1)J_{n+ \frac{1}{2}} (z)^{2} = \frac{2z}{π} . (9.3.6)(b)

7) For real ν, J_{ν}(z) has an infinite number of real zeros (see [Watson, 1962], p.478). For real ν > −1, J_{ν}(z) has only real zeros (Lommel, see [Watson, 1962], p.482, p.483). If j(ν, 1), j(ν, 2) … are the positive zeros of J_{ν}(z) (ν > −1) arranged in ascending order of magnitude, then

(9.3.7)(a)  j(ν, k) = kπ + \frac{π(2ν − 1)}{4} + O(k^{−1}) as k → ∞.

8) (Siegel, see [Watson, 1962], p.484) J_{ν}(z) is not an algebraic number when ν is a rational number and z is an algebraic number other than zero. In particular, this proves Bourget’s conjecture that J_{ν}(z) and J_{ν+m}(z) (m ∈ ℕ_{+}) have no common zeros other than zero.

The Fourier transform of the characteristic function of the unit ball is explicitly calculated by means of the Bessel function:

Formula 9.4 (see Example (1.1.4)). We put Ω := B(0, 1) = {x = (x_{1}, …, x_{n}) ∈ ℝ^{n} :

∑_{j=1}^{n} x_{j}^{2} < 1}. Then we have a formula

(9.4)  ℱχ_{Ω}(ζ) = (2π)^{\frac{n}{2}} \frac{J\frac{n}{2} (t)}{t^{\frac{n}{2}}},

where ζ = (ζ_{1}, …, ζ_{n}) ∈ ℂ^{n} and we fix a branch t := (ζ_{1}^{2} + ⋯ + ζ_{n}^{2})^{\frac{1}{2}} ∈ ℂ.

Now let us illustrate four methods to find (9.4). (9.4.1) (Radon transform). (See (3.13)). Because ℱχ_{Ω}(ζ) is O(n) invariant, we have

\begin{aligned}ℱχ_{Ω}(ζ) & = ℱχ_{Ω}(t, 0, …, 0) \\ & = ∫_{Ω} e^{\sqrt{−1}x_{1}t}dx_{1}dx_{2} ⋯ dx_{n} \\ & = ∫_{−1}^{1} e^{\sqrt{−1}x_{1}t}(1 − x_{1}^{2})^{\frac{n−1}{2}} \frac{π^{\frac{n−1}{2}}}{Γ (\frac{n−1}{2} + 1)}dx \\ & = \frac{π^{\frac{n−1}{2}}}{Γ (\frac{n+1}{2})} \frac{\sqrt{π}Γ (\frac{n+1}{2})}{(\frac{t}{2})^{\frac{n}{2}}}J\frac{n}{2} (t) \\ & = (2π)^{\frac{n}{2}} \frac{J\frac{n}{2} (t)}{t^{\frac{n}{2}}}. \end{aligned}  (9.1.2)

(9.4.2) (Polar coordinates).

ℱχ_{Ω}(ζ) = ∫_{0}^{1} ∫_{S^{n−1}} e^{\sqrt{−1}r(ζ,ω)}r^{n−1}dr dω
= ∫_{0}^{1}(2π)^{\frac{n}{2}} \frac{J_{\frac{n}{2} −1}(rt)}{(rt)^{\frac{n}{2} −1}}r^{n−1}dr  (9.3.4)(a)

= \frac{1}{t^{n}} ∫_{0}^{t}(2π)^{\frac{n}{2}} \frac{J_{\frac{n}{2} −1}(x)}{x^{\frac{n}{2} −1}}x^{n−1}dx

= \frac{(2π)^{\frac{n}{2}}}{t^{n}} ∫_{0}^{t} x^{\frac{n}{2}} J_{\frac{n}{2} −1}(x)dx

= \frac{(2π)^{\frac{n}{2}}}{t^{n}} t^{\frac{n}{2}} J\frac{n}{2} (t)  (9.3.3)(a)

= (2π)^{\frac{n}{2}} \frac{J\frac{n}{2} (t)}{t^{\frac{n}{2}}}.

(9.4.3) (Differential equation). In polar coordinates, we set

x = r · ω (r ∈ ℝ, ω ∈ S^{n−1}).

Then r \frac{∂}{∂r} is a smooth vector field on ℝ^{n} and we have

r \frac{∂}{∂r} χ_{Ω} = −2δ(r^{2} − 1),

where δ is the Dirac’s delta function of one valuable. Thus we have

(r^{2} − 1)r \frac{∂}{∂r} χ_{Ω} = 0

as an equation of distributions. Passing to its Fourier transform, we have

(Δ_{ℝ^{n}} + 1) (n + t\frac{∂}{∂t}) ℱχ_{Ω}(tη) = 0

where t ∈ ℝ and η ∈ S^{n−1}. In the spherical coordinate y = tη (t ∈ ℝ^{×}, η ∈ S^{n−1}) on ℝ^{n}, the Laplacian Δ_{ℝ^{n}} takes the form Δ_{ℝ^{n}} = \frac{∂^{2}}{∂t^{2}} + \frac{n−1}{t} \frac{∂}{∂t} + \frac{1}{t^{2}} Δ_{S^{n−1}}. Since ℱχ_{Ω} is O(n)- invariant, ℱχ_{Ω}(tη) does not depend on η ∈ S^{n−1}, for which we write F (t). Let θ = t\frac{d}{dt}. Then we have

(θ + n − 2)(θ^{2} + nθ + t^{2})F (t) = (θ^{2} + (n − 2)θ + t^{2})(θ + n)F (t) = 0

If n ≥ 3, then it follows that

(θ^{2} + nθ + t^{2})F (t) = 0

because it has no singularity at the origin. From (1.1.1)(a), F (0) = the volume of Ω =

\frac{π^{\frac{n}{2}}}{Γ(\frac{n}{2} +1)} and thus we conclude F (t) = (2π)^{\frac{n}{2}} \frac{J_{\frac{n}{2}} (t)}{t^{\frac{n}{2}}}.

(9.4.4) (Holomorphic continuation). From the formula χ_{Ω} = (1 − r^{2})_{+}^{λ}|_{λ=0} (see notation

(2.1)), ℱ((1 − r^{2})_{+}^{λ})|_{λ=0} gives a formula of ℱχ_{Ω}(ζ). A formula for ℱ((1 − r^{2})_{+}^{λ}) is obtained

from that of ℱ((1 + r^{2})^{λ}) for Re λ < −\frac{n}{2} by using holomorphic continuation with respect to λ and r in [Gel’fand-Shilov, 1959].

Bibliography

[Agranovsky, Preprint] M.L.Agranovsky, On the stability of the spectrum in the Pompeiu

problem, Preprint. [Alexandrov, 1962] A.Alexandrov, A characteristic property of spheres, Annali di Matem-

atica 58 (1962), 303-315. [Arnold, 1972] V.I.Arnold, Modes and quasimodes, Funct. Anal. and its Appl. 6 (1972),

94-101. [Aviles, 1986] P.Aviles, Symmetry theorems related to Pompeiu’s problem, Amer. J. Math.

108-5 (1986), 1023-1036. [van den Ban-Schlichtkrull, in preparation] E.van den Ban and H.Schlichtkrull, The most

continuous part of the Plancherel decomposition for a reductive smmetric space, in preparation.

[Berenstein, 1976] C.A.Berenstein, On the converse to Pompeiu problems, Notas e Com-

municacōes de Matemática (Univ. Fed. de Bernambuco) 73 (1976). [Berenstein, 1980] , An inverse spectral theorem and its relation to the Pompeiu prob-

lem, J. Analyse Math. 37 (1980), 128 - 144. [Berenstein, 1987] , Spectral synthesis on symmetric spaces, Contemp. Math., 63

(1987), 1-25, Integral geometry (Brunswick, Maine, 1984). [Berenstein-Gay, 1989] C.A.Berenstein and R.Gay, Le probleme de Pompeiu local, J. Anal-

yse Math. 52 (1989), 133-166. [Berenstein-Ray-Yger, 1990] C.A.Berenstein, R.Gay and A.Yger, Inversion of the local

Pompeiu transform, J. Analyse Math. 54 (1990), 259-287. [Berenstein-Shahshahani, 1983] C.A.Berenstein and M.Shahshahani, Harmonic analysis

and the Pompeiu problem, Amer. J. Math. 105-5 (1983), 1217-1229. [Berenstein-Tarabusi, 1991] C.A.Berenstein and E.C.Tarabusi, Inversion formulas for the

k-dimensional Radon transform in real hyperbolic spaces, Duke Math. J. 62 (1991), 613-631.

[Berenstein-Taylor, 1976] C.A.Berenstein and B.A.Taylor, The “three-squares” theorem for

continuous functions, Arch. Rational Mech. Anal. 63 (1976), 253 - 259, (1977). [Berenstein-Yang, 1982] C.A.Berenstein and P.Yang, Overdetermined Neumann problem in

the unit disk, Advances in Math. 44 (1982), 1-17. [Berenstein-Yang, 1987] , An inverse Neumann problem, J. reine angew. Math. 382

(1987), 1-21. [Berenstein-Zalcman, 1976] C.A.Berenstein and L.Zalcman, Pompeiu’s problem on spaces

of constant curvature, J. Analyse Math. 30 (1976), 113 - 130. [Berenstein-Zalcman, 1980] , Pompeiu’s problem on symmetric spaces, Comment.

Math. Helv. 55-4 (1980), 593 - 621. [Brion, 1987] M.Brion, Classification des espaces homogenes spheriques, Compositio Math.

63-2 (1987), 189-208.

[Brion-Luna-Vust, 1986] M.Brion, D.Luna and Th.Vust, Espaces homogenes spheriques, Invent. Math. 84-3 (1986), 617–632.

[Brown-Kahane, 1982] L.Brown and J.P.Kahane, A note on the Pompeiu problem for convex domains, Math. Ann. 259-1 (1982), 107 - 110.

[Brown-Schnitzer-Shields, 1968] L.Brown, F.Schnitzer and A.L.Shields, A note on a problem of D. Pompeiu, Math.Z. 105 (1968), 59-61.

[Brown-Schreiber-Taylor, 1973] L.Brown, B.M.Schreiber and B.A.Taylor, Spectral synthesis and the Pompeiu problem, Ann. Inst. Fourier (Grenoble) 23 (1973), 125 - 154.

[Caffarelli, 1977] L.A.Caffarelli, The regularity of free boundaries in higher dimensions, Acta Math. 139 (1977), 155-184.

[Carey-Kaniuth-Moran, 1991] A.L.Carey, E.Kaniuth and W.Moran, The Pompeiu problem for groups, Math. Proc. Camb. Phil. Soc. 109 (1991), 45-58.

[Chern, 1959] S.S.Chern, Integral formulas for hypersurfaces in Euclidean space and their applications to uniqueness theorems, J. Math. and Mech. 8 (1959), 947-955.

[Chern-Lashof, 1958] S.S.Chern and R.K.Lashof, On the total curvature of immersed manifolds, II, Michigan Math. J. 5 (1958), 5-12.

[Delsarte, 1958] J.Delsarte, Note sur une propriété nouvelle des fonctions harmoniques, C. R. Acad. Sci. Paris 246 (1958), 1358-1360.

[DeMar-Davis, 1966] R.F.DeMar and P.J.Davis, A complex Pompeiu problem, Duke Math. J. 33 (1966), 91-101.

[Dickson, 1957] D.G.Dickson, Expansions in Series of Solutions of Linear Difference- Differential and Infinite Order Differential Equations with Constant Coefficients, vol. 23, Memoirs of A. M. S., 1957.

[Duflo, 1986] M.Duflo, Open problems suggested by M.Duflo, Open Problems in Representation Theory (T.Oshima, Univ. of Tokyo, ed.), pp. 1-5, Proceedings of the conference “Analysis on homogeneous spaces” held at Katata, 1986.

[Eberlein-O’Neil, 1973] P.Eberlein, B.O’Neil, Visibility manifolds, Pac.J.Math. 46 (1973), 45-110.

[Erdélyi et al., 1953] A.Erdélyi, W.Magnus, F.Oberhettinger and F.G.Tricomi, Higher Transcendental Functions, (Bateman Manuscript Project), vol. 2, McGraw-Hill, New York, 1953.

[Flensted-Jensen, 1986] M.Flensted-Jensen, Analysis on Non-Riemannian Symmetric Spaces, Conference Board, No.61, A.M.S., 1986.

[Fujiwara-Tanikawa-Yukita, 1978] D.Fujiwara, M.Tanikawa and S.Yukita, The spectrum of the Laplacian and boundary perturbation 1, Proc.Japan Acad. 54 (1978), 87-91.

[Funk, 1916] P.Funk, Über eine geometrische Anwendung der Abelschen Integralgleichung, Math. Ann. 77 (1916), 129-135.

[Garofalo-Segala, 1991] N.Garofalo and F.Segala, New results on the Pompeiu problem, Trans. Amer. Math. Soc. 325-1 (1991), 273-286.

[Gel’fand-Graev, 1955] I.M.Gel’fand and M.I.Graev, The analogue of Plancherel’s formula for the clasical groups, Tr. Mosk. Mat. O.-va 4 (1955), 375-404 (Russian); Transl. II. Ser., Amer. Math. Soc.9 (1958) 123-154.

[Gel’gand-Graev, 1959] , Geometry of homogeneous spaces, representations of groups in homogeneous spaces, and related questions of integral geometry, Tr. Mosk. Mat. O.-va 8 (1959), 321-390 (Russian); Transl. II. Ser., Amer. Math. Soc. 37 (1964) 351-429.

[Gel’fand-Graev-Shapiro, 1967] I.M.Gel’fand, M.I.Graev and Z.Ya.Shapiro, Integral geometry on k-dimensional planes, Funkts. Anal. Prilozh. 1 (1967), 15-31 (Russian); Funct. Anal. Appl. 1 (1967) 14-27.

[Gel’fand-Shilov, 1959] I.M.Gel’fand and G.E.Shilov, Generalized Functions, Properties

and operations, vol. 1, Moscow, 1959 (Russian); Transl. (1964) Academic Press. [Gel’fand-Graev-Vilenkin, 1962] I.M.Gel’fand, M.I.Graev and N.Ya.Vilenkin, Generalized

Functions, Integral Geometry and Representation Theory, vol. 5, Moscow, 1962 (Russian); Transl. (1966) Academic Press.

[Gindikin-Karpelevič, 1962] S.G.Gindikin and F.I.Karpelevič, Plancherel measure of Riemannian symmetric spaces of non-positive curvature, Dokl. Akad. Nauk SSSR 145 (1962), 252-255.

[Gindikin, 1992] S.G.Gindikin, Some notes on the Radon transform and integral geometry,

Mh. Math. 113 (1992), 23-32. [Guillemin-Sternberg, 1984] V.Guillemin and S.Sternberg, Multiplicity free spaces, J. Dif-

ferential Geometry 19 (1984), 31-56. [Gradshteyn-Ryzhik, 1980] I.S.Gradshteyn-I.M.Ryzhik, Table of Integrals, Series, and

Products, Enlarged edition by A.Jeffrey, Academic Press, San Diego, 1980. [Harish-Chandra, 1954] Harish-Chandra, The Plancherel formula for complex semisimple

Lie groups, Trans. Amer. Math. Soc. 76 (1954), 485-528. [Helgason, 1959] S.Helgason, Differential operators on homogeneous spaces, Acta Math.

102 (1959), 239-299. [Helgason, 1965] , Radon-Fourier transforms on symmetric spaces and related group

representations, Bull. Amer. Math. Soc. 71 (1965), 757-763. [Helgason, 1970] , A duality for symmetric spaces with applications to group repre-

sentations, Advances in Math. 5 (1970), 1-154. [Helgason, 1976] , A duality for symmetric spaces with applications to group repre-

sentations, II. Differential equations and eigenspace representations, Advances in Math. 22 (1976), 187-219.

[Helgason, 1978] , Differential Geometry, Lie Groups and Symmetric Spaces, Pure

and Appl. Math., vol. 80, Academic Press, 1978. [Helgason, 1984] , Groups and Geometric Analysis, Pure and Appl. Math., vol. 113,

Academic Press, 1984. [Helgason, 1990] , The totally-goedesic Radon transform on constant curvature

spaces, Contemp. Math., 113 (1990), 141-149.

[Herz, 1962] C.S.Herz, Fourier transforms related to convex sets, Ann. Math. 75-1 (1962), 81-92.

[Hunt, 1956] G.A.Hunt, A theorem of Élie Cartan, Proc. Amer. Math. Soc. 7 (1956), 307-308.

[John, 1934] F.John, Bestimmung einer Funktion aus ihren Integralen über gewisse Mannigfaltigkeiten, Math. Ann. 100 (1934), 488-520.

[John, 1955] , Plane waves and spherical means, Wiley(Interscience), New York, 1955.

[Kashiwara et al., 1978] M.Kashiwara, A.Kowata, K.Minemura, K.Okamoto, T.Oshima and M.Tanaka, Eigenfunctions of invariant differential operators on a symmetric space, Ann. of Math. 107 (1978), 1 - 39.

[Kobayashi-Nomizu, 1969] S.Kobayashi and K.Nomizu, Foundations of Differential Geometry, vol. 2, John Wiley & Sons, Inc., 1969.

[Kobayashi, 1986] T.Kobayashi, Null varieties for convex domains, Reports on unitary representation seminar 6 (1986), 1-18, Proceedings of the conference on “Unitary representation theory” held at Toba. (Japanese)

[Kobayashi, 1987] , On characteristic functions of domains and the zeros of their Fourier-Laplace transforms, Master’s dissertation II, University of Tokyo (1987). (Japanese)

[Kobayashi-a, 1989] , Asymptotic behaviours of the null variety for a convex domain in a non-positively curved space form, Jour. Fac. Sci. Univ. Tokyo 36-3 (1989), 389-478.

[Kobayashi-b, 1989] , How can we ‘view’ from silhouettes ?, Su-Semi 9 (1989), 82-87, Nihon-Hyôronsha, Tokyo, (to be reprinted in “Gendai Sugaku no Ayumi” vol.4, 1992). (Japanese)

[Kobayashi, 1992] , Perturbations of domains in the Pompeiu problem, preprint (1992).

[Kobayashi, in preparation] , Characterization of convexity by the null variety, in preparation.

[Kobayashi-Ono, 1990] T.Kobayashi and K.Ono, Note on Hirzebruch’s proportionality principle, Jour. Fac. Sci. Univ. Tokyo 37-1 (1990), 71-87.

[Koornwinder, 1981] T.H.Koornwinder, Invariant differential operators on non-reductive homogeneous spaces, Publ. Math. Centrum, Amsterdam, Report ZW 153 (1981).

[Krämer, 1979] M.Krämer, Sphärische Untergruppen in kompakten zusammenhängenden Liegruppen, Compositio Math. 38-2 (1979), 129-153.

[Matsuki, 1990] T.Matsuki, Orbits on flag manifolds, Proceedings of the International Congress of Mathematicians II (1990), 807-813 (I.Satake, ed.).

[Micheletti, 1972] A.M.Micheletti, Perturbazione dello spettro dell’operatore di Laplace, in relazione ad una variazione del campo, Ann. Scuola Norm. Sup. Pisa 26 (1972), 151-169.

[Milnor, 1963] J.Milnor, Morse Theory, Annals of Math. Studies, No. 51 Princeton Uni-

versity Press, Princeton, N.J., 1963. [Pompeiu, 1929-a] D.Pompeiu, Sur certains systèms d’équations linéaires et sur une pro-

priété intégrale des fonktions de plusieurs variables, C. R. Acad. Sci. Paris 188 (1929), 1138-1139.

[Pompeiu, 1929-b] , Sur une propriété intégrales des fonctions de deux variables réelles, Bull. Sci. Acad. Royale Belgique 15 (1929), 265-269.

[Radon, 1917] J.Radon, Über die Bestimmung von Funktionen durch ihre Integralwerte längs gewisser Mannigfaltigkeiten, Ber. Verh. Sächs. Akad. Wiss. Leipzig. Math.-Nat. kl. 69 (1917), 262-277.

[Rana, 1983] I.K.Rana, A Pompeiu problem for locally-finite measures on ℝ^{n}, Rend. Mat.

(7) 3-2 (1983), 291-300. [Sasaki, 1980] T.Sasaki, Minkowski’s problems and real Monge-Ampere equations, monthly

mathmatics 1-7 (1980), 522-532. (Japanese) [Schlichtkrull, 1984] H.Schlichtkrull, Hyperfunctions and Harmonic Analysis on Symmetric

Spaces, Birkhäuser, 1984. [Schmid, 1985] W. Schmid, Boundary value problems for group invariant differential equa-

tions, Ast7erisque, hors s’erie, Élie Cartan et les mathématiques d’aujourd’hui. [Scott-Sitaram, 1988] D.R.Scott and A.Sitaram, Some remarks on the Pompeiu problem

for groups, Proc. Amer. Math. Soc. 104-4 (1988), 1261-1266. [Serrin, 1971] J.Serrin, A symmetry problem in potential theory, Archiv. for Rat. Mech

and Anal. 43 (1971), 304-318. [Servedio, 1973] Servedio, Prehomogeneous vector spaces and varieties, Trans. Amer.

Math. Soc. 176 (1973), 421-444. [Shahshani-Sitaram, 1987] M.Shahshahani and A.Sitaram, The Pompeiu problem in ex-

terior domains in symmetric spaces, Contemp. Math., 63 (1987), 267-277, Integral geometry (Brunswick, Maine, 1984).

[Sitaram, 1984] A.Sitaram, Some remarks on measures on noncompact semisimple Lie

groups, Pacific J. Math. 110 (1984), 429-434. [Thompson-Schonbek, 1980] K.W.Thompson and T.Schonbek, A problem of the Pompeiu

type, Amer. Math. Monthly 87-1 (1980), 32 - 36. [Tricomi, 1947] F.Tricomi, Suglizeri delle funzioni di cui si conosce una rappresentazione

asintotica, Ann. Mat. Pura Appl. 4 (1947), 283-300. [Uhlenbeck, 1976] K.Uhlenbeck, Generic properties of eigengunctions, Amer. J. Math. 98

(1976), 1059-1078.

[Vinberg, 1963] È.B.Vinberg, The theory of convex homogeneous cones, Trans. Moscow. Math. Soc. 12 (1963), 340-403.

[Vinberg-Kimel’fel’d, 1978] È.B.Vinberg and B.N.Kimel’fel’d, Homogeneous domains on flag manifolds and spherical subsets of semisimple Lie groups, Funkts. Anal. Prilozh. 12-3 (1978), 12 -19 (Russian); Funct. Anal. Appl. 12 (1978) 168-174.

[Wallach, 1973] N.R.Wallach, Harmonic Analysis in Homogeneous Spaces, Dekker, New

York, 1973. [Wallach, 1988] , Real Reductive Groups I, Pure and Appl. Math., vol. 132, Academic

Press, 1988. [Wallach, 1990] , Lectures at University of Tokyo, Fall, 1990. [Warner, F., 1981] F.Warner, Foundations of Differentiable Manifolds and Lie Groups,

GTM, vol. 94, Springer Verlag, New York, 1981. [Warner, G., 1972] G.Warner, Harmonic Analysis on Semisimple Lie Groups I, Springer

Verlag, Berlin, 1972. [Watson, 1962] G.N.Watson, A Treatise on the Theory of Bessel Functions, Second edition,

Cambridge at the university press, 1962. [Weinberger, 1971] H.F.Weinberger, Remark on the preceding paper of Serrin, Archiv. for

Rat. Mech. and Anal. 43 (1971), 319-320. [Williams, 1976] S.A.Williams, A partial solution of the Pompeiu problem, Math. Ann.

223-2 (1976), 183-190. [Williams, 1981] , Analyticity of the boundary for Lipschitz domains without the

Pompeiu property, Indiana Univ. Math. J. 30-3 (1981), 357 - 369. [Yau, 1982] S.T.Yau, Seminar on Differential Geometry, Problem Section (S.T.Yau, ed.),

Annals of Math. Studies, Princeton, N.J., 1982. [Yosida, 1938] K.Yosida, A theorem concerning the semisimple Lie groups, Tohoku Math.

J. 44 (1938), 81-84. [Zalcman, 1972] L.Zalcman, Analyticity and the Pompeiu problem, Arch. Rational Mech.

Anal. 47 (1972), 237-254. [Zalcman, 1980] , Offbeat integral geometry, Amer. Math. Monthly 87-3 (1980), 161

- 175. [Zeilberger, 1978] D.Zeilberger, Pompeiu’s problem on discrete space, Proc. Nat. Acad.

Sci. U.S.A. 75-8 (1978), 3555-3556.