0. Introduction

Interesting classes of (𝔤, K)-modules are often described naturally in terms of cohomologically induced representations in various settings such as unitary highest weight modules, the theory of dual reductive pairs, discrete series for semisimple symmetric spaces, and etc. These have been stimulating the study of algebraic properties of derived functor modules. Now an almost satisfactory theory on derived functor modules including a functorial property about unitarizability, has been developed in the ‘good’ range of parameters. However, actual interesting families of unitary representations contain possibly singular parameters, so we are still faced with subtle problems such as finding conditions for non-vanishing, irreducibility, or pairwise inequivalence of these (𝔤, K)- modules (see Problem(0.6)′). This paper treats such delicate algebraic properties of Zuckerman’s derived functor modules with singular parameters which arise from discrete series for indefinite Stiefel manifolds U(p, q; 𝔽)/U(p−m, q; 𝔽) (p ≥ 2m, 𝔽 = ℝ, ℂ or ℍ). Some of them are isomorphic to “unipotent” representations in the philosophy of Arthur (e.g. [1], §5), and some are out of the range given by Vogan [29] for his unitarizability theorem. We should also remark that the unitary dual of a pseudo-orthogonal group U(p, q; 𝔽) has not been classified yet, except in lower rank cases.

Let G be a connected real reductive linear Lie group and H be a closed subgroup which is reductive in G. A homogeneous space G/H carries a G-invariant measure, so we have a natural unitary representation of G on the Hilbert space L^{2}(G/H). Harish-Chandra modules of finite length realized in L^{2}(G/H) are called discrete series for G/H. These play a fundamental role in harmonic analysis on G/H. If H is noncompact,

Typeset by AMS-TEX

Received by the editors 6 March, 1990 and, in revised form 30 April 1990.

Partially supported by Grant-in-Aid for Scientific Research (No.01740081).

discrete series for G/H do not necessarily enter the Plancherel decomposition in L^{2}(G) and then should correspond to a relatively singular part of the unitary dual of G.

If H is an open subgroup of the fix point group of an involutive automorphism σ of G, G/H is a typical example of a homogeneous space of reductive type and called a semisimple symmetric space. Much progress on discrete series for semisimple symmetric spaces has been made in the last ten years (see e.g. [2], [8], [9], [20], [21], [24], [33]). In particular, discrete series for G/H is not empty iff

(0.1)  rank G/H = rank K/H ∩ K,

where K is a σ-stable maximal compact subgroup in G.

Suppose that H has a direct decomposition H_{1} × H_{2} with H_{1} compact. Correspondingly to the G-equivariant H_{1}-principal bundle

H_{1} → G/H_{2} → G/H,

we have

L^{2}(G/H) = L^{2}(G/H; 1) ↪ L^{2}(G/H_{2}) ≃ ⨁_{τ} dim τ L^{2}(G/H; τ),

where the sum is taken over the unitary dual of the compact group H_{1} and L^{2}(G/H; τ) denotes the space of the square integrable sections of the vector bundle over G/H associated to τ.

Throughout this section, we will write G/H, G/H_{2} as a symbol of semisimple symmetric spaces with the rank condition (0.1), homogeneous spaces in the above setting, respectively. Since we allow the case H_{1} = {e} and H_{2} = H, G/H_{2} represents a wider class of homogeneous spaces than G/H in this notation. Typical examples of G ⊃ H = H_{1} × H_{2} are given in (0.10), and if G/H is any other classical irreducible semisimple symmetric space then H_{1} is one dimensional (or zero dimensional).

UNITARY REPRESENTATIONS IN L^{2}(U(p, q; 𝔽)/U(p − m, q; 𝔽)) 3

It is natural to extend the study of harmonic analysis from L^{2}(G/H) to L^{2}(G/H_{2}), in which more general unitary representations of G are expected to be realized. Discrete series for G/H_{2} have not been studied very well before, except in Schlichtkrull’s original work [23] generalizing Flensted-Jensen’s construction [8]. Building on work of Schlichtkrull and Oshima-Matsuki [20], we construct a wider class of discrete series for G/H_{2} whose parameters are possibly more singular.

Roughly, our construction is as follows: In a complexification G_{ℂ} of G, other real forms K^{r} ⊂ G^{r} ⊃ H^{r} ≈ H_{1}^{r} × H_{2}^{r} of K_{ℂ} ⊂ G_{ℂ} ⊃ H_{ℂ} ≈ H_{1}_{ℂ} × H_{2}_{ℂ} are defined so that H^{r} is compact [9]. Let P^{r} = M^{r}A^{r}N^{r} be a minimal parabolic subgroup of G^{r}, ρ ≡ ρ(𝔫^{r}) be half the sum of roots as usual, and X_{j} (1 ≤ j ≤ l, finite) be the closed K^{r}-orbits in G^{r}/P^{r}. Denote by A the sheaf of analytic functions, by B that of hyperfunctions. For (δ, V) ∈ M̂^{r}, ν ∈ Â^{r}, we define F to be the A or B valued principal series by

F(G^{r}/P^{r}; δ ⊗ ν) := {F ∈ F(G^{r}; V) ; F (gman) = δ(m)^{−1}a^{−ν−ρ}F (g) for m ∈ M^{r}, a ∈ A^{r}, n ∈ N^{r}}.

Then naturally we have a G^{r}-invariant nondegenerate bilinear form

(0.2)  B: B(G^{r}/P^{r}; δ^{∗} ⊗ (−ν)) × A(G^{r}/P^{r}; δ ⊗ ν) → ℂ,

by B(F, v) := ∫_{H^{r}}⟨F (k), v(k)⟩ dk. If v is a cyclic vector in A(G^{r}/P^{r}; δ ⊗ ν) with trivial H_{2}^{r}-action, then B induces an injective mapping

(0.3)  P_{v}: B(G^{r}/P^{r}; δ^{∗} ⊗ (−ν)) → A(G^{r}/H_{2}^{r}),

by P_{v}(F)(g) := B(F (g · ), v) = B(F, g · v). Then P_{v} is an analogue of the Poisson transform which respects the left G^{r}-action. Let

B_{K^{r}}^{j}(δ ⊗ ν) := {F ∈ B(G^{r}/P^{r}; δ ⊗ ν) ; supp F ⊂ X_{j}, F transforms according to

finite dimensional representations of K_{ℂ} under the action of K^{r}}.

Now we are ready to state a construction of discrete series for G/H_{2}:

Proposition 0.4. Assume that

B_{K^{r}}^{j}(G^{r}/P^{r}; δ^{∗} ⊗ (−ν)) ≠ 0, (0.5)(a) (0.5)(b) A(G^{r}/P^{r}; δ ⊗ ν) contains a H_{2}^{r}-fixed cyclic vector v, (0.5)(c) ⟨ν, α⟩ > 0 for any α ∈ Σ(𝔤^{r}, 𝔞^{r}).

Then the image of P_{v} (B_{K^{r}}^{j}(δ^{∗} ⊗ (−ν))) (1 ≤ j ≤ l) by Flensted-Jensen duality (a kind of holomorphic continuation) gives discrete series for L^{2}(G/H_{2}).

We remark that if H_{1} = {e}, H_{2} = H and if δ = 1, then the above construction exhausts all discrete series for semisimple symmetric spaces G/H (in particular, those for group manifolds G ≃ G × G/ diag(G)) (see Fact(1.5.1),(1.5.2)).

Since the proof of Proposition(0.4) is similar to [23] and [20], and since the proof requires a lot of analytic notations in estimating square integrability, we do not give it in this paper. Instead, we investigate algebraic properties of these unitary representations realized in L^{2}(G/H_{2}). First of all, we must find the explicit conditions for (0.5)(a),(b), which are satisfied if the parameter ν is sufficiently ‘regular’.

To obtain the explicit condition for (0.5)(b) is non-trivial but fairly easy: It is well known that this condition is automatically satisfied under (0.5)(c) if H_{1}^{r} = {e} due to Helgason, Kostant [10], [16]. This condition is also known if dim H_{1}^{r} = 1 (see [19], Theorem 1.2, see also an explicit c-function obtained in [25](with a miswriting in Sp(n, ℝ) case), [26]). For a general H_{1}^{r}, we can calculate it by standard arguments using Gindikin-Karpalevič method and the explicit knowledge of the socle filtration of principal series for rank one simple Lie groups (see [5]). The arguments are independent and we shall report it in another paper.

The most important and non-trivial part is (0.5)(a). So our interest in this paper will be concentrated on the following algebraic properties involving (0.5)(a):

Problem 0.6. Find each of the explicit conditions, deciding whether or not, (𝔤, K^{r})- modules B_{K^{r}}^{j}(δ^{∗} ⊗ (−ν)) (1 ≤ j ≤ l) vanish, are irreducible, are pairwise inequivalent and etc.

Thanks to the duality theorem due to Hecht, Miličic, Schmid and Wolf ([4], [11], see also [2] II.6), the modules B_{K^{r}}^{j}(δ^{∗} ⊗ (−ν)) are isomorphic to the duals of a family of (𝔤, K)-modules defined by Zuckerman. Then we may reformulate Problem (0.6) into a more familiar one:

Problem 0.6′. Find an explicit condition guaranteeing

i) R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 if S := \frac{1}{2} dim(K/L ∩ K).

ii) R_{𝔮}^{j}(ℂ_{λ}) = 0 for any j ≠ S. iii) R_{𝔮}^{S}(ℂ_{λ}) is irreducible or zero. iv) R_{𝔮}^{S}(ℂ_{λ}) ≃ R_{𝔮′}^{S′}(ℂ_{λ′}).

Here 𝔮 = 𝔩 + 𝔲 is a θ-stable parabolic subalgebra, ℂ_{λ} is a metapletic (𝔩, (L ∩ K)^{∼}) unitary character, and R_{𝔮}^{j}(ℂ_{λ}) (j ∈ ℕ) are Zuckerman’s cohomologically induced (𝔤, K)- modules (see §1.3 for our normalization).

Remark 0.7. Our discrete series for G/H_{2} are originally the dual of cohomologically induced representations from finite dimensional representations according to δ ∈ M̂^{r}. If we use further induction by stages and the Borel-Weil-Bott theorem for compact groups, we reduce them to those induced from characters. Let 𝔱_{0} be a maximal abelian subalgebra of 𝔨_{0}^{⊥(𝔥_{2})_{0}}, a subspace of 𝔨_{0} orthogonal to (𝔥_{2})_{0}. Then the discrete series constructed in Proposition(0.4) correspond to the modules R_{𝔮}^{S}(ℂ_{λ}) where 𝔮 is defined by a generic element of 𝔱_{0}. The choice of closed K^{r}-orbits X_{j} (1 ≤ j ≤ l) corresponds to the choice of polarizations. We will find that the conditions (0.5)(a),(b) and (c) impose on λ some mild positivity conditions with respect to 𝔮 (see List(0.9), Theorems in §2).

The difficulty in Problem(0.6)′ arises from singular parameters. Let us explain it. Suppose that 𝔮 = 𝔩 + 𝔲 is a θ-stable parabolic subalgebra and let ℂ_{λ} be a metapletic (𝔩, (L ∩ K)^{∼})-module. Three kinds of dominance conditions on λ are defined:

(0.8)(a) λ + ρ_{𝔩} is dominant for 𝔲, (0.8)(b) λ is dominant for 𝔲, (0.8)(c) μ_{λ} := λ + ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨) is Δ^{+}(𝔨)-dominant.

We say (0.8)(a) (weekly) good and (0.8)(b) (weakly) fair according to [33] (more precisely, see Definition(1.2.1)).

General theory guarantees nice behaviors of the modules R_{𝔮}^{j}(ℂ_{λ}) when ℂ_{λ} is in the good range where R_{𝔮}^{j}(ℂ_{λ}) has a regular ℨ(𝔤)-infinitesimal character (see Fact(1.4.1)). Indeed, suppose that a unitary character ℂ_{λ} is in the good range. Then,

R_{𝔮}^{j}(ℂ_{λ}) = 0 if j ≠ S,

R_{𝔮}^{S}(ℂ_{λ}) is nonzero, irreducible, and unitarizable.

Suppose one allows ℂ_{λ} to be in the (weakly) fair range, the cohomology group R_{𝔮}^{S}(ℂ_{λ}) tends to be reducible or vanish (see Fact(1.4.2)). The condition (0.8)(c) is sufficient for the non-vanishing of R_{𝔮}^{S}(ℂ_{λ}). The normality of the moment map (see §6) is sufficient for the irreducibility (or vanishing) of R_{𝔮}^{S}(ℂ_{λ}). For example, these sufficient conditions apply to Proposition 6.41 in [32] which is used in classifying the unitary dual of GL(2n, ℝ). (This part of the book [32] suggests some importance of the study in the strip: fair but not good.) However, unfortunately none of these conditions is necessary in general.

Suppose one allows ℂ_{λ} to wander outside the (weakly) fair range, the cohomology group R_{𝔮}^{S}(ℂ_{λ}) may be non-unitary, and cohomology may turn up in other dimensions as well.

Here is a sketchy list of the ranges of the parameters of the discrete series constructed in Proposition(0.4) in terms of Zuckerman’s derived functor modules (see Remark(0.7) for precise meaning). Semisimple Lie groups G ≃ G × G/ diag(G), semisimple symmetric spaces G/H, and our homogeneous spaces G/H_{2} are assumed to satisfy the rank condition rank G = rank K, (0.1), and (0.1) respectively.

the parameter of discrete series R_{𝔮}^{S}(ℂ_{λ})

G  G/H  G/H_{2}
ℂ_{λ} is fair  Yes  Yes  No
ℂ_{λ} is good  Yes  No  No
μ_{λ} is Δ^{+}(𝔨)-dominant  Yes  No  No

List 0.9.

Here “Yes” means that the corresponding condition is always satisfied, and otherwise “No”.

Recall that the construction in Proposition(0.4) exhausts all discrete series for G and G/H (see Fact(1.5.1),(1.5.2)). Counter examples (i.e. “No”) in List(0.9) for G/H_{2} are given as follows:

Suppose that G/H_{2} is an indefinite Stiefel manifold

(0.10)  G/H_{2} = U(p, q; 𝔽)/U(p − m, q; 𝔽),  (p ≥ 2m, 𝔽 = ℝ, ℂ or ℍ),

which is one of the most interesting and typical homogeneous spaces in our setting. (In this case H_{1} = U(m; 𝔽).) Then Theorem 1 ∼ 3 in §2 says that the discrete series for G/H_{2} in Proposition(0.4) contains a module R_{𝔮}^{S}(ℂ_{λ}) such that

i) ℂ_{λ} is outside the weakly fair range when 𝔽 = ℂ or ℍ with any p, q, m. ii) μ_{λ} is not dominant for Δ^{+}(𝔨) when 𝔽 = ℍ with any p, q, m; when 𝔽 = ℂ with p ≠ 2m;

when 𝔽 = ℝ with p − q − 2m ≥ 3.

The purpose of this paper is to make a detailed study on Problem(0.6)′ when G/H_{2} is of the form (0.10), by seeking for useful techniques which are applicable to derived functor modules with singular parameters and aiming at some understanding of the unitary dual. Most of our results of this paper involve a special class of θ-stable parabolic subalgebra: A Levi part is of the form L ≃ 𝕋^{l} × U(p − k, q; 𝔽). (l = k if 𝔽 = ℂ, ℍ and 2l = k if 𝔽 = ℝ.) Our approach here is algebraic and does not depend on the fact that the representations are realized in L^{2}-functions; even H = H_{1} × H_{2} will not appear explicitly in the proof. We shall sometimes deal with Problem(0.6)′ for less special derived functor modules at the same time in the proof (e.g. §5).

Remark 0.11.

i) The discrete series which were first constructed by Flensted-Jensen for G/H, by Schlichtkrull for G/H_{2} have the property that μ_{λ} is Δ^{+}(𝔨)-dominant, which corresponds to the fact that B_{K^{r}}^{j}(δ^{∗} ⊗ (−ν)) contains a measure valued function. We call these discrete series Flensted-Jensen type, Schlichtkrull type, respectively. Among discrete series of Schlichtkrull type, the representations which have stable Langlands parameters are explicitly stated in Theorem 8.2 ∼ 8.4 in [23] (with some minor misprints). Some of them were “new” unitary representations. It is known that Flensted-Jensen type discrete series do not always exhaust all discrete series for L^{2}(G/H) ([8] §8, [20]). We notice that if G ⊃ H ⊃ H_{2} is of the form (0.10) and if 𝔽 = ℍ, the shortage of Flensted-Jensen type for G/H happens only if p ≫ q + 2m, while that of Schlichtkrull type for G/H_{2} happens for any p, q and m. (This is similar if 𝔽 = ℂ.)

ii) Clearly, discrete series are unitarizable with respect to their L^{2}-norm. On the other hand, unitarizability of Zuckerman’s derived functor modules has been proven algebraically in the weakly good range [29],[34]. If they are induced from characters, the assumption can be weakened to the weakly fair range ([29], Theorem 7.1). Then the latter stronger version covers the range of discrete series for semisimple symmetric spaces G/H (see List(0.10)). One of the reasons why we reduce our discrete series to cohomologically induced representations from characters (see Remark(0.7)) is to compare with these algebraic results.

It has also been observed by many people that the unitarizability of R_{𝔮}^{S}(ℂ_{λ}) special parabolic subalgebras 𝔮 still holds outside the fair range with small spectrum λ (see [7] and the references therein). It seems that our unitarizability results in Part(6) of Theorems 1 and 2 contain some new cases, namely, the θ-stable parabolic subalgebras 𝔮 which we treat here are more general.

iii) None of Problem(0.6)′ is trivial even if ℂ_{λ} is in the (weakly) fair range: For example, Problem(0.6) has been recently investigated by Bien, Vogan, Matsuki and Oshima ([2], [21], [33]) for discrete series for G/H (semisimple symmetric spaces). J.F.Adams [1] also deals with some part of Problem(0.6)′ in this range when 𝔮 is “holomorphic” in studying unitary highest weight modules.

Now let us explain briefly the methods of this paper to deal with Problem(0.6)′. To find the condition that assures R_{𝔮}^{S}(ℂ_{λ}) ≠ 0, there are various techniques such as

i) Coherent continuation based on a) the τ-invariant (Vogan’s U_{α} calculus) (e.g. [27], [1]),

b) a precise knowledge of the non-vanishing exponents in the asymptotic behavior of spherical functions on semisimple symmetric spaces [22],

c) small parameters out of the fair range. ii) Generalized Blattner formula.

The analytic approach (i-b) depends on the fact that these modules are realized in eigenspaces on symmetric spaces. This idea was first introduced by Oshima and Matsuki to check the non-vanishing of some part of the discrete series for G/H (unpublished). We do not go into details here. A formulation with a detailed proof in a generalized setting G/H_{2} and actual calculations in a special setting may be seen in [13], which was our first approach to this subject.

We take the approach of neither (i-a) nor (i-b) in this paper, because both of them are only partially successful in covering all of the singular parameters in our Theorems 1 to 3 in §2.

The technique (i-c) is new and appears valuable because of its simplicity. The idea is illustrated in §5 for G = Sp(p, q) and more general θ-stable parabolic subalgebras. The point here is to concentrate on the range of most singular and small parameters (which we write B standing for ‘bounded blocks’) by forgetting all about dominant conditions. We will find that the picture of the set

{λ; R_{𝔮}^{S}(ℂ_{λ}) ≠ 0, λ is fair with respect to 𝔮}

depends heavily on the choice of polarizations of θ-stable parabolic subalgebras with a fixed Levi part.

As for the method (ii), notoriously complicated cancellations of many terms in a generalized Blattner formula have prevented us from an explicit calculation of a general K-type except some few cases such as · G is small. · a K-type is special (e.g. with a highest weight μ_{λ} under (0.8)(c)). · a θ-stable parabolic subalgebra 𝔮 is special (e.g. quasi abelian (see [7]) or corresponds to unitary highest weight modules) and a K-type is less special. Another novelty here is to get nice information on all K-types in our setting in §4. Let

us introduce the idea briefly. Put Θ_{λ} := ∑_{j}(−1)^{j}R_{𝔮}^{j}(ℂ_{λ}) as a virtual K-module. We stratify K̂ by a subspace 𝔱 of a Cartan subalgebra 𝔱^{c} of 𝔨, and add all the multiplicities (∼ dimensions) of K-types occurring in Θ_{λ} belonging to each stratum parametrized by δ ∈ 𝔱^{∗}. (If R_{𝔮}^{S}(ℂ_{λ}) is a so called ladder representation, the parameter space 𝔱 is one dimensional in our definition.) The resulting function M(𝔮, λ, δ) (precisely, see §4.1) vanishes for all δ if Θ_{λ} = 0 by definition. Then we present an explicit formula for M(𝔮, λ, δ) (Propositions in §4.3-5) in terms of the determinant of a matrix whose entries are polynomials of λ and δ, from which we can determine the explicit condition describing whether R_{𝔮}^{S}(ℂ_{λ}) vanishes or not under the assumption R_{𝔮}^{j}(ℂ_{λ}) = 0 (j ≠ S). The special value of M(𝔮, λ, δ) for the ‘minimal’ δ := μ̃_{λ} (see (4.3.4),(4.4.4),(4.5.4) for definition) is also found. It will play an important role in showing pairwise inequivalence among our discrete series in §8. This method here is applicable not only when μ_{λ} is not Δ^{+}(𝔨)-dominant but also when λ is small and out of the fair range.

We also mention that a related non-vanishing condition (necessity part) has been proven generally by Matsuki [21] for discrete series for semisimple symmetric spaces (i.e. H_{1} = {e} case). His proof is beautiful but depends on the fact that they are realized in L^{2}-eigenspaces on symmetric spaces.

To prove the irreducibility of R_{𝔮}^{S}(ℂ_{λ}) in the fair range in our settings, we can use the known theory of D-modules except the case G = Sp(p, q) (see Fact(6.2.4)). When 𝔤 = 𝔰𝔭(n, ℂ), we shall follow Vogan’s method in [33], Example 5.10. The main step of the proof is to find a very special direction to which a translation of the twisted differential operators on a generalized flag variety coming from U(𝔤) behaves reasonably. Although there is no new ideas in §6, we shall give a complete proof of this main step (see Theorem(6.3.1), a slight refinement of Theorem 5.11 in [33]) which is formulated more generally than what we need.

Here is an outline of the contents. In §1 we fix some notations and recall related facts. They are restrictive and set the scene for §2. The main results are Theorem 1, 2 and 3 in §2. The rest of the paper is devoted to prove them (sometimes with suitable generalization). Since the parameters outside the fair range are peculiar to our discrete series, our concern will be mainly with 𝔽 = ℂ or ℍ. §3 contains further useful notations and preliminary results on translation functors that we need in the proof. This is essentially standard material included for the benefit of the reader. §4, 5 and 6 are independent of each other and the reader can go directly to any of them. §4 considers the necessary and sufficient condition that R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 by a K-type formula. §5 provides another method to prove quickly the sufficient condition for the non-vanishing of Θ_{λ} in a more generalized situation. (We illustrate this idea when 𝔽 = ℍ case.) §6 gives the proof of irreducibility result in the fair range when 𝔤 = 𝔰𝔭(n, ℂ) with special parabolic subalgebras. §7 proves some vanishing results R_{𝔮}^{j}(ℂ_{λ}) = 0 (j ≠ S) outside the fair range with small spectrum. The proof for pairwise inequivalence of these modules is given in §8.

This paper consists of three lectures which were delivered at Conference on “Eigenfunctions on symmetric spaces and representations of Lie groups” held at RIMS Kyoto on July 1987, Summer school at University of Industrial Technology on August 1987, and Lie groups and representations Seminar at University of Tokyo on January 1989. The results here except §5 were announced in [14] with a sketch of the proof. The author expresses his sincere gratitude to Professor Toshio Oshima for the suggestion of generalizing Schlichtkrull’s results which was a motivation of the initial work [13]. Moreover, our results in §4 are inspired by the idea of controlling the multiplicities of ‘small’ K-types, which I learned from him. It is a pleasure to thank Professor Toshihiko Matsuki who is generously willing to allow me to publish our proof for §5 (quite different from his) here first. I would like to thank Professor Henrik Schlichtkrull who read quite carefully the original manuscript together with Dr. Jesper Bang-Jensen and sent me a list of errors and comments, and Mr. Hiroyuki Ochiai who examined parts of the manuscript. Thanks are also due to the referee for his careful and kind comments. I am also grateful to Professor Mogens Flensted-Jensen for his lectures in Japan in 1986 which stimulated my interest in this field, Professor David Vogan, Dr. Susana Salamanca-Riba and Professor Joseph A. Wolf for their interest in this work, and Dr. Itaru Terada for the instruction of TEX.

1. Notation

In this section we set up notation. See [28] and [32] for general references. Let H be a connected real linear reductive Lie group, with real Lie algebra 𝔥_{0} and complexified Lie algebra 𝔥. Since H is contained in a complex Lie group, we denote by H_{ℂ} the connected complex Lie subgroup with Lie algebra 𝔥. The center of the universal enveloping algebra U(𝔥) is written as ℨ(𝔥). In what follows analogous notation will be applied to Lie groups denoted by other Roman upper case letters without comment.

We will use the standard notation ℕ, ℤ, ℝ, ℂ and ℍ. Here ℕ means the set of non-negative integers and ℍ means the ℝ-algebra of quarternionic numbers. We denote by ℕ_{+} the set of positive integers. For x ∈ ℝ, we write [x] := sup{n ∈ ℤ; n ≤ x}, the Gaussian integer of x.

1.1. θ-stable parabolic subalgebra

Let G be a connected real linear reductive Lie group. Let K ⊂ G be a maximal compact subgroup and fix a Cartan involution θ so that 𝔤_{0} = 𝔨_{0} + 𝔭_{0} is a Cartan decomposition of 𝔤_{0}. Fix a nondegenerate bilinear form ⟨ , ⟩ on 𝔤 invariant under G and θ, which is positive definite on 𝔭_{0} and negative definite on 𝔨_{0}. This form will be restricted to subspaces and transferred to dual vector spaces without change of notation. If the restriction of ⟨ , ⟩ to each subspace 𝔞, 𝔟 with 𝔞 ⊂ 𝔟 is non-degenerate, we look upon

(1.1.1)  𝔞^{∗} ⊂ 𝔟^{∗}

through this bilinear form.

Fix an abelian subalgebra 𝔱_{0} of 𝔨_{0}. Define L to be the centralizer of 𝔱_{0} in G. Let 𝔲 be the vector space spanned by positive eigenspaces of a fixed generic element of \sqrt{−1}𝔱_{0}^{∗}. Then 𝔮 := 𝔩 + 𝔲 gives a Levi decomposition and 𝔮 is called a θ-stable parabolic subalgebra ([28] Definition 5.2.1). Note that L coincides with the normalizer of 𝔮 in G and is a connected reductive Lie group. Clearly 𝔱 is contained in the center of 𝔩, and in this paper we shall usually assume that 𝔱 satisfies

(1.1.2)  𝔱 coincides with the center of 𝔩.

We use this assumption only for simplifying notation. In most situations we will let

(1.1.3)  S := dim(𝔲 ∩ 𝔨) = \frac{1}{2} dim K/L ∩ K.

Write ℂ_{2ρ(𝔲)} for the determinant character of L on 𝔲 whose differential is given by

(1.1.4)  2ρ(𝔲)(X) = trace(ad(X)_{|𝔲})  (X ∈ 𝔩).

Let 𝔥 be a Cartan subalgebra of 𝔩. Then 𝔱 ⊂ 𝔥 and 𝔥 is also a Cartan subalgebra of 𝔤. We sometimes view ρ(𝔲) as an element of 𝔥^{∗} or of 𝔱^{∗} under the assumption(1.1.2) according to notation(1.1.1). Let L^{∼} ≡ L^{∼G} be the metapletic two-fold cover of L (Duflo; see [32] Definition 5.7) defined by the square root ρ(𝔲) of 2ρ(𝔲). (The definition of L^{∼} is independent of the particular choice of the nilradical 𝔲). Write ζ for the non-trivial element of the kernel of the covering map L^{∼} → L. Then the evaluation of ρ(𝔲) at ζ is by definition −1. A metapletic representation of L^{∼} is the one that is −1 on ζ.

There is a natural bijection between metapletic representations of L^{∼} and representations of L by the assignment

(1.1.5)  τ ↦ τ ⊗ ℂ_{−ρ(𝔲)}

for each metapletic representation τ.

We denote by ℂ_{λ} the one-dimensional metapletic representation of L^{∼} whose differential is given by λ ∈ 𝔩^{∗}. (Then automatically λ ∈ 𝔱^{∗} in the sense of (1.1.1) under the assumption (1.1.2).) We use the same notation ℂ_{λ} when it is a character of L or L^{∼}.

1.2. good range, fair range

Take a Cartan subalgebra 𝔥 of 𝔤 and let W (𝔤, 𝔥) be the Weyl group of the root system Δ(𝔤, 𝔥). We usually assume 𝔥_{0} is a fundamental Cartan subalgebra. This means 𝔥_{0} = θ𝔥_{0} and 𝔥_{0} = 𝔱_{0}^{c} + 𝔞_{0}^{c} ≡ (𝔥_{0} ∩ 𝔨_{0}) + (𝔥_{0} ∩ 𝔭_{0}) such that 𝔱_{0}^{c} is a Cartan subalgebra of 𝔨_{0}. A maximal ideal of ℨ(𝔤) is identified with a W (𝔤, 𝔥) orbit in 𝔥^{∗}:

Hom_{ℂ−algebra}(ℨ(𝔤), ℂ) ≃ 𝔥^{∗}/_{∼W (𝔤,𝔥)}

via the Harish-Chandra isomorphism ℨ(𝔤) ≃ S(𝔥)^{W (𝔤,𝔥)} which involves a shift by ρ(Δ^{+}(𝔤, 𝔥)).

We follow the terminologies below from [33] Definition 2.5. Suppose that 𝔥 is contained in 𝔩 in the setting of §1.1.

Definition 1.2.1. Let W be a metapletic (𝔩, (L ∩ K)^{∼})-module which has a ℨ(𝔩)- infinitesimal character represented by γ ∈ 𝔥^{∗}. We say that W is in the good range if

(1.2.2)  Re⟨α, γ⟩ > 0  for each α ∈ Δ(𝔲, 𝔥).

In this case we also say that γ is in the good range with respect to 𝔮 ⊂ 𝔤. Clearly, this condition is invariant under the action of the Weyl group W (𝔩, 𝔥).

Suppose that [𝔩, 𝔩] acts by zero on W. We say that W is in the fair range if

(1.2.2)  Re⟨α, γ_{|𝔱}⟩ > 0  for each α ∈ Δ(𝔲, 𝔥)

(notation §1.1). W (or γ) is called weakly good (respectively weakly fair) if the weak inequalities ( ≥) hold.

One should notice that good implies fair if [𝔩, 𝔩] acts by zero on W. Conversely, there exists in general a strip of the fair range that is not in the good range. Roughly, the size of this strip corresponds to that of a Levi part of 𝔮 and it is empty when 𝔮 is a Borel subalgebra.

1.3. cohomological parabolic induction

As an algebraic analogue of Dolbeault cohomology on a homogeneous complex manifold G/L, Zuckerman introduced the cohomological parabolic induction (we follow [32] Definition 6.20)

R_{𝔮}^{j} ≡ (R_{𝔮}^{𝔤})^{j} (j ∈ ℕ),

which is a covariant functor from the category of metapletic (𝔩, (L ∩ K)^{∼})-modules to the category of (𝔤, K)-modules.

With notation as before, fix a Cartan subalgebra 𝔥 ⊂ 𝔩. The definition here differs from [28] Definition 6.3.1 only by a ρ-shift. In our normalization, if a metapletic (𝔩, (L ∩ K)^{∼})-module W has ℨ(𝔩)-infinitesimal character γ ∈ 𝔥^{∗}, then R_{𝔮}^{j}(W) has ℨ(𝔤)- infinitesimal character γ in the Harish-Chandra parametrization.

1.4. results from Zuckerman and Vogan

Retain notations as in §1.1-2. The following theorem is due to Zuckerman and Vogan (see [28],[32],[35]).

Fact 1.4.1. In the setting of §1.1, suppose that W is a metapletic (𝔩, (L∩K)^{∼})-module.

1) Assume that W is weakly good. a) R_{𝔮}^{j}(W) = 0 for all j ≠ S.

UNITARY REPRESENTATIONS IN L^{2}(U(p, q; 𝔽)/U(p − m, q; 𝔽)) 17

b) R_{𝔮}^{S}(W) is an irreducible (𝔤, K)-module or zero if W is irreducible.

c) R_{𝔮}^{S}(W) is unitarizable if W is unitary.

2) If W (≠ 0) is good, then R_{𝔮}^{S}(W) ≠ 0.

3) If W is good and if R_{𝔮}^{S}(W) is unitary, then W is unitarizable. When W is fair, it is known that some of the properties in the above theorem fail:

Fact 1.4.2. Retain notations as in Fact(1.4.1) and assume that [𝔩, 𝔩] acts by zero on W.

1) Assume that W is weakly fair.

a) (true) R_{𝔮}^{j}(W) = 0 for all j ≠ S.

b) (false) R_{𝔮}^{S}(W) is an irreducible (𝔤, K)-module or zero.

c) (true) R_{𝔮}^{S}(W) is unitarizable if W is unitary.

2) (false) If W is fair, then R_{𝔮}^{S}(W) ≠ 0.

Part(1)(a),(c) is due to Vogan [29], Theorem7.1. A counter example for (1)(b) is given in [31] when 𝔤 is of type C_{4}. As for (2), see also §4 and §5.

Note that if W is fair R_{𝔮}^{j}(W) does not necessarily have a regular ℨ(𝔤)-infinitesimal character and does not always have nice behavior under translation.

1.5. results from Harish-Chandra and Oshima-Matsuki We do not use directly the results cited here in later sections. However, they will

help us to understand what we are doing in this paper. Retain notations in §1.1-3. Cohomologically induced representations from θ-stable parabolic subalgebras are convenient in describing ‘discrete series’:

Fact 1.5.1 (Harish-Chandra). Any discrete series for G is of the form R_{𝔮}^{S}(ℂ_{λ}) with 𝔱 a Cartan subalgebra of 𝔤 contained in 𝔨 and with ℂ_{λ} in the good range. (In this case 𝔮 is a Borel subalgebra of 𝔤.)

Fact 1.5.2 (Oshima-Matsuki [20], cf. [9], [33] Definition(2.8)). Let σ be an involution of G commuting with the Cartan involution θ, and H be an open subgroup of the fixed points of σ. Then any discrete series for a semisimple symmetric space G/H is of the form R_{𝔮}^{S}(ℂ_{λ}) with 𝔱 a Cartan subalgebra of 𝔨 contained in {X ∈ 𝔨; σ(X) = −X} and with ℂ_{λ} in the fair range.

Remark 1.5.3. Several remarks are in order. First, the existence of such an abelian subspace 𝔱 is equivalent to the famous rank condition rank G = rank K (respectively, rank G/H = rank K/H∩K), which is known to be the necessary and sufficient condition for the existence of discrete series. Second, Oshima and Matsuki do not use derived functor modules directly as alluded to in the Introduction. Third, we have mentioned neither the necessary evenness condition on λ (e.g. (2.7.4)(c)) nor the condition for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0. Finally, there exist now algebraic proofs of the unitarizability of these modules (Fact(1.4.2)).

2. Main results

In this section, we state our main theorems.

2.1. G = Sp(p, q)

Let G = Sp(p, q) where p, q ≥ 1 and let θ be a Cartan involution of 𝔤 corresponding to the maximal compact subgroup K = Sp(p) × Sp(q) naturally embedded in a matrix group G.

Let 𝔥_{0} be a fundamental Cartan subalgebra of 𝔤_{0}; then 𝔥_{0} is contained in 𝔨_{0}. Choose coordinates {f_{i}; 1 ≤ i ≤ p + q} on 𝔥^{∗} such that the root systems of 𝔤 and 𝔨 for 𝔥 are given by,

Δ(𝔤, 𝔥) = {±(f_{i} ± f_{j}), ±2f_{l}; 1 ≤ i < j ≤ p + q, 1 ≤ l ≤ p + q},
Δ(𝔨, 𝔥) = {±(f_{i} ± f_{j}) ; 1 ≤ i < j ≤ p or p + 1 ≤ i < j ≤ q}
∪ {±2f_{l}; 1 ≤ l ≤ p + q}.

Let {H_{i}} ⊂ 𝔥 be the dual basis for {f_{i}} ⊂ 𝔥^{∗}. Fix an integer r such that 1 ≤ r ≤ p. Set

𝔱 := ∑_{j=1}^{r} ℂH_{j} (⊂ 𝔥).

Let 𝔩 be the centralizer of 𝔱 in 𝔤, and L be the centralizer of 𝔱 in G. Then L is θ-stable in G, isomorphic to 𝕋^{r} × Sp(p − r, q) with complexified Lie algebra 𝔩.

Let 𝔮 ≡ 𝔮(r) = 𝔩 + 𝔲 (Levi decomposition) be a θ-stable parabolic subalgebra of 𝔤, whose nilpotent radical 𝔲 is defined by the following roots for 𝔥.

Δ(𝔲, 𝔥) := {f_{i} ± f_{j}, 2f_{l}; 1 ≤ i ≤ r, i < j ≤ p + q, 1 ≤ l ≤ r}.

20 TOSHIYUKI KOBAYASHI

Then ρ(𝔲) = ∑_{j=1}^{r}(p + q + 1 − j)f_{j}, S ≡ dim_{ℂ}(𝔲 ∩ 𝔨) = r(2p − r). Set Q := p + q − r(> 0). Let

λ := ∑_{i=1}^{r} λ_{i}f_{i} (λ_{i} ∈ ℂ)

be an element of 𝔱^{∗}, which will be sometimes regarded as an element of 𝔥^{∗}.

2.2. main theorem for G = Sp(p, q) Theorem 1. Retain notations in §2.1.

0) Any θ-stable parabolic subalgebra of 𝔤 with Levi part 𝔩 is conjugate to 𝔮 = 𝔩 + 𝔲 by an element of K.

1) ℂ_{λ} lifts to a metapletic (𝔩, (L∩K)^{∼})-module if and only if λ_{i} ∈ ℤ for all i (1 ≤ i ≤ r). From Part (2) to (6), we always assume

(2.2.1)  λ_{i} ∈ ℤ (1 ≤ i ≤ r).

2) If λ satisfies

(2.2.2)  λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r−1} ≥ |λ_{r}| and λ_{r} ≥ −Q,

then R_{𝔮}^{S−j}(ℂ_{λ}) = 0, for any j ≠ 0.

3) Under the assumption (2.2.2), R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 if and only if

(2.2.3)  \{ \begin{matrix}λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r−1} ≥ |λ_{r}| and λ_{r} ≥ −Q, \\ λ_{i} ≠ λ_{j}  (i ≠ j) \\ λ_{r−2q} ≥ Q + 1  (when r > 2q). \end{matrix}

Here we impose the third condition only when r > 2q.

If (2.2.3) is satisfied, then R_{𝔮}^{S}(ℂ_{λ}) has ℨ(𝔤)-infinitesimal character

(λ_{1}, …, λ_{r}, Q, Q − 1, …, 1) ∈ 𝔥^{∗}

in the Harish-Chandra parametrization.

4) The set {R_{𝔮}^{S}(ℂ_{λ}) ; λ satisfies (2.2.3)} consists of non-zero (𝔤, K)-modules which are pairwise inequivalent.

5) If λ satisfies (2.2.3) and λ_{r} > 0, then R_{𝔮}^{S}(ℂ_{λ}) is a non-zero irreducible (𝔤, K)-module.

6) Assume that r is an even number. Set r = 2m (0 < 2m ≤ p). If λ satisfies (2.2.3) and λ_{r−1} + λ_{r} > 0, then there is an injective (𝔤, K)-homomorphism

R_{𝔮}^{S}(ℂ_{λ}) ⟶ L^{2}(Sp(p, q)/Sp(p − m, q))

into discrete series for an indefinite quarternionic Stiefel manifold G/H_{2} = Sp(p, q)/Sp(p − m, q). In particular, the corresponding derived functor modules are non-zero unitarizable.

Remark 2.2.4. We can derive the unitarizability of R_{𝔮}^{S}(ℂ_{λ}) from Part(6) in the above theorem under the assumption (2.2.3). In fact, the case λ_{r−1} + λ_{r} = 0 with r = 2m corresponds to “limit of discrete series” for Sp(p, q)/Sp(p − m, q) and is still unitarizable. As for an odd integer r, we can deduce the unitarizability from the even integer r + 1 case. Actually, we choose sufficiently large λ_{1} and make use of Fact(1.4.1)(3), Lemma(3.2.1)(2).

Remark 2.2.5. If we apply our result to the case where r = 1 under (2.2.3), we have the known result on the unitarizability, the non-vanishing, and an explicit K-type formula (see also §4.3) of R_{𝔮}^{S}(ℂ_{λ}) due to Enright, Parthasarathy, Wallach, and Wolf (see [7], §9). In this case (r = 1), 𝔮 is quasi abelian and R_{𝔮}^{S}(ℂ_{λ}) is a ladder representation in their terminology. Our assumption(2.2.3) is nothing but λ_{1} ≥ −Q when r = 1, which is equivalent to “z < 0” after a careful comparison of ρ-shift (This range is referred as “especially curious” in [7], p126).

2.3. G = U(p, q)

Let G = U(p, q) where p, q ≥ 1 and let θ be a Cartan involution corresponding to the maximal compact subgroup K = U(p) × U(q) naturally embedded in a matrix group G.

Let 𝔥_{0} be a fundamental Cartan subalgebra of 𝔤_{0}; then 𝔥_{0} is contained in 𝔨_{0}. Choose coordinates {f_{i}; 1 ≤ i ≤ p + q} on 𝔥^{∗} such that the root systems of 𝔤 and 𝔨 for 𝔥 are given by

Δ(𝔤, 𝔥) = {±(f_{i} − f_{j}) ; 1 ≤ i < j ≤ p + q},
Δ(𝔨, 𝔥) = {±(f_{i} − f_{j}) ; 1 ≤ i < j ≤ p or p + 1 ≤ i < j ≤ p + q}.

Let {H_{i}} ⊂ 𝔥 be the dual basis of {f_{i}} ⊂ 𝔥^{∗}. Fix two nonnegative integers r and s such that 1 ≤ r + s ≤ p. Set

𝔱 := ∑_{i=1}^{r+s} ℂH_{i} + ℂ ∑_{i=1}^{p+q} H_{i} (⊂ 𝔥).

Then the centralizer L of 𝔱 in G is isomorphic to 𝕋^{r+s} × U(p − r − s, q).

Let 𝔮 ≡ 𝔮(r, s) = 𝔩 + 𝔲 be a θ-stable parabolic subalgebra of 𝔤 with nilpotent radical 𝔲 given by,

Δ(𝔲, 𝔥) := {f_{i} − f_{j} ; 1 ≤ i < j ≤ r + s}
∪ {f_{i} − f_{j} ; 1 ≤ i ≤ r, r + s + 1 ≤ j ≤ p + q}
∪ {−f_{i} + f_{j} ; r + 1 ≤ i ≤ r + s, r + s + 1 ≤ j ≤ p + q}.

Then S = \frac{1}{2}(r + s)(2p − r − s − 1). Set Q := \frac{1}{2}(p + q − r − s − 1) (≥ 0).

Finally, let

λ := ∑_{i=1}^{r+s} λ_{i}f_{i} + \frac{−r + s}{2} ∑_{i=1}^{p+q} f_{i} ∈ 𝔱^{∗} (λ_{i} ∈ ℂ).

2.4. main theorem for G = U(p, q)

Theorem 2. Retain notations in §2.3 and fix r, s.

0) There are r + s + 1 K-conjugacy classes of θ-stable parabolic subalgebras of 𝔤 with Levi part 𝔩. A complete system of representatives is given by

{𝔮(a, b) ; a, b ∈ ℕ, a + b = r + s}.

1) ℂ_{λ} lifts to a metapletic (𝔩, (L ∩ K)^{∼})-module if and only if λ_{i} ∈ ℤ + Q for all i (1 ≤ i ≤ r + s). From Part (2) to (6), we assume

(2.4.1)  λ_{i} ∈ ℤ + Q  (1 ≤ i ≤ r + s).

2) If λ satisfies

(2.4.2)  \{ \begin{matrix}λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r} ≥ λ_{r+1} ≥ ⋯ ≥ λ_{r+s}, \\ λ_{r} ≥ −Q  when r > 0, \\ λ_{r+1} ≤ Q  when s > 0, \end{matrix}

then R_{𝔮}^{S−j}(ℂ_{λ}) = 0 for any j ≠ 0.

3) Under the assumption (2.4.2), R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 if and only if

(2.4.3)  \{ \begin{matrix}λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ λ_{r+1} > λ_{r+2} > ⋯ > λ_{r+s}, \\ λ_{r−q} ≥ Q + 1  when r > q, \\ λ_{r} ≥ −Q  when r > 0, \\ λ_{r+q+1} ≤ −Q − 1  when s > q, \\ λ_{r+1} ≤ Q  when s > 0. \end{matrix}

If (2.4.3) is satisfied, then R_{𝔮}^{S}(ℂ_{λ}) has ℨ(𝔤)-infinitesimal character

(λ_{1}, …, λ_{r+s}, Q, Q − 1, …, −Q) + \frac{−r + s}{2}(1, 1, …, 1) ∈ 𝔥^{∗}.

4) The set {R_{𝔮}^{S}(ℂ_{λ}) ; λ satisfies (2.4.3)} consists of non-zero (𝔤, K)-modules which are pairwise inequivalent.

5) If λ satisfies (2.4.3) and λ_{r} ≥ 0 ≥ λ_{r+s}, then R_{𝔮}^{S}(ℂ_{λ}) is a non-zero irreducible (𝔤, K)-module.

6) Assume that r = s. Set m := r (= s). If λ satisfies (2.4.3) and λ_{r} > λ_{r+1}, then there is an injective (𝔤, K)-homomorphism

R_{𝔮}^{S}(ℂ_{λ}) ⟶ L^{2}(U(p, q)/U(p − m, q))

into discrete series for an indefinite complex Stiefel manifold U(p, q)/U(p − m, q). In particular, the corresponding derived functor modules are non-zero unitarizable.

Remark 2.4.4. The unitarizability of R_{𝔮}^{S}(ℂ_{λ}) under the assumption (2.4.3) (without assuming r = s) is derived from Part(6) in the above theorem by reasoning similar to Remark(2.2.4)

Remark 2.4.5. Our parabolic subalgebra 𝔮 ≡ 𝔮(r, s) is holomorphic ([1], Definition 1.6) iff r = 0 or s = 0. 𝔮 ≡ 𝔮(r, s) is quasi abelian ([7], p.109) iff (r, s) = (1, 0), (1, 1) or (0, 1). Another algebraic approach of some part of our results may be seen in the above cases (cf. Part(ii) and (iii) in Remark(0.11)).

2.5. G = SO_{0}(p, q)

Let G = SO_{0}(p, q) where p, q ≥ 1 and let θ be a Cartan involution corresponding to the maximal compact subgroup K = SO(p) × SO(q) naturally embedded in a matrix group G.

Let 𝔥_{0} = (𝔥_{0} ∩ 𝔨_{0}) + (𝔥_{0} ∩ 𝔭_{0}) ≡ 𝔱_{0}^{c} + 𝔞_{0}^{c} be a fundamental Cartan subalgebra of 𝔤. If both p and q are odd, then dim_{ℂ} 𝔞^{c} = 1, otherwise dim_{ℂ} 𝔞^{c} = 0 and 𝔥 = 𝔱^{c}. Choose coordinates \{f_{i}; 1 ≤ i ≤ [\frac{p + q}{2} ]\} on 𝔥^{∗} such that the root systems of 𝔤 and 𝔨 are given by

\begin{matrix}Δ(𝔤, 𝔥) = & \{±(f_{i} ± f_{j}) ; 1 ≤ i < j ≤ [\frac{p + q}{2} ]\} \\ ∪ (\{±f_{l}; 1 ≤ l ≤ [\frac{p + q}{2} ]\} (p + q:odd) ) \end{matrix}

Δ(𝔨, 𝔱^{c}) =\{±(f_{i} ± f_{j}) ; 1 ≤ i < j ≤ [\frac{p}{2}]\}

∪ \{±(f_{i} ± f_{j}) ; [\frac{p + q}{2} ] − [\frac{q}{2}] + 1 ≤ i < j ≤ [\frac{p + q}{2} ]\}

∪ (\{±f_{l}; 1 ≤ l ≤ [\frac{p}{2}]\} (p:odd) )

∪ (\{±f_{l}; [\frac{p + q}{2} ] − [\frac{q}{2}] + 1 ≤ l ≤ [\frac{p + q}{2} ]\} (q:odd) ) ,

respectively. Let {H_{i}} ⊂ 𝔥 be the dual basis for {f_{i}} ⊂ 𝔥^{∗}. Fix an integer r such that 1 ≤ r ≤ [\frac{p}{2}]. Set 𝔱 := ∑_{j=1}^{r} ℂH_{j} (⊂ 𝔱^{c} ⊂ 𝔥). Then the centralizer L of 𝔱 in G is isomorphic to 𝕋^{r} × SO_{0}(p − 2r, q). Define two elements of 𝔱^{∗} (⊂ (𝔱^{c})^{∗} ⊂ 𝔥^{∗}) by,

μ := ∑_{i=1}^{r}(\frac{p + q}{2} − i)f_{i}, μ′ := μ − (p + q − 2r)f_{r} ∈ 𝔱^{∗}.

26 TOSHIYUKI KOBAYASHI

Let 𝔮 = 𝔩 + 𝔲 (resp. 𝔮′ := 𝔩 + 𝔲′) be a θ-stable parabolic subalgebra of 𝔤 with nilpotent radical 𝔲 (resp. 𝔲′) given by,

Δ(𝔲, 𝔥) := {α ∈ Δ(𝔤, 𝔥) ; ⟨α, μ⟩ > 0},
Δ(𝔲′, 𝔥) := {α ∈ Δ(𝔤, 𝔥) ; ⟨α, μ′⟩ > 0}.

Then ρ(𝔲) = μ (resp. ρ(𝔲′) = μ′) when restricted to 𝔥 and S = r(p − r − 1). Set Q := \frac{p + q}{2} − r − 1(≥ −\frac{1}{2}). Finally, let

λ := ∑_{i=1}^{r} λ_{i}f_{i}, λ′ := λ − 2λ_{r}f_{r} ∈ 𝔱^{∗} (λ_{i} ∈ ℂ).

2.6. main theorem for G = SO_{0}(p, q) Theorem 3. Retain notations in §2.5.

0) A complete system of representatives of the K-conjugacy classes of θ-stable parabolic subalgebras of 𝔤 with Levi part 𝔩 is given by,

a) {𝔮} when p ≠ 2r,

b) {𝔮}, {𝔮′} when p = 2r.

1) The following three conditions on λ are equivalent:

a) ℂ_{λ} lifts to a metapletic (𝔩, (L ∩ K)^{∼})-module.

b) ℂ_{λ′} lifts to a metapletic (𝔩, (L ∩ K)^{∼})-module.

c) λ_{i} ∈ ℤ + Q for all i (1 ≤ i ≤ r). From Part (2) to (6), we always assume

(2.6.1)  λ_{i} ∈ ℤ + Q (1 ≤ i ≤ r).

2) If λ satisfies

(2.6.2)  λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r} ≥ 0,

then R_{𝔮}^{S−j}(ℂ_{λ}) = R_{𝔮′}^{S−j}(ℂ_{λ′}) = 0 for any j ≠ 0.

3) Under the assumption (2.6.2), the following three conditions are equivalent. a) R_{𝔮}^{S}(ℂ_{λ}) ≠ 0.

b) R_{𝔮′}^{S}(ℂ_{λ′}) ≠ 0.

c) λ satisfies

(2.6.3)  \{ \begin{matrix}λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ 0, \\ λ_{r−q} ≥ Q + 1  (when r > q). \end{matrix}

In this case, R_{𝔮}^{S}(ℂ_{λ}) and R_{𝔮′}^{S}(ℂ_{λ′}) have the same ℨ(𝔤)-infinitesimal character

(λ_{1}, …, λ_{r}, Q, Q − 1, …, Q − [Q]) ∈ 𝔥^{∗}.

4) When p ≠ 2r, R_{𝔮}^{S}(ℂ_{λ}) ≃ R_{𝔮′}^{S}(ℂ_{λ′}) and the set

{R_{𝔮}^{S}(ℂ_{λ}) ; λ satisfies (2.6.3)}

consists of pairwise inequivalent (𝔤, K)-modules. When p = 2r, the set

{R_{𝔮}^{S}(ℂ_{λ}), R_{𝔮′}^{S}(ℂ_{λ′}) ; λ satisfies (2.6.3)}

consists of pairwise inequivalent (𝔤, K)-modules.

5) If λ satisfies (2.6.3), then the (𝔤, K)-modules R_{𝔮}^{S}(ℂ_{λ}) and R_{𝔮′}^{S}(ℂ_{λ′}) are non-zero and irreducible.

6) If λ satisfies (2.6.3) and λ_{r} > 0, then there is an injective (𝔤, K)-homomorphism

R_{𝔮}^{S}(ℂ_{λ}) ⟶ L^{2}(SO_{0}(p, q)/SO_{0}(p − r, q)), when p ≠ 2r,
R_{𝔮}^{S}(ℂ_{λ}) ⊕ R_{𝔮′}^{S}(ℂ_{λ′}) ⟶ L^{2}(SO_{0}(p, q)/SO_{0}(p − r, q)), when p = 2r,

into discrete series for an indefinite real Stiefel manifold SO_{0}(p, q)/SO_{0}(p − r, q).

Remark 2.6.4. As opposed to the quarternionic case or the complex case, our result for real pseudo-orthogonal groups, on the unitarizability of derived functor modules, is completely contained in general theory because the parameter is always in the fair range.

2.7. list and figures of various conditions on parameters

For reference below we list various conditions on λ explicitly. We denote by (a) the case where G = Sp(p, q), by (b) the case where G = U(p, q), and by (c) the case where G = SO(p, q) respectively.

Q = p + q − r, (2.7.0)(a)

Q = \frac{1}{2}(p + q − r − s − 1), (2.7.0)(b)

Q = \frac{1}{2}(p + q) − r − 1. (2.7.0)(c)

The condition that ℂ_{λ} is in the weakly fair range amounts to

λ_{1} ≥ ⋯ ≥ λ_{r} ≥ 0, (2.7.1)(a) λ_{1} ≥ ⋯ ≥ λ_{r} ≥ 0 ≥ λ_{r+1} ≥ ⋯ ≥ λ_{r+s}, (2.7.1)(b) λ_{1} ≥ ⋯ ≥ λ_{r} ≥ 0. (2.7.1)(c)

The condition that ℂ_{λ} is in the weakly good range amounts to

λ_{1} ≥ ⋯ ≥ λ_{r} ≥ Q, (2.7.2)(a) λ_{1} ≥ ⋯ ≥ λ_{r} ≥ Q and − Q ≥ λ_{r+1} ≥ ⋯ ≥ λ_{r+s}, (2.7.2)(b) λ_{1} ≥ ⋯ ≥ λ_{r} ≥ max(0, Q). (2.7.2)(c)

The condition that μ_{λ} is dominant for Δ^{+}(𝔨) amounts to

λ_{1} > ⋯ > λ_{r} ≥ p − q − r + 1, (2.7.3)(a)

\{ \begin{matrix}λ_{1} > ⋯ > λ_{r} ≥ \frac{p − q − r − s + 1}{2}, \\ λ_{r+1} > ⋯ > λ_{r+s} ≥ −\frac{p − q − r − s + 1}{2}, \end{matrix} (2.7.3)(b)

λ_{1} > ⋯ > λ_{r} ≥ \frac{p − q − 2r}{2} . (2.7.3)(c)

Let r = 2m in (a), r = s = m in (b) and r = m in (c), respectively. The condition that R_{𝔮}^{S}(ℂ_{λ}) is discrete series for a semisimple symmetric space G/H = U(p, q; 𝔽)/U(m; 𝔽) × U(p − m, q; 𝔽) amounts to the following condition (conversely, discrete series for G/H are exhausted): λ_{i} ∈ ℤ + Q, ℂ_{λ} is in the fair range, and

(2.7.4)(a)  λ_{2i−1} = λ_{2i} + 1  (1 ≤ i ≤ m),
(2.7.4)(b)  λ_{i} = −λ_{2r+1−i}  (1 ≤ i ≤ m),
(2.7.4)(c)  λ_{i+1} − λ_{i} ∈ 2ℤ + 1 (1 ≤ i ≤ m − 1).

Example 2.7.5. Retain notation in Theorem 2 (6) and choose r = s = m = 1. Let G = U(p, q) ⊃ H = U(1) × U(p − 1, q) ⊃ H_{2} = U(p − 1, q) with p + q = 9, 2 ≤ p ≤ 8. In particular, rank G/H = 1.

\begin{matrix}· & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · \\ · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & × & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & • & • & × & • & • & • & • & • & • & • & • & • \\ · & · & · & · & • & • & • & • & × & • & • & • & • & • & • & • & • \\ · & · & · & • & • & • & • & • & • & × & • & • & • & • & • & • & • \\ · & · & • & • & • & • & • & • & • & • & × & • & • & • & • & • & • \\ · & · & • & • & • & • & • & • & • & • & • & × & • & • & • & • & • \end{matrix}

Figure 2.7.6. (Q = \frac{1}{2}(p + q − 3) = 3)

• : R_{𝔮}^{S}(ℂ_{λ}) (≠ 0) is discrete series for G/H_{2} given in Theorem 2 (6); (λ_{1}, λ_{2}) ∈ ℤ^{2}, λ_{1} > λ_{2}, −3 ≤ λ_{1} and λ_{2} ≤ 3.

× : R_{𝔮}^{S}(ℂ_{λ}) (≠ 0) is discrete series for G/H; λ_{1} = −λ_{2} ∈ ℕ_{+}.

μ_{λ} is Δ^{+}(𝔨)-dominant (Schlichtkrull type) ⇔ λ_{1} > 2, p − 5 ≤ λ_{1} and λ_{2} ≤ −p + 5,
ℂ_{λ} is weakly fair (cf. Fact(1.4.2))  ⇔ λ_{1} ≥ 0 ≥ λ_{2},
ℂ_{λ} is good (cf. Fact(1.4.1))  ⇔ λ_{1} > 3 > −3 > λ_{2},
R_{𝔮}^{S}(ℂ_{λ}) has singular infinitesimal character ⇔ λ_{1}(or λ_{2}) ∈ {0, ±1, ±2, ±3}.

The figure for G/H_{2} = U(p, q)/U(p − 1, q) (2 ≤ p, 1 ≤ q) is similar via the usual identification √\overline{−1}𝔱_{0}^{∗} ≃ ℝ^{2}. Roughly, λ_{1} − λ_{2} (resp. λ_{1} + λ_{2}) determines the asymptotic behavior (resp. the right action of H_{1} = U(1)) of the functions on G/H_{2} contained in a (𝔤, K)-module R_{𝔮}^{S}(ℂ_{λ}).

Example 2.7.9. Retain notation in Theorem 1 (6) and choose r = 2m = 2. Let G = Sp(p, q) ⊃ H = Sp(1) × Sp(p − 1, q) ⊃ H_{2} = Sp(p − 1, q) with p + q = 5, 2 ≤ p ≤ 4. In particular, rank G/H = 1.

\begin{matrix}· & · & · & · & · & · & · & · & · & · & · & · & × & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & × & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & × & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & × & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & × & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & × & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & × & • & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · \\ · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · \end{matrix}

Figure 2.7.10. (Q = p + q − 2 = 3)

• : R_{𝔮}^{S}(ℂ_{λ}) (≠ 0) is discrete series for G/H_{2} given in Theorem 1 (6); (λ_{1}, λ_{2}) ∈ ℤ^{2}, λ_{1} > |λ_{2}|, λ_{2} ≥ −3.

× : R_{𝔮}^{S}(ℂ_{λ}) (≠ 0) is discrete series for G/H; λ_{1} = λ_{2} + 1 ∈ ℕ_{+}.

μ_{λ} is Δ^{+}(𝔨)-dominant  ⇔ λ_{1} > λ_{2} ≥ 2p − 6,
ℂ_{λ} is weakly fair  ⇔ λ_{1} ≥ λ_{2} ≥ 0,
ℂ_{λ} is good  ⇔ λ_{1} > λ_{2} > 3,
R_{𝔮}^{S}(ℂ_{λ}) has singular infinitesimal character ⇔ λ_{1} = ±λ_{2} or λ_{i} ∈ {0, ±1, ±2, ±3}.

The curious condition λ_{r−2q} ≥ Q + 1 in (2.2.3) occurs only if r > 2q ≥ 2. Choose r = 2m = 4. Let G = Sp(p, q) ⊃ H = Sp(2) × Sp(p − 2, q) ⊃ H_{2} = Sp(p − 2, q) with p + q = 7, 4 ≤ p ≤ 6. In particular, rank G/H = 2. The cross section cut out by the (λ_{2}, λ_{4}) plane makes this figure visible, revealing the essential structure.

\begin{matrix}· & · & · & · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · \\ · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · \end{matrix}

Figure 2.7.11. (p, q) = (6, 1), Q = p + q − 4 = 3

\begin{matrix}· & · & · & · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & • & • & • & • & • & • & • & • & • & • \\ · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · \\ · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · & · \end{matrix}

Figure 2.7.12. (p, q) = (5, 2) or (4, 3), Q = p + q − 4 = 3

Let us explain what Figures (2.7.11) and (2.7.12) means. For each (λ_{2}, λ_{4}) corresponding to a black point • there, we have discrete series R_{𝔮}^{S}(ℂ_{λ}) for G/H_{2} = Sp(p, q)/Sp(p−2, q) with parameter λ = (λ_{1}, λ_{2}, λ_{3}, λ_{4}) whenever (λ_{1}, λ_{3}) ∈ ℤ^{2} satisfies λ_{1} > λ_{2} > λ_{3} > λ_{4}. There exist infinitely many such (λ_{1}, λ_{3}) for each (λ_{2}, λ_{4}) corresponding to a black point •. Among them, there exists a unique choice of (λ_{1}, λ_{3}) corresponding to discrete series for a semisimple symmetric space G/H = Sp(p, q)/Sp(2)× Sp(p − 2, q), namely λ_{1} := λ_{2} + 1, λ_{3} := λ_{4} + 1. In this case where r = 2m = 4, the condition λ_{r−2q} ≥ Q + 1 appears iff q = 1, which distinguishes Figure(2.7.11) (q = 1) from Figure(2.7.12) (q ≥ 2).

2.8. remarks

Suppose we are in the setting of Theorem 1, i.e. G = Sp(p, q). We explain what is easy and what is non-trivial by existing theories. The later sections are devoted to the proof for only non-trivial parts.

Part (0) and (1) are obvious. From Fact(1.4.2)(1-a), the non trivial part in (2) is when ℂ_{λ} is outside the weakly fair range, i.e. 0 > λ_{r} ≥ −Q. This case is proved in §7.1. We remark that if −Q > λ_{r}, then it may happen that the cohomology R_{𝔮}^{S−j}(ℂ_{λ}) ≠ 0 for j = 0, 1.

As for Part (3), it is easy to understand the conditions λ_{i} ≠ λ_{j} because they correspond to the ‘compact simple root condition’ (see Lemma(3.2.1)(1)). The last condition λ_{r−2q} ≥ Q + 1 is mysterious. Understanding this condition is the main subject of §4 (K-type method) and §5.

The irreducibility results (5) in the weakly fair range in Theorems 2 and 3 are derived from existing D-module theory (Fact(6.2.4)). So we shall restrict ourselves to the case 𝔤 = 𝔰𝔭(n, ℂ) in §6. It is likely that all the discrete series in Part(6) are irreducible even if the fair range condition λ_{r} > 0 is dropped. I have checked only the case where p = r by Vogan’s U_{α}-calculus.

Finally, the condition λ_{r} ≥ −Q corresponds to the condition that there exists a cyclic H_{2}^{r}-fixed vector in some principal series (see (0.2)(b)), while the condition λ_{r−1}+λ_{r} > 0 corresponds to square integrability. As we said in the Introduction, the proof of Part (6) is not treated in this paper.

3. Further notations and preliminary results

3.1. Jantzen-Zuckerman’s translation functor

Let G be a linear connected reductive Lie group as in §1 and H ⊂ G be a Cartan subgroup. When ξ ∈ 𝔥 is in the weight lattice for 𝔤 we write

(3.1.1)(a)  F (𝔤, ξ).

for the unique irreducible finite dimensional representation of 𝔤 of extremal weight ξ. Analogously, if this representation lifts to G, we denote the lift by

(3.1.1)(b)  F (G, ξ).

This is unambiguous because G is connected. The lifting condition is that ξ is in the weight lattice in H of G, that is, the the subgroup of 𝔥^{∗} consisting of weights of finite dimensional representations of G. With a little abuse of notation, we also write F (G_{ℂ}, ξ) for the holomorphic representation of G_{ℂ} with extremal weight ξ ∈ 𝔥^{∗} in §6, and F (L^{∼}, ξ) for the metapletic representation F (L, ξ − ρ(𝔲)) ⊗ ℂ_{ρ(𝔲)} in the setting of §1.1.

If X is a ℨ(𝔤)-finite 𝔤-module and γ is a character of ℨ(𝔤), we define

P_{γ}(X) := {x ∈ X; (z − γ(z))^{n}x = 0 for some n > 0, and all z ∈ ℨ(𝔤)}.

Suppose ξ ∈ 𝔥^{∗} is in the weight lattice, and γ ∈ 𝔥^{∗}. The Jantzen-Zuckerman translation functor is defined by

ψ_{γ}^{γ+ξ}(X) := P_{γ+ξ}(P_{γ}(X) ⊗ F (G, ξ)).

3.2. induction by stages

The spectral sequence of induction by stages sometimes clarifies critical points in proof. We shall frequently need the following well-known

Lemma 3.2.1 (see [28] Proposition 6.3.6). Let 𝔮 = 𝔩 + 𝔲 ⊂ 𝔭 = 𝔪 + 𝔫 be θ-stable parabolic subalgebras of 𝔤 with 𝔩 ⊂ 𝔪. Fix a metapletic (𝔩, (L ∩ K)^{∼G})-module W with respect to 𝔩 ⊂ 𝔤 (see §1.1 for notation). Notice that W ⊗ ℂ_{−ρ(𝔫)} is a metapletic (𝔩, (L ∩ K)^{∼M})-module with respect to 𝔩 ⊂ 𝔪. Then there is a first quadrant spectral sequence of (𝔤, K)-modules:

(R_{𝔭}^{𝔤})^{i} (ℂ_{ρ(𝔫)} ⊗ (R_{𝔮∩𝔪}^{𝔪})^{j} (W ⊗ ℂ_{−ρ(𝔫)})) ⇒ (R_{𝔮}^{𝔤})^{i+j} (W).

If ℂ_{ρ(𝔫)} lifts to a character of M, then we can write this simply as

(R_{𝔭}^{𝔤})^{i} ((R_{𝔮∩𝔪}^{𝔪})^{j} (W)) ⇒ (R_{𝔮}^{𝔤})^{i+j} (W).

Finally, suppose that W has ℨ(𝔩)-infinitesimal character γ ∈ 𝔥^{∗}. As a special case of the above spectral sequence, we have

1) Assume that M/L is compact and that W is finite dimensional. If γ is not regular and integral for 𝔪, then (R_{𝔮}^{𝔤})^{j} (W) = 0 for all j.

2) If γ is in the weakly good range with respect to 𝔭 ⊂ 𝔤, then

(R_{𝔭}^{𝔤})^{s} (ℂ_{ρ(𝔫)} ⊗ (R_{𝔮∩𝔪}^{𝔪})^{j} (W ⊗ ℂ_{−ρ(𝔫)})) ≃ (R_{𝔮}^{𝔤})^{s+j} (W).

where s := dim(𝔫 ∩ 𝔨).

3.3. definition of A (λ ▷ λ′)

Let 𝔩 be the centralizer in 𝔤 of a semisimple abelian subspace 𝔱 and fix a Cartan subalgebra 𝔥 ⊂ 𝔩 and a positive system Δ^{+}(𝔩, 𝔥). For λ, λ′ ∈ 𝔥^{∗} such that λ′ − λ is contained in the weight lattice of 𝔤, we introduce a finite subset A(Δ^{+}(𝔩); λ ▷ λ′) ≡ A (λ ▷ λ′) of 𝔥^{∗} as follows: For μ ∈ 𝔥^{∗}, μ is an element of A (λ ▷ λ′) iff

(3.3.1)(a) μ − λ is dominant for Δ^{+}(𝔩, 𝔥), (3.3.1)(b) μ − λ occurs as a weight of F (𝔤, λ′ − λ) for 𝔥,

(3.3.1)(c)  μ + ρ_{𝔩} ∈ W (𝔤, 𝔥) · (λ′ + ρ_{𝔩}).  (Weyl group orbit)

Inspecting the definition, one finds

σ · A(Δ^{+}(𝔩); λ ▷ λ′) = A(σ · Δ^{+}(𝔩); σ · λ ▷ σ · λ′),

for σ ∈ W (𝔩, 𝔥).

In later applications we will usually choose λ, λ′ ∈ 𝔥^{∗} so that

(3.3.2)  A (λ ▷ λ′) ⊂ 𝔱^{∗}

as well as λ, λ′ ∈ 𝔱^{∗} in the notation(1.1.1).

3.4. A (λ ▷ λ′) and derived functor modules

We describe a standard lemma for the connection between the translation principle and derived functor modules in terms of A (λ ▷ λ′). The smaller the cardinality of A (λ ▷ λ′) is, the less complicated the effect of translation functors on cohomologically induced representations becomes. This is the reason why we shall often use translation away from one wall to another wall avoiding regular infinitesimal characters.

Lemma(3.4.1) has two typical applications in later sections, in which we can proceed by induction on appropriate strings of λ’s in proving some properties on R_{𝔮}^{i}(ℂ_{λ}). One is to show R_{𝔮}^{S−i}(ℂ_{λ}) = 0 (i ≠ 0) where ℂ_{λ} is not necessarily in the fair range (see §7). The starting point of induction here is of course in the fair (or good) range. The other is to find a sufficient condition for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 (see §5), which is non-trivial even if ℂ_{λ} is fair. The starting point here is sometimes outside the fair range.

Lemma 3.4.1 (cf. [28] Chapter 7 §4). Suppose we are given two metapletic (𝔩, (L ∩ K)^{∼})-modules ℂ_{λ}, ℂ_{λ′} with λ, λ′ ∈ 𝔱^{∗}.

1) If R_{𝔮}^{j}(ℂ_{λ}) = R_{𝔮}^{j}(F (L^{∼}, μ)) = 0 for all j ∈ ℤ and all μ ∈ A (λ ▷ λ′) \ {λ′}, then R_{𝔮}^{j}(ℂ_{λ′}) = 0 for all j ∈ ℤ.

2) If A (λ ▷ λ′) = {λ′}, then

ψ_{λ+ρ_{𝔩}}^{λ′+ρ_{𝔩}} (R_{𝔮}^{j}(ℂ_{λ})) = R_{𝔮}^{j}(ℂ_{λ′}).

In particular, R_{𝔮}^{j}(ℂ_{λ}) = 0 implies R_{𝔮}^{j}(ℂ_{λ′}) = 0 for each fixed j ∈ ℤ.

3) Suppose that A (λ ▷ λ′) consists of two elements. Say, let A (λ ▷ λ′) = {μ, λ′}. Assume that R_{𝔮}^{j}(ℂ_{λ}) = R_{𝔮}^{j}(F (L^{∼}, μ)) = 0 for all j ≠ S. Then,

a) R_{𝔮}^{j}(ℂ_{λ′}) = 0 for j ≠ S − 1, S.

b) If λ′ ∉ μ + Δ(U(𝔲), 𝔥) and if

(3.4.2)  Hom_{(𝔤,K)}(R_{𝔮}^{S−1}(ℂ_{λ′}), R_{𝔮}^{S}(F (L^{∼}, μ))) = 0,

then R_{𝔮}^{j}(ℂ_{λ′}) = 0 for j ≠ S. b)′ If λ′ ∉ μ + Δ(U(𝔲), 𝔥) and if μ − λ is a unique weight of F (G, λ′ − λ) with the property (3.3.1)(a), then there is a long exact sequence of (𝔤, K)-modules:

⋯ → R_{𝔮}^{p−1}(ℂ_{λ′}) → R_{𝔮}^{p}(F (L^{∼}, μ)) → ψ_{λ+ρ_{𝔩}}^{λ′+ρ_{𝔩}} (R_{𝔮}^{p}(ℂ_{λ})) → R_{𝔮}^{p}(ℂ_{λ′}) → R_{𝔮}^{p+1}(F (L^{∼}, μ)) → · · · .

c) If μ ∉ λ′ + Δ(U(𝔲), 𝔥) (especially, if λ′ ∈ μ + Δ(U(𝔲), 𝔥)), then R_{𝔮}^{j}(ℂ_{λ′}) = 0 for j ≠ S.

Remark 3.4.3. In Part (3) of the above lemma, we allow the multiplicity of a weight μ − λ in F (G, λ′ − λ) to be greater than one. Observe that the multiplicity of λ′ − λ is always one.

Proof. We can prove this Lemma by an argument similar to that in Chapter 7 §4 [28], where the ℨ(𝔤)-infinitesimal characters λ+ρ_{𝔩} and λ′ +ρ_{𝔩} are assumed that one is regular and another lies in a generic point in the walls. So we review the argument there with a sketch of a necessary modification in our setting.

We first recall from Lemma(7.2.3) in [28] the following fact: Let F be a finite dimensional representation of G, and

(3.4.4)  {0} = F_{0} ⊂ F_{1} ⊂ ⋯ ⊂ F_{n} = F

be a (𝔮, L ∩ K) stable filtration with trivial induced action of 𝔲 on F_{i}/F_{i−1}. Then there is a natural spectral sequence of (𝔤, K)-modules

(3.4.5)  R_{𝔮}^{p}(W ⊗ F_{i}/F_{i−1}) ⇒ R_{𝔮}^{p}(W) ⊗ F.

Assume that W ≃ ℂ_{λ} (λ ∈ 𝔱^{∗} ⊂ 𝔩^{∗}), F = F (G, λ′ − λ) and F_{i}/F_{i−1} ≃ F (L, μ_{i}) with μ_{i} ∈ 𝔥^{∗} dominant for Δ^{+}(𝔩, 𝔥). Applying the projection P_{λ′+ρ_{𝔩}} on the both sides of (3.4.5), we obtain

(3.4.6)  P_{λ′+ρ_{𝔩}} (R_{𝔮}^{p}(F (L^{∼}, λ + μ_{i}))) ⇒ ψ_{λ+ρ_{𝔩}}^{λ′+ρ_{𝔩}} (R_{𝔮}^{p}(ℂ_{λ})) .

But P_{λ′+ρ_{𝔩}} (R_{𝔮}^{p}(F (L^{∼}, λ + μ_{i}))) ≠ 0 only if λ+ρ_{𝔩}+μ_{i} = w·(λ′+ρ_{𝔩}) for some w ∈ W (𝔤, 𝔥), thus only if λ + μ_{i} ∈ A (λ ▷ λ′) .

Recall that R_{𝔮}^{j}(W) = 0 for any (𝔩, (L∩K)^{∼}) module W and any j > S ([28] Corollary 6.3.21). Now a standard spectral sequence argument, combined with the following claim, completes the proof of the lemma. □

Recall that F_{|𝔩} ≃ ⊕_{i=1}^{n}F (L, μ_{i}) as a (𝔩, L ∩ K)-module.

Claim 3.4.7. With notation as above, assume ν_{1}, ν_{2} ∈ {μ_{1}, …, μ_{n}} satisfy ν_{2} ∉ ν_{1} + Δ(U(𝔲)). Then

U(𝔮)F (L, ν_{1}) ⫋ U(𝔮)(F (L, ν_{1}) + F (L, ν_{2})).

In particular, we can choose a filtration (3.4.4) so that i < j whenever μ_{i} = ν_{1} and μ_{j} = ν_{2}.

Proof of Claim. Consider an (𝔩, L ∩ K) surjective homomorphism

U(𝔲) ⊗ F (L, ν_{1}) → U(𝔲)F (L, ν_{1}) ≃ U(𝔮)F (L, ν_{1}) ⊂ F,

given by u⊗v ↦ u·v for u ∈ U(𝔲) and v ∈ F (L, ν_{1}). The highest weight of an irreducible finite dimensional 𝔩 module occurring in U(𝔲) ⊗ F (L, ν_{1}) is of the form ν_{1} + δ with some δ ∈ Δ(U(𝔲)). Therefore this remains true for the (𝔩, L∩K)-module U(𝔮)F (L, ν_{1}). From our assumption, we conclude that F (L, ν_{2}) does not occur in U(𝔮)F (L, ν_{1}). □

3.5. some symbols

Let k, m ∈ ℂ, and n ∈ ℕ. Write

[k; m, n] := (k, k + m, …, k + (n − 1)m) ∈ ℂ^{n}.

We also use the following notations:

⟨n⟩ := [n; −1, n] = (n, n − 1, …, 1),
k^{n} := [k; 0, n] = (k, k, …, k).

From definition we have

a_{1}[k_{1}; m_{1}, n] + a_{2}[k_{2}; m_{2}, n] = [a_{1}k_{1} + a_{2}k_{2}; a_{1}m_{1} + a_{2}m_{2}, n],
[k; m, n_{1} + n_{2}] = [k; m, n_{1}] ⊕ [k + n_{1}m; m, n_{2}],

for a_{i}, k_{i}, k, m_{i}, m ∈ ℂ and n_{i}, n ∈ ℕ (i = 1, 2).

When λ = (λ_{1}, …, λ_{n}) ∈ ℂ^{n}, we write ^{t}λ for (λ_{n}, …, λ_{1}), the transpose of λ.

The Weyl group W (C_{n}) ≃ 𝔖_{n} ⋉ ℤ_{2}^{n} acts by permuting and changing the signs of the coordinates of ℂ^{n}. For a fixed p (1 ≤ p ≤ n), 1×W (C_{n−p}), W (A_{n}) ≃ 𝔖_{n} and so on are regarded as subgroups of W (C_{n}) in an obvious way. These actions are sometimes restricted to an invariant subspace of ℂ^{n}.

4. Some explicit formulas on K multiplicities

4.1. preliminaries

In this section, we shall give some explicit formulas about K-types occurring in ∑_{i}(−1)^{i}R_{𝔮}^{S−i}(ℂ_{λ}). If the modules R_{𝔮}^{S−i}(ℂ_{λ}) vanish except in a single degree i = 0, this formula determines immediately whether R_{𝔮}^{S}(ℂ_{λ}) = 0 or not. A condition guaranteeing the vanishing of R_{𝔮}^{S−i}(ℂ_{λ}) (i ≠ 0) outside the fair range (cf. Fact(1.4.2)) will be given in §7 when R_{𝔮}^{S}(ℂ_{λ}) corresponds to discrete series for U(p, q; 𝔽)/U(p − m, q; 𝔽) (see §2). The setting of this section will be slightly more general in the range of ℂ_{λ} than the results in §2.

Suppose we are in the setting of §1.1. Let 𝔥 = 𝔱^{c} + 𝔞^{c} be a fundamental Cartan subalgebra of 𝔤 with 𝔥 ⊂ 𝔩. As usual we write half the sum of roots as ρ_{c} ≡ ρ(𝔨, 𝔱^{c}), ρ(𝔲) ≡ ρ(𝔲, 𝔥) and ρ(𝔲 ∩ 𝔨) ≡ ρ(𝔲 ∩ 𝔨, 𝔱^{c}), respectively. Let μ_{λ} := (λ + ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨))_{|𝔱^{c}} ∈ 𝔱^{c}^{∗}. We remark that our definition of R_{𝔮}^{S−i}(ℂ_{λ}) here follows [32] Definition 6.20 and differs from [28] Definition 6.3.1 by a ρ(𝔲)-shift (see §1.3). Recall that S = dim(𝔲 ∩ 𝔨). Then the Blattner formula due to Hecht-Schmid and generalized by Vogan ([28] Theorem 6.3.12) gives

(4.1.1) ∑_{i}(−1)^{i} dim Hom_{K} (π, R_{𝔮}^{S−i}(ℂ_{λ}))

= ∑_{j}(−1)^{j} dim Hom_{L∩K} (H^{j}(𝔲 ∩ 𝔨, π), S(𝔲 ∩ 𝔭) ⊗ ℂ_{μ_{λ}})

for π ∈ K̂, where S(V) denotes a symmetric tensor algebra of a vector space V. When μ_{λ} is Δ^{+}(𝔨) dominant, it is easy to see that the multiplicity of the particular K-type π = F (K, μ_{λ}) (notation (3.1.1)) in ∑_{i}(−1)^{i}R_{𝔮}^{S−i}(ℂ_{λ}) equals one, from formula (4.1.1), whence ∑_{i}(−1)^{i}R_{𝔮}^{S−i}(ℂ_{λ}) ≠ 0. However, for actual calculation with more general λ or π, too many cancellations occur, so that it is hard to tell even whether the formula(4.1.1) vanishes or not. T.Oshima showed me an unpublished note (Lemma(4.2.6)) on how to calculate the Blattner formula for some particular small K type (which frequently coincides with one of the minimal K-types in Vogan’s sense, cf. [28]) occurring in the discrete series for a semisimple symmetric space G/H = SO(p, q)/SO(m)×SO(p−m, q) even if μ_{λ} is not Δ^{+}(𝔨) dominant. Inspired by his technique, we are able to control all K-types in terms of some alternating polynomial functions (see (4.2.2)). To do this we also need another idea about ‘collecting K-types’. Communications with H.Schlichtkrull clarified this idea in the following general setting. Now let us explain it roughly.

Retain notations in §1.1-3. (So 𝔱_{0} is a fixed abelian subalgebra of 𝔨_{0} and 𝔥_{0} is a fundamental Cartan subalgebra of 𝔩_{0} (and also of 𝔤_{0}) such that 𝔱 ⊂ 𝔱^{c} = 𝔨 ∩ 𝔥 ⊂ 𝔥 ⊂ 𝔩.) We may assume that a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲 is defined by

Δ(𝔲, 𝔥) = {α ∈ Δ(𝔤, 𝔥) ; ⟨α, ν⟩ ≥ 1},
Δ(𝔩, 𝔥) = {α ∈ Δ(𝔤, 𝔥) ; ⟨α, ν⟩ = 0},

with a normalized generic element ν ∈ √\overline{−1}𝔱_{0}^{∗}.

We choose a positive system of Δ(𝔨, 𝔱^{c}) so that Δ(𝔲 ∩ 𝔨) ⊂ Δ^{+}(𝔨). Set Θ_{λ} := ∑_{i}(−1)^{i}R_{𝔮}^{S−i}(ℂ_{λ}) and π = F (K, μ). The information of each K-type occurring in Θ_{λ} (e.g. [Θ_{λ}_{|K} : π] = ∑_{j}(−1)^{j} dim Hom_{K}(π, R_{𝔮}^{S−j}(ℂ_{λ})) is so detailed that we can hardly expect that it is a nice function of μ ∈ (𝔱^{c})^{∗} and λ ∈ 𝔱^{∗}. In fact, such a function may behave somewhat irregularly when λ and μ are small enough. So our idea is to investigate more course information about K-types:

Set

K̂(Θ_{λ}) := {π ∈ K̂; [Θ_{λ}_{|K} : π] ≠ 0},
K̂(δ) := {π ∈ K̂; the highest weight μ of π satisfies μ_{|𝔱} = δ},

for λ ∈ 𝔱^{∗} and δ ∈ 𝔱^{∗}. If λ does not satisfy the integrality conditions so that ℂ_{λ} lifts to a metapletic (𝔩, (L ∩ K)^{∼})-character, we set K̂(Θ_{λ}) := {∅}. Similarly K̂(δ) := {∅} if δ does not lie in the obvious lattice in √\overline{−1}𝔱_{0}^{∗}. Clearly,

Θ_{λ}_{|K} ≡ 0 ⟺ K̂(Θ_{λ}) = ∅,

K̂ = ∐_{δ∈𝔱^{∗}} K̂(δ).

Writing P(K̂) for the totality of subsets in K̂, we have a map

𝔱^{∗} × 𝔱^{∗} ∋ (λ, δ) ↦ K̂(Θ_{λ}) ∩ K̂(δ) ∈ P(K̂).

Take a positive-valued function d: K̂ → ℕ_{+} (e.g. d(π) ≡ 1, or d(π) := dim π). Now we define a map M: 𝔱^{∗} × 𝔱^{∗} → ℤ by

(4.1.2)  M(λ, δ) := ∑_{π∈K̂(δ)} d(π) [Θ_{λ}_{|K} : π] = ∑_{π∈K̂(Θ_{λ})∩K̂(δ)} d(π) [Θ_{λ}_{|K} : π].

If R_{𝔮}^{i}(ℂ_{λ}) vanishes except in a single degree, then it follows from definition that

M(λ, δ) = 0 for all δ ∈ 𝔱^{∗} ⟺ R_{𝔮}^{i}(ℂ_{λ}) = 0 for all i.

What we would expect now is that, with a suitable choice of d: K̂ → ℕ_{+}, this M(λ, δ) is a restriction of a nice polynomial function of δ and λ so that we can tell explicitly whether or not M(λ, δ) = 0 for all δ ∈ 𝔱^{∗}. One of the simplest examples is a ladder representation which has K-highest weights lying along a single line ([7]) and in this case dim 𝔱 = 1. We will give a beautiful formula of M(λ, δ) with a special choice of d in some settings (see (4.1.8) for precise definition). The formula is expressed as a determinant of a certain matrix whose entries are polynomials of λ and δ, and the matrix reduces to a scalar if dim 𝔱 = 1. First of all, let us check that Definition(4.1.2) always makes sense. This is guaranteed by the following

Proposition 4.1.3. Retain notations as above. Then,

♯ (K̂(Θ_{λ}) ∩ K̂(δ)) < ∞.

We note that K̂(δ) is possibly an infinite set, as is K̂(Θ_{λ}). The proof of Proposition(4.1.3) is based on a uniform estimate over K̂(δ) as used in calculating the Blattner formula. That is,

Lemma 4.1.4. With notations as above, fix δ, b ∈ √\overline{−1}𝔱_{0}^{∗}. Recall that ν is a fixed element of √\overline{−1}𝔱_{0}^{∗}, defining a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲. If π ∈ K̂(δ) satisfies Hom_{T} (H^{j}(𝔲 ∩ 𝔨, π), S^{m}(𝔲 ∩ 𝔭) ⊗ ℂ_{b}) ≠ 0, then

m ≤ ⟨δ − b, ν⟩ − j.

Postponing the proof of Lemma(4.1.4) for a while, we first prove Proposition(4.1.3).

Proof of Proposition(4.1.3). Let π_{μ} ∈ K̂(δ). From (4.1.1) and Lemma(4.1.4),

|[Θ_{λ} : π_{μ}]| ≤ ∑_{j} dim Hom_{L∩K}(H^{j}(𝔲 ∩ 𝔨, π), ⨁_{m=0}^{∞} S^{m}(𝔲 ∩ 𝔭) ⊗ ℂ_{μ_{λ}})

≤ ∑_{j} dim Hom_{L∩K}(H^{j}(𝔲 ∩ 𝔨, π), ⨁_{m=0}^{⟨δ−μ_{λ},ν⟩} S^{m}(𝔲 ∩ 𝔭) ⊗ ℂ_{μ_{λ}}).

Therefore, we have

K̂(Θ_{λ}) ∩ K̂(δ) ⊂ ⋃_{j,τ} {π ∈ K̂; Hom_{L∩K}(H^{j}(𝔲 ∩ 𝔨, π), τ) ≠ 0},

where τ runs over the finite subset of { \atop L ∩ K} occurring in ⨁_{m=0}^{⟨δ−μ_{λ},ν⟩} S^{m}(𝔲 ∩ 𝔭) ⊗ ℂ_{μ_{λ}}. From Kostant’s Borel-Weil-Bott theorem ([15], see also [28] Corollary 3.2.16), the highest weight of an irreducible representation of L ∩ K occurring in H^{j}(𝔲 ∩ 𝔨, π) is of the form

w · (μ + ρ_{c}) − ρ_{c} ∈ (𝔱^{c})^{∗},

with w ∈ W_{K}^{𝔩∩𝔨} and l(w) = j. Here

W_{K}^{𝔩∩𝔨} := {w ∈ W (𝔨, 𝔱^{c}) ; Δ^{+}(w) ⊂ Δ(𝔲 ∩ 𝔨)},
Δ^{+}(w) := Δ^{+}(𝔨) ∩ w · Δ^{−}(𝔨),
l(w) = ♯Δ^{+}(w).

Write η ∈ (𝔱^{c})^{∗} for the highest weight of τ ∈ { \atop L ∩ K} (recall that L ∩ K is of maximal rank in K). Then the equation w · (μ + ρ_{c}) − ρ_{c} = η determines μ uniquely for each w and τ. Thus, for any τ ∈ { \atop L ∩ K}, we have

∑_{j} ♯{π ∈ K̂; Hom_{L∩K}(H^{j}(𝔲 ∩ 𝔨, π), τ) ≠ 0} ≤ ♯W_{K}^{𝔩∩𝔨} < ∞,

whence ♯ (K̂(Θ_{λ}) ∩ K̂(δ)) < ∞. □

Remark 4.1.5. Define a subset of K̂ by

K̂_{(ν)}(δ) := {π ∈ K̂; the highest weight μ of π satisfies ⟨μ, ν⟩ = ⟨δ, ν⟩}.

Then K̂_{(ν)}(δ) ⊃ K̂(δ). The above proof actually has shown the following statement:

♯ (K̂(Θ_{λ}) ∩ K̂_{(ν)}(δ)) < ∞.

This might be used to give some variations of the definition of M(λ, δ) in (4.1.2).

Proof of Lemma(4.1.4). As in the above proof, any 𝔱-weight occurring in H^{j}(𝔲 ∩ 𝔨, π) is of the form

(w · (μ + ρ_{c}) − ρ_{c})_{|𝔱}

with w ∈ W_{K}^{𝔩∩𝔨} and l(w) = j. Since μ ∈ (𝔱^{c})^{∗} is Δ^{+}(𝔨) dominant, there are nonnegative real numbers {n_{α}; α ∈ Δ^{+}(𝔨)} such that μ − w · μ = ∑_{α∈Δ^{+}(𝔨)} n_{α}α. We also have ρ_{c} −w·ρ_{c} = ∑_{α∈Δ^{+}(w)} α from definition of Δ^{+}(w). Thus, taking an inner product with ν ∈ √\overline{−1}𝔱_{0}^{∗} ⊂ (𝔱^{c})^{∗}, we have

⟨δ, ν⟩ = ⟨μ, ν⟩ = ⟨(μ − w · μ) + (ρ_{c} − w · ρ_{c}) + w · (μ + ρ_{c}) − ρ_{c}, ν⟩

= ⟨ ∑_{α∈Δ^{+}(𝔨)} n_{α}α + ∑_{α∈Δ^{+}(w)} α + w · (μ + ρ_{c}) − ρ_{c}, ν⟩

≥ j + ⟨w · (μ + ρ_{c}) − ρ_{c}, ν⟩. (4.1.6)

In the last inequality, we have used Δ^{+}(𝔨) ⊂ Δ(𝔩) ∪ Δ(𝔲) and Δ^{+}(w) ⊂ Δ(𝔲 ∩ 𝔨). On the other hand, any 𝔱-weight of S^{m}(𝔲 ∩ 𝔭) ⊗ ℂ_{b} is of the form

∑_{α∈Δ(𝔲∩𝔭)}m_{α}α_{|𝔱} + b

with some m_{α} ∈ ℕ satisfying ∑ m_{α} = m. In particular,

(4.1.7)  ⟨ ∑_{α∈Δ(𝔲∩𝔭)}m_{α}α_{|𝔱} + b, ν⟩ ≥ m + ⟨b, ν⟩.

From (4.1.6) and (4.1.7), Hom_{T} (H^{j}(𝔲 ∩ 𝔨, π), S^{m}(𝔲 ∩ 𝔭) ⊗ ℂ_{b}) ≠ 0 implies m + ⟨b, ν⟩ ≤ ⟨δ, ν⟩ − j. □

From now on, let us restrict ourselves to the following settings: Suppose that K has a direct decomposition K = K_{1} × K_{2} and 𝔱 ⊂ 𝔨_{1}. We write 𝔱^{c} = 𝔱_{1}^{c} + 𝔱_{2}^{c} according to 𝔨 = 𝔨_{1} + 𝔨_{2}. Clearly, we have 𝔱 ⊂ 𝔱_{1}^{c}. Let π = π_{1} ⊠ π_{2} ∈ K̂ ≃ K̂_{1} × K̂_{2}. Putting d(π) := dim π_{2} in (4.1.2), we introduce a multiplicity function

(4.1.8)  M(𝔮, λ, δ) := ∑_{π∈K̂(δ)} dim(π_{2}) [Θ_{λ}_{|K} : π],

for λ, δ ∈ 𝔱^{∗}(⊂ (𝔱_{1}^{c})^{∗}).

The rest of this section will be devoted to showing that this M(𝔮, λ, δ) behaves as a nice polynomial of δ under some hypotheses on λ (see §4.3-5 for results) when G = U(p, q; 𝔽). We finish this subsection with another formula for M(𝔮, λ, δ).

Set

(4.1.9)  m(𝔮, λ, π_{1}) := ∑_{i=0}^{S}(−1)^{i} dim Hom_{K_{1}}(π_{1}, R_{𝔮}^{S−i}(ℂ_{λ})).

Since H^{j}(𝔲 ∩ 𝔨, π) ≃ H^{j}(𝔲 ∩ 𝔨_{1}, π_{1}) ⊠ π_{2} as L ∩ K ≃ (L ∩ K_{1}) × K_{2} module, we have

(4.1.10) m(𝔮, λ, π_{1}) = ∑_{j} ∑_{m=0}^{∞}(−1)^{j} dim Hom_{L∩K_{1}}(H^{j}(𝔲 ∩ 𝔨_{1}, π_{1}), S^{m}(𝔲 ∩ 𝔭) ⊗ ℂ_{μ_{λ}}).

If δ^{(1)} ∈ (𝔱_{1}^{c})^{∗} is the highest weight of π_{1} ∈ K̂_{1}, we sometimes write m(𝔮, λ, δ^{(1)}) for m(𝔮, λ, π_{1}). Now we have

(4.1.11)  M(𝔮, λ, δ) = ∑_{π_{1}} m(𝔮, λ, π_{1}),

where the sum is taken over the set:

{π_{1} ∈ K̂_{1} ; the restriction of the highest weight of π_{1} to 𝔱 is δ}.

4.2. some alternating polynomials

Fix positive integers n and l. Define a polynomial of t by

(4.2.1)  a(t, n) =^{def} \frac{Γ(n + t)}{Γ(n)Γ(1 + t)} = \frac{(t + n − 1) ⋯ (t + 1)}{(n − 1)!}.

and polynomials of x = (x_{1}, …, x_{l}) and y = (y_{1}, …, y_{l}) by

F (n, l; x, y) =^{def} det (a(x_{i} + y_{j}; n)_{1≤i,j≤l}) , (4.2.2)(a)

d(n, l; x) =^{def} det (a(x_{i} − j; n)_{1≤i,j≤l}) . (4.2.2)(b)

By definition we have

(4.2.3)  d(n, l; x) = F (n, l; x, (−1, −2, …, −l)).

If t ∈ ℤ and t ≥ −n + 1, then

(4.2.4)  a(t, n) = dim S^{t}(ℂ^{n}),

where dim S^{t}(ℂ^{n}) := 0 for t < 0. The point of (4.2.4) is that it covers small negative integers because a(t, n) = 0 for −n+1 ≤ t ≤ −1. This will be useful for later applications (see §4.3-5).

Lemma 4.2.5. As a polynomial of x and y,

F (n, l; x, y) ≡ 0 if and only if n < l.

The following lemma is due to T.Oshima and J.Sekiguchi. We include here its proof, which is also due to them, for the sake of completeness. The author is grateful to them.

Lemma 4.2.6(Oshima-Sekiguchi).

d(n, l; x) = \{ \begin{matrix}\frac{∏^{l} ; i=1 ∏^{n−l} ; k=1(k + x_{i} − 1)}{∏^{l} ; j=1(n − l + j − 1)!} × ∏_{i>j}(x_{i} − x_{j}) & if n ≥ l, \\ 0 : & if n < l. \end{matrix}

In particular, for fixed n, l ∈ ℕ_{+} and x ∈ ℂ^{l},

d(n, l; x) ≠ 0 iff \{ \begin{matrix}n & ≥ l, \\ x_{i} & ∉ \{0, −1, …, 1 − n + l\} for any i, \\ x_{i} & ≠ x_{j} for any i ≠ j. \end{matrix}

Proof of Lemma 4.2.5. Since F (n, l; x, y) is an alternating polynomial function of (say) x, it is divisible by the simplest alternating polynomial ∏_{i>j}(x_{i} − x_{j}). On the other hand, F (n, l; x, y) is of degree ≤ n−1 with respect to x_{1} (expand the determinant along the first column), while ∏_{i>j}(x_{i}−x_{j}) is of degree l−1. Hence the ‘if’ part of the lemma. The ‘only if’ part is guaranteed by the special values given in Lemma(4.2.6). □

Proof of Lemma(4.2.6). From definition, we have

a(t, n) − a(t − 1, n) = \{ \begin{matrix}a(t, n − 1) & if n ≥ 2, \\ 0 & if n = 1. \end{matrix} (4.2.7)(a)

a(t, n + 1) = \frac{n + t}{n}a(t, n) (4.2.7)(b)

We put a(t, n) ≡ 0 if n ∈ −ℕ, then (4.2.7)(a) can be rewritten for n ∈ ℤ:

(4.2.8)  a(x_{i} − j, n) = a(x_{i} − (j + 1), n) + a(x_{i} − j, n − 1).

If we apply (4.2.8) to all j (1 ≤ j ≤ l − 1) and iterate on (4.2.8) then we conclude

d(n, l; x) = \{ \begin{matrix}det (a(x_{i} − j, n − l + j)_{1≤i,j≤l}) & if n ≥ l, \\ 0 & if n < l. \end{matrix}

From now on, assume that n ≥ l. Iteration on (4.2.7)(b) yields

a(x_{i} − j, n − l + j) = (∏^{n−l} ; k=1 \frac{k + x_{i} − 1}{k + j − 1}) a(x_{i} − j, j)

= (∏^{n−l} ; k=1(k + x_{i} − 1)) \frac{(x_{i} − 1) ⋯ (x_{i} − j + 1)}{(n − l + j − 1)!}.

UNITARY REPRESENTATIONS IN L^{2}(U(p, q; 𝔽)/U(p − m, q; 𝔽)) 51

Therefore

d(n, l; x) = \frac{∏^{l} ; i=1 ∏^{n−l} ; k=1(k + x_{i} − 1)}{∏^{l} ; j=1(n − l + j − 1)!} det ((x_{i} − 1) ⋯ (x_{i} − j + 1)_{1≤i,j≤l})

= \frac{∏^{l} ; i=1 ∏^{n−l} ; k=1(k + x_{i} − 1)}{∏^{l} ; j=1(n − l + j − 1)!} det ((x_{i}^{j−1})_{1≤i,j≤l})

Vandermonde’s determinant equals the simplest alternating polynomial, which leads to the desired formula. □

Finally we prove the following lemma for use in §4.8. Lemma 4.2.8. Let

(4.2.9)  d′(n, l; x) := det (a(x_{i} + j; n)_{1≤i,j≤l}) . (cf. (4.2.2)(b))

Then we have

d′(n, l; x) = (−1)^{l(n−1)}d(n, l; −x − [n; 0, l]).

(Recall notation(3.5).) In particular,

d′(n, l; x) ≠ 0 iff \{ \begin{matrix}n & ≥ l, \\ x_{i} & ∉ \{−l − 1, −l − 2, …, −n\} for any i, \\ x_{i} & ≠ x_{j} for any i ≠ j. \end{matrix}

Proof. This is a direct consequence of combinatorial reciprocity:

(4.2.10)  a(t, n) = (−1)^{n−1}a(−t − n, n).

The last statement is a translation of Lemma(4.2.6) via (4.2.10). □

4.3. result in quaternionic case

Retain notations in §4.1-2 and §2.1. We have 𝔱^{c} = 𝔥 ≃ ℂ^{p+q} in this case. Then,

ρ_{c}  = [p; −1, p] ⊕ [q; −1, q]  = (p, p − 1, …, 1, q, q − 1, …, 1),
ρ(𝔲) = [p + q; −1, r] ⊕ [0; 0, p + q − r] = (p + q, …, p + q − r + 1, 0, …, 0),
ρ(𝔲 ∩ 𝔨) = [p; −1, r] ⊕ [0; 0, p + q − r] = (p, p − 1, …, p − r + 1, 0, …, 0).

Thus we have,

ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨) = [−p + q; 1, r] ⊕ [0; 0, p + q − r]
= (−p + q, −p + q + 1, …, −p + q + r − 1, 0, …, 0).

For λ := (λ_{1}, …, λ_{r}, 0, …, 0) ∈ 𝔥^{∗}, we set

μ_{λ} := λ + ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨) = (b_{1}, …, b_{r}, 0, …, 0),

where

(4.3.1)  b_{i} := λ_{i} − p + q + i − 1  (1 ≤ i ≤ r).

Proposition 4.3.2. Retain notations as above. Let λ_{i} ∈ ℤ (1 ≤ i ≤ r), δ ∈ ℤ^{r}. Recall that we assign b = (b_{1}, …, b_{r}) to λ by (4.3.1) and that Q = p + q − r. Assume that

Q ≥ λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q, (4.3.3)(a) δ_{1} ≥ ⋯ ≥ δ_{r} ≥ 0. (4.3.3)(b)

Define y ≡ y(λ) = (y_{1}, …, y_{r}) ∈ ℤ^{r}, z ≡ z(δ) = (z_{1}, …, z_{r}) ∈ ℤ^{r} by y_{i} := i − b_{i} = −λ_{i} + p − q + 1 and z_{j} := δ_{j} − j. Set

k := max[{i; 1 ≤ i ≤ r, b_{i} ≥ 0} ∪ {0}]. (4.3.4)(a) μ̃_{λ} := (b_{1}, …, b_{k}, 0, …, 0) ∈ ℤ^{r} ⊂ 𝔱^{∗} (4.3.4)(b)

Then the following holds.

1) M(𝔮, λ, δ) = F (2q, r; y(λ), z(δ)).

2) Assume moreover that ∑_{i=1}^{r} δ_{i} = ∑_{i=1}^{k} b_{i}. Then a) If δ ≠ μ̃_{λ}, then M(𝔮, λ, δ) = 0.

b) If δ = μ̃_{λ}, then M(𝔮, λ, δ) = d(2q, r − k; y_{k+1}(λ), …, y_{r}(λ)).

3) M(𝔮, λ, δ) = 0 if ∑_{i=1}^{r} δ_{i} < ∑_{i=1}^{k} b_{i}. Remark 4.3.5. In the above Proposition the assumption Q ≥ λ_{1} in (4.3.3)(a) can be relaxed to Q ≥ λ_{k+1} in (2) and can be dropped in (3).

The proof of this Proposition together with Remark(4.3.5) will be given in §4.7 after some preparations in §4.6. We observe that r − 2q > 0 iff r − 2q > k under the assumption (4.3.3)(a). Indeed, b_{r−2q} = λ_{r−2q} − Q − 1 when r > 2q.

Since y_{i}(λ) = i − b_{i} ≥ 1 (k + 1 ≤ i ≤ r), combining Proposition(4.3.2) with Lemma(4.2.5-6), we have Corollary 4.3.6. Retain the same notation as in Proposition(4.3.2). If λ ∈ ℤ^{r}⊕0^{p+q−r} satisfies (4.3.3)(a) and if R_{𝔮}^{S−i}(ℂ_{λ}) = 0 for i ≠ 0, then the following two conditions are equivalent:

a) R_{𝔮}^{S}(ℂ_{λ}) = 0.

b) 2q < r.

If we use induction by stages (see an argument in Remark(5.1.5) in the next section), we have immediately Corollary 4.3.6′. Retain the same notation as in Proposition(4.3.2). If λ ∈ ℤ^{r} ⊕ 0^{p+q−r} satisfies

λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q,

and if R_{𝔮}^{S−i}(ℂ_{λ}) = 0 for i ≠ 0, then the following two conditions are equivalent: a) R_{𝔮}^{S}(ℂ_{λ}) = 0.

b) 2q < r and λ_{r−2q} ≤ Q. Remark 4.3.8. μ_{λ} is a K-dominant weight only if k = r (see (4.3.4)(a) for definition). (It is always the case when λ is ‘sufficiently’ regular.) This is the reason why we use μ̃_{λ}(∈ 𝔱^{∗}) instead of μ_{λ}.

4.4. result in complex case

Retain notations in §4.1-2 and §2.1. Note that 𝔱^{c} = 𝔥 in this case.

We abbreviate [k; m, n] as [k; m] when this vector is understood to be contained in ℂ^{n}. According to the direct decomposition:

𝔥^{∗} ≃ ℂ^{p+q} = ℂ^{r} ⊕ ℂ^{s} ⊕ ℂ^{p−r−s} ⊕ ℂ^{q}

defined by basis {f_{i}; 1 ≤ i ≤ p + q}, we have

ρ_{c} = [\frac{p − 1}{2}; −1] ⊕ [\frac{−p + 2s − 1}{2}; −1] ⊕ [\frac{p − 2r − 1}{2}; −1] ⊕ [\frac{q − 1}{2}; −1],

ρ(𝔲) = [\frac{p + q − 1}{2}; −1] ⊕ [\frac{−p − q + 2s − 1}{2}; −1] ⊕ [\frac{−r + s}{2}; 0] ⊕ [\frac{−r + s}{2}; 0],

ρ(𝔲 ∩ 𝔨) = [\frac{p − 1}{2}; −1] ⊕ [\frac{−p + 2s − 1}{2}; −1] ⊕ [\frac{−r + s}{2}; 0] ⊕ [0; 0].

Thus, ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨)

= [\frac{−p + q + 1}{2}; 1] ⊕ [\frac{p − q − 2s + 1}{2}; 1] ⊕ [\frac{r − s}{2}; 0] ⊕ [\frac{−r + s}{2}; 0].

For λ := (λ_{1}, …, λ_{r+s}, 0, …, 0) + [\frac{−r + s}{2}; 0, p + q], we set

μ_{λ} := λ + ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨) = (b_{1}, …, b_{r+s}) ⊕ [0; 0, p − r − s] ⊕ [−r + s; 0, q],

where

(4.4.1)  \{ \begin{matrix}b_{i} & := λ_{i} + \frac{−p + q − r + s − 1}{2} + i  (1 ≤ i ≤ r) \\ b_{r+i} & := λ_{r+i} + \frac{p − q − r − s − 1}{2} + i  (1 ≤ i ≤ s) \end{matrix}

Proposition 4.4.2. Retain notations as above. Let λ_{i} ∈ ℤ + Q (1 ≤ i ≤ r + s), δ ∈ ℤ^{r+s}. Recall that we defined Q = \frac{1}{2}(p + q − r − s − 1). Assume that

Q ≥ λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q, (4.4.3)(a) Q ≥ λ_{r+1} > λ_{r+2} > ⋯ > λ_{r+s} ≥ −Q, (4.4.3)(b) δ_{1} ≥ δ_{2} ≥ ⋯ ≥ δ_{r} ≥ 0 ≥ δ_{r+1} ≥ δ_{r+2} ≥ ⋯ ≥ δ_{r+s}. (4.4.3)(c)

We define y^{(1)} ≡ y^{(1)}(λ) = (y_{1}^{(1)}, …, y_{r}^{(1)}) ∈ ℤ^{r}, y^{(2)} ≡ y^{(2)}(λ) = (y_{1}^{(2)}, …, y_{s}^{(2)}) ∈ ℤ^{s}, z^{(1)} ≡ z^{(1)}(δ) = (z_{1}^{(1)}, …, z_{r}^{(1)}) ∈ ℤ^{r} and z^{(2)} ≡ z^{(2)}(δ) = (z_{1}^{(2)}, …, z_{s}^{(2)}) ∈ ℤ^{s} by

y_{i}^{(1)} := i − b_{i} = −λ_{i} − \frac{1}{2}(−p + q − r + s − 1) (1 ≤ i ≤ r), y_{i}^{(2)} := i − b_{r+i} = −λ_{r+i} − \frac{1}{2}(p −

q − r − s − 1) (1 ≤ i ≤ s), z_{j}^{(1)} := δ_{j} − j (1 ≤ j ≤ r) and z_{j}^{(2)} := δ_{r+j} − j (1 ≤ j ≤ s). Set

\{ \begin{matrix}k := max[\{i; 1 ≤ i ≤ r, b_{i} ≥ 0\} ∪ \{0\}], \\ l := max[\{i; 1 ≤ i ≤ s, 0 ≥ b_{r+s+1−i}\} ∪ \{0\}], \end{matrix} (4.4.4)(a)

μ̃_{λ} := (b_{1}, …, b_{k}, 0, …, 0, b_{r+s−l+1}, …, b_{r+s}, 0, …, 0) ∈ 𝔱^{∗} (4.4.4)(b)

Then the following holds:

1) M(𝔮, λ, δ) = F (q, r; y^{(1)}(λ), z^{(1)}(δ)) · F (q, s; −y^{(2)}(λ), −z^{(2)}(δ)).

2) Assume moreover that ∑_{i=1}^{r} δ_{i} = ∑_{i=1}^{k} b_{i} and that ∑_{i=1}^{s} δ_{r+i} = ∑_{i=1}^{l} b_{r+s−l+i}. a) If δ ≠ μ̃_{λ}, then M(𝔮, λ, δ) = 0.

b) If δ = μ̃_{λ}, then

M(𝔮, λ, δ) = d(q, r − k; y_{k+1}^{(1)}, …, y_{r}^{(1)}) · d′(q, s − l; −y_{1}^{(2)}, …, −y_{s−l}^{(2)}).

3) M(𝔮, λ, δ) = 0 if ∑_{i=1}^{r} δ_{i} < ∑_{i=1}^{k} b_{i} or if ∑_{i=1}^{s} δ_{r+i} > ∑_{i=1}^{l} b_{r+s−l+i}.

Remark 4.4.5. In the above Proposition, the assumption Q ≥ λ_{1} in (4.4.3)(a) (respectively, λ_{r+s} ≥ −Q in (4.4.3)(b)) can be relaxed to Q ≥ λ_{k+1} (respectively, λ_{r+s−l} ≥ −Q) in (2) and can be dropped in (3).

The argument parallels Proposition(4.3.2) in the quarternionic case; we will explicitly give only the necessary steps in §4.8. We observe that r − q > 0 iff r − q > k under the assumption (4.4.3)(a) and that s − q > 0 iff s − q > l under the assumption (4.4.3)(b). Indeed, b_{r−q} = λ_{r−q} − Q − 1 when r > q and b_{r+q+1} = λ_{r+q+1} + Q + 1 when s > q.

Since y_{i}^{(1)}(λ) = i − b_{i} ≥ 1 (k + 1 ≤ i ≤ r), −y_{i}^{(2)}(λ) = −i + b_{r+i} ≥ −(s − l) (1 ≤ i ≤ s − l), combining Proposition(4.4.2) with Lemma(4.2.5),(4.2.6) and (4.2.8), we have

Corollary 4.4.6. Retain the same notations in Proposition(4.4.2). If (λ_{1}, …, λ_{r+s}) ∈ ℤ^{r+s} + [Q; 0, r + s] satisfies

λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q,
Q ≥ λ_{r+1} > λ_{r+2} > ⋯ > λ_{r+s},

and if R_{𝔮}^{S−i}(ℂ_{λ}) = 0 for i ≠ 0, then the following two conditions are equivalent:

a) R_{𝔮}^{S}(ℂ_{λ}) = 0.

b) q < r and λ_{r−q} ≤ Q or q < s and λ_{r+q+1} ≥ −Q.

4.5. result in real case

Retain notations in §4.1-2 and §2.5. Note that 𝔱^{c} = 𝔥 iff pq is even. Set p′ := [\frac{p}{2}] and q′ := [\frac{q}{2}]. We have

ρ_{c} = [\frac{p}{2} − 1; −1, p′] ⊕ [\frac{q}{2} − 1; −1, q′],
ρ(𝔲)_{|𝔱^{c}} = [\frac{p + q}{2} − 1; −1, r] ⊕ [0; 0, p′ + q′ − r],
ρ(𝔲 ∩ 𝔨) = [\frac{p}{2} − 1; −1, r] ⊕ [0; 0, p′ + q′ − r],

Thus, ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨)_{|𝔱^{c}} = [\frac{−p + q}{2} + 1; 1, r] ⊕ [0; 0, p′ + q′ − r].

For λ := (λ_{1}, …, λ_{r}, 0, …, 0), λ′ := λ − 2λ_{r}f_{r} ∈ (𝔱^{c})^{∗}(⊂ −𝔥^{∗}), we set

μ_{λ} : = λ + ρ(𝔲) − 2ρ(𝔲 ∩ 𝔨) = (b_{1}, …, b_{r}, 0, …, 0) ∈ (𝔱^{c})^{∗},
μ′_{λ′} : = λ′ + ρ(𝔲′) − 2ρ(𝔲′ ∩ 𝔨)= (b′_{1}, …, b′_{r}, 0, …, 0) ∈ (𝔱^{c})^{∗},

where

(4.5.1)  \{ \begin{matrix}b_{i} : & = λ_{i} + \frac{−p + q}{2} + i  (1 ≤ i ≤ r), \\ b_{i} & = b′_{i} (1 ≤ i ≤ r − 1), b_{r} = −b′_{r}. \end{matrix}

Proposition 4.5.2. Retain notations as above. Let λ_{i} ∈ ℤ + Q (1 ≤ i ≤ r), δ ∈ ℤ^{r}. Recall that Q = \frac{1}{2}(p + q) − r − 1. Assume that

Q ≥ λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q, (4.5.3)(a)

(4.5.3)(b)  δ_{1} ≥ δ_{2} ≥ ⋯ ≥ δ_{r} ≥ 0  (p ≠ 2r)
(4.5.3)(b′)  δ_{1} ≥ δ_{2} ≥ ⋯ ≥ δ_{r−1} ≥ |δ_{r}|  (p = 2r)

We define y ≡ y(λ) = (y_{1}, …, y_{r}) ∈ ℤ^{r}, z ≡ z(δ) = (z_{1}, …, z_{r}) ∈ ℤ^{r} by y_{i} := i−b_{i} =

−λ_{i} − \frac{p − q}{2} (1 ≤ i ≤ r), z_{j} := δ_{j} − j (1 ≤ j ≤ r). Set

k := max[{i; 1 ≤ i ≤ r, b_{i} ≥ 0} ∪ {0}]. (4.5.4)(a) μ̃_{λ} := (b_{1}, …, b_{k}, 0, …, 0). (4.5.4)(b)

Then the following holds:

1) M(𝔮, λ, δ) = F (q, r; y(λ), z(δ)).

2) Assume moreover that ∑_{i=1}^{r} δ_{i} = ∑_{i=1}^{k} b_{i}. Then a) If δ ≠ μ̃_{λ}, then M(𝔮, λ, δ) = 0.

b) If δ = μ̃_{λ}, then M(𝔮, λ, δ) = d(q, r − k; y_{k+1}(λ), …, y_{r}(λ)).

3) M(𝔮, λ, δ) = 0 if ∑_{i=1}^{r} δ_{i} < ∑_{i=1}^{k} b_{i}. Similar statements are valid for λ′.

Corollary 4.5.5. Retain the same notations in Proposition(4.5.2). If (λ_{1}, …, λ_{r}) ∈ ℤ^{r} + [Q; 0, r] satisfies

λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q,

and if R_{𝔮}^{S−i}(ℂ_{λ}) = R_{𝔮′}^{S−i}(ℂ_{λ′}) = 0 for i ≠ 0, then the following three conditions are equivalent:

a) R_{𝔮}^{S}(ℂ_{λ}) = 0.

a′) R_{𝔮′}^{S}(ℂ_{λ′}) = 0.

b) q < r and λ_{r−q} ≤ Q.

4.6. some auxiliary lemmas

In this subsection, we collect elementary lemmas used in the proof of §4.3-5. Our strategy is to reduce an alternating sum in a generalized Blattner formula to the determinant of some matrix. The main point is that only a (possibly small) symmetric group contained in a Weyl group appears in the actual calculation. In particular the elements in a Weyl group which reverse the signature of coordinates do not appear under suitable inequalities of vectors. This is what we describe here.

Fix p, r ∈ ℕ with 1 ≤ r ≤ p. Lemma 4.6.1. Let a = (a_{1}, …, a_{r}, 0, …, 0) ∈ ℤ^{p}. Assume that

(4.6.2)  a_{j} ≥ −2p + 2r − 1 for any j (1 ≤ j ≤ r).

If

(4.6.3)  w · (δ + ⟨p⟩) − ⟨p⟩ = a

for some w ∈ 𝔖_{p} ⋉ ℤ_{2}^{p}, δ = (δ_{1}, …, δ_{p}) ∈ ℕ^{p}, then we have

\{ \begin{matrix}w & ∈ 𝔖_{r} × 1_{p−r} (⊂ 𝔖_{r} × 𝔖_{p−r} ⊂ 𝔖_{p} ⋉ ℤ_{2}^{p}), \\ δ_{j} & = 0  (r + 1 ≤ j ≤ p). \end{matrix}

Proof. The equation(4.6.3) is written as

(4.6.3′) w · (δ_{1} + p, δ_{2} + p − 1, …, δ_{p} + 1)

= (a_{1} + p, a_{2} + p − 1, …, a_{r} + p + 1 − r, p − r, p − r − 1, …, 1).

Therefore there are at least p−r elements among {δ_{j}+p+1−j; 1 ≤ j ≤ p} whose absolute values are not greater than p − r. Since δ_{j} ≥ 0, this implies w ∈ (𝔖_{r} ⋉ ℤ_{2}^{r}) × 1_{p−r} and δ_{j} = 0 (r + 1 ≤ j ≤ p).

Now we must show that w does not reverse any signs. The condition (4.6.2) assures that a_{j} + p + 1 − j ≥ −(p − r), while δ_{j} + p + 1 − j ≥ p + 1 − j ≥ p + 1 − r for 1 ≤ j ≤ r. Hence a_{j} + p + 1 − j must be positive and w ∈ 𝔖_{r} × 1_{p−r}, which completes the proof. Note that we have also proved at the same time that a_{j} + p + 1 − j ≥ p + 1 − r, that is, a_{j} ≥ −r + j (1 ≤ j ≤ r). □

Lemma 4.6.4. Let b = (b_{1}, …, b_{r}) ∈ ℤ^{r} satisfy

b_{1} ≥ b_{2} ≥ ⋯ ≥ b_{k} ≥ 0, (4.6.5)(a) 0 > b_{j} (k + 1 ≤ j ≤ r), (4.6.5)(b)

for some k (0 ≤ k ≤ r). Set

(4.6.6)  n := − ∑_{j=k+1}^{r} b_{j} ∈ ℕ.

Assume that there are w ∈ 𝔖_{r}, c = (c_{1}, …, c_{r}) ∈ ℤ^{r}, δ = (δ_{1}, …, δ_{r}) ∈ ℤ^{r} such that

w · (δ + ⟨r⟩) − ⟨r⟩ = b + c, (4.6.7) c_{j} ≥ 0 (1 ≤ j ≤ k), (4.6.8)(a) δ_{1} ≥ δ_{2} ≥ ⋯ ≥ δ_{r} ≥ 0 (4.6.8)(b)

Then the following holds.

1) ∑_{j=1}^{r} c_{j} = ∑_{j=1}^{r}(δ_{j} − b_{j}) ≥ n.

2) If ∑_{j=1}^{r} c_{j} = n or equivalently if ∑_{j=1}^{r} δ_{j} = ∑_{j=1}^{k} b_{j}, then

\{ \begin{matrix}w ∈ 1_{k} × 𝔖_{r−k}, \\ δ_{j} = b_{j} (1 ≤ j ≤ k), δ_{j} = 0 (k + 1 ≤ j ≤ r), \\ c_{j} = 0 (1 ≤ j ≤ k), c_{k+j} = −b_{k+j} + j − w · j (1 ≤ j ≤ r − k). \end{matrix}

3) If ∑_{j=1}^{r} c_{j} > n, then ∑_{j=1}^{r} δ_{j} > ∑_{j=1}^{k} b_{j}. In particular, we have

δ ≠ (b_{1}, …, b_{k}, 0, …, 0).

Proof. From definition(4.6.6) of n and the equation(4.6.7), we have

(4.6.9)  ∑_{j=1}^{r} δ_{j} = ∑_{j=1}^{r}(b_{j} + c_{j}) = ∑_{j=1}^{k} b_{j} + ∑_{j=1}^{r} c_{j} − n.

Using inequalities (4.6.8)(a),(b), we have

(4.6.10)  ∑_{j=1}^{k} b_{j} ≤ ∑_{j=1}^{k}(b_{j} + c_{j}) ≤ ∑_{j=1}^{k} δ_{j} ≤ ∑_{j=1}^{r} δ_{j}.

Here the middle inequality is because b + c − δ = w · (δ + ⟨r⟩) − (δ + ⟨r⟩) and because the components of δ + ⟨r⟩ are strictly decreasing.

The statement (3) is clear from (4.6.9). Substitution of (4.6.9) into (4.6.10) gives ∑_{j=1}^{r} c_{j} ≥ n, namely, (1).

Finally assume ∑_{j=1}^{r} c_{j} = n. Then all the terms in (4.6.10) must be equal, which implies

\{ \begin{matrix}w ∈ 𝔖_{k} × 𝔖_{r−k}, \\ δ_{j} = 0  (k + 1 ≤ j ≤ r), \\ c_{j} = 0  (1 ≤ j ≤ k). \end{matrix}

Then both of the first k coordinates of δ + ⟨r⟩ and b + c + ⟨r⟩ are strictly decreasing, so that w ∈ 1_{k} × 𝔖_{r−k}. Now the remaining assertions in (2) are clear. □

The following result is a direct consequence of Lemma(4.6.1) and Lemma(4.6.4).

Proposition 4.6.11. Let b = (b_{1}, …, b_{r}, 0, …, 0) ∈ ℤ^{p}. Assume that

\{ \begin{matrix}b_{1} ≥ b_{2} ≥ ⋯ ≥ b_{k} ≥ 0 \\ 0 > b_{j} ≥ −2p + 2r − 1 (k + 1 ≤ j ≤ r), \end{matrix}

for some k (0 ≤ k ≤ r). Suppose that there are w ∈ 𝔖_{p}⋉ℤ_{2}^{p}, c = (c_{1}, …, c_{r}, 0, …, 0) ∈ ℤ^{p} and δ = (δ_{1}, …, δ_{p}) ∈ ℤ^{p} such that

\{ \begin{matrix}w · (δ + ⟨p⟩) − ⟨p⟩ = b + c, \\ c_{j} ≥ 0  (1 ≤ j ≤ r), \\ δ_{1} ≥ δ_{2} ≥ ⋯ ≥ δ_{p} ≥ 0. \end{matrix}

Set n := − ∑_{j=k+1}^{r} b_{j}. Then the following holds.

1) ∑_{j=1}^{r} c_{j} ≥ n, ∑_{j=1}^{r} δ_{j} ≥ ∑_{j=1}^{k} b_{j}, w ∈ 𝔖_{r} × 1_{p−r}, and δ_{j} = 0 (r + 1 ≤ j ≤ p).

2) If ∑_{j=1}^{r} c_{j} = n or equivalently if ∑_{j=1}^{r} δ_{j} = ∑_{j=1}^{k} b_{j}, then,

\{ \begin{matrix}w ∈ 1_{k} × 𝔖_{r−k}, \\ δ_{j} = b_{j} (1 ≤ j ≤ k), δ_{j} = 0 (k + 1 ≤ j ≤ r), \\ c_{j} = 0 (1 ≤ j ≤ k), c_{k+j} = −b_{k+j} + j − w · j (1 ≤ j ≤ r − k). \end{matrix}

3) If ∑_{j=1}^{r} c_{j} > n, then ∑_{j=1}^{r} δ_{j} > ∑_{j=1}^{k} b_{j}. In particular,

δ ≠ (b_{1}, …, b_{k}, 0, …, 0).

4.7. proof for quarternionic case

In this subsection, we complete the proof of Proposition(4.3.2). Suppose we are in the setting of §4.3. First recall (cf. §2.1) that the root system of 𝔲 for 𝔱^{c} = 𝔥 is given by

Δ(𝔲 ∩ 𝔭, 𝔥) := {f_{i} ± f_{j} ; 1 ≤ i ≤ r, p + 1 ≤ j ≤ p + q},

and that L ∩ K ≃ 𝕋^{r} × Sp(p − r) × Sp(q) acts on 𝔲 ∩ 𝔭 ≃ ℂ^{2rq} by

⨁_{i=1}^{r} (F (𝕋^{r}, f_{i}) ⊠ 1 ⊠ F (Sp(q), (1, 0, …, 0))) .

Assume that λ ∈ ℤ^{r} satisfies

λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q ≡ −p − q + r.

Then μ_{λ} ≡ (b_{1}, …, b_{r}, 0, …, 0) with b_{j} ≡ λ_{j} − p + q + j − 1 satisfies

(4.7.1)  b_{1} ≥ b_{2} ≥ ⋯ ≥ b_{r} ≥ −2p + 2r − 1

Define k (0 ≤ k ≤ r) by (4.3.4).

Lemma 4.7.2. Retain notation as above, let λ ∈ ℤ^{r}, δ ∈ ℤ^{p} with the same hypotheses on λ and δ_{1} ≥ ⋯ ≥ δ_{p} ≥ 0.

1) If (δ_{r+1}, …, δ_{p}) = (0, …, 0), then

(4.7.3)  m(𝔮, λ, δ) = ∑_{w,{c_{i}}} sgn(w) ∏_{i=1}^{r} dim S^{c_{i}}(ℂ^{2q}).

(See §4.1 for notation.) Here the sum is taken over the set satisfying

w ∈ 𝔖_{r}, c = (c_{1}, …, c_{r}) ∈ ℕ^{r}, (4.7.4)(a)

w · ((δ_{1}, …, δ_{r}) + ⟨r⟩) − ⟨r⟩ = b + c. (4.7.4)(b)

2) If

(4.7.5)  (δ_{r+1}, …, δ_{p}) ≠ (0, …, 0) or ∑_{i=1}^{r} δ_{i} < ∑_{i=1}^{k} b_{i},

then m(𝔮, λ, δ) = 0.

Proof. As an L ∩ K_{1} ≃ 𝕋^{r} × Sp(p − r)-module,

S(𝔲 ∩ 𝔭) ≃ ⨁_{c∈ℕ^{r}} (∏_{i=1}^{r} dim S^{c_{i}}(ℂ^{2q}).) · (c_{1}, …, c_{r}) ⊠ 1,

where the multiplier in the right hand stands for the multiplicity of each representation and c = (c_{1}, …, c_{r}) ∈ ℤ^{r} denotes an additive character of 𝕋^{r}. Thus for each π_{1} ∈ K̂_{1},

(4.7.6) Hom_{L∩K_{1}}(H^{j}(𝔲 ∩ 𝔨_{1}, π_{1}), S(𝔲 ∩ 𝔭) ⊗ ℂ_{μ_{λ}})

≃ ⨁_{c∈ℕ^{r}} (∏_{i=1}^{r} dim S^{c_{i}}(ℂ^{2q})) · Hom_{𝕋^{r}×Sp(p−r)}(H^{j}(𝔲 ∩ 𝔨_{1}, π_{1}), (b + c) ⊠ 1).

We will regard c as an element of ℕ^{p} by putting c = (c_{1}, …, c_{r}, 0, …, 0) without comment. Appealing to Kostant’s Borel-Weil theorem

dim Hom_{𝕋^{r}×Sp(p−r)}(H^{j}(𝔲 ∩ 𝔨_{1}, π_{1}), (b + c) ⊠ 1)

is the number of elements w of

W_{K_{1}}^{𝔩∩𝔨_{1}} := {w ∈ W (𝔨_{1}) ; Δ^{+}(𝔨_{1}) ∩ w · Δ^{−}(𝔨_{1}) ⊂ Δ(𝔲 ∩ 𝔨_{1})}

such that

l(w) = j, (4.7.7)(a) w · (δ + ⟨p⟩) − ⟨p⟩ = b + c. (4.7.7)(b)

Assume that (4.7.7)(b) holds. From (4.7.1) and c_{i} ≥ 0, we are able to apply Proposition(4.6.11), so that we conclude w ∈ 𝔖_{r} × 1_{p−r}, δ_{i} = 0 (r + 1 ≤ i ≤ p) and ∑_{i=1}^{r} δ_{i} ≥ ∑_{i=1}^{k} b_{i}. Thus the statement (2) is clear. Now Part (1) is followed from (4.7.6) since 𝔖_{r} × 1_{p−r} ⊂ W_{K_{1}}^{𝔩∩𝔨_{1}}. □

Proof of Proposition(4.3.2). First notice that we have

(4.7.8)  M(𝔮, λ, δ) = m(𝔮, λ, δ ⊕ [0; 0, p − r]),

from Lemma(4.7.2). Therefore the third statement is plain in view of Lemma(4.7.2)(2).

1) . Fix w ∈ 𝔖_{r} and define c ∈ ℤ^{r} by the formula(4.7.4)(b). Then putting j := w·i (1 ≤ i ≤ r), we have

c_{i} = −b_{i} + (δ_{j} + r − j + 1) − (r − i + 1)
= −b_{i} + (δ_{j} − j) + i
= −λ_{i} + p − q + 1 + δ_{j} − j
≥ −(p + q − r) + p − q + 1 − r

= 1 − 2q, (4.7.9)

for any i (1 ≤ i ≤ r). In the third inequality we used (4.3.3)(a). Then this inequality(4.7.9) assures the condition for (4.2.4) so that we have

a(c_{i}, 2q) = dim S^{c_{i}}(ℂ^{2q}).

Therefore, combining (4.7.3) with (4.7.8), we get

(4.7.10)  M(𝔮, λ, δ) = ∑_{w∈𝔖_{r}} sgn(w) (∏_{i=1}^{r} a(i − b_{i} + δ_{w·i} − w · i, 2q))
= F (2q, r; y(λ), z(δ)) (Definition(4.2.2)(a)).

2) Assume ∑_{i=1}^{r} δ_{i} = ∑_{i=1}^{k} b_{i}. From Lemma(4.6.4)(2), each term in the right hand side of (4.7.3) is non-zero only if δ = (b_{1}, …, b_{k}, 0, …, 0) and w ∈ 1_{k} × 𝔖_{r−k}. From (4.7.8), the first statement follows. If δ = (b_{1}, …, b_{k}, 0, …, 0), then we have

M(𝔮, λ, δ) = ∑_{w∈1_{k}×𝔖_{r−k}} sgn(w) ( ∏_{i=k+1}^{r} a(i − b_{i} − w · i, 2q))
= d(2q, r − k; y_{k+1}(λ), …, y_{r}(λ)),

which is the desired result. □

We remark that in the proof of the second statement i−b_{i}−w·i ≥ 1−2q (k+1 ≤ i ≤ r) if Q ≥ λ_{k+1}. In this way Remark(4.3.5) is justified.

4.8. proof for complex case

Set v ∈ ℂ^{p}(≃ (𝔱_{1}^{c})^{∗}) by

v := [\frac{p − 1}{2}; −1, r] ⊕ [\frac{−p + 2s − 1}{2}; −1, s] ⊕ [\frac{p − 2r − 1}{2}; −1, p − r − s]

= [p; −1, r] ⊕ [s; −1, s] ⊕ [p − r; −1, p − r − s] + [\frac{−p − 1}{2}; 0, p].

We need first the next Lemma as a slight variant of Proposition(4.6.11).

Lemma 4.8.1. Let b = (b_{1}, …, b_{r}, b_{r+1}, …, b_{r+s}, 0, …, 0) ∈ ℤ^{p}. Assume that

b_{1} ≥ b_{2} ≥ ⋯ ≥ b_{k} ≥ 0, (4.8.2)(a)

(4.8.2)(b)  0 > b_{i} ≥ −p + s + i  (k + 1 ≤ i ≤ r),
(4.8.2)(c)  p − r − s − 1 + i ≥ b_{r+i} > 0  (1 ≤ i ≤ s − l),

0 ≥ b_{r+s+1−l} ≥ ⋯ ≥ b_{r+s}, (4.8.2)(d)

66 TOSHIYUKI KOBAYASHI

for some k (0 ≤ k ≤ r) and l (0 ≤ l ≤ s). Suppose that there are w ∈ 𝔖_{p}, c = (c_{1}, …, c_{r+s}, 0, …, 0) ∈ ℤ^{p}, δ = (δ_{1}, …, δ_{p}) ∈ ℤ^{p} such that

w · (δ + v) − v = b + c, (4.8.3)(a) c_{i} ≥ 0 (1 ≤ i ≤ r), c_{i} ≤ 0 (r + 1 ≤ i ≤ r + s), (4.8.3)(b)

δ_{1} ≥ δ_{2} ≥ ⋯ ≥ δ_{r} ≥ δ_{r+s+1} ≥ δ_{r+s+2} ≥ ⋯ ≥ δ_{p} ≥ δ_{r+1} ≥ δ_{r+2} ≥ ⋯ ≥ δ_{r+s}. (4.8.3)(c)

Set n := − ∑_{i=k+1}^{r} b_{i}, m := ∑_{i=1}^{s−l} b_{r+i} ∈ ℕ. Then the following holds:

1) w ∈ 𝔖_{r}×𝔖_{s}×1_{p−r−s}, δ_{i} = 0 (r+s+1 ≤ i ≤ p), ∑_{i=1}^{r} c_{i} ≥ n, and ∑_{i=1}^{s} c_{r+i} ≤ −m.

2) If

(4.8.4)  ∑_{i=1}^{r} c_{i} = n and ∑_{i=1}^{s} c_{r+i} = −m,

then,

\{ \begin{matrix}w ∈ 1_{k} × 𝔖_{r−k} × 𝔖_{s−l} × 1_{l} × 1_{p−r−s}, \\ δ_{i} = b_{i} (1 ≤ i ≤ k, r + s − l + 1 ≤ i ≤ r + s), \\ δ_{i} = 0 (k + 1 ≤ i ≤ r + s − l, r + s + 1 ≤ i ≤ p) \\ c_{i} = 0 (1 ≤ i ≤ k, r + s − l + 1 ≤ i ≤ r + s), \\ c_{k+i} = −b_{k+i} + i − w′ · i (1 ≤ i ≤ r − k), \\ c_{r+i} = −b_{r+i} + i − w′′ · i (1 ≤ i ≤ s − l). \end{matrix}

Here w′ ∈ 𝔖_{r−k} and w′′ ∈ 𝔖_{s−l} are the obvious restrictions of w.

3) If ∑_{i=1}^{r} c_{i} > n, then ∑_{i=1}^{r} δ_{i} > ∑_{i=1}^{k} b_{i}. If ∑_{i=1}^{s} c_{r+i} < −m, then ∑_{i=1}^{s} δ_{r+i} > ∑_{i=1}^{l} b_{r+s−l+i}. In particular, if (4.8.4) is not satisfied, then

δ ≠ (b_{1}, …, b_{k}) ⊕ [0; 0, r + s − k − l] ⊕ (b_{r+s+1−l}, …, b_{r+s}) ⊕ [0; 0, p − r − s].

Proof. We may replace v by [p; −1, r]⊕[s; −1, s]⊕[p−r; −1, p−r −s] without changing notations. The condition(4.8.2)(b)(c) and (4.8.3)(b) assure

(b + c + v)_{i} ≡ b_{i} + c_{i} + p + 1 − i  ≥ s + 1 (1 ≤ i ≤ r)
(b + c + v)_{r+i} ≡ b_{r+i} + c_{r+i} + s + 1 − i ≤ p − r (1 ≤ i ≤ s)
{(b + c + v)_{r+s+i}; 1 ≤ i ≤ p − r − s} = {s + 1, s + 2, …, p − r}

From (4.8.3)(a)(c), b + c + v consists of distinct entries, which implies w ∈ 𝔖_{r} × 𝔖_{s} × 1_{p−r−s}, δ_{i} = 0 (r + s + 1 ≤ i ≤ p).

Now we apply (4.6.4) to the first r block (respectively the second s block) and then the lemma follows. □

Suppose we are in the setting of §4.4. Recall that the root system of 𝔲 for 𝔱^{c} = 𝔥 is given by (see §2.3)

Δ(𝔲 ∩ 𝔭, 𝔥) := {f_{i} − f_{j} ; 1 ≤ i ≤ r, p + 1 ≤ j ≤ p + q}
∪ {−f_{i} + f_{j} ; r + 1 ≤ i ≤ r + s, p + 1 ≤ j ≤ p + q}.

and that L ∩ K ≃ 𝕋^{r+s} × U(p − r − s) × U(q) acts on 𝔲 ∩ 𝔭 ≃ ℂ^{(r+s)q} by

⨁_{i=1}^{r} (F (𝕋^{r+s}, f_{i}) ⊠ 1 ⊠ F (U(q), (−1, 0, …, 0)))

⊕ ⨁_{i=1}^{s} (F (𝕋^{r+s}, −f_{r+i}) ⊠ 1 ⊠ F (U(q), (1, 0, …, 0))) .

Assume that

\{ \begin{matrix}λ_{i} ∈ ℤ + Q, \\ λ_{1} > λ_{2} > ⋯ > λ_{r} ≥ −Q. \\ Q ≥ λ_{r+1} > λ_{r+2} > ⋯ > λ_{r+s} \end{matrix}

Then μ_{λ} = (b_{1}, …, b_{r+s}, 0, …, 0) (Definition (4.4.1)) satisfies

\{ \begin{matrix}b_{1} ≥ b_{2} ≥ ⋯ ≥ b_{r} ≥ −p + s + r, \\ p − r − s ≥ b_{r+1} ≥ b_{r+2} ≥ ⋯ ≥ b_{r+s}. \end{matrix}

Now the proof of Proposition(4.4.2) is done in the same way as the previous section based on the following lemma (cf. Lemma(4.7.2)). We omit the proof.

Lemma 4.8.5. Retain notation as above, let λ ∈ ℤ^{r+s} + [Q; 0, r + s], δ ∈ ℤ^{p} with the same hypotheses on λ and δ_{1} ≥ δ_{2} ≥ ⋯ ≥ δ_{r} ≥ δ_{r+s+1} ≥ δ_{r+s+2} ≥ ⋯ ≥ δ_{p} ≥ δ_{r+1} ≥ δ_{r+2} ≥ ⋯ ≥ δ_{r+s}. Then

1) If δ_{r+s+1} = ⋯ = δ_{p} = 0, then

(4.8.6) m(𝔮, λ, δ) = ( ∑_{w′,{c_{i}}} sgn(w′) ∏_{i=1}^{r} dim S^{c_{i}}(ℂ^{q}))

× ( ∑_{w′′,{c_{r+i}}} sgn(w′′) ∏_{i=1}^{s} dim S^{−c_{r+i}}(ℂ^{q}))

Here the sums are taken over the sets satisfying

w′ ∈ 𝔖_{r}, c_{i} ∈ ℕ (1 ≤ i ≤ r), (4.8.7)(a) w′ · ((δ_{1}, …, δ_{r}) + ⟨r⟩) − ⟨r⟩ = (b_{1} + c_{1}, …, b_{r} + c_{r}), (4.8.7)(b)

w′′ ∈ 𝔖_{s}, c_{r+i} ∈ −ℕ (1 ≤ i ≤ s), (4.8.8)(a)

w′′ · (δ_{r+1}, …, δ_{r+s}) + ⟨s⟩) − ⟨s⟩ = (b_{r+1} + c_{r+1}, …, b_{r+s} + c_{r+s}), (4.8.8)(b)

respectively.

2) If (δ_{r+s+1}, …, δ_{p}) ≠ (0, …, 0) or ∑_{i=1}^{r} δ_{i} < ∑_{i=1}^{k} b_{i} or ∑_{i=1}^{s} δ_{r+i} > ∑_{i=1}^{l} b_{r+s−l+i}, then m(𝔮, λ, δ) = 0.

4.9. proof for real case

L ∩ K ≃ 𝕋^{r} × SO(p − r) × SO(q) acts on 𝔲 ∩ 𝔭 ≃ ℂ^{rq} by

⨁_{i=1}^{r} (F (𝕋^{r}, f_{i}) ⊠ 1 ⊠ F (SO(q), (−1, 0, …, 0))) .

The proof of Proposition(4.5.2) reduces to the same calculation for the quarternionic case, owing to the following

Lemma 4.9.1. Let a = (a_{1}, …, a_{r}, 0, …, 0) ∈ ℤ^{p′}. Assume that

(4.9.2)  a_{i} ≥ −p + 2r + 1 for any i (1 ≤ i ≤ r).

If

w · (δ + [\frac{p}{2} − 1; −1, p′]) − [\frac{p}{2} − 1; −1, p′] = a

for some w ∈ W (SO(p)), δ = (δ_{1}, …, δ_{p′}) ∈ ℕ^{p′} such that δ_{1} ≥ ⋯ ≥ δ_{p′} ≥ 0 when p is odd, δ_{1} ≥ ⋯ ≥ δ_{p′−1} ≥ |δ_{p′}| when p is even, then we have w ∈ 𝔖_{r} × 1_{p′−r} and δ_{i} = 0 (r + 1 ≤ i ≤ p′).

The proof of this Lemma is similar to that of Lemma(4.6.1) and so we omit it. The condition(4.9.2) guarantees a_{i} + \frac{p}{2} − i ≥ −(\frac{p}{2} − r − 1) (1 ≤ i ≤ r) and is satisfied by putting a_{i} = b_{i} + c_{i} (notation(4.5.1)) when λ_{i} ≥ −Q and c_{i} ∈ ℕ.

5. An alternative proof of the sufficiency for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0

In studying our discrete series for G/H_{2}, we are obliged to deal with the parameters outside the fair range. Usually, we move from good parameters toward bad (e.g. fair or non-fair) ones. But this is a negative way of thinking. In this section, a positive application of bad parameters is offered: making use of direct information for possibly non-fair parameters.

Retain notation in §1. Suppose that ℂ_{λ} is in the fair range. Then (𝔤, K)-modules R_{𝔮}^{j}(ℂ_{λ}) vanish except in the single degree j = S, but the remaining module may also vanish as we saw in the previous section. A recent (unpublished) method (cf. §0) to check the sufficiency for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 due to Matsuki and Oshima is based on a computation of very special ‘small’ K-types case-by-case. They explained to me that it sometimes requires a long calculation and some preparations on finite dimensional representations and that the special case of this section (in particular 𝔽 = ℍ case) is extremely tedious because of a non-triviality of a partition function for 𝔲 ∩ 𝔭 as well as a complicated polarization of a parabolic subalgebra.

In the present case ℂ_{λ} is in the fair range. Nevertheless, treating the parameters outside the fair range at the same time leads us to another simple proof of the sufficient condition for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0.

Here is our idea:

i) nice information may be buried outside the fair range. ii) the modules with small (but not necessarily fair) parameters form a closed universe.

Actually we make use of the difference between the fair range and the condition that μ_{λ} is Δ^{+}(𝔨, 𝔱^{c}) dominant when λ is small.

5.1. theorem: sufficient condition for R_{𝔮}^{S}(ℂ_{λ}) ≠ 0

Let G = Sp(p, q) and 𝔥_{0} be a fundamental Cartan subalgebra of 𝔤_{0}. Take bases of 𝔥 and 𝔥^{∗} as in §2.1. Fix r and s with 0 < r ≤ p, 0 ≤ s ≤ q and put

𝔱 := ∑_{i=1}^{r} ℂH_{i} + ∑_{i=1}^{s} ℂH_{p+i} ⊂ 𝔥, (5.1.1)(a)

Q := p + q − r − s, (5.1.1)(b) (5.1.1)(c) L : the centralizer of 𝔱 in G (≃ 𝕋^{r+s} × Sp(p − r, q − s)).

First of all, let us parametrize θ-stable parabolic subalgebras of 𝔤 with Levi part 𝔩. Let (m_{M}, …, m_{1}), (n_{N}, …, n_{1}) be partitions of r and s respectively, that is, r = ∑_{j=1}^{M} m_{j}, s = ∑_{j=1}^{N} n_{j} with m_{j}, n_{j} ∈ ℕ_{+}. Assume that M = N or M = N + 1. Then a θ-stable parabolic subalgebra 𝔮 = 𝔩 + 𝔲 is defined by giving the nilradical 𝔲 so that

(5.1.2)  \{ \begin{matrix}ρ(𝔲)_{|m_{j}} & = [Q + ∑_{k=1}^{j} m_{k} + ∑_{k=1}^{j−1} n_{k}; −1, m_{j}], \\ ρ(𝔲)_{|n_{j}} & = [Q + ∑_{k=1}^{j} m_{k} + ∑_{k=1}^{j} n_{k}; −1, n_{j}]. \end{matrix}

Here we identify 𝔱^{∗} with ℂ^{r+s} via the basis {H_{i}; 1 ≤ i ≤ r, p + 1 ≤ i ≤ p + s} as usual. We denote by ρ(𝔲)_{|m_{j}}, ρ(𝔲)_{|n_{j}} the obvious restriction of ρ(𝔲) ∈ 𝔥^{∗} to each subspace ℂ^{m_{j}} ≃ ∑_{i=1}^{m_{j}} ℂH_{m_{M} +···+m_{j+1}+i} (⊂ 𝔥), ℂ^{n_{j}} ≃ ∑_{i=1}^{n_{j}} ℂH_{p+n_{N} +···+n_{j+1}+i} (⊂ 𝔥), respectively. See §3.5 for the notation of [k; m, n]. Conversely, any θ-stable parabolic subalgebra 𝔮 with Levi part 𝔩 is obtained in this manner up to conjugation by an element of K and up to exchange of p by q.

In the setting above, 𝔮 ∩ 𝔨 is a parabolic subalgebra of 𝔨 and we have (with similar notations as above):

\{ \begin{matrix}ρ(𝔲 ∩ 𝔨)_{|m_{j}} & = [p − r + ∑_{k=1}^{j} m_{k}; −1, m_{j}], \\ ρ(𝔲 ∩ 𝔨)_{|n_{j}} & = [q − s + ∑_{k=1}^{j} n_{k}; −1, n_{j}]. \end{matrix}

In the present case ρ(𝔲), ρ(𝔲 ∩ 𝔨) ∈ 𝔱^{∗} (⊂ 𝔥^{∗} in the notation (1.1.1)). Set

𝔱_{ℤ}^{∗} : = {λ ∈ 𝔱^{∗}; ⟨λ, α^{∨}⟩ ∈ ℤ for all α ∈ Δ(𝔲, 𝔥) }
= {λ = (λ_{1}, …, λ_{r+s}) ∈ 𝔱^{∗}; λ_{i} ∈ ℤ}
B : = {λ = (λ_{1}, …, λ_{r+s}) ∈ 𝔱^{∗}; −Q ≤ λ_{i} ≤ Q}
F : = {λ ∈ 𝔱^{∗}; ⟨λ, α⟩ > 0 for all α ∈ Δ(𝔲, 𝔥) }
B_{ℤ} : = B ∩ 𝔱_{ℤ}^{∗},  F_{ℤ} := F ∩ 𝔱_{ℤ}^{∗}.

Note that a weight λ ∈ 𝔱^{∗} lifts to a metapletic (𝔩, (L ∩ K)^{∼})-module iff λ ∈ 𝔱_{ℤ}^{∗}. Definition 5.1.3. Suppose we are given 𝔮 ⊂ 𝔤 as above and λ ∈ 𝔱^{∗}. Set

r′ := #{i; 1 ≤ i ≤ r, λ_{i} ≤ Q},
s′ := #{i; p + 1 ≤ i ≤ p + s, λ_{i} ≤ Q},
p′ := p − r + r′, q′ := q − s + s′,
𝔱′′_{0} := ∑_{i=1}^{r−r′} ℝ√\overline{−1}H_{i} + ∑_{i=1}^{s−s′} ℝ√\overline{−1}H_{p+i},
𝔤′_{0} := the semisimple part of the centralizer of 𝔱′′_{0} in 𝔤_{0}
≃ 𝔰𝔭(p′, q′) ⊂ 𝔤_{0},
𝔮′ := 𝔤′ ∩ 𝔮 ⊃ 𝔤.

In extremal cases, (𝔤′_{0}, 𝔮′) = (𝔤_{0}, 𝔮) if λ ∈ B, and (𝔤′_{0}, 𝔮′) = (𝔩_{0}, 𝔩) if λ ∈ F is sufficiently ‘regular’. We shall use similar notations for (𝔤′_{0}, 𝔮′) by adding′. For example, 𝔱′ = ∑_{i=1}^{r′} ℂH_{r−r′+i} + ∑_{i=1}^{s′} ℂH_{p+s−s′+i}, Q′ = (p − r + r′) + (q − s + s′) − r′ − s′ = Q, and etc.

Now we are ready to state the main result in this section.

Theorem 5.1.4. Let G = Sp(p, q), 𝔮 be a θ-stable parabolic subalgebra of 𝔤 given by (5.1.2), and λ ∈ F_{ℤ}. Let (𝔤′_{0}, 𝔮′) be associated to (𝔤_{0}, 𝔮, λ) by Definition(5.1.3). Then R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 if −ρ(𝔲′) + 2ρ(𝔲′ ∩ 𝔨′) ∈ B′.

We emphasize again that the conclusion was essentially found first by Matsuki and Oshima. In fact, if R_{𝔮}^{S}(ℂ_{λ}) is realized as discrete series for a semisimple symmetric space (in this case, the parameter λ is so degenerate as in (2.7.4)(a) and all of m_{j}, n_{j}’s are even), the above sufficient condition in Theorem(5.1.4) coincides with the necessary one given in [21] after some calculations. We remark that our formulation based on Definition(5.1.3) helps us to understand the somewhat complicated and different description given in [21] in this case.

If ℂ_{λ} is in the good range with respect to 𝔮, then 𝔮′ = 𝔩 and −ρ(𝔲′) + 2ρ(𝔲′ ∩ 𝔨′) = 0 ∈ B′, whence Theorem(5.1.4) implies a well known result R_{𝔮}^{S}(ℂ_{λ}) ≠ 0. The opposite extremal case is essential. Recall that (𝔤′_{0}, 𝔮′) = (𝔤_{0}, 𝔮) if λ ∈ B in Definition(5.1.3). Then the following is a special case of Theorem(5.1.4) by assuming λ ∈ B.

Theorem 5.1.4′ (special case). With notation as above, if −ρ(𝔲) + 2ρ(𝔲 ∩ 𝔨) ∈ B, then R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 for all λ ∈ B ∩ F_{ℤ}.

Since the essence of our proof for Theorem(5.1.4) lies in this special case, we shall prove only this case. Indeed, we shall iterate translations among small singular parameters, where the components (strictly) larger than Q are always stable, so that the proof for Theorem(5.1.4) parallels exactly to that for Theorem(5.1.4)′.

Remark 5.1.5. We can also reduce Theorem(5.1.4) to Theorem(5.1.4)′ directly. In fact, according to the direct decomposition 𝔱 = 𝔱′′ + 𝔱′, we write λ = λ′′ + λ′, and ρ(𝔲) = ρ′′ + ρ′. If λ ∈ F, then ρ′ = ρ(𝔲′) in this quarternionic case. Assume that

⟨λ, α_{|𝔱′′}⟩ > 0  for any α ∈ Δ(𝔲) with α_{|𝔱′′} ≠ 0,
λ_{i} ≥ −Q for any i (1 ≤ i ≤ r + s).

(This condition is clearly weaker than λ ∈ F.) Then we have

(R_{𝔮}^{𝔤})^{j+S−S′} (ℂ_{λ}) ≠ 0 ⟺ (R_{𝔮}^{𝔤})^{j+S−S′} (ℂ_{λ+ξρ′′}) ≠ 0 (∀ξ ∈ ℕ)
⟺ (R_{𝔮′}^{𝔤′})^{j} (ℂ_{λ′}) ≠ 0

The first equivalence is derived from Lemma(3.4.1)(2). Indeed, we can easily find a sequence λ^{(0)} = λ, λ^{(1)}, …, λ^{(n)} = λ + ξρ′′ so that

A (λ^{(i−1)} ▷ λ^{(i)}) = {λ^{(i)}}, A (λ^{(i)} ▷ λ^{(i−1)}) = {λ^{(i−1)}} for i = 1, 2, …, n.

The second equivalence is from Lemma(3.2.1)(2) and Fact(1.4.1)(2). Hence we have, in particular, (R_{𝔮}^{𝔤})^{S} (ℂ_{λ}) ≠ 0 ⇔ (R_{𝔮′}^{𝔤′})^{S′} (ℂ_{λ′}) ≠ 0. Since Q′ = Q, we have λ′ ∈ B and we are done.

Remark 5.1.6. Let λ̃ ≡ λ̃(𝔮) := −ρ(𝔲) + 2ρ(𝔲 ∩ 𝔨). We write the coordinates of λ̃ as λ̃_{i} (1 ≤ i ≤ r + s) by restricting it to 𝔱. Then elementary arithmetic shows the equivalence of the following four conditions:

λ̃ ∈ B. (5.1.7)(a) Q ≥ λ̃_{i} > −Q for all i (1 ≤ i ≤ r + s). (5.1.7)(b) Q ≥ λ̃_{i} for all i (1 ≤ i ≤ r + s). (5.1.7)(c)

\{ \begin{matrix}∑_{k=1}^{j} m_{k} ≤ 2(q − s) + ∑_{k=1}^{j−1} n_{k} (1 ≤ j ≤ M), \\ ∑_{k=1}^{j} n_{k} ≤ 2(p − r) + ∑_{k=1}^{j} m_{k} (1 ≤ j ≤ N). \end{matrix} (5.1.7)(d)

The implication (5.1.7)(c) ⇒ (5.1.7)(b) looks strange, but it turns out that the estimate on m_{j} and n_{j} blocks (see after (5.1.2) for notation) plays a ‘complementary’ role to each other with reverse signs.

5.2. key lemmas

Definition 5.2.1. For each subset S of B_{ℤ}, we define S̃ ⊂ B_{ℤ} to be the smallest set such that

S̃ ⊃ S (5.2.2)(a) λ ∈ S̃, λ′ ∈ 𝔱_{ℤ}^{∗} and A (λ ▷ λ′) ⊂ S̃ ∪ {λ′} ⇒ λ′ ∈ S̃. (5.2.2)(b)

By definition A (λ ▷ λ′) is a subset of 𝔥^{∗}. However, as is easy to see, if λ, λ′ ∈ B_{ℤ}(⊂ 𝔱^{∗}), then A (λ ▷ λ′) ⊂ B_{ℤ}, where we regard 𝔱^{∗} as a subspace of 𝔥^{∗} as usual. Finally, set

N ≡ N (𝔮) := {λ ∈ B_{ℤ}; R_{𝔮}^{i}(ℂ_{λ}) = 0 for all i}.

Here are two key lemmas.

Lemma 5.2.3. With notation as above,

1) ˜ S̃ = S̃ for any subset S of B_{ℤ}.

2) Ñ = N.

Lemma 5.2.4. ˜{λ} = B_{ℤ} if λ ∈ B ∩ F_{ℤ}.

Proof of Lemma(5.2.3). Part (1) is clear from definition and Part (2) is a direct consequence of the relation between translation functor and cohomologically parabolic induction in Lemma(3.4.1)(1). □

Admitting Lemma(5.2.4) for a while, we shall complete the proof of Theorem(5.1.4)′.

Proof of Theorem(5.1.4)′. If λ̃ ≡ −ρ(𝔲) + 2ρ(𝔲 ∩ 𝔨) ∈ B, then λ̃ ∉ N because

∑_{i}(−1)^{i} dim Hom_{K}(1, R_{𝔮}^{S−i}(ℂλ̃)) = 1,

as is easy to see by the generalized Blattner formula. Thus N ⫋ B_{ℤ}, which implies B ∩ F_{ℤ} ⊂ F_{ℤ} \ N because of Lemma(5.2.3) and Lemma(5.2.4). On the other hand, if λ ∈ F_{ℤ} then ℂ_{λ} is in the fair range and R_{𝔮}^{i}(ℂ_{λ}) = 0 for all i ≠ S. Therefore R_{𝔮}^{S}(ℂ_{λ}) ≠ 0 for any λ ∈ B ∩ F_{ℤ}. □

5.3. proof of the combinatorial part

We are now left with the proof of Lemma(5.2.4), whose statement depends only on r +s and Q, and dose not depend on the particular partitions of r and s. Set n := r +s. Let 𝔱_{ℝ}^{∗} := {λ = (λ_{1}, …, λ_{n}) ∈ 𝔱^{∗}; λ_{i} ∈ ℝ}. Each connected component of

{λ = (λ_{1}, …, λ_{n}) ∈ 𝔱_{ℝ}^{∗}; λ_{i} ≠ λ_{j} (i ≠ j), λ_{i} ≠ 0} ⊂ 𝔱_{ℝ}^{∗}

is a Weyl chamber for W (C_{n}).

Claim(5.3.1). Let C ⊂ 𝔱_{ℝ}^{∗} be a Weyl chamber. If λ ∈ B_{ℤ} ∩ C, then ˜\{λ\} ⊃ B_{ℤ} ∩ \overline{C}.

Proof. Fix λ′ ∈ B_{ℤ} ∩ C. Then we can easily find a sequence λ = λ^{(0)}, λ^{(1)}, …, λ^{(k)} = λ′ such that A (λ^{(i−1)} ▷ λ^{(i)}) = {λ^{(i)}} (1 ≤ i ≤ k). Now the property(5.2.2) shows λ′ ∈ ˜{λ}. □

Claim(5.3.2). Let λ = (n, n − 1, …, 1). Then ˜{λ} contains λ′ in each case below. a) λ′ = (n, n − 1, …, j, j + 1, …, 1) (1 ≤ j ≤ n − 1)

b) λ′ = (n, n − 1, …, 2, −1)

Proof. A (λ ▷ λ′′) = {λ′′} and A (λ′′ ▷ λ′) = {λ, λ′} if we choose λ′′ as follows: a) λ′′ := (n, n − 1, …, j, j, …, 1), b) λ′′ = (n, n − 1, …, 2, 0). Now the claim follows from the property(5.2.2). □ Claim(5.3.1) and Claim(5.3.2) lead us to

(5.3.3) ˜{λ} = B_{ℤ}

for each fixed λ ∈ B_{ℤ} ∩ C, and for each Weyl chamber C. Now Lemma(5.2.4) is proved.

6. Proof of irreducibility results

6.1. irreducibility in the fair range

In this section, we prove a result about irreducibility of certain series of cohomologically induced representations. Suppose we are in the setting of §1.1-3. One might expect that

(6.1.1) If a metapletic representation ℂ_{λ} of (𝔩, (L ∩ K)^{∼}) is in the weakly fair range, then the derived functor module R_{𝔮}^{S}(ℂ_{λ}) is irreducible as a (𝔤, K)-module or zero.

This is known to be false in general as alluded to in Fact(1.4.2)(2). However, it still holds in some interesting cases:

0) ℂ_{λ} is in the weakly good range (see Fact(6.2.4)(c)).

i) 𝔤 is of type A (see Fact(6.2.4)(b)). ii) (𝔤_{0}, 𝔮) arises in a natural description of discrete series for a semisimple symmetric

space ([33], see Fact(1.5.2) for notation).

In this section we shall prove a positive answer for (6.1.1) when ℂ_{λ} is in the fair range and when iii) 𝔤 is of type C and [𝔩, 𝔩] is also of type C.

To prove this, we need a slight refinement (Theorem(6.3.1)) of [33] Theorem 5.11, which assures a nice behavior under translation in an extremely special direction when applied to some Harish-Chandra bimodules: namely, twisted differential operators coming from U(𝔤) when 𝔤 = 𝔰𝔭(n, ℂ).

Our approach parallels exactly Vogan’s proof there. But we isolate the necessary steps explicitly since it is complicated in detail (ex. (6.7.7)).

6.2. twisted differential operators

We introduce some notation used throughout this section §6. For materials of this subsection, we refer to [2], [3], [6], [30] and [33].

Fix a connected complex reductive (algebraic) group G_{ℂ} with Lie algebra 𝔤. We identify Harish-Chandra bimodules with actual Harish-Chandra modules for G_{ℂ} (regarded as a real group) by a Chevalley anti-automorphism. Fix a Cartan subalgebra 𝔥 and a parabolic subalgebra 𝔮 of 𝔤 with Levi decomposition 𝔮 = 𝔩 + 𝔲 so that 𝔩 contains 𝔥. Fix a weakly fair weight γ ∈ 𝔥^{∗}. Write R_{γ}(𝔩 : 𝔩) for the one-dimensional Harish-Chandra bimodule for L_{ℂ} with ℨ(𝔩 ⊗ ℂ)-infinitesimal character (γ, γ) ∈ 𝔥^{∗} ⊕ 𝔥^{∗}. That is, its bimodule structure is given by,

X · y = y · X = γ(X_{1})y

for y ∈ R_{γ}(𝔩 : 𝔩), X = X_{1} + X_{2} ∈ 𝔱 ⊕ [𝔩, 𝔩] = 𝔩 (notation (1.1.2)). Define

(6.2.1)  R_{γ}(𝔩 : 𝔤) = Ind(Q_{ℂ} ↑ G_{ℂ})(R_{γ}(𝔩 : 𝔩)),

a Harish-Chandra bimodule for G_{ℂ}, which has a unique G_{ℂ}-fixed vector. Here we denote by Ind(Q_{ℂ} ↑ G_{ℂ}) normalized parabolic induction for Harish-Chandra bimodules.

Note that R_{γ}(𝔩 : 𝔤) may be identified with a G_{ℂ}-equivariant twisted ring of differential operators on the generalized flag variety G_{ℂ}/Q_{ℂ}. Then R_{γ}(𝔩 : 𝔤) has a natural algebra structure with

(6.2.2)  d : U(𝔤) → R_{γ}(𝔩 : 𝔤)

an algebra homomorphism that is nonzero on the unique G_{ℂ}-fixed vector. Define

(6.2.3)(a)  I_{γ}(𝔩 : 𝔤) : = Ker(d : U(𝔤) → R_{γ}(𝔩 : 𝔤))
(6.2.3)(b)  A_{γ}(𝔩 : 𝔤) : = Image(d : U(𝔤) → R_{γ}(𝔩 : 𝔤))
≃ U(𝔤)/I_{γ}(𝔩 : 𝔤).

If 𝔮 is a Borel subalgebra, we write simply I(γ) for I_{γ}(𝔥 : 𝔤), which is nothing but the ideal in U(𝔤) generated by the corresponding maximal ideal in ℨ(𝔤). From this view point, we put I(w · γ) = I(γ) (w ∈ W (𝔤, 𝔥)) so that I(γ) is defined for all γ ∈ 𝔥^{∗}. Here is the irreducibility result due to J.Bernstein that we need (see [30] Proposition 16.8,

[2] Proposition III.2.2.2 and I.5.6, see also [33] Proposition 5.7).

Fact 6.2.4. Retain notations as above.

a) Suppose that W is a one dimensional metapletic (𝔩, (L ∩ K)^{∼}) representation having ℨ(𝔩)-infinitesimal character γ in the weakly fair range. The algebra R_{γ}(𝔩 : 𝔤) acts on R_{𝔮}^{S}(W), and the resulting (R_{γ}(𝔩 : 𝔤), K)-module is irreducible or zero.

b) If the moment map

π: T^{∗}(G_{ℂ}/Q_{ℂ}) ≃ G_{ℂ} ×_{Q_{ℂ}} 𝔮^{⊥} → G · (𝔮^{⊥}) ⊂ 𝔤^{∗}

is birational and has a normal image, then A_{γ}(𝔩 : 𝔤) = R_{γ}(𝔩 : 𝔤).

c) If γ is in the weakly good range, then A_{γ}(𝔩 : 𝔤) = R_{γ}(𝔩 : 𝔤).

If A_{γ}(𝔩 : 𝔤) = R_{γ}(𝔩 : 𝔤), then Part (a) implies the irreducibility (or vanishing) of R_{𝔮}^{S}(ℂ_{λ}) as a (𝔤, K)-module. It is known that the assumption in Fact(6.2.4)(b) is satisfied if 𝔤 = 𝔤𝔩(n, ℂ) and 𝔮 is any parabolic (see [17]) or if 𝔤 = 𝔰𝔬(n, ℂ) and 𝔮 is of the form in §2.5 (see [12] for birationality and [18] for normality, see also [2] III.3.2), whence Part (5) of Theorem 2, 3. But unfortunately, the birationality fails (although the image of π is normal) when 𝔤 = 𝔰𝔭(n, ℂ) and 𝔮 is of the form in §2.1.

This should be compared with the fact that the normality frequently fails (although π is birational) in the setting of Fact(1.5.2) (discrete series for semisimple symmetric spaces) if 𝔤 = 𝔰𝔭(n, ℂ). Thus, throughout the rest of this section we shall restrict ourselves to the case where 𝔤 = 𝔰𝔭(n, ℂ) and 𝔮 is a general parabolic subalgebra and to the case of singular ℨ(𝔤)-infinitesimal characters.

6.3. theorem

Theorem 6.3.1 (cf. [33] Theorem 5.11). Suppose 𝔤 is 𝔰𝔭(n, ℂ). Let 𝔥 be the standard Cartan subalgebra identified with ℂ^{n} as usual. Fix non-negative integers N, r such that 1 ≤ N ≤ n − r. Write l = n − N − r. Let 𝔮 be the standard parabolic subalgebra with Levi factor

𝔩 = 𝔤𝔩(N, ℂ) ⊕ 𝔤𝔩(1, ℂ)^{r} ⊕ 𝔰𝔭(l, ℂ).

Suppose γ is the ℨ(𝔩)-infinitesimal character of a one-dimensional representation of 𝔩:

(6.3.2)(a)  γ = (λ_{0} + N − 1, λ_{0} + N − 2, …, λ_{0}, λ_{1}, …, λ_{r}, l, l − 1, …, 1).

Assume that

(6.3.2)(b)  λ_{j} ∈ ℤ for any j, l ≥ λ_{0} ≥ ⋯ ≥ λ_{r} ≥ 0 and λ_{0} > 0.

Set ξ := (l − λ_{0})^{N} ⊕ 0^{n−N} = (l − λ_{0}, …, l − λ_{0}, 0, …, 0). Suppose M is an irreducible 𝔤-module annihilated by I_{γ+ξ}(𝔩 : 𝔤). Then ψ_{γ+ξ}^{γ}M is an irreducible 𝔤-module or zero.

Remark 6.3.3. Theorem 5.11 in [33] is the case where N = 2. But the above theorem with N = 1 may also apply to the irreducibility results in [33] owing to the Borel-Weil-Bott theorem for U(2). In order to prove Part(5) of Theorem 1 (§2.2), we need the case where N = 1.

6.4. irreducibility result Corollary 6.4.1. Let G be a real form of Sp(n, ℂ), K a maximal compact subgroup of G and θ the corresponding Cartan involution of G. Let 𝔭 = 𝔪 + 𝔫 be a θ-stable parabolic subalgebra (see §1.1) of 𝔤 with a Levi factor

𝔪 ≃ 𝔤𝔩(n_{1}, ℂ) ⊕ 𝔤𝔩(n_{2}, ℂ) ⊕ ⋯ ⊕ 𝔤𝔩(n_{k}, ℂ) ⊕ 𝔰𝔭(l, ℂ),

for some 0 ≤ l ≤ n and n_{j} ∈ ℕ_{+} such that ∑_{j=1}^{k} n_{j} = n − l. Fix a Cartan subalgebra 𝔥 of 𝔪. Let W be a metapletic (𝔪, (M ∩K)^{∼})-character with ℨ(𝔪)-infinitesimal character γ = (γ_{1}, …, γ_{n−l}, l, l − 1, …, 1) ∈ 𝔥^{∗}. Assume that γ_{j} ∈ ℤ and

γ_{1} ≥ γ_{2} ≥ ⋯ ≥ γ_{n−l} > 0.

Then R_{𝔭}^{s}(W) is an irreducible (𝔤, K)-module or zero.

This includes the irreducibility assertions in Part (5) of Theorem 1. In fact, it is obtained by applying the above corollary to the case when n_{1} = ⋯ = n_{k} = 1.

6.5. Vogan’s idea on the translation principle for A_{γ}(𝔩 : 𝔤)

We shall review shortly a technique due to Vogan [33] which shows irreducibility of certain series of derived functor modules. The main tools for reduction to good parameters are a combination of induction by stages and translation principle. Translation is used in a pair of translation functors for Harish-Chandra modules and Harish-Chandra bimodules.

Let R be a complex algebra, endowed with Harish-Chandra bimodule structure for G_{ℂ} through an algebra homomorphism

(6.5.1)  d : U(𝔤) → R.

Let M be an R-module which is ℨ(𝔤)-finite via (6.5.1). Let F be a finite dimensional representation of 𝔤 with extremal weight −ξ on which the adjoint action of 𝔤 exponentiates to G_{ℂ}. A formal argument (the Jacobson density theorem) (see [33] Corollary 3.9) shows that

(6.5.2) If M is an irreducible R-module, then ψ_{γ+ξ}^{γ}M is an irreducible (ψ_{(γ+ξ,γ+ξ)}^{(γ,γ)}R)-module or zero.

Retain notations in §6.2. Let γ := λ + ρ_{𝔩} ∈ 𝔥^{∗} be in the weakly fair range. Let us apply (6.5.2) to d : U(𝔤) ↠ A_{λ+ρ_{𝔩}}(𝔩 : 𝔤). We expect a reasonable behavior under translation (see Theorem(6.3.1)):

(6.5.3)  ψ_{(λ+ρ_{𝔩}+ξ,λ+ρ_{𝔩}+ξ)}^{(λ+ρ_{𝔩},λ+ρ_{𝔩})}A_{λ+ρ_{𝔩}+ξ}(𝔩 : 𝔤) = A_{λ+ρ_{𝔩}}(𝔩 : 𝔤).

But, in the (weakly) fair range, the behavior of A_{λ+ρ_{𝔩}}(𝔩 : 𝔤) under translation is not as good as those of R_{𝔮}^{S}(ℂ_{λ}) and R_{λ+ρ_{𝔩}}(𝔩 : 𝔤). Vogan pointed out that (6.5.3) is guaranteed by the existence of Harish-Chandra bimodules C_{λ} and C_{λ+ξ} satisfying (6.5.4)(a) ψ_{(λ+ρ_{𝔩}+ξ,λ+ρ_{𝔩}+ξ)}^{(λ+ρ_{𝔩},λ+ρ_{𝔩})}C_{λ+ξ} = C_{λ}. (6.5.4)(b) Each of C_{λ+ξ} and C_{λ} is generated by its unique G_{ℂ}-fixed vector as a Harish-Chandra bimodule. (6.5.4)(c) There are Harish-Chandra bimodule maps

C_{λ+ξ} → R_{λ+ρ_{𝔩}+ξ}(𝔩 : 𝔤)
C_{λ} → R_{λ+ρ_{𝔩}}(𝔩 : 𝔤)

that are non-zero on the G_{ℂ}-fixed vectors.

With a special choice of ξ and C_{λ} in §6.7, we shall check these conditions in §6.8-10.

6.6. notations about GL(n, ℂ) and Sp(n, ℂ)

Since we shall treat only complex reductive Lie groups in §6.6-10, we write GL(n) for GL(n, ℂ) and etc, for simplicity.

First let G_{ℂ} = GL(n). Take a Cartan subalgebra 𝔥 of 𝔤 so that 𝔥 consists of n × n diagonal matrices with complex entries. Diagonal coordinates induce 𝔥^{∗} ≃ ℂ^{n} as usual. Let B be a Borel subgroup of G_{ℂ} consisting of upper triangular matrices. Let γ = (γ_{1}, …, γ_{n}) ∈ 𝔥^{∗}. Define a Harish-Chandra bimodule for G_{ℂ} = GL(n) by,

[γ] ≡ [γ_{1}, …, γ_{n}] = unique irreducible subquotient of R_{γ}(𝔥 : 𝔤)

containing the G_{ℂ}-fixed vectors.

We remark that [γ] = J(2γ) in the notation of [30] §11 when regarded as an irreducible spherical representation of G_{ℂ} (as a real group).

Assume that the sequence (Re γ_{j}) is decreasing. With the above parametrization, we have (see [30] Theorem 11.5, Lemma 11.11): (6.6.1)(a) [γ] is the unique irreducible quotient of R_{γ}(𝔥 : 𝔤). (6.6.1)(b) [γ] is finite dimensional iff γ_{j} − γ_{j+1} ∈ ℕ_{+} for any j. (6.6.1)(c) [γ] is one dimensional iff γ = [c; −1, n] (§3.5) for some c ∈ ℂ.

Next we look upon GL(n) as a reductive subgroup in Sp(n). Since GL(n) is of maximal rank in Sp(n), 𝔥 is also a Cartan subalgebra in 𝔰𝔭(n). Let B̃(⊃ B) be the Borel subgroup of Sp(n), making ⟨n⟩ = (n, n − 1, …, 1) dominant for Δ(𝔟̃, 𝔥).

Definition 6.6.2. An (ordered) partition of n is a sequence

π = (p_{1}, …, p_{s})

of non-negative integers, such that ∑_{j=1}^{s} p_{j} = n. Denote by P (π) the parabolic subgroup of GL(n) containing B and having a Levi factor

(6.6.3)  L(π) =^{def} GL(p_{1}) × GL(p_{2}) × ⋯ × GL(p_{s}).

Similarly we denote by P̃(π) the parabolic subgroup of Sp(n) containing B̃ with the same Levi factor L(π).

UNITARY REPRESENTATIONS IN L^{2}(U(p, q; 𝔽)/U(p − m, q; 𝔽)) 85

6.7. definition of C_{λ}

Suppose that we are given r, l ∈ ℕ_{+}, λ_{j} ∈ ℤ (0 ≤ j ≤ r) such that, l ≥ λ_{0} ≥ λ_{1} ≥ ⋯ ≥ λ_{r} ≥ 0, (6.7.1)(a) λ_{0} > 0. (6.7.1)(b) Let {Λ_{j}}_{1≤j≤s+1} be the totality of distinct values in {λ_{j} ; 0 ≤ j ≤ r} ∪ {0} such that

(6.7.2)(a)  Λ_{1} > ⋯ > Λ_{s} > Λ_{s+1} = 0.

We shall ignore Λ_{s+1} unless λ_{r} = 0. The next definition serves to simplify the notation somewhat:

(6.7.2)(b)  Λ_{0} := l + 1.

Set

(6.7.3) m(j) := ♯{k; λ_{k} = Λ_{j}, 1 ≤ k ≤ r} ∈ ℕ (1 ≤ j ≤ s + 1).

Clearly ∑_{j=1}^{s+1} m(j) = r ≥ s − 1 (or ≥ s when λ_{r} = 0). Let

λ^{(0)} = [λ_{0} + N − 1; −1, N] := (λ_{0} + N − 1, λ_{0} + N − 2, …, λ_{0}).

Define

(6.7.4)(a)  ν^{(j)} : = [Λ_{j−1} − 1; −1, Λ_{j−1} − Λ_{j}]
= (Λ_{j−1} − 1, Λ_{j−1} − 2, …, Λ_{j}) (1 ≤ j ≤ s).

If Λ_{s} ≥ 2, we also define

(6.7.4)(b)  ν^{(s+1)} := [Λ_{s} − 1; −1, Λ_{s} − 1] ≡ ⟨Λ_{s} − 1⟩
= (Λ_{s} − 1, …, 2, 1).

With this notation, if {λ_{1}, …, λ_{r}} ∋ 1 (equivalently, if Λ_{s} = 1), then (ν^{(1)}, ν^{(2)}, …, ν^{(s)}) is a grouping of (l, l − 1, …, 2, 1). If {λ_{1}, …, λ_{r}} ∌ 1 (equivalently, if Λ_{s} ≥ 2), then (ν^{(1)}, ν^{(2)}, …, ν^{(s)}, ν^{(s+1)}) is a grouping of (l, l − 1, …, 2, 1). If {λ_{1}, …, λ_{r}} ∋ 0 (equivalently, if λ_{r} = Λ_{s+1} = 0 and m(s + 1) > 0), then (Λ_{1}^{m(1)}, Λ_{2}^{m(2)}, …, Λ_{s}^{m(s)}, 0^{m(s+1)}) is a grouping of (λ_{1}, λ_{2}, …, λ_{r}). If {λ_{1}, …, λ_{r}} ∌ 0 (equivalently, if λ_{r} ≠ 0), then (Λ_{1}^{m(1)}, Λ_{2}^{m(2)}, …, Λ_{s}^{m(s)}) is a grouping of (λ_{1}, λ_{2}, …, λ_{r}).

Put

(6.7.5)  π := (N, 1^{m(1)}, Λ_{0} − Λ_{1}, …, 1^{m(s)}, Λ_{s−1} − Λ_{s}, Λ_{s} − 1, 1^{m(s+1)}),

an ordered partition of n = N + r + l. Then according to notation (6.6.3), a reductive subgroup L(π) of Sp(n) is given by,

GL(N) × ∏^{s} ; j=1 (GL(1)^{m(j)} × GL(Λ_{j−1} − Λ_{j})) × GL(Λ_{s} − 1) × GL(1)^{m(s+1)}.

More precisely, if {λ_{1}, …, λ_{r}} ∋ 1 (equivalently, if Λ_{s} = 1), then the above GL(Λ_{s} − 1) and the below [ν^{(s+1)}] should be omitted. Similarly, if λ_{r} > 0 (equivalently, if m(s+1) = 0), then the above GL(1)^{m(s+1)} and the below [Λ_{s+1}]^{m(s+1)} should be omitted. We define a Harish-Chandra bimodule for L(π) by

(6.7.6)  σ_{λ} =^{def} [λ^{(0)}] ⊗ ⨂_{j=1}^{s} ([Λ_{j}]^{m(j)} ⊗ [ν^{(j)}]) ⊗ [ν^{(s+1)}] ⊗ [Λ_{s+1}]^{m(s+1)},

and a Harish-Chandra bimodule for Sp(n) by

(6.7.7)  C_{λ} =^{def} Ind(P̃(π) ↑ Sp(n))(σ_{λ}).

Finally, take

(6.7.8)  ξ := (l − λ_{0})^{N} ⊕ 0^{n−N}.

We define σ_{λ+ξ} and C_{λ+ξ} similarly by changing only λ^{(0)}. We shall prove Theorem(6.3.1) by checking (6.5.4)(a)-(c) for these C_{λ}, C_{λ+ξ} in the subsequent three subsections.

The somewhat complicated notation of Λ_{i}, m(i), ν^{(j)} and π will be used throughout §6. The following figure would be helpful useful to memorize the definition.

•  ^{•}^{•}  ^{•}^{•}  ^{•}^{•}  ^{•}  ^{•}  ^{•}^{•}  ^{•}^{•}  ^{•} •  •^{Λ_{1}^{m(1)}}  ^{ν^{(1)}}  ^{Λ_{2}^{m(2)}}  ^{ν^{(2)}}  ^{Λ_{s}^{m(s)}}  ^{ν^{(s)}}  ^{ν^{(s+1)}} _{Λ_{s+1}_{m(s+1)}}^{...}

Figure 6.7.9

6.8. verification of (6.5.4)(a)

The proof of (6.5.4)(a) depends on the following

Lemma 6.8.1. In the setting of (6.7.1)-(6.7.5),(6.7.8), set

γ := (λ^{(0)}, Λ_{1}^{m(1)}, ν^{(1)}, Λ_{2}^{m(2)}, ν^{(2)}, …, Λ_{s}^{m(s)}, ν^{(s)}, ν^{(s+1)}, 0^{m(s+1)}).

88 TOSHIYUKI KOBAYASHI

Let μ ∈ 𝔥^{∗} ≃ ℂ^{n} be a weight in F (Sp(n), −ξ). Assume that μ is dominant for Δ^{+}(𝔩(π), 𝔥),

γ + ξ + μ = w · γ for some w ∈ W (𝔰𝔭(n)).

Then

μ = −ξ.

Write ρ_{𝔩(π)} = ρ(Δ^{+}(𝔩(π))). We note that the above conclusion is equivalent to

A(Δ^{+}(𝔩(π)); γ + ξ − ρ_{𝔩(π)} ▷ γ − ρ_{𝔩(π)}) = {γ − ρ_{𝔩(π)}}

in terms of our definition in §3.3 for a pair 𝔩(π) ⊂ 𝔰𝔭(n).

Assuming this lemma for a while, we verify (6.5.4)(a). Proof of (6.5.4)(a). Set γ = λ + ρ_{𝔩} as in §6.5.

ψ_{(γ+ξ,γ+ξ)}^{(γ,γ)}C_{λ+ξ}

=P_{(γ,γ)} (Ind(P̃(π) ↑ Sp(n))(σ_{λ+ξ}) ⊗ End(F (𝔤, −ξ)))
≃P_{(γ,γ)} (Ind(P̃(π) ↑ Sp(n))(σ_{λ+ξ} ⊗ End(F (𝔤, −ξ)))) .

As the above Lemma(6.8.1) assures that A ((γ + ξ − ρ_{𝔩(π)}, γ + ξ − ρ_{𝔩(π)}) ▷ (γ − ρ_{𝔩(π)}, γ − ρ_{𝔩(π)}))

={(γ − ρ_{𝔩(π)}, γ − ρ_{𝔩(π)})}

for 𝔩(π) ⊗_{ℝ} ℂ ⊂ 𝔰𝔭(n, ℂ) ⊗_{ℝ} ℂ, the last Harish-Chandra bimodule is isomorphic to Ind(P̃(π) ↑ Sp(n))(σ_{λ}) = C_{λ} (cf. Lemma(3.4.1)(2)). Hence (6.5.4)(a). □

Lemma(6.8.1) is deduced from the following two elementary claims.

Claim 6.8.2. Let γ = (γ_{1}, …, γ_{n}), ξ = L^{N} ⊕ 0^{n−N} ∈ ℤ^{n} satisfy γ_{k} ≥ 0 for any k and L ≥ 0. Let μ ∈ ℤ^{n} be a weight of F (Sp(n), −ξ). Assume that

(6.8.3)  γ + ξ + μ = w · γ

for some w, an element of the Weyl group W (𝔰𝔭(n)) ≃ 𝔖_{n} ⋉ ℤ_{2}^{n}. Then the following holds.

−L ≤ μ_{k} ≤ 0 for any k (1 ≤ k ≤ n). (6.8.4)(a)

∑_{k=1}^{n} μ_{k} = −NL. (6.8.4)(b)

(6.8.4)(c) (6.8.3) holds for some w ∈ 𝔖_{n}. (6.8.4)(d) If N + 1 ≤ k ≤ n and γ_{k} ≤ min_{1≤j≤N} γ_{j}, then μ_{k} = 0.

Proof. As every weight of F (Sp(n), −ξ) lies in the convex hull of the extremal weights, we have

(6.8.5)  −L min(N, #I) ≤ ∑_{j∈I} μ_{j} ≤ L min(N, #I),

for any subset I ⊂ {1, …, n}. In particular, applying the above inequality to I = {1, …, n}, we have

∑_{k=1}^{n}(γ + ξ + μ)_{k} = ∑_{k=1}^{n} γ_{k} + NL + ∑_{k=1}^{n} μ_{k} ≥ ∑_{k=1}^{n} γ_{k} + NL − NL = ∑_{k=1}^{n} γ_{k}.

On the other hand, since γ_{k} ≥ 0 for any k, we have

∑_{k=1}^{n}(γ + ξ + μ)_{k} = ∑_{k=1}^{n}(w · γ)_{k} ≤ ∑_{k=1}^{n} γ_{k},

from (6.8.3). Thus the both sides must be equal, so that we have (6.8.4)(b),(c). Now (6.8.4)(a) follows from (6.8.4)(b) by applying (6.8.5) to I = {1, …, n}\{k} and I = {k}.

It remains to check (6.8.4)(d). From (6.8.4)(a), we have

(γ + ξ + μ)_{j} = \{ \begin{matrix}γ_{j} + L + μ_{j} ≥ γ_{j} & if 1 ≤ j ≤ N, \\ γ_{j} + μ_{j} ≤ γ_{j} & if N + 1 ≤ j ≤ n. \end{matrix}

Fix k such that γ_{k} ≤ min_{1≤j≤N}γ_{j}. Then

{j; 1 ≤ j ≤ n, (γ + ξ + μ)_{j} ≥ γ_{k}}
={1, 2, …, N} ∪ {j; N + 1 ≤ j ≤ n, (γ + μ)_{j} ≥ γ_{k}}
⊂{1, 2, …, N} ∪ {j; N + 1 ≤ j ≤ n, γ_{j} ≥ γ_{k}}
={j; 1 ≤ j ≤ n, γ_{j} ≥ γ_{k}}.

Since γ + ξ + μ is a permutation of γ ((6.8.4)(c)), both sets must coincide. Thus we have

γ_{j} ≥ γ_{k} ⇒ γ_{j} + μ_{j} ≥ γ_{k}

for any j with N + 1 ≤ j ≤ n. Therefore if we assume moreover N + 1 ≤ k ≤ n, then we have γ_{k} + μ_{k} ≥ γ_{k} by taking j = k. Hence μ_{k} = 0 from (6.8.4)(a). □

Claim 6.8.6. Suppose we are in the setting of Claim(6.8.2) with the same hypotheses. Set J := {k; γ_{k} > min_{1≤j≤N}γ_{j}} and J′ := {k; N + 1 ≤ k ≤ n, k ∉ J}. Assume moreover that

(6.8.7)  min_{j∈J} μ_{j} ≥ min_{j∈J′} μ_{j},

If J′ is not empty, then μ = −ξ.

Proof. First we show that μ_{j} = 0 (N + 1 ≤ j ≤ n) if J′ ≠ ∅. Indeed, μ_{j} = 0 when j ∈ J′ by (6.8.4)(d). If J′ ≠ ∅, this implies μ_{j} = 0 (j ∈ J) because of (6.8.4)(a) and (6.8.7). Therefore μ_{j} = 0 for j ∈ J ∪ J′ = {N + 1, N + 2, …, n}. Now (6.8.4)(a),(b) determines μ_{1} = −L. □

6.9. verification of (6.5.4)(b)

Retain notations in §6.7. Set

n(j) := \{ \begin{matrix}N + m(1) + Λ_{0} − Λ_{1} & (j = 1), \\ m(j) + Λ_{j−1} − Λ_{j} & (2 ≤ j ≤ s). \end{matrix}

Proof of (6.5.4)(b). Define the partition π̃ of n by

π̃ := (n(1), …, n(s), Λ_{s} − 1, 1^{m(s+1)}),

and associate a parabolic subgroup P̃(π̃) of Sp(n) (Definition(6.6.2)) with the Levi factor

L(π̃) = (∏_{j=1}^{s} G_{j}) × GL(Λ_{s} − 1) × GL(1)^{m(s+1)}.

Here G_{j} ≃ GL(n(j)) stands for the obvious j-th block of L(π̃). Set

C_{λ}^{(j)} := \{ \begin{matrix}Ind(P̃(π) ∩ G_{1} ↑ G_{1})([λ^{(0)}] ⊗ [Λ_{1}]^{m(1)} ⊗ [ν^{(1)}]) & (j = 1) \\ Ind(P̃(π) ∩ G_{j} ↑ G_{j})([Λ_{j}]^{m(j)} ⊗ [ν^{(j)}]) & (2 ≤ j ≤ s). \end{matrix}

Then appealing to induction by stages, we have

C_{λ} = Ind(P̃(π̃) ↑ Sp(n)) (( ⊗^{s}_{j=1}C_{λ}^{(j)}) ⊗ [ν^{(s+1)}] ⊗ [Λ_{s+1}]^{m(s+1)}) .

As C_{λ}^{(j)} is an irreducible bimodule for G_{j} due to Barbasch and Vogan ([30] Proposition 12.2), each Harish-Chandra bimodule map for G_{j}

U(𝔤_{1})/I(λ^{(0)}, Λ_{1}, …, Λ_{1}, ν^{(1)}) → C_{λ}^{(1)}
U(𝔤_{j})/I(Λ_{j}, …, Λ_{j}, ν^{(j)}) → C_{λ}^{(j)} (2 ≤ j ≤ s)

is surjective. From Theorem 5.5 of [33] combined with the exactness of the functor Ind(P̃(π̃) ↑ Sp(n)), the bimodule map

U(𝔰𝔭(n))/I(γ) → C_{λ}

is also surjective (γ is defined in (6.3.2)(a)). Therefore C_{λ} is generated by a unique (up to scalar) Sp(n)-fixed vector as a Harish-Chandra bimodule for Sp(n).

The similar argument is available for λ + ξ. More precisely, one should replace π̃ by its refinement according to n(1) = N + (m(1) + Λ_{0} − Λ_{1}). Thus we have shown the condition (6.5.4)(b). □

6.10. verification of (6.5.4)(c)

Retain notations in Theorem(6.3.1) and §6.6-7. Suppose we are in the setting of (6.7.1-4). Define another ordered partition of n = N + r + l by

π′ = (N, 1^{m(1)}, …, 1^{m(s+1)}, Λ_{0} − Λ_{1}, …, Λ_{s} − Λ_{s+1})
= (N, 1^{r}, Λ_{0} − Λ_{1}, …, Λ_{s} − Λ_{s+1}),

a permutation of π (Definition(6.7.5)). We define a Harish-Chandra bimodule for L(π′) by

(σ′)_{λ} := [λ^{(0)}] ⊗ ( ⊗^{s+1}_{j=1}[Λ_{j}]^{m(j)}) ⊗ ( ⊗^{s+1}_{j=1}[ν^{(j)}]),

and a Harish-Chandra bimodule for Sp(n) by

(C′)_{λ} := Ind(P̃(π′) ↑ Sp(n))((σ′)_{λ}).

We will construct Harish-Chandra bimodule maps for Sp(n):

C_{λ} → (C′)_{λ} (6.10.1) (C′)_{λ} → R_{γ}(𝔩 : 𝔤) (6.10.2)

that are non-zero on Sp(n)-fixed vectors.

As for the first map (6.10.1), it is proved on GL(n) level. In fact, put

γ̃ := (^{t}λ^{(0)}, Λ_{1}^{m(1)},^{t}ν^{(1)}, …, Λ_{s}^{m(s)},^{t}ν^{(s)},^{t}ν^{(s+1)}, Λ_{s+1}^{m(s+1)}) ∈ ℂ^{n},
γ̃′ := (^{t}λ^{(0)}, Λ_{1}^{m(1)}, …, Λ_{s}^{m(s)}, Λ_{s+1}^{m(s+1)},^{t}ν^{(1)}, …,^{t}ν^{(s)},^{t}ν^{(s+1)}) ∈ ℂ^{n},

Then there is a commutative diagram of intertwining operators between Harish-Chandra bimodules (see Definition(6.6.2)):

Ind(B ↑ GL(n))(Rγ̃(𝔥 : 𝔥)) ⟶ Ind(B ↑ GL(n))(R_{}_{γ̃′}(𝔥 : 𝔥))
∪  ∪
Ind(P (π) ↑ GL(n))(σ_{λ}) ⟶ Ind(P (π′) ↑ GL(n))((σ′)_{λ}).

The point here is that these horizontal maps are non-zero on GL(n)-fixed vectors ([28] Chapter 4, see also [30] Theorem 11.5). By inducing it to Sp(n), the induced Harish-Chandra bimodule map

C_{λ} ≃ Ind(P̃((n)) ↑ Sp(n)) Ind(P (π) ↑ GL(n))(σ_{λ}) → (C′)_{λ} ≃ Ind(P̃((n)) ↑ Sp(n)) Ind(P (π′) ↑ GL(n))((σ′)_{λ})

is non-zero on the Sp(n)-fixed vector.

As for the second map (6.10.2) we use induction by stages according to

L(π′) ⊂ L_{ℂ} := GL(N) × GL(1)^{r} × Sp(l) ⊂ Sp(n).

Then the desired result is obtained from the corresponding one for L_{ℂ}, that is,

Ind(P̃(π′) ∩ L_{ℂ} ↑ L_{ℂ})((σ′)_{λ}) → R_{γ}(𝔩 : 𝔩)

is a surjective Harish-Chandra bimodule map for L_{ℂ}.

The argument for λ + ξ is quite similar.

6.11. proof of Corollary(6.4.1)

Suppose we are in the setting of Corollary(6.4.1). Define the parabolic subalgebra 𝔮 = 𝔩 + 𝔲 of 𝔤 such that 𝔮 ⊂ 𝔭, 𝔥 ⊂ 𝔩 ⊂ 𝔪 and 𝔩 ≃ 𝔤𝔩(n_{1}) ⊕ 𝔤𝔩(1)^{n−n_{1}−l} ⊕ 𝔰𝔭(l).

We proceed by induction on n. Put 𝔤′ := 𝔤𝔩(n_{1}) ⊕ 𝔰𝔭(n − n_{1}), s′ := dim(𝔫 ∩ 𝔨 ∩ 𝔤′). We will show that R_{𝔭}^{s}(ℂ_{λ}) is irreducible as U(𝔤)-module assuming the irreducibility in the 𝔰𝔭(n − n_{1}) case. If γ_{n_{1}} ≥ l, then γ is weakly good with respect to 𝔭 + 𝔤′ ⊂ 𝔤 and so the inductive assumption assures that (R_{𝔭}^{𝔤})^{s} (ℂ_{λ}) ≃ (R_{𝔭+𝔤′}^{𝔤})^{s−s′} ((R_{𝔭∩𝔤′}^{𝔤′})^{s′} (ℂ_{λ})) is irreducible (or zero) (Fact(1.4.1)(1), Lemma(3.2.1)). Thus we shall concentrate on the case where γ_{n_{1}} < l.

First recall that our assumption on γ = λ + ρ_{𝔪} implies that γ is in the weakly fair range with respect to both 𝔮 ⊂ 𝔤 and 𝔭 ⊂ 𝔤. We have a surjective Harish-Chandra bimodule map for M_{ℂ} (cf. (6.6.1)):

Ind(M_{ℂ} ∩ Q_{ℂ} ↑ M_{ℂ}) (R_{γ}(𝔩 : 𝔩)) → R_{γ}(𝔪 : 𝔪).

Applying the exact functor Ind(P_{ℂ} ↑ G_{ℂ}) with trivial 𝔫 action, we have a surjective Harish-Chandra bimodule map for G_{ℂ}:

R_{γ}(𝔩 : 𝔤) → R_{γ}(𝔪 : 𝔤).

Here we used R_{γ}(𝔩 : 𝔤) ≃ Ind(P_{ℂ} ↑ G_{ℂ}) Ind(M_{ℂ} ∩ Q_{ℂ} ↑ M_{ℂ}) (R_{γ}(𝔩 : 𝔩)). Thus the U(𝔤)-action on R_{𝔭}^{s}(ℂ_{λ}) factors through algebra homomorphisms

U(𝔤) → R_{γ}(𝔩 : 𝔤) ↠ R_{γ}(𝔪 : 𝔤).

Now we apply Theorem(6.3.1) to r = n_{2} +⋯+n_{k}, N = n_{1} and M = R_{𝔭}^{s}(ℂ_{λ+ξ}). (As γ is a ℨ(𝔪)-infinitesimal character of a one dimensional representation, γ is of the form (6.3.2)(a) with (6.3.2)(b).) Choose ξ as in Theorem(6.3.1), then R_{𝔭}^{s}(ℂ_{λ+ξ}) is irreducible by the inductive assumption as we saw, in the case γ_{n_{1}} ≥ l. Therefore if we show

(6.11.1)  ψ_{γ+ξ}^{γ}R_{𝔭}^{s}(ℂ_{λ+ξ}) = R_{𝔭}^{s}(ℂ_{λ}),

then we are done. From Lemma(3.4.1)(2), it suffices to show

μ = −ξ,

if μ ∈ 𝔥^{∗} ≃ ℂ^{n} is a weight in F (Sp(n), −ξ) satisfying

μ is dominant for Δ^{+}(𝔪, 𝔥),
γ + ξ + μ = w · γ for some w ∈ W (𝔰𝔭(n)).

This is derived from Lemma(6.8.1) by the following observation: With notation in Lemma(6.8.1), reorder π, γ as

π′ := (N, 1^{m(1)+···+m(s)+m(s+1)}, Λ_{0} − Λ_{1}, …, Λ_{s−1} − Λ_{s}, Λ_{s} − 1),
γ′ := (λ^{(0)}, λ_{1}, …, λ_{r}, ν^{(1)}, …, ν^{(s)}, ν^{(s+1)}).

The same statement for γ′, 𝔩(π′) as in Lemma(6.8.1) holds (use an inner automorphism of Sp(n)) and its assumption is satisfied under (6.11.2) because 𝔩(π′) ⊂ 𝔪. (Recall that (ν^{(1)}, ν^{(2)}, …, ν^{(s)}, ν^{(s+1)}) is a grouping of (l, l − 1, …, 2, 1).) Hence (6.11.1) holds and Corollary(6.4.1) is proved.

7. Proof of vanishing results outside the fair range

This section is devoted to Part(2) of Theorems 1 and 2 (see §2): the vanishing result, when j ≠ S ≡ dim(𝔲 ∩ 𝔨), for the derived functor modules R_{𝔮}^{j}(ℂ_{λ}) with small parameter λ such that ℂ_{λ} lies outside the fair range. When G = SO_{0}(p, q), the condition (2.6.2) implies that ℂ_{λ} (resp. ℂ_{λ′}) lies in the weakly fair range with respect to 𝔮 (resp. 𝔮′). Then Part(2) of Theorem 3 is a direct consequence of Fact(1.4.2)(1-a). When G = U(p, q), we can apply Lemma(3.4.1)(2) and Fact(1.4.2)(1-a). This is fairly easy and we shall give a preliminary Lemma(7.1.1). The rest of this section (§7.2-4) will be devoted to the case G = Sp(p, q). The non-trivial part there is only the crossing of the wall λ_{r} = 0.

7.1. proof in complex case

When G = U(p, q), the iteration of the following lemma and Lemma(3.4.1)(2) reduces the vanishing of R_{𝔮}^{j}(ℂ_{λ}) (j ≠ S) under the condition (2.4.2) to that under the condition

λ_{1} > λ_{2} > ⋯ > λ_{r} > Q ≥ λ_{r+1} ≥ ⋯ ≥ λ_{r+s}.

Similarly, this is then reduced to the vanishing under the condition

λ_{1} > λ_{2} > ⋯ > λ_{r} > Q > −Q > λ_{r+1} > ⋯ > λ_{r+s}.

Now the vanishing of R_{𝔮}^{j}(ℂ_{λ}) (j ≠ S) in this last case is well known (see Fact(1.4.1)(1-a) or Fact(1.4.2)(1-a)). So it suffices to show

Lemma(7.1.1). Let G = U(p, q) and 𝔮 = 𝔮(r, s) = 𝔩 + 𝔲 (r + s ≤ p) be a θ-stable parabolic subalgebra. Retain notations as in §2.3. Let λ = (λ_{1}, …, λ_{r+s}) ∈ 𝔱^{∗}. Fix i (1 ≤ i ≤ r + s) and set λ′ := λ + f_{i} ∈ 𝔱^{∗}. Assume

λ_{i} ≥ −Q. (7.1.2)(a) λ_{j} ≠ λ_{i} + 1 for any j (1 ≤ j ≤ r + s). (7.1.2)(b)

Then A (λ′ ▷ λ) = {λ} (see §3.3 for definition).

Proof. Fix a positive system Δ^{+}(𝔩, 𝔥) so that ρ_{𝔩} = (\overbrace{0, …, 0}^{r+s}, Q, Q − 1, …, −Q). We shall prove A(Δ^{+}(𝔩, 𝔥); λ′ ▷ λ) = {λ}. (Then we tell a fortiori A(Δ^{+}(𝔩, 𝔥); λ′ ▷ λ) is independent of a particular choice of Δ^{+}(𝔩, 𝔥). This is the reason why we omit Δ^{+}(𝔩, 𝔥) in the statement of Lemma(7.1.1).)

Suppose μ = (μ_{1}, …, μ_{p+q}) ∈ A(Δ^{+}(𝔩, 𝔥); λ′ ▷ λ). As μ − λ′ is a weight of a p + q dimensional representation F (𝔤𝔩(p + q, ℂ), −f_{i}) = F (𝔤𝔩(p + q, ℂ), −f_{1}), we can find k (1 ≤ k ≤ p + q) such that

(7.1.3)  μ = λ′ − f_{k} = λ + f_{i} − f_{k}.

Thus μ − λ′ is dominant for Δ^{+}(𝔩, 𝔥) iff 1 ≤ k ≤ r + s or k = p + q.

Let us observe that k ≠ p + q. If μ were to be λ + f_{i} − f_{p+q}, then (7.1.2)(a) and (7.1.3) would imply

#{j; (μ + ρ_{𝔩})_{j} = −Q − 1} = #{j; (λ + ρ_{𝔩})_{j} = −Q − 1} + 1,

contradicting to the fact that μ + ρ_{𝔩} ∈ 𝔖_{p+q} · (λ + ρ_{𝔩}). Therefore we have 1 ≤ k ≤ r + s and μ = λ + f_{i} − f_{k} ∈ 𝔱^{∗}. Then μ + ρ_{𝔩} ∈ 𝔖_{p+q} · (λ + ρ_{𝔩}) implies μ ∈ 𝔖_{r+s} · λ. Now (7.1.2)(b) assures k = i and μ = λ + f_{i} − f_{i} = λ. Hence A(Δ^{+}(𝔩, 𝔥); λ′ ▷ λ) = {λ}. □

7.2. vanishing result in quaternionic case

The rest of this section will be devoted to proving Theorem 1 (2). That is, Theorem 7.2.1. Let G = Sp(p, q) and fix an integer r (1 ≤ r ≤ p). Set Q := p + q − r and let 𝔮 ≡ 𝔮(r) be a θ-stable parabolic subalgebra as in §2.1. Retain the notation there. If λ = (λ_{1}, …, λ_{r}) ∈ ℤ^{r} ⊂ 𝔱^{∗} satisfies

(7.2.2)  λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{r−1} ≥ |λ_{r}|, λ_{r} ≥ −Q,

then R_{𝔮}^{j}(ℂ_{λ}) = 0 for any j ≠ S.

Remark 7.2.3. The assumption λ_{r} ≥ −Q in (7.2.2) is crucial. In fact, it does happen that R_{𝔮}^{S−i}(ℂ_{λ}) ≠ 0 (i = 0, 1) when λ_{r} < −Q. (I checked this when p = r by using Vogan’s U_{α} calculus.)

7.3. maximal parabolic case

First we prove Theorem 7.2.1 in the case where r = 1 so that 𝔮 is a maximal parabolic subalgebra in 𝔤. This is a main part in the proof.

Lemma 7.3.1. Theorem(7.2.1) holds when r = 1.

Proof. We write λ(n) := n ∈ 𝔱^{∗} ≃ ℂ for n ∈ ℤ. Then ℂ_{λ(n)} is in the weakly fair range iff n ≥ 0. Because of the vanishing theorem, R_{𝔮}^{j}(ℂ_{λ(n)}) = 0 (j ≠ S), in the weakly fair range (Fact(1.4.2)(1-a)), we should concentrate our attention on 0 > n ≥ −Q. Then thanks to Lemma(3.4.1), it suffices to treat only the case where n = −1 because

A (λ(−i) ▷ λ(−i − 1)) = {λ(−i − 1)} (−1 ≥ i ≥ −Q + 1).

(See §3.3 for notation.) Let us prove the vanishing result for λ = λ(−1). For this, we apply Lemma(3.4.1)(3-a,b) and Claim(3.4.7). But we give a little detailed explanation of it for the benefit of the reader. First observe that

(7.3.2)(a)  A (λ(1) ▷ λ(0)) = A (λ(−1) ▷ λ(0)) = {λ(0)},
(7.3.2)(b)  A (λ(0) ▷ λ(1)) = A (λ(0) ▷ λ(−1)) = {λ(1), λ(−1)},

and that R_{𝔮}^{j}(ℂ_{λ(1)}) = R_{𝔮}^{j}(ℂ_{λ(0)}) = 0 for j ≠ S. From (7.3.2)(a), we have

ψ_{λ(1)+ρ_{𝔩}}^{λ(0)+ρ_{𝔩}}(R_{𝔮}^{j}(ℂ_{λ(1)})) = R_{𝔮}^{j}(ℂ_{λ(0)}),
ψ_{λ(−1)+ρ_{𝔩}}^{λ(0)+ρ_{𝔩}}(R_{𝔮}^{j}(ℂ_{λ(−1)})) = R_{𝔮}^{j}(ℂ_{λ(0)}).

Applying Claim(3.4.7) with F = F (Sp(p, q), λ(1)) (≃ ℂ^{2p+2q}), we have a filtration of 𝔮-modules:

{0} = F_{0} ⊂ F_{1} ⊂ F_{2} ⊂ F_{3} = F

such that F_{1} ≃ ℂ_{λ(1)}, F_{2}/F_{1} ≃ 1 ⊠ F (Sp(p − 1, q), (1, 0, …, 0)) and F/F_{2} ≃ ℂ_{λ(−1)}. From (7.3.2)(b) and Lemma(3.4.1)(3-b)′, we have a long exact sequence of (𝔤, K)- modules:

0 → R_{𝔮}^{S−1}(ℂ_{λ(−1)}) → R_{𝔮}^{S}(ℂ_{λ(1)}) → ψ_{λ(0)+ρ_{𝔩}}^{λ(1)+ρ_{𝔩}}(R_{𝔮}^{S}(ℂ_{λ(0)})) → R_{𝔮}^{S}(ℂ_{λ(−1)}) → 0,

and R_{𝔮}^{j}(ℂ_{λ(−1)}) = 0 for j ≠ S − 1, S. Since R_{𝔮}^{S}(ℂ_{λ(1)}) is (nonzero) irreducible by Corollary(6.4.1), we have either

R_{𝔮}^{S−1}(ℂ_{λ(−1)}) = 0,

or

R_{𝔮}^{S−1}(ℂ_{λ(−1)}) ≃ R_{𝔮}^{S}(ℂ_{λ(1)}).

But the latter case is impossible. Indeed, applying ψ_{λ(1)+ρ_{𝔩}}^{λ(0)+ρ_{𝔩}} = ψ_{λ(−1)+ρ_{𝔩}}^{λ(0)+ρ_{𝔩}} to the latter (false) isomorphism, we would have R_{𝔮}^{S−1}(ℂ_{λ(0)}) = R_{𝔮}^{S}(ℂ_{λ(0)}). Since ℂ_{λ(0)} is in the weakly fair range, we have R_{𝔮}^{S−1}(ℂ_{λ(0)}) = 0, while R_{𝔮}^{S}(ℂ_{λ(0)}) ≠ 0 by Corollary(4.3.6). This is a contradiction. Therefore R_{𝔮}^{j}(ℂ_{λ(−1)}) = 0 for all j ≠ S. This completes the proof. □

7.4. general parabolic case Proof of Theorem(7.2.1). Fix λ = (λ_{1}, …, λ_{r}) satisfying (7.2.2). The spectral sequence of induction by stages (Lemma(3.2.1)) corresponding to

L := 𝕋^{r} × Sp(p − r, q) ⊂ M := 𝕋^{r−1} × Sp(p − r + 1, q) ⊂ G = Sp(p, q)

collapses to

(7.4.1)  (R_{𝔮(r−1)}^{𝔤})^{i} (R_{𝔮∩𝔪}^{𝔪})^{S′} (ℂ_{λ}) ≃ R_{𝔮}^{i+S′}(ℂ_{λ}),

from the vanishing result Lemma(7.3.1) applied to a maximal parabolic subalgebra 𝔮 ∩ 𝔪 ⊂ 𝔪 and Q = (p − r + 1) + q − 1 = p + q − r. Here S′ = dim(𝔲 ∩ 𝔨 ∩ 𝔪). Take a weight μ = (μ_{1}, …, μ_{r−1}, λ_{r}) ∈ ℤ^{r} ⊂ 𝔱^{∗} such that

μ_{1} > μ_{2} > ⋯ > μ_{r−1} > max(|λ_{r}|, Q).

Since R_{𝔮∩𝔪}^{S′}(ℂ_{μ}) is in the good range with respect to 𝔮(r−1) ⊂ 𝔤, we have R_{𝔮}^{j}(ℂ_{μ}) = 0 for j ≠ S. We can easily find a sequence λ^{(0)} = μ, λ^{(1)}, …, λ^{(n)} = λ so that

A (λ^{(i−1)} ▷ λ^{(i)}) = {λ^{(i)}} for i = 1, 2, …, n.

Hence an inductive argument shows R_{𝔮}^{j}(ℂ_{λ}) = 0 for all j ≠ S from Lemma(3.4.1). □

8. Proof of the inequivalence results

In this section we shall show the results on pairwise inequivalence among R_{𝔮}^{S}(ℂ_{λ})’s, i.e. Part (4) of the Theorems in §2. Our method is based on the K-spectrum calculated in §4. In view of the ℨ(𝔤)-infinitesimal characters, the non-trivial cases are when G = Sp(p, q) or when G = SO_{0}(p, q) with p = 2r.

8.1. quarternionic case

Suppose we are in the setting of §2.2. It suffices to compare the modules with the same ℨ(𝔤)-infinitesimal character to prove Part (4) of Theorem 1. That is, we must show:

Proposition 8.1.1. Let G = Sp(p, q) and fix an integer r (1 ≤ r ≤ p). Retain the notation in §2.1. Let λ = (λ_{1}, …, λ_{r−1}, λ_{r}) ∈ ℤ^{r} ⊂ 𝔱^{∗} satisfy

(8.1.2)  \{ \begin{matrix}λ_{1} > λ_{2} > ⋯ > λ_{r−1} > λ_{r} > 0, \\ λ_{r−2q} ≥ Q + 1(≡ p + q − r + 1) when r > 2q. \end{matrix}

If we set λ′ = (λ_{1}, …, λ_{r−1}, −λ_{r}), then R_{𝔮}^{S}(ℂ_{λ}) ≄ R_{𝔮}^{S}(ℂ_{λ′}) as a (𝔤, K)-module.

Proof. Let l (0 ≤ l ≤ r − 1) be an integer such that λ_{l} > Q ≥ λ_{l+1}. We may assume λ_{l} >> Q so that λ_{l} − p + q + l − 1 ≥ 0. We may also assume Q ≥ λ_{r}. This is possible because the general statement is derived from the corresponding one in this special case by reduction to absurdity (use the translation principle). In our present situation, R_{𝔮}^{i}(ℂ_{λ}) = R_{𝔮}^{i}(ℂ_{λ′}) = 0 when i ≠ S by Theorem(7.2.1). Hence, it suffices to find μ ∈ 𝔱^{∗} such that M(𝔮, λ, μ) ≠ M(𝔮, λ′, μ) (see §4.1 for notation). We define b, b′ ∈ ℤ^{r} by (4.3.1) and k, k′ ∈ ℕ by (4.3.4) corresponding to λ, λ′ respectively. Set δ = (b_{1}, …, b_{k}, 0, …, 0), δ′ = (b′_{1}, …, b′_{k′}, 0, …, 0) ∈ ℤ^{r}. Let us show

M(𝔮, λ, δ′) < M(𝔮, λ′, δ′).

As b_{l} = λ_{l}−p+q+l−1 ≥ 0 from our assumption, we have k, k′ ≥ l and Q ≥ λ_{k+1}, λ′_{k′+1} respectively. We divide now into two cases according as δ = δ′ or not. (I) δ ≠ δ′.

In this case we have ∑_{i=1}^{r} δ′_{i} < ∑_{i=1}^{r} δ_{i} = ∑_{i=1}^{k} b_{i}. Hence, M(𝔮, λ, δ′) = 0 < M(𝔮, λ′, δ′) from Proposition(4.3.2)(2)-(3) (see also Remark(4.3.5)).

(II) δ = δ′. In this case we have k = k′. Combining Proposition(4.3.2)(2) with the explicit formula of d(n, l; x) in Lemma(4.2.6) and putting P := p − q + 1, then we have

\frac{M(𝔮, λ, δ)}{M(𝔮, λ′, δ′)} = \frac{d(2q, r − k; −λ_{k+1} + P, …, −λ_{r−1} + P, −λ_{r} + P)}{d(2q, r − k; −λ_{k+1} + P, …, −λ_{r−1} + P, λ_{r} + P)}

= \frac{∏^{2q−r+k} ; j=1(j − λ_{r} + p − q) ∏_{k+1≤j<r}(λ_{j} − λ_{r})}{∏^{2q−r+k} ; j=1(j + λ_{r} + p − q) ∏_{k+1≤j<r}(λ_{j} + λ_{r})}

< 1.

Hence M(𝔮, λ, δ′) = M(𝔮, λ, δ) < M(𝔮, λ′, δ′). □

8.2. orthogonal case Suppose we are in the setting of §2.6 and §4.5. Assume p = 2r and (2.6.3). Put

δ := μ_{λ}_{|𝔱^{c}} = (λ_{1} + \frac{q}{2} − r + 1, …, λ_{r−1} + \frac{q}{2} − 1, λ_{r} + \frac{q}{2}) ∈ (𝔱_{1}^{c})^{∗},
δ′ := μ′_{λ′}_{|𝔱^{c}} = (λ_{1} + \frac{q}{2} − r + 1, …, λ_{r−1} + \frac{q}{2} − 1, −λ_{r} − \frac{q}{2}) ∈ (𝔱_{1}^{c})^{∗},

Then δ ≠ δ′ and

M(𝔮, λ, δ) = M(𝔮′, λ′, δ′) = 1
M(𝔮, λ, δ′) = M(𝔮′, λ′, δ) = 0

This follows from Proposition(4.5.2), but is easy to check directly because μ_{λ}, μ′_{λ′} are Δ^{+}(𝔨) dominant in this case. Since R_{𝔮}^{S−j}(ℂ_{λ}) = R_{𝔮′}^{S−j}(ℂ_{λ′}) = 0 when j ≠ 0, we have R_{𝔮}^{S}(ℂ_{λ}) ≄ R_{𝔮′}^{S}(ℂ_{λ′}).

References

[1] J.Adams, Unitary highest weight modules, Adv. in Math. 63 (1987), 113-137.

[2] F.Bien, Spherical D-modules and representations of reductive Lie groups, Ph.D. dissertation, M.I.T., Cambridge, Massachusetts (1986).

[3] W.Borho and J-L.Brylinski, Differential operators on homogeneous spaces I, Invent.Math. 69 (1982), 437-476.

[4] J.T.Chang, Remarks on localization and standard modules: The duality theorem on a generalized flag variety, preprint, M.S.R.I., Berkeley.

[5] D.Collingwood, Representations of rank one Lie groups, Research notes in Math., Pitman Publishing, London, 1985.

[6] M.Duflo, Sur la classification des ideaux primitifs dans l’algebre enveloppante d’une algebre de Lie semi-simple, Ann. of Math. 105 (1977), 107-120.

[7] T.J.Enright, R.Parthasarathy, N.R.Wallach, and J.A.Wolf, Unitary derived functor module with small spectrum, Acta.Math. 154 (1985), 105-136.

[8] M.Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. of Math. 111 (1980), 253-311.

[9] , Analysis on non-Riemannian symmetric spaces, Conference Board, No.61, A.M.S., 1986.

[10] S.Helgason, A duality for symmetric spaces with applications to group representations, Adv. in Math. 5 (1970), 1-154.

[11] H.Hecht, D.Miličic, W.Schmid and J.A.Wolf, Localization and standard modules for real semisimple Lie groups, Invent.Math. 90 (1987), 297-332.

[12] W.Hesselink, Polarizations in the classical groups, Math.Z. 160 (1978), 217-234.

[13] T.Kobayashi, Unitary representations realized in sections of vector bundles over semisimple symmetric spaces (in Japanese), Master’s dissertation I, Univ.of Tokyo (1987).

[14] , Construction of discrete series for vector bundles over semisimple symmetric spaces (in Japanese), RIMS Kyoto Kokyuroku 642 (1988), 134-156.

[15] B.Kostant, Lie algebra cohomology and the generalized Borel-Weil theorem, Ann. of Math. 74 (1961), 329-387.

[16] , Irreducibility of certain series of representations, Publication of 1971 Summer School in Math., (ed. I.M.Gel’fand), Bolyai-Janos Math.Soc., Budapest.

[17] H.Kraft and C.Procesi, Closures of conjugacy classes of matrices are normal, Invent.Math. 53 (1979), 227-247.

[18] , On the geometry of conjugacy classes in the classical groups, Comment.Math.Helv. 57 (1982), 539-602.

[19] A.Magnus, Non-spherical principal series representations of a semisimple Lie group, Memoirs No.216, A.M.S., 1979.

[20] T.Matsuki and T.Oshima, A description of discrete series for semisimple symmetric spaces, Advanced Studies in Pure Math. 4 (1984), 331-390.

[21] T.Matsuki, A description of discrete series for semisimple symmetric spaces II, Advanced Studies in Pure Math. 14 (1988), 531-540.

[22] T.Oshima, Asymptotic behavior of spherical functions on semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 561-601.

[23] H.Schlichtkrull, A series of unitary irreducible representations induced from a symmetric subgroup of a semisimple Lie group, Invent.Math. 68 (1982), 497-516.

[24] , The Langlands parameters of Flensted-Jensen’s discrete series for semisimple symmetric spaces, J.Funct.Anal. 50 (1983), 133-150.

[25] , One dimensional K-types in finite dimensional representations of semisimple Lie groups. A generalization of Helgason’s theorem, Math.Scand. 54 (1984), 279-294.

[26] N.Shimeno, Eigenspaces of invariant differential operators on a homogeneous line bundle on a Riemannian symmetric space, to appear in J.Fac.Sci.Univ.Tokyo, 37-1.

[27] D.Vogan, Irreducible characters of semisimple Lie groups. I, Duke Math.J. 46 (1979), 61-108.

[28] , Representations of real reductive Lie groups, Birkhäuser, 1981.

[29] , Unitarizability of certain series of representations, Ann. of Math. (1984), 141-187.

[30] , The unitary dual of GL(n) over an archimedean field, Invent.Math. (1986), 449-505.

[31] , The orbit method and primitive ideals for semisimple Lie algebras, Lie algebras and related topics, vol. 5, CMS Conference Proceedings, 1986, pp. 281-316.

[32] , Unitary representations of reductive Lie groups, Princeton University Press, Princeton, New Jersey, 1987.

[33] , Irreducibility of discrete series representations for semisimple symmetric spaces, Advanced Studies in Pure Math. 14 (1988), 191-221.

[34] N.R.Wallach, On the unitarizability of derived functor modules, Invent.Math. 78 (1984), 131-141.

[35] , Real reductive groups I, Academic Press, 1988.

[36] G.Zuckerman, Tensor products of finite and infinite dimensional representations of semisimple Lie groups, Ann. of Math. 106 (1977), 295-308.

Toshiyuki Kobayashi Department of Mathematics Faculty of Science University of Tokyo Hongo, Tokyo 113 Japan.

From April 1, 1991,

Department of Mathematics College of Arts and Sciences University of Tokyo Komaba, Meguro, Tokyo 153 Japan.