Propagation of multiplicity-free property for holomorphic vector bundles^{∗}

Toshiyuki KOBAYASHI Research Institute for Mathematical Sciences Kyoto University

       E-mail address: toshi@kurims.kyoto-u.ac.jp

Abstract

We prove a propagation theorem of multiplicity-free property from fibers to spaces of global sections for holomorphic vector bundles, which yields various multiplicity-free results in representation theory for both finite and infinite dimensional cases.

The key geometric condition in our theorem is an orbit-preserving anti-holomorphic diffeomorphism on the base space, which brings us to the concept of visible actions on complex manifolds.

Mathematics Subject Classifications (2000): Primary: 22E46, Secondary 32M10, 32M05, 46E22.

Key words: multiplicity-free representation, reproducing kernel, unitary representation, homogeneous space, holomorphic bundle, visible action

Contents

1 Introduction 2

2 Complex geometry and multiplicity-free theorem 4 3 Proof of Theorem 2.2 8

∗This work was partially supported by Grant-in-Aid for Scientific Research 18340037, Japan Society for the Promotion of Science.

4 Visible actions on complex manifolds 14

5 Multiplicity-free theorem for associated bundles 18

6 Proof of Proposition 5.2 23

1 Introduction

Propagation of unitarity from fibers to spaces of sections (more generally, stalks to cohomologies) is one of the most fundamental results in representation theory. Propagation theory of unitarity was established by Mackey [12] for induced representations in 1950s and by Vogan [17] and Wallach [18] for cohomologically induced representations in 1980s.

Multiplicity-freeness is another important concept in representation theory that generalizes irreducibility. The goal of this paper is to provide a propagation theorem of multiplicity-freeness from fibers to spaces of sections for holomorphic vector bundles.

Let us begin by recalling the definition of multiplicity-freeness for unitary representations. By a theorem of Mautner–Teleman, any unitary representation π of a locally compact group G can be decomposed into the direct integral of irreducible unitary representations:

π ≃ ∫ ⊕ ; Ĝ m_{π}(τ)τdμ(τ),

where Ĝ is the set of equivalence classes of irreducible unitary representations, μ is a measure on Ĝ, and m : Ĝ → ℕ ∪ {∞} is a measurable function that stands for ‘multiplicity’. We shall say that π is multiplicity-free if the ring of continuous G-intertwining endomorphisms is commutative. This condition implies that m is not greater than one almost everywhere with respect to μ.

Multiplicity-free representations arise in broad range of mathematics such as expanding functions (Taylor series, Fourier expansion, spherical harmonics, …) and classical identities (the Cappeli identity, many formulae of special functions, …), though we may not be aware of even the fact that the representation is there. Multiplicity-free representations are a special class of representations, by which one could expect beautiful and powerful applications to other fields, and on which one could expect a simple and detailed study.

To state our main results, let H be a Lie group, and V → D an Hequivariant holomorphic vector bundle. We naturally have a representation of H on the space O(D, V) of global holomorphic sections. Then, the first form of our multiplicity-free theorem is stated briefly as follows (see Theorem 2.2 for details):

Theorem 1.1. Any unitary representation of H which is realized in O(D, V) is multiplicity-free if the H-equivariant bundle V → D satisfies the following three conditions:

(1.1)

(Fiber) For every x ∈ D, the isotropy representation of H_{x} on the fiber V_{x} is multiplicity-free.

(1.2)

(Base space) There exists an anti-holomorphic bundle endomorphism σ, which preserves every H-orbit on the base space D.

(1.3) (Compatibility) See (2.2.3).

The compatibility condition (1.3) is less important because it is often automatically fulfilled by the choice of σ (for example, see Remark 5.2.3; see also [9, Appendix]). Thus, for the propagation of multiplicity-free property from fibers V_{x} to the space O(D, V) of sections, the geometric condition (1.2) on the base space D is crucial.

The geometric condition (1.2) is made clear by the concept of visible actions on complex manifolds (more precisely, S-visible actions; see Definition 4.2). Thus, the second form of our multiplicity-free theorem is formalized in Theorem 4.3 in terms of S-visible actions. Here, S is a totally real slice of the H-action on the base space. If H acts transitively, then even irreducibility propagates (see Proposition 2.5). In general, the smaller the codimension of generic H-orbits on D is, the smaller the slice S we may take and the more likely the multiplicity-free assumption on fibers (see (4.3.2)) tends to be fulfilled. It is noteworthy that coisotropic actions (or multiplicity-free actions) on symplectic manifolds in the sense of Guillemin and Sternberg [3] or Huckleberry and Wurzbacher [4] and polar actions on Riemannian manifolds (see Podestà–Thorbergsson [14], for example) are relevant to visible actions on complex manifolds. The relation among these three concepts is discussed in [8, Section 4].

The third form of our multiplicity-free theorem is formalized in the setting where the bundle V → D is associated to a principal bundle K → P → D and to a representation (μ, V) of the structure group K. This is Theorem 5.3. This form is intended for actual applications, in particular, for branching problems (decompositions of irreducible representations when restricted to subgroups).

Our multiplicity-free theorem has a wide range of applications for both finite and infinite dimensional cases, for both discrete and continuous spectra, and for both classical and exceptional cases. For example, Theorem 5.3 explains the multiplicity-free property of the Plancherel formula for line bundles (and also for some vector bundles) over Riemannian symmetric spaces, the Hua–Kostant–Schmid K-type formula [8, 15], and the canonical representation in the sense of Vershik–Gelfand–Graev (e.g. [1]). These are examples of infinite dimensional multiplicity-free representations. Our theorem also gives a geometric explanation of the complete list of multiplicity-free tensor product of (finite dimensional) representations for GL(n) which was recently found by Stembridge [16] by combinatorial method.

This paper plays the central role in a series of our papers [7, 8, 9, 10, 11]. The paper [7, 8] discussed various applications including the aforementioned multiplicity-free theorems by using Theorem 5.3. Classification results on visible actions on complex manifolds are discussed in subsequent papers [10, 11].

Acknowledgement. I owe much to Professor J. Faraut for enlightening discussions, in particular, for explaining the idea of [2] in an early stage of this work. Substantial part of this research was made during my visit to Harvard University. I express my gratitude to Professor W. Schmid who gave me a wonderful atmosphere of research.

2 Complex geometry and multiplicity-free theorem

This section gives a first form of our multiplicity-free theorem. We may regard it as a propagation theorem of multiplicity-free property from fibers to spaces of sections in the setting where there may exist infinitely many orbits on base spaces. The main result of this section is Theorem 2.2. We shall reformulate it by means of visible actions in Section 4, and furthermore present its group theoretic version in Section 5.

2.1 Equivariant holomorphic vector bundle

Let V = ⨿_{x∈D}V_{x} → D be a Hermitian holomorphic vector bundle over a connected complex manifold D. We denote by O(D, V) the space of holomorphic sections of V → D. It carries a Fréchet topology by the uniform convergence on compact sets.

Suppose a Lie group H acts on the bundle V → D by automorphisms. This means that the action of H on the total space, denoted by L_{h} : V → V, and the action on the base space, denoted simply by h : D → D, x ↦ h·x, are both biholomorphic for h ∈ H, and that the induced linear map L_{h} : V_{x} → V_{h·x} between the fibers is unitary for any x ∈ D. In particular, we have a unitary representation of the isotropy subgroup H_{x} := {h ∈ H : h · x = x} on the fiber V_{x}.

The action of H on the bundle V → D gives rise to a continuous representation on O(D, V) by the pull-back of sections, namely, s ↦ L_{h}s(h^{−1}· ) for h ∈ H and s ∈ O(D, V).

Definition 2.1. Suppose π is a unitary representation of H defined on a Hilbert space H. We will say π is realized in O(D, V) if there is an injective continuous H-intertwining map from H into O(D, V).

Let {U_{α}} be trivializing neighborhoods of D, and g_{αβ} : U_{α} ∩ U_{β} → GL(n, ℂ) be the transition functions for the holomorphic vector bundle V → D. Then, the anti-holomorphic vector bundle \overline{V} → D is defined to be the complex vector bundle with the transition functions \overline{g_{αβ}}. We denote by \overline{O}(D, \overline{V}) the space of anti-holomorphic sections for \overline{V} → D.

Suppose σ is an anti-holomorphic diffeomorphism of D. Then the pullback σ^{∗}V = ⨿_{x∈D}V_{σ(x)} is an anti-holomorphic vector bundle over D. In turn, \overline{σ^{∗}V} → D is a holomorphic vector bundle over D. The fiber at x ∈ D is identified with \overline{V_{σ(x)}}, the complex conjugate vector space of V_{σ(x)} (see Section 3.1).

The holomorphic vector bundle \overline{σ^{∗}V} is isomorphic to V if and only if σ lifts to an anti-holomorphic endomorphism σ̃ of V. In fact, such σ̃ induces a conjugate linear isomorphism σ̃_{x} : V_{x} → V_{σ(x)}, which then defines a ℂ-linear isomorphism

(2.1.1)  Ψ_{x} : V_{x} → (\overline{σ^{∗}V})_{x}, v ↦ \overline{σ̃_{x}(v)}

via the identification (\overline{σ^{∗}V})_{x} ≃ \overline{V_{σ(x)}}. Then, Ψ : V → \overline{σ^{∗}V} is an isomorphism of holomorphic vector bundles such that its restriction to the base space D is the identity. For simplicity, we shall use the letter σ in place of σ̃. For a Hermitian vector bundle V, by a bundle endomorphism σ, we mean that σ_{x} is furthermore isometric (or equivalently, Ψ_{x} is unitary) for any x ∈ D.

2.2 Multiplicity-free theorem (first form)

The following is a first form of our multiplicity-free theorem:

Theorem 2.2. Let V → D be a Hermitian holomorphic vector bundle, on which a Lie group H acts by automorphisms. Assume:

(2.2.1) the isotropy representation of H_{x} on the fiber V_{x} is multiplicity-free for any x ∈ D.

We write its irreducible decomposition as V_{x} = ⨁_{i=1}^{n(x)} V_{x}^{(i)}. Assume furthermore

that there exists an anti-holomorphic bundle endomorphism σ satisfying the following two conditions: for any x ∈ D,

(2.2.2) there exists h ∈ H such that σ(x) = h · x, and
(2.2.3) σ_{x}(V_{x}^{(i)}) = L_{h}(V_{x}^{(i)}) for any i (1 ≤ i ≤ n(x)).

Then, any unitary representation that is realized in O(D, V) is multiplicityfree.

We shall give a proof of Theorem 2.2 in Section 3.

Remark 2.2.1. 1) The conditions (2.2.1) – (2.2.3) of Theorem 2.2 is local in the sense that the same conclusion holds if D′ is an H-invariant open subset of D, and if the conditions (2.2.1) – (2.2.3) are satisfied for x ∈ D′. This is clear because the restriction map O(D, V) → O(D′, V|_{D′}) is injective and continuous.

2) The proof in Section 3 shows that one can replace H_{x} with its arbitrary subgroup H′_{x} in (2.2.1). (Such a replacement makes (2.2.1) stronger, and (2.2.3) weaker.)

In the following two subsections, we explain special cases of Theorem 2.2.

2.3 Line bundle case

We begin with the observation that the assumptions (2.2.1) and (2.2.3) are automatically fulfilled if V_{x} is irreducible, in particular, if V → D is a line bundle. Hence, we have:

Corollary 2.3. In the setting of Theorem 2.2, assume V → D is a line bundle. If there exists an anti-holomorphic bundle endomorphism satisfying (2.2.2), then any unitary representation that is realized in O(D, V) is multiplicity-free.

This case was announced in [6] with a sketch of proof, and its applications are extensively discussed in [9] for the branching problems (i.e. the decomposition of the restriction of unitary representations) with respect to reductive symmetric pairs.

2.4 Trivial bundle case

If the vector bundle is the trivial line bundle V = D × ℂ, then any antiholomorphic diffeomorphism on D lifts to an anti-holomorphic endomorphism of V by (x, u) ↦ (σ(x), ū). Hence, we have:

Corollary 2.4. If there exists an anti-holomorphic diffeomorphism σ of D satisfying (2.2.2), then any unitary representation which is realized in O(D) is multiplicity-free.

This result was previously proved in Faraut and Thomas [2] under the assumption that σ^{2} = id.

2.5 Propagation of irreducibility

The strongest condition on group actions is transitivity. Transitivity on base spaces guarantees that even irreducibility propagates from fibers to spaces of sections. The following result is due to S. Kobayashi [5].

Proposition 2.5. In the setting of Theorem 2.2, suppose that H acts transitively on D and that H_{x} acts irreduciblly on V_{x} for some (equivalently, for any) x ∈ D. Then, there exists at most one unitary representation π that can be realized in O(D, V). In particular, such π is irreducible if exists.

Proof. This is an immediate consequence of Lemma 3.3 and Proposition 3.4 (n(x) = 1 case) below, which will be used in the proof of Theorem 2.2 in Section 3.

We note that the condition (2.2.2) is much weaker than the transitivity of the group H on D. Our geometric condition (2.2.2) brings us to the concept of visible actions, which we shall discuss in Section 4.

3 Proof of Theorem 2.2

This section is devoted entirely to the proof of Theorem 2.2.

3.1 Some linear algebra

We begin carefully with basic notations.

Given a complex Hermitian vector space V, we define a complex Hermitian vector space \overline{V} as a collection of the symbol \overline{v} (v ∈ V) equipped with a scalar multiplication av̄ := \overline{\overline{a}v} for a ∈ ℂ, and with an inner product (ū, v̄) := (v, u).

The complex dual space V^{∨} is identified with \overline{V} by \overline{V} ⟶^{∼} V^{∨}, v̄ ↦ (·, v). In particular, we have a natural isomorphism of complex vector spaces:

(3.1.1)  V ⊗ \overline{V} ⟶^{∼} End(V).

Given a unitary map A : V → W between Hermitian vector spaces, we define a unitary map \overline{A} : \overline{V} → \overline{W} by \overline{v} ↦ \overline{Av}. Then the induced map A ⊗ \overline{A} : V ⊗ \overline{V} → W ⊗ \overline{W} gives rise to a complex linear isomorphism:

(3.1.2)  A_{♯} : End(V) → End(W).

Then, it is readily seen from the unitarity of A that

(3.1.3)  A_{♯}(id_{V}) = id_{W} .

In particular, if an endomorphism of V is diagonalized with respect to an orthogonal direct sum decomposition V = ⨁_{i=1}^{n} V^{(i)}, then we have the following formula of A_{♯}:

(3.1.4)  A_{♯} (∑_{i=1}^{n} λ_{i} id_{V^{(i)}}) = ∑_{i=1}^{n} λ_{i} id_{A(V^{(i)})} (λ_{1}, …, λ_{n} ∈ ℂ).

3.2 Reproducing kernel for vector bundles

This subsection summarizes some basic results on reproducing kernels for holomorphic vector bundles. The results here are standard for the trivial bundle case.

Suppose we are given a continuous embedding H ↪ O(D, V) of a Hilbert space H into the Fréchet space O(D, V) of holomorphic sections of the holomorphic vector bundle V → D. The continuity implies in particular that for each y ∈ D the point evaluation map:

H → V_{y}, f ↦ f(y)

is continuous. Then, by the Riesz representation theorem, there exists uniquely K_{H}(·, y) ∈ H ⊗ \overline{V_{y}} such that

(3.2.1)  (f, K_{H}(·, y))_{H} = f(y) for any f ∈ H.

We take an orthonormal basis {φ_{ν}} of H, and expand K_{H} as

(3.2.2)  K_{H}(·, y) = ∑_{ν} a_{ν}(y)φ_{ν}(·).

It follows from (3.2.1) that the coefficient a_{ν}(y) is given by

a_{ν}(y) = (K_{H}(·, y), φ_{ν}(·))_{H} = \overline{φ_{ν}(y)},

and the expansion of K_{H} converges in H. By the continuity H ↪ O(D, V) again, (3.2.2) converges uniformly on each compact set for each fixed y ∈ D. Thus, K_{H}(x, y) is given by the formula:

(3.2.3)  K_{H}(x, y) ≡ K(x, y) = ∑_{ν} φ_{ν}(x)\overline{φ_{ν}(y)} ∈ V_{x} ⊗ \overline{V_{y}} ,

and defines a smooth section of the exterior tensor product bundle V ⊠ \overline{V} → D × D which is holomorphic in the first argument and anti-holomorphic in the second. We will say K_{H} is the reproducing kernel of the Hilbert space H ⊂ O(D, V).

For the convenience of the reader, we pin down basic properties of reproducing kernels for holomorphic vector bundles in a way that we use later.

Lemma 3.2. 1) Let K_{i}(x, y) be the reproducing kernels of Hilbert spaces H_{i} ⊂ O(D, V) with inner products ( , )_{H_{i}}, respectively, for i = 1, 2. If K_{1} ≡ K_{2}, then the two subspaces H_{1} and H_{2} coincide and the inner products ( , )_{H_{1}} and ( , )_{H_{2}} are the same.

2) If K_{1}(x, x) = K_{2}(x, x) for any x ∈ D, then K_{1} ≡ K_{2}.

Proof. 1) Let us reconstruct the Hilbert space H from the reproducing kernel K. For each y ∈ D and v^{∗} ∈ V_{y}^{∗} := \overline{V_{y}}^{∨}, we define ψ(y, v^{∗}) by

ψ(y, v^{∗}) := ⟨K(·, y), v^{∗}⟩ ∈ H.

Here, ⟨ , ⟩ denotes the canonical pairing between \overline{V_{y}} and \overline{V_{y}}^{∨}. We claim that the ℂ-span of {ψ(y, v^{∗}) : y ∈ D, v^{∗} ∈ V_{y}^{∗}} is dense in H. This is because (f, ψ(y, v^{∗}))_{H} = ⟨f(y), v^{∗}⟩ for any f ∈ H by (3.2.1). Thus, the Hilbert space H is reconstructed from the pre-Hilbert structure

(3.2.4)  (ψ(y_{1}, v_{1}^{∗}), ψ(y_{2}, v_{2}^{∗}))_{H} := ⟨K(y_{2}, y_{1}), v_{2}^{∗} ⊗ \overline{v_{1}^{∗}}⟩.

2) We denote by \overline{D} the complex manifold endowed with the conjugate complex structure on the same real manifold D. Then, \overline{V} → \overline{D} is a holomorphic vector bundle, and we can form a holomorphic vector bundle V ⊠\overline{V} → D×\overline{D}. In turn, K_{i}(·, ·) are regarded as its holomorphic sections. As the diagonal embedding ι : D → D × \overline{D}, z ↦ (z, z) is totally real, our assumption (K_{1} − K_{2})|_{ι(D)} ≡ 0 implies K_{1} − K_{2} ≡ 0 by the unicity theorem of holomorphic functions.

3.3 Equivariance of the reproducing kernel

Next, suppose that the Hermitian holomorphic vector bundle V → D is H-equivariant and that (π, H) is a unitary representation of H realized in O(D, V). Let K_{H} be the reproducing kernel of the embedding H ↪ O(D, V). We shall see how the unitarity of (π, H) is reflected in the reproducing kernel K_{H}.

We regard K_{H}(x, x) ∈ V_{x} ⊗ \overline{V_{x}} as an element of End(V_{x}) via the isomorphism (3.1.1). Then, we have:

Lemma 3.3. With the notation (3.1.2) applied to L_{h} : V_{x} → V_{h·x}, we have

K_{H}(h · x, h · x) = (L_{h})_{♯}K_{H}(x, x) for any h ∈ H.

In particular, K_{H}(x, x) ∈ End_{H_{x}}(V_{x}) for any x ∈ D.

Proof. Let {φ_{ν}} be an orthonormal basis of H. Since (π, H) is a unitary representation, {π(h)^{−1}φ_{ν}} is also an orthonormal basis of H for every fixed h ∈ H. Because the formula (3.2.3) of the reproducing kernel is valid for any orthonormal basis, we have

(3.3.1)  K_{H}(x, y) = ∑_{ν}(π(h)^{−1}φ_{ν})(x)\overline{(π(h)^{−1}φ_{ν})(y)}

= ∑_{ν} L_{h^{−1}}φ_{ν}(h · x)\overline{L_{h^{−1}}φ_{ν}(h · y)}

= (L_{h^{−1}} ⊗ \overline{L_{h^{−1}}})K_{H}(h · x, h · y)

for any x, y ∈ D. Hence, (L_{h} ⊗ \overline{L_{h}})K_{H}(x, y) = K_{H}(h · x, h · y) and Lemma follows.

3.4 Diagonalization of the reproducing kernel

The reproducing kernel for a holomorphic vector bundle is a matrix valued section as we have defined in (3.2.3). The multiplicity-free property of the isotropy representation on the fiber diagonalizes the reproducing kernel:

Proposition 3.4. Suppose (π, H) is a unitary representation of H realized in O(D, V). Assume that the isotropy representation of H_{x} on the fiber V_{x} decomposes as a multiplicity-free sum of irreducible representations of H as

V_{x} = ⨁_{i=1}^{n} V_{x}^{(i)}. (Here, n ≡ n(x) may depend on x ∈ D.) Then, the reproducing

kernel is of the form

K_{H}(x, x) = ∑_{i=1}^{n} λ^{(i)}(x) id_{V_{x}^{(i)}}

for some complex numbers λ^{(1)}(x), …, λ^{(n)}(x).

Proof. A direct consequence of Lemma 3.3 and Schur’s lemma.

3.5 Construction of an anti-linear isometry J

In the setting of Theorem 2.2, suppose that σ is an anti-holomorphic bundle endomorphism. We define a conjugate linear map

(3.5.1)  J : O(D, V) → O(D, V), f ↦ σ^{−1} ∘ f ∘ σ,

namely, Jf(x) := σ^{−1}(f(σ(x)) for x ∈ D.

Lemma 3.5. If the conditions (2.2.1) – (2.2.3) are satisfied, then J is an isometry from H onto H for any unitary representation (π, H) realized in O(D, V).

Proof. We define a Hilbert space H̃ := J(H), equipped with the inner product

(Jf_{1}, Jf_{2})H̃ := (f_{2}, f_{1})_{H} for f_{1}, f_{2} ∈ H.

Let us show that the reproducing kernel KH̃ for H̃ coincides with K_{H}. To see this, we take an orthonormal basis {φ_{ν}} of H. Then, {Jφ_{ν}} is an orthonormal basis of H̃, and therefore

KH̃(x, y) = ∑_{ν} Jφ_{ν}(x)\overline{Jφ_{ν}(y)}

= ∑_{ν} σ_{x}^{−1} (φ_{ν}(σ(x))) \overline{σ_{y}^{−1} (φ_{ν}(σ(y)))}

= (σ_{x}^{−1} ⊗ \overline{σ_{y}^{−1}}) K_{H}(σ(x), σ(y)).

For x = y, this formula can be restated as

(3.5.2)  KH̃(x, x) = (σ_{x}^{−1})_{♯}K_{H}(σ(x), σ(x))

with the notation (3.1.2) applied to the unitary map σ_{x}^{−1} : V_{σ(x)} → V_{x}. We fix x ∈ D, and take h ∈ H such that σ(x) = h · x (see (2.2.2)). Then,

(3.5.3)  KH̃(x, x) = (σ_{x}^{−1})_{♯}K_{H}(h · x, h · x) = (σ_{x}^{−1})_{♯}(L_{h})_{♯}K_{H}(x, x).

Here, the last equality follows from Lemma 3.3.

Since the action of H_{x} on V_{x} is multiplicity-free, it follows from Proposition 3.4 that there exist complex numbers λ^{(i)}(x) such that

K_{H}(x, x) = ∑_{i} λ^{(i)}(x) id_{V_{x}^{(i)}} .

Then by (3.1.3) we have

(3.5.4)  (L_{h})_{♯}K_{H}(x, x) = ∑_{i} λ^{(i)}(x) id_{L_{h}(V_{x}^{(i)})} .

Furthermore, since σ_{x}^{−1}(L_{h}(V_{x}^{(i)})) = V_{x}^{(i)} by the assumption (2.2.3), it follows from (3.1.4) that

(3.5.5)  (σ_{x}^{−1})_{♯} (∑_{i} λ^{(i)}(x) id_{L_{h}(V_{x}^{(i)})}) = ∑_{i} λ^{(i)}(x) id_{V_{x}^{(i)}} .

Combining (3.5.3), (3.5.4) and (3.5.5), we get

KH̃(x, x) = K_{H}(x, x).

Then, by Lemma 3.2, the Hilbert space H̃ coincides with H and

(Jf_{1}, Jf_{2})_{H} = (Jf_{1}, Jf_{2})H̃ = (f_{2}, f_{1})_{H} for f_{1}, f_{2} ∈ H.

This is what we wanted to prove.

Remark 3.5.1. In terms of the bundle isomorphism Ψ : V → \overline{σ^{∗}V} (see (2.1.1)), J is given by (Jf)(x) = Ψ_{x}^{−1}(\overline{f(σ(x))}). We note

J^{2} = id on O(D, V)

if σ^{2} = id_{V}, or equivalently, if σ^{2} = id_{D} and \overline{Ψ_{σ(x)}} ∘ Ψ_{x} = id_{V_{x}} for any x ∈ D. However, we do not use this condition to prove Theorem 2.2.

3.6 Proof of Theorem 2.2

As a final step, we need the following lemma which was proved in [2] under the assumption that J^{2} = id and that V → D is the trivial line bundle. For the sake of completeness, we give a proof here.

Lemma 3.6. For A ∈ End_{H}(H), the adjoint operator A^{∗} is given by

(3.6.1)  A^{∗} = JAJ^{−1}.

Proof. We divide the proof into two steps. Step 1 (self-adjoint case). We may and do assume that A − I is positive definite because neither the assumption nor the conclusion changes if we replace A by A + cI (c ∈ ℝ). Here, we note that A + cI is positive definite if c is greater than the operator norm ∥A∥.

From now, assume A ∈ End_{H}(H) is a self-adjoint operator such that A−I is positive definite. We introduce a pre-Hilbert structure on H by

(3.6.2)  (f_{1}, f_{2})_{H_{A}} := (Af_{1}, f_{2})_{H} for f_{1}, f_{2} ∈ H.

Since A − I is positive definite, we have

(f, f)_{H} ≤ (f, f)_{H_{A}} ≤ ∥A∥(f, f)_{H} for f ∈ H.

Therefore, H is still complete with respect to the new inner product ( , )_{H_{A}}. The resulting Hilbert space will be denoted by H_{A}.

If f_{1}, f_{2} ∈ H and g ∈ H, then

(π(g)f_{1}, π(g)f_{2})_{H_{A}} = (Aπ(g)f_{1}, π(g)f_{2})_{H}
= (π(g)Af_{1}, π(g)f_{2})_{H} = (Af_{1}, f_{2})_{H} = (f_{1}, f_{2})_{H_{A}}.

Therefore, π also defines a unitary representation on H_{A}. Applying Lemma 3.5 to both H_{A} and H, we have

(Af_{1}, f_{2})_{H} = (f_{1}, f_{2})_{H_{A}} = (Jf_{2}, Jf_{1})_{H_{A}} = (AJf_{2}, Jf_{1})_{H} = (Jf_{2}, A^{∗}Jf_{1})_{H} = (Jf_{2}, JJ^{−1}A^{∗}Jf_{1})_{H} = (J^{−1}A^{∗}Jf_{1}, f_{2})_{H}.

Hence, A = J^{−1}A^{∗}J. Step 2 (general case). Suppose A ∈ End_{H}(H). Then A^{∗} also commutes with π(g) (g ∈ H) because π is unitary. We put B := \frac{1}{2}(A + A^{∗}) and C := \frac{\sqrt{−1}}{2}(A^{∗}−A). Then, both B and C are self-adjoint operators commuting with π(g) (g ∈ H). It follows from Step 1 that B^{∗} = JBJ^{−1} and C^{∗} = JCJ^{−1}. Since J is conjugate-linear, we have (\sqrt{−1}C)^{∗} = J(\sqrt{−1}C)J^{−1}. Hence, A = B + \sqrt{−1}C also satisfies A^{∗} = JAJ^{−1}.

Proof of Theorem 2.2. Let A, B ∈ End_{H}(H). By Lemma 3.6, we have

AB = J^{−1}(AB)^{∗}J = J^{−1}B^{∗}JJ^{−1}A^{∗}J = BA.

Therefore, the ring End_{H}(H) is commutative.

4 Visible actions on complex manifolds

This section analyzes the geometric condition (2.2.2) on the complex manifold D. We shall introduce the concept of S-visible actions, with which Theorem 2.2 is reformulated in a simpler manner (see Theorem 4.3).

4.1 Visible actions on complex manifolds

Suppose a Lie group H acts holomorphically on a connected complex manifold D.

Definition 4.1. We say the action is S-visible if there exist a subset S of D such that

(4.1.1) D′ := H · S is open in D,

and an anti-holomorphic diffeomorphism σ of D′ satisfying the following two conditions:

(4.1.2) σ|_{S} = id,
(4.1.3) σ preserves every H-orbit in D′.

Remark 4.1.1. The above condition is local in the sense that we may replace S by its subset S′ in Definition 4.1 as far as H · S′ is open in D. Remark 4.1.2. By the definition of D′, it is obvious that

(4.1.4) S meets every H-orbit in D′.

Thus, Definition 4.1 is essentially the same with strong visibility in the sense of [8, Definition 3.3.1]. In fact, the difference is only an additional requirement that S is a smooth submanifold in [8]. We note that if S is a smooth submanifold in Definition 4.1, then S is totally real by the condition (4.1.2), and consequently, the H-action becomes visible (see [8, Theorem 4.3]).

4.2 Compatible automorphism

Retain the setting of Definition 4.1. Suppose σ is an anti-holomorphic diffeomorphism of D′. Twisting the original H-action by σ, we can define another holomorphic action of H on D′ by

D′ → D′, x ↦ σ(h · σ^{−1}(x)).

If this action can be realized by H, namely, if there exists a group automorphism σ̃ of H such that

σ̃(h) · x = σ(h · σ^{−1}(x)) for any x ∈ D′,

we say σ̃ is compatible with σ. This condition is restated simply as

(4.2.1)  σ̃(h) · σ(y) = σ(h · y) for any y ∈ D′.

Definition 4.2. We say an S-visible action has a compatible automorphism of the transformation group H if there exists an automorphism σ̃ of the group H satisfying the condition (4.2.1).

We remark that the condition (4.1.3) follows from (4.1.1) and (4.1.2) if there exists σ̃ satisfying (4.2.1). In fact, any H-orbit in D′ is of the form H · x for some x ∈ S, and then

σ(H · x) = σ̃(H) · σ(x) = H · x

by (4.1.2) and (4.2.1).

Suppose V → D is an H-equivariant holomorphic vector bundle. If there is a compatible automorphism σ̃ of H with an anti-holomorphic diffeomorphism σ on D, then we have the following isomorphism:

(\overline{σ^{∗}V})_{h·y} ≃ \overline{V}_{σ(h·y)} = \overline{V}_{σ̃(h)·σ(y)} for h ∈ H and y ∈ D.

Therefore, we can let H act equivariantly on the holomorphic vector bundle \overline{σ^{∗}V} → D by defining the left translation on \overline{σ^{∗}V} as

L_{h}^{σ} : (\overline{σ^{∗}V})_{y} → (\overline{σ^{∗}V})_{h·y}

via the identification with the left translation \overline{L_{σ̃(h)}} : \overline{V}_{σ(y)} → \overline{V}_{σ̃(h)·σ(y)}. Then, the two H-equivariant holomorphic vector bundles V and \overline{σ^{∗}V} are isomorphic if and only if σ lifts to an anti-holomorphic bundle endomorphism σ (we use the same letter) which respects the H-action in the sense that

(4.2.2)  L_{σ̃(h)} ∘ σ = σ ∘ L_{h} on V for any h ∈ H.

4.3 Propagation of multiplicity-free property

By using the concept of S-visible actions, we give a second form of our main theorem as follows:

Theorem 4.3. Let V → D be an H-equivariant Hermitian holomorphic vector bundle. Assume the following three conditions are satisfied:

(4.3.1)

(Base space) The action on the base space D is S-visible with a compatible automorphism of the group H (Definition 4.2).

(4.3.2)

(Fiber) The isotropy representation of H_{x} on V_{x} is multiplicity-free for any x ∈ S.

We write its irreducible decomposition as

V_{x} = ⨁_{i=1}^{n(x)} V_{x}^{(i)}.

(4.3.3)

(Compatibility) σ lifts to an anti-holomorphic endomorphism (we use the same letter σ) of the H-equivariant Hermitian holomorphic vector bundle V such that

(4.3.3)(a)  σ_{x}(V_{x}^{(i)}) = V_{x}^{(i)} for 1 ≤ i ≤ n(x), x ∈ S.

Then, any unitary representation which is realized in O(D, V) is multiplicityfree.

The difference from the previous conditions (2.2.1) and (2.2.3) in Theorem 2.2 is that the conditions (4.3.2) and (4.3.3)(a) concern only with the slice S, while we had to deal with the whole D (or at least its open subset) in Theorem 2.2.

Remark 4.3.1. We can sometimes find a slice S such that the isotropy subgroup H_{x} is independent of generic x ∈ S. Bearing this in mind, we set

H_{S} := ⋂_{x∈S} H_{x}
= {g ∈ H : gx = x for any x ∈ S}.

Theorem 4.3 still holds if we replace H_{x} with H_{S} (see also Remark 2.2.1(2)).

Proof. We shall reduce Theorem 4.3 to Theorem 2.2 by using the H-equivariance of the bundle endomorphism σ. Let us show that the conditions (2.2.1), (2.2.2) and (2.2.3) are satisfied for the H-invariant open subset D′ := H · S of D.

First we observe that the condition (4.1.3) implies (2.2.2) because σ(x) ∈ σ(H · x) = H · x for any x ∈ D′.

Next, take any element x ∈ D′ and we write x = h · x_{0} (h ∈ H, x_{0} ∈ S). We set

V_{x}^{(i)} := L_{h}(V_{x_{0}}^{(i)}) (1 ≤ i ≤ n(x_{0})).

Through the group isomorphism H_{x_{0}} ⟶^{∼} H_{x}, l ↦ hlh^{−1} and the left translation L_{h} : V_{x_{0}} → V_{x}, we get the isomorphism between the two isotropy representations, H_{x_{0}} → GL(V_{x_{0}}) and H_{x} → GL(V_{x}), because L_{hlh^{−1}} = L_{h}∘L_{l}∘L_{h}^{−1} (l ∈ H_{x_{0}}). In particular, the direct sum

V_{x} = ⨁_{i=1}^{n(x_{0})} V_{x}^{(i)}

gives a multiplicity-free decomposition of irreducible representations of H_{x}. Hence, the condition (2.2.1) is satisfied for all x ∈ D′.

Finally, we set g := σ̃(h)h^{−1} ∈ H. As σ(x_{0}) = x_{0}, we have

σ(x) = σ(h · x_{0}) = σ̃(h) · σ(x_{0}) = σ̃(h) · x_{0} = g · x.

Besides, we have for any i (1 ≤ i ≤ n(x) = n(x_{0})),

σ_{x}(V_{x}^{(i)}) = σ_{x}(L_{h}(V_{x_{0}}^{(i)}))
= L_{σ̃(h)}(σ_{x_{0}}(V_{x_{0}}^{(i)}))  by (4.2.2)
= L_{σ̃(h)} (V_{x_{0}}^{(i)})  by (4.3.3)(a)
= L_{σ̃(h)h^{−1}}L_{h}(V_{x_{0}}^{(i)})
= L_{g}(V_{x}^{(i)}).

Hence, the condition (2.2.3) holds for any x ∈ D′. Therefore, all the assumptions of Theorem 2.2 are satisfied for the open subset D′. Now, Theorem 4.3 follows from Theorem 2.2 and Remark 2.2.1 (1).

5 Multiplicity-free theorem for associated bundles

This section provides a third form of our multiplicity-free theorem (see Theorem 5.3). It is intended for actual applications to group representation theory, especially to branching problems. The idea here is to reformalize the geometric condition of Theorem 4.3 (second form) in terms of the representation of the structure group of an equivariant principal bundle.

Theorem 5.3 is used as a main machinery in [7, 8] (referred to as [7, Theorem 1.3] and [8, Theorem 2], of which we have postponed the proof to this article) for various multiplicity-free theorems including the following cases:

• tensor product representations of GL(n) [7, Theorem 3.6],

• branching problems for GL(n) ↓ GL(n_{1}) × GL(n_{2}) × GL(n_{3}) ([7, Theorem 3.4]),

• Plancherel formulae for vector bundles over Riemannian symmetric spaces ([8, Theorems 21 and 30]).

5.1 Automorphisms on equivariant principal bundles

We begin with the setting where a Hermitian holomorphic vector bundle V over a connected complex manifold D is given as the associated bundle V ≃ P ×_{K} V to the following data (P, K, μ, V):

K is a Lie group,

ϖ : P → D is a principal K-bundle,

V is a finite dimensional Hermitian vector space, μ : K → GL_{ℂ}(V) is a unitary representation.

Suppose that a Lie group H acts on P from the left, commuting with the right action of K. Then H acts also on the Hermitian vector bundle V → D by automorphisms.

We take p ∈ P, and set x := ϖ(p) ∈ D. If h ∈ H_{x}, then ϖ(hp) = h · x = x = ϖ(p). Therefore, there is a unique element of K, denoted by i_{p}(h), such that

(5.1.1)  hp = p i_{p}(h).

The correspondence h ↦ i_{p}(h) gives rise to a Lie group homomorphism i_{p} : H_{x} → K. We set

(5.1.2)  H_{(p)} := i_{p}(H_{x}).

Then, H_{(p)} is a subgroup of K.

Definition 5.1. By an automorphism of the H-equivariant principal Kbundle ϖ : P → D, we mean that there exist a diffeomorphism σ : P → P and Lie group automorphisms σ : K → K and σ : H → H (by a little abuse of notation, we use the same letter σ) such that

(5.1.3) σ(hpk) = σ(h)σ(p)σ(k) (h ∈ H, k ∈ K, p ∈ P).

The condition (5.1.3) immediately implies:

(5.1.4) σ induces an action (denoted again by σ) on P/K ≃ D,

(5.1.5) the induced action σ on D is compatible with σ ∈ Aut(H) (see (4.2.1) for the definition).

We write P^{σ} for the set of fixed points by σ, that is,

P^{σ} := {p ∈ P : σ(p) = p}.

Then, we have:

Lemma 5.1. σ(H_{(p)}) = H_{(p)} if p ∈ P^{σ}.

Proof. Take h ∈ H_{x}. Applying σ to the equations h · x = x (∈ D) and hp = pi_{p}(h) (∈ P), we have σ(h) · x = x and σ(h)p = pσ(i_{p}(h)) from (5.1.3). Hence, σ(h) ∈ H_{x} and i_{p}(σ(h)) = σ(i_{p}(h)). Therefore, σ(H_{x}) ⊂ H_{x} and σ(H_{(p)}) ⊂ H_{(p)}. Likewise, σ^{−1}(H_{x}) ⊂ H_{x} and σ^{−1}(H_{(p)}) ⊂ H_{(p)}. Hence, we have proved σ(H_{x}) = H_{x} and σ(H_{(p)}) = H_{(p)}.

5.2 Multiplicity-free theorem

For a representation μ of K, we denote by μ^{∨} the contragredient representation of μ. It is isomorphic to the conjugate representation \overline{μ} if μ is unitary.

Proposition 5.2. Retain the setting of Subsection 5.1. Assume that there exist an automorphism σ of the H-equivariant principal K-bundle ϖ : P → D such that

(5.2.1) the induced action of σ on D is anti-holomorphic,

and a subset B of P^{σ} satisfying the following two conditions:

(5.2.2) HBK contains a non-empty open subset of P.

The restriction μ|_{H_{(b)}} is multiplicity-free as an H_{(b)}-module for any b ∈ B.

(5.2.3)

We write its irreducible decomposition as μ|_{H_{(b)}} ≃ ⨁_{i=1}^{n} ν_{b}^{(i)}. Further, we assume: (5.2.4) (a) μ ∘ σ ≃ μ^{∨} as K-modules.

(5.2.4) (b) For any b ∈ B and i, ν^{(i)} ∘ σ ≃ ν^{(i)}^{∨} as H_{(b)}-modules.

Then, any unitary representation of H that is realized in O(D, V) is multiplicityfree.

The proof of Proposition 5.2 is given in Section 6

Remark 5.2.1. Loosely, the conditions (5.2.2) and (5.2.3) mean that the holomorphic bundle V → D cannot be ‘too large’, with respect to the transformation group H. The remaining condition (5.2.4) is often automatically fulfilled (e.g. Corollary 5.4).

Remark 5.2.2. As in Remark 2.2.1, Proposition 5.2 still holds if H_{(b)} is replaced by its arbitrary subgroup H′_{(b)} for each b ∈ B in (5.2.3) and (5.2.4) (b).

Remark 5.2.3. For a connected compact Lie group K, the condition (5.2.4) (a) is satisfied for any finite dimensional representation μ of K if we take σ ∈ Aut(K) to be a Weyl involution. We recall that σ is a Weyl involution if there exists a Cartan subalgebra 𝔱 of the Lie algebra 𝔨 of K such that dσ = − id on 𝔱. It is noteworthy that any simply-connected compact Lie group admits a Weyl involution.

5.3 Multiplicity-free theorem (third form)

In the assumption of Proposition 5.2, the subgroups H_{(b)} may depend on b (see (5.2.3) and (5.2.4) (b)). For actual applications, we give a weaker but simpler form by taking just one subgroup M instead of a family of subgroups H_{(b)}.

For a subset B of P, we define the following subgroup M_{H}(B) of K:

M_{H}(B) := {k ∈ K : for each b ∈ B, there is h ∈ H such that hb = bk} (5.3.1)
= ⋂_{b∈B} K_{Hb} ,

where K_{Hb} denotes the isotropy subgroup at Kb in the left coset space H\P, which is acted on by K from the right. Then M_{H}(B) is σ-stable if B ⊂ P^{σ}, as is readily seen from (5.1.3).

Theorem 5.3. Assume that there exist an automorphism σ of the H-equivariant principal K-bundle ϖ : P → D satisfying (5.2.1) and a subset B of P^{σ} with the following three conditions (5.3.2) – (5.3.4): Let M := M_{H}(B).

(5.3.2) HBK contains a non-empty open subset of P.

(5.3.3) The restriction μ|_{M} is multiplicity-free.

We shall write its irreducible decomposition as μ|_{M} ≃ ⨁_{i=1}^{n} ν^{(i)}.

(5.3.4) (a) μ ∘ σ ≃ μ^{∨} as representations of K.

(5.3.4) (b) ν^{(i)} ∘ σ ≃ ν^{(i)}^{∨} as representations of M for any i (1 ≤ i ≤ n).

Then, any unitary representation of H which is realized in O(D, V) is multiplicityfree.

Remark 5.3.1. Theorem 5.3 still holds if we replace M with an arbitrary σ-stable subgroup of M_{H}(B) to verify the conditions (5.3.3) and (5.3.4) (b).

Assuming Proposition 5.2, we first complete the proof of Theorem 5.3.

Proof of Theorem 5.3. In view of Proposition 5.2 and Remark 5.2.2, it is sufficient to show M_{H}(B) ⊂ H_{(b)} for any b ∈ B.

To see this, take any k ∈ M_{H}(B). By the definition (5.3.1), there exists h ∈ H such that hb = bk. Then, h ∈ H_{ϖ(b)}. Since i_{b}(h) ∈ K is characterized by the property hb = b i_{b}(h) (see (5.1.1)), k coincides with i_{b}(h). Hence, k = i_{b}(h) ∈ i_{b}(H_{ϖ(b)}) = H_{(b)} (see (5.1.2)). Thus, we have proved M_{H}(B) ⊂ H_{(b)} for any b ∈ B.

5.4 Line bundle case

In general, the condition (5.3.2) tends to be fulfilled if B is large, while the condition (5.3.3) tends to be fulfilled if B is small (namely, if M is large). However, we do not have to consider the condition (5.3.3) if V → D is a line bundle. Hence, by taking B to be maximal, that is, by setting B := P^{σ}, we get:

Corollary 5.4. Suppose we are in the setting of Subsection 5.1. Suppose furthermore that K is connected and dim μ = 1. Assume that there exists an automorphism σ of the H-equivariant principal K-bundle ϖ : P → D satisfying (5.2.1) and the following two conditions:

(5.4.1) dσ = − id on the center 𝔠(𝔨) of the Lie algebra 𝔨 of K.

(5.4.2) HP^{σ}K contains a non-empty open subset of P.

Then, any unitary representation which can be realized in O(D, V) is multiplicityfree.

Proof of Corollary. As we mentioned, we apply Theorem 5.3 with B := P^{σ}. The condition (5.3.3) is trivially satisfied because dim μ = 1.

Let us show μ ∘ σ = μ^{∨}. We write K = [K, K] · C, where [K, K] is the commutator subgroup and C = exp(𝔠(𝔨)). Since [K, K] is semisimple, it acts trivially on the one dimensional representations μ ∘ σ and μ^{∨}. By (5.4.1), μ ∘ σ(e^{X}) = μ(e^{−X}) = μ^{∨}(e^{X}) for any X ∈ 𝔠(𝔨). Hence μ ∘ σ = μ^{∨} both on [K, K] and C. Therefore, the condition (5.3.4) (a) holds. Then, (5.3.4) (b) also holds. Therefore, Corollary follows from Theorem 5.3.

5.5 Multiplicity-free branching laws

So far, we have not assumed that P has a group structure. Now, we consider the case that P is a Lie group which we denote by G, and that H and K are closed subgroups of G. This framework enables us to apply Theorem 5.3 to the restriction of representations of G (constructed on G/K) to its subgroup H. Applications of Corollary 5.5 include multiplicity-free branching theorems of highest weight representations for both finite and infinite dimensional cases (see [7, 8, 9]).

We denote the centralizer of B in H ∩ K by

Z_{H∩K}(B) := {l ∈ H ∩ K : lbl^{−1} = b for any b ∈ B}.

Corollary 5.5. Suppose D = G/K carries a G-invariant complex structure, and V = G ×_{K} V is a G-equivariant holomorphic vector bundle over D associated to a unitary representation μ : K → GL(V). We assume there exist an automorphism σ of the Lie group G stabilizing H and K such that the induced action on D = G/K is anti-holomorphic, and a subset B of G^{σ} satisfying the conditions (5.3.2), (5.3.3), and (5.3.4) (a) and (b) for P := G and M := Z_{H∩K}(B). Then, any unitary representation of H which can be realized in the G-module O(D, V) is multiplicity-free.

Proof. Since Z_{H∩K}(B) is contained in M_{H}(B) by the definition (5.3.1), Corollary 5.5 is a direct consequence of Theorem 5.3 and Remark 5.3.1.

6 Proof of Proposition 5.2

This section gives a proof of Proposition 5.2 by showing that all the conditions of Theorem 4.3 are fulfilled. Then, the proof of our third form (Theorem 5.3) will be completed.

6.1 Verification of the condition (4.3.1)

Suppose we are in the setting of Proposition 5.2. Then, HBK contains a non-empty open subset of P, and consequently ϖ(HBK) contains a nonempty open subset, say W, of D. By taking the union of H-translates of W, we get an H-invariant open subset D′ := H · W of D. We set

S := D′ ∩ ϖ(B).

Then, D′ = H · S. Besides, σ|_{S} = id because B ⊂ P^{σ}. Thus, the H-action on D is S-visible with a compatible automorphism σ of H by (5.1.3) in the sense of Definition 4.2. Thus, the condition (4.3.1) holds for D′.

6.2 Verification of the condition (4.3.2)

Next, let us prove that V_{x} is multiplicity-free as an H_{x}-module for all x in S.

Let V ≃ P ×_{K} V be the associated bundle, and P ×V → V, (p, v) ↦ [p, v] by the natural quotient map. For p ∈ P we set x := ϖ(p) ∈ D. Then, we can identify the fiber V_{x} with V by the bijection

(6.2.1)  ι_{p} : V ⟶^{∼} V_{x}, v ↦ [p, v].

Via the bijection (6.2.1) and a group homomorphism i_{p} : H_{x} → H_{(p)}, the isotropy representation of H_{x} on V_{x} factors through the representation μ : H_{(p)} → GL(V), namely, the following diagram commutes for any l ∈ H_{x}:

(6.2.2)  \begin{aligned}V & ⟶_{ι_{p}}^{∼} & V_{x} \\ ^{μ(i_{p}(l))}↓ & & ↓^{L_{l}} \\ V & ⟶_{ι_{p}}^{∼} & V_{x} \end{aligned}

Now, suppose x ∈ S. We take b ∈ B such that x = ϖ(b).

According to (5.2.3), we decompose V as a multiplicity-free sum of irreducible representations of H_{(b)}, for which we write

(6.2.3)  μ = ⨁_{i=1}^{n} ν_{b}^{(i)}, V = ⨁_{i=1}^{n} V_{b}^{(i)}.

Then, it follows from (6.2.2) that if we set V_{x}^{(i)} := ι_{b}(V_{b}^{(i)}), then

(6.2.4)  V_{x} = ⨁_{i=1}^{n} V_{x}^{(i)}

is an irreducible decomposition as an H_{x}-module. Hence, (4.3.2) is verified.

6.3 Verification of the condition (4.3.3)

Third, let us construct an isomorphism Ψ : V → \overline{σ^{∗}V}. According to the assumption (5.2.4) (a), there exists a K-intertwining isomorphism, denoted by ψ : V → \overline{V}, between the two representations μ and \overline{μ ∘ σ}. As the vector bundle V → D is associated to the data (P, K, μ, V), so is the vector bundle σ^{∗}V → D to the data (P, K, \overline{μ ∘ σ}, \overline{V}). Hence the map

P × V → P × \overline{V} , (p, v) ↦ (p, ψ(v))

induces the bundle isomorphism

(6.3.1)  Ψ : V ⟶^{∼} \overline{σ^{∗}V}.

In other words, the conjugate linear map defined by

(6.3.2)  φ : V → V, v ↦ \overline{ψ(v)}

satisfies

μ(σ(k)) ∘ φ = φ ∘ μ(k) for k ∈ K.

Hence, we can define an anti-holomorphic endomorphism of V by

V → V, [p, v] ↦ [σ(p), φ(v)].

This endomorphism, denoted by the same letter σ, is a lift of the antiholomorphic map σ : D → D, and satisfies (4.2.2) because of (5.1.3).

Besides, for x = ϖ(p), we have

(6.3.3)  ι_{σ(p)} ∘ φ = σ_{x} ∘ ι_{p} .

Finally, let us verify the condition (4.3.3)(a). Step 1. First, let us show

(6.3.4)  φ(V_{b}^{(i)}) = V_{b}^{(i)} for 1 ≤ i ≤ n.

Bearing the inclusion H_{(b)} ⊂ K in mind, we consider the representation \overline{μ ∘ σ} : K → GL(\overline{V}) and its subrepresentation realized on ψ(V_{b}^{(i)}) (⊂ \overline{V}) as an H_{(b)}-module. Then, this is isomorphic to (ν_{b}^{(i)}, V_{b}^{(i)}) as H_{(b)}-modules because ψ : V → \overline{V} intertwines the two representations μ and \overline{μ ∘ σ} of K. On the other hand, it follows from the irreducible decomposition (6.2.3) that the representation \overline{μ ∘ σ} when restricted to the subspace \overline{V_{b}^{(i)}} is isomorphic

to \overline{ν_{b}^{(i)} ∘ σ} as H_{(b)}-modules. By our assumption (5.2.4) (b), ν_{b}^{(i)} is isomorphic

to \overline{ν_{b}^{(i)} ∘ σ}, which occurs in \overline{V} exactly once. Therefore, the two subspaces

ψ(V_{b}^{(i)}) and \overline{V_{b}^{(i)}} must coincide. Hence, we have (6.3.4) by (6.3.2). Step 2. Next we show that (4.3.3)(a) holds for x = ϖ(b) if b ∈ B. We note that σ(b) = b and σ(x) = x. Then, it follows from (6.3.3) and (6.3.4) that

σ_{x} ∘ ι_{b}(V_{b}^{(i)}) = ι_{σ(b)} ∘ φ(V_{b}^{(i)}) = ι_{σ(b)}(V_{b}^{(i)}) = ι_{b}(V_{b}^{(i)}).

Since V_{x}^{(i)} = ι_{b}(V_{b}^{(i)}), we have proved σ_{x}(V_{b}^{(i)}) = V_{b}^{(i)}.

Hence, (4.3.3)(a) holds.

Thus, all the conditions of Theorem 4.3 hold for D′. Therefore, Proposition 5.2 follows from Theorem 4.3 and Remark 2.2.1 (2). Hence, the proof of Theorem 5.3 is completed.

References

[1] G. van Dijk and S. C. Hille, Canonical representations related to hyperbolic spaces, J. Funct. Anal., 147 (1997), 109–139.

[2] J. Faraut and E. G. F. Thomas, Invariant Hilbert spaces of holomorphic functions, J. Lie Theory, 9 (1999), 383–402.

[3] V. Guillemin and S. Sternberg, Multiplicity-free spaces, J. Differential Geom., 19 (1984), 31–56.

[4] A. T. Huckleberry and T. Wurzbacher, Multiplicity-free complex manifolds, Math. Ann., 286 (1990), 261–280.

[5] S. Kobayashi, Irreducibility of certain unitary representations, J. Math. Soc. Japan, 20 (1968), 638–642.

[6] T. Kobayashi, Multiplicity-free theorem in branching problems of unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (ed. K. Mimachi), (1997), 9–17.

[7] , Geometry of multiplicity-free representations of GL(n), visible actions on flag varieties, and triunity, Acta Appl. Math., 81 (2004), 129–146.

[8] , Multiplicity-free representations and visible actions on complex manifolds, Publ. RIMS, 41 (2005), 497–549 (a special issue of Publications of RIMS commemorating the fortieth anniversary of the founding of the Research Institute for Mathematical Sciences).

[9] , Multiplicity-free theorems of the restriction of unitary highest weight modules with respect to reductive symmetric pairs, to appear in Progr. Math., Birkhäuser.

[10] , Visible actions on symmetric spaces, preprint.

[11] , A generalized Cartan decomposition for the double coset space (U(n_{1}) × U(n_{2}) × U(n_{3}))\U(n)/(U(p) × U(q)), preprint.

[12] G. W. Mackey, Induced representations of locally compact groups I, Annals of Math., 55 (1952), 101–139.

[13] K.-H. Neeb, On some classes of multiplicity free representations, Manuscripta Math., 92 (1997), 389–407.

[14] F. Podestà and G. Thorbergsson, Polar and coisotropic actions on Kähler manifolds, Trans. Amer. Math. Soc., 354 (2002), 1759–1781.

[15] W. Schmid, Die Randwerte holomorphe Funktionen auf hermetisch symmetrischen Raumen, Invent. Math., 9 (1969-70), 61–80.

[16] J. R. Stembridge, Multiplicity-free products of Schur functions, Ann. Comb., 5 (2001), 113–121.

[17] D. A. Vogan, Jr., Unitarizability of certain series of representations, Ann. Math., 120 (1984), 141-187.

[18] N. R. Wallach, On the unitarizability of derived functor modules, Invent. Math., 78 (1984), 131–141.

[19] J. Wolf, Representations that remain irreducible on parabolic subgroups. Differential geometrical methods in mathematical physics (Proc. Conf., Aixen-Provence/Salamanca, 1979), pp. 129–144, Lecture Notes in Math., 836, Springer, Berlin, 1980.