A generalized Cartan decomposition for the double coset space

(U(n_{1}) × U(n_{2}) × U(n_{3}))\U(n)/(U(p) × U(q))

Toshiyuki Kobayashi ^{∗} Research Institute for Mathematical Sciences, Kyoto University

Abstract

Motivated by recent developments on visible actions on complex manifolds, we raise a question whether or not the multiplication of three subgroups L, G′ and H surjects a Lie group G in the setting that G/H carries a complex structure and contains G′/G′ ∩ H as a totally real submanifold.

Particularly important cases are when G/L and G/H are generalized flag varieties, and we classify pairs of Levi subgroups (L, H) such that LG′H = G, or equivalently, the real generalized flag variety G′/H ∩ G′ meets every L-orbit on the complex generalized flag variety G/H in the setting that (G, G′) = (U(n), O(n)). For such pairs (L, H), we introduce a herringbone stitch method to find a generalized Cartan decomposition for the double coset space L\G/H, for which there has been no general theory in the non-symmetric case. Our geometric results provides a unified proof of various multiplicity-free theorems in representation theory of general linear groups.

∗Partly supported by Grant-in-Aid for Exploratory Research 16654014, Japan Society of the Promotion of Science Keywords and phrases: Cartan decomposition, double coset space, multiplicity-free representation, semisimple Lie group, homogeneous space, visible action, flag variety 2000MSC : primary 22E4; secondary 32A37, 43A85, 11F67, 53C50, 53D20 e-mail address: toshi@kurims.kyoto-u.ac.jp

Contents

1 Introduction and statement of main results . . . . . . . . . . . . 2 2 Symmetric case . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3 Non-symmetric case 1: min(n_{1}, n_{2}, n_{3}) = 1 . . . . . . . . . . . . 8 4 Non-symmetric case 2: min(n_{1}, n_{2}, n_{3}) ≥ 2 . . . . . . . . . . . . 12 5 Non-symmetric case 3: min(p, q) = 1 . . . . . . . . . . . . . . . . 13 6 Proof of Theorem A . . . . . . . . . . . . . . . . . . . . . . . . . 15 7 Visible actions on generalized flag varieties . . . . . . . . . . . . 24 8 Applications to representation theory . . . . . . . . . . . . . . . 25

1 Introduction and statement of main results

1.1. Our object of study is the double coset space L\G/H, where L ⊂ G ⊃ H are a triple of reductive Lie groups.

In the ‘symmetric case’ (namely, both (G, L) and (G, H) are symmetric pairs), the theory of the Cartan decomposition G = LBH or its variants gives an explicit description of the double coset decomposition L\G/H (e.g., [3, 4, 14, 15, 16, 17]). However, in the general case where one of the pairs (G, L) and (G, H) is non-symmetric, there is no known structure theory on the double cosets L\G/H even for a compact Lie group.

Motivated by the recent works of ‘visible actions’ (Definition 7.1) on complex manifolds [9, 11] and multiplicity-free representations (e.g. the classification of multiplicity-free tensor product representations of GL(n), see [8, 19]), we have come to realize the importance of understanding the double cosets L\G/H in the non-symmetric case such as

(G, L, H) = (U(n), U(n_{1})×U(n_{2})×⋯×U(n_{k}), U(m_{1})×⋯×U(m_{l})), (1.1)

where n = n_{1} + ⋯ + n_{k} = m_{1} + ⋯ + m_{l}.

In this article, we initiate the study of the double cosets L\G/H in the ‘non-symmetric and visible case’ by taking (1.1) as a test case, and develop new techniques in finding an explicit decomposition for L\G/H.

For this, first we single out triples that give rise to visible actions. Theorem A gives a classification of the triples (L, G, H) such that G has the decomposition G = LG′H where G′ = O(n), or equivalently, any L-orbit on the complex generalized flag variety G/H ≃ B_{m_{1},...,m_{l}}(ℂ^{n}) (see (1.5)) intersects with its real form B_{m_{1},...,m_{l}}(ℝ^{n}). The proof uses an idea of invariant theory arising from quivers. The classification includes some interesting nonsymmetric cases such as k = 3, l = 2 and min(n_{1} +1, n_{2} +1, n_{3} +1, m_{1}, m_{2}) = 2. Then, the L-action on G/H (and likewise, the H-action on G/L) becomes visible in the sense of [9] (see Definition 7.1).

Second, we confine ourselves to these triples, and prove an analog of the Cartan decomposition

G = LBH

by finding an explicit subset B in G′ such that generic points of B form a J-transversal totally real slice (Definition 7.1) for the L-action on G/H of minimal dimension (see Theorem B). The novelty of our method in the non-symmetric case is an idea of ‘herringbone stitch’ (see Section 3).

1.2. To explain the perspectives of our generalization of the Cartan decomposition, let us recall briefly classic results on the double cosets L\G/H in the symmetric case. A prototype is a theorem due to H. Weyl: let K be a connected compact Lie group, and T a maximal toral subgroup. Then,

any element of K is conjugate to an element of T.  (1.2)

We set G := K ×K, A := {(t, t^{−1}) : t ∈ T } and identify K with the subgroup diag(K) := {(k, k) : k ∈ K} of G. Then, the statement (1.2) is equivalent to the double coset decomposition:

G = KAK.  (1.3)

In the above case, (G, K) = (K×K, diag(K)) forms a compact symmetric pair. More generally, the decomposition (1.3) still holds for a Riemannian symmetric pair (G, K) by taking A ≃ 𝕋^{k} (G/K: compact type) or A ≃ ℝ^{k} (G/K: non-compact type) where k = rank G/K. Such a decomposition is known as the Cartan decomposition for symmetric spaces.

A further generalization of the Cartan decomposition has been developed over the decades under the hypothesis that both (G, L) and (G, H) are symmetric pairs. For example, Hoogenboom [4] gave an analog of the Cartan decomposition for L\G/H = (U(l) × U(m))\U(n)/(U(p) × U(q)) (l + m = p + q = n) by finding a toral subgroup 𝕋^{k} as its representatives, where k = min(l, m, p, q). This result is generalized by Matsuki [16], showing that there exists a toral subgroup B in G such that

G = LBH  (1.4)

if G is compact (see Fact 2.1). Analogous decomposition also holds in the case where G is a non-compact reductive Lie group and L is its maximal compact subgroup, by taking a non-compact abelian subgroup B of dimension rank_{ℝ} G/H (see Flensted-Jensen [3]).

1.3. Before explaining a new direction of study in the non-symmetric case, we pin down some remarkable aspects on the Cartan decomposition in the symmetric case from algebraic, geometric, and analytic viewpoints.

Algebraically, finding nice representatives of the double coset is relevant to the reduction theory, or the theory of normal forms. For example, the Cartan decomposition (1.3) for (G, K) = (GL(n, ℝ), O(n)) corresponds to the diagonalization of symmetric matrices by orthogonal transformations. The case G = G′ × G′, L = H = diag(G′) with G′ = GL(n, ℂ) is equivalent to the theory of Jordan normal forms.

Geometrically, (1.4) means that every L-orbit on the (pseudo-)Riemannian symmetric space G/H meets the flat totally geodesic submanifold B/B ∩ H. The decomposition (1.4) is also used in the construction of a G-equivariant compactification of the symmetric space G/H (see [2] for a survey on various compactifications).

Analytically, the Cartan decomposition is particularly important in the analysis of asymptotic behavior of global solutions to G-invariant differential equations on the symmetric space G/H (e.g. [3, 4, 6]).

1.4. Now, we consider the non-symmetric case L ⊂ G ⊃ H. Unlike the symmetric case, we cannot expect the existence of an abelian subgroup B such that LBH contains an interior point of G in general, as is easily observed by the argument of dimensions (e.g. [6, Introduction]). Instead, we raise here the following question:

Question 1.1. Does there exist a ‘nice’ subgroup G′ such that LG′H contains an open subset of G?

For a compact G, one may strengthen Question 1.1 as follows:

Question 1.1′. Does there exist a ‘nice’ subgroup G′ such that G = LG′H?

The decomposition G = LG′H means that the double coset space L\G/H can be controlled by a subgroup G′.

What is a ‘nice’ subgroup G′? In contrast to the previous case that G/H carries a G-invariant Riemannian structure and that the abelian subgroup G′ = B (see (1.4)) gives a flat totally geodesic submanifold of G/H, we are interested in the case that G/H carries a G-invariant complex structure and that the subgroup G′ gives a totally real submanifold G′/G′ ∩ H of G/H. In the latter case, the L-action on G/H is said to be previsible ([9, Definition 3.1.1]) if LG′H contains an open subset of G.

1.5. Let us state our main results. Suppose we are in the setting (1.1) and consider Question 1.1′ for

G′ := O(n).

In this setting, we shall give a necessary and sufficient condition for the multiplication map L × G′ × H → G to be surjective.

In order to clarify its geometric meaning, let B_{m_{1},...,m_{l}}(ℂ^{n}) denote the complex (generalized) flag variety:

{(V_{1}, …, V_{l}) : {0} = V_{0} ⊂ V_{1} ⊂ V_{2} ⊂ ⋯ ⊂ V_{l−1} ⊂ V_{l} = ℂ^{n},
dim V_{i} = m_{1} + ⋯ + m_{i} (1 ≤ i ≤ l)}.  (1.5)

Likewise, the real (generalized) flag variety B_{m_{1},...,m_{l}}(ℝ^{n}) is defined and becomes a totally real submanifold of B_{m_{1},...,m_{l}}(ℂ^{n}).

Theorem A. Let k, l ≥ 2 and n = n_{1} +⋯+n_{k} = m_{1} +⋯+m_{l} be partitions of n by positive integers. Let

(G, L, H) = (U(n), U(n_{1}) × U(n_{2}) × ⋯ × U(n_{k}), U(m_{1}) × ⋯ × U(m_{l})).

We set N := min(n_{1}, …, n_{k}) and M := min(m_{1}, …, m_{l}). Then the following five conditions are equivalent:

i) G = LG′H. Here, G′ := O(n).

ii) B_{m_{1},...,m_{l}}(ℝ^{n}) × B_{n_{1},...,n_{k}}(ℝ^{n}) meets every G-orbit on B_{m_{1},...,m_{l}}(ℂ^{n}) × B_{n_{1},...,n_{k}}(ℂ^{n}) by the diagonal action.

ii)′ B_{m_{1},...,m_{l}}(ℝ^{n}) meets every L-orbit on B_{m_{1},...,m_{l}}(ℂ^{n}).

ii)′′ B_{n_{1},...,n_{k}}(ℝ^{n}) meets every H-orbit on B_{n_{1},...,n_{k}}(ℂ^{n}).

iii) One of the following conditions holds:

0) k = 2,  l = 2,
I) k = 3, N = 1, l = 2,
II) k = 3, N ≥ 2, l = 2, M = 2,
III)  l = 2, M = 1,
I′) k = 2,  l = 3, M = 1,
II′) k = 2, N = 2, l = 3, M ≥ 2,
III′) k = 2, N = 1.

Remark 1.5.1. Both (G, L) and (G, H) are symmetric pairs if and only if (k, l) = (2, 2), namely, (G, L, H) is in Case 0.

Remark 1.5.2. The condition (ii) implies that the L-action on G/H is previsible (see Definition 7.1). We shall see in Section 7 that this action is (strongly) visible, too (see also [9, Corollary 17]).

Remark 1.5.3. A holomorphic action of a complex reductive group on a complex manifold D is called spherical if its Borel subgroup has an open orbit on D. We shall see in Section 8 that the condition (ii) in Theorem A is equivalent to:

iv) B_{m_{1},...,m_{l}}(ℂ^{n}) × B_{n_{1},...,n_{k}}(ℂ^{n}) is a spherical variety of G_{ℂ} := GL(n, ℂ).

See Littelmann [13] for the statement (iv) in the case k = l = 2, namely, Case 0 in (iii).

Remark 1.5.4. An isometric action of a compact Lie group L on a Riemannian manifold is called polar if there exists a submanifold that meets every L-orbit orthogonally. Among Cases 0∼III′ in Theorem A, the L-action on G/H is polar if and only if (G, L, H) is in Case 0 (see [1]).

1.6. Suppose one of (therefore, all of) the equivalent conditions in Theorem A is satisfied. As a finer structural result of the double coset decomposition L\G/H, we shall construct a fairly simple subset B of G′ = O(n) such that the multiplication map L × B × H → G is still surjective, according to Cases 0 ∼ III of Theorem A. We omit Cases I′ ∼ III′ below because these are essentially the same with Cases I ∼ III. For Case I below, we may and do assume n_{1} = 1 without loss of generalities.

Theorem B (generalized Cartan decomposition). Let

(G, L, H) = (U(n), U(n_{1}) × ⋯ × U(n_{k}), U(p) × U(q)).

Then, there exists B ⊂ O(n) such that G = LBH, where B is of the following form:

B ≃ \begin{cases}𝕋^{min(n_{1},n_{2},p,q)} & (Case 0), \\ 𝕋^{min(p,q,n_{2},n_{3})} · 𝕋^{min(p,q,n_{2}+1,n_{3}+1)} & (Case I), \\ 𝕋^{2} · 𝕋 · 𝕋^{2} & (Case II), \\ \underbrace{𝕋 · · · · · 𝕋}_{k−1} & (Case III). \end{cases}

Here, 𝕋^{a} · 𝕋^{b} means a subset of the form {xy ∈ G : x ∈ 𝕋^{a}, y ∈ 𝕋^{b}} for some toral subgroups 𝕋^{a} and 𝕋^{b}.

We note that B is no longer a subgroup of G in Cases I, II and III.

1.7. This article is organized as follows. First, we give a proof of Theorem B (generalized Cartan decomposition G = LBH) by constructing explicitly the subset B of O(n). This is done in Theorems 2.2 (Case 0), 3.1 (Case I), 4.1 (Case II), and 5.1 (Case III), respectively by using an idea of herringbone stitch. This also gives a proof of the implication (iii) ⇒ (i) in Theorem A. The remaining implications of Theorem A are proved in Section 6. An application to representation theory is discussed in Section 8.

Theorem A was announced and used in [8, Theorem 3.1] and [9, Theorem 16] and its proof was postponed until this article. Theorem B was presented in the Oberwolfach workshop on “Finite and Infinite Dimensional Complex Geometry and Representation Theory”, organized by A. T. Huckleberry, K.-H. Neeb, and J. A. Wolf, February 2004. The author thanks the organizers for a wonderful and stimulating atmosphere of the workshop.

2 Symmetric case

This section reviews a well-known fact on the Cartan decomposition for the symmetric case. The results here will be used in the non-symmetric case (Sections 3, 4, and 5) as a ‘stitch’ (see Diagram 3.1, for example). Theorem 2.2 corresponds to Theorem B in Case 0, which is proved here.

First, we recall from [4, Theorem 6.10], [15, Theorem 1] the following:

Fact 2.1. Let G be a connected, compact Lie group with Lie algebra 𝔤. Suppose that τ and θ are two involutive automorphisms of G, and that L and H are open subgroups of G^{τ} and G^{θ}, respectively. We take a maximal abelian subspace 𝔟 in 𝔤^{−τ,−θ} := {X ∈ 𝔤 : τX = θX = −X}, and write B for the connected abelian subgroup with Lie algebra 𝔟. Then G = LBH.

Now, let us consider the setting:

(G, L, H) = (U(n), U(n_{1}) × U(n_{2}), U(p) × U(q)),  (2.1)

where n = n_{1}+n_{2} = p+q. Then, both (G, L) and (G, H) are symmetric pairs. In fact, if we set I_{p,q} := diag(1, …, 1, −1, …, −1) and define an involution θ by θ(g) := I_{p,q}gI_{p,q}^{−1}, then H = G^{θ}. Likewise, L = G^{τ} if we set τ(g) := I_{n_{1},n_{2}}gI_{n_{1},n_{2}}^{−1}.

We set

l := min(n_{1}, n_{2}, p, q)  (2.2)

and define an abelian subspace:

𝔟 := ∑_{i=1}^{l} ℝ(E_{i,n+1−i} − E_{n+1−i,i}).

Then, 𝔟 is a maximal abelian subspace in 𝔤^{−τ,−θ}, and B := exp(𝔟) is a toral subgroup of O(n). Now, applying Fact 2.1, we obtain:

Theorem 2.2. G = LBH.

3 Non-symmetric case 1: min(n_{1}, n_{2}, n_{3}) = 1

In Sections 3, 4, and 5, we give an explicit decomposition formula for the double coset space L\G/H in Cases I, II, and III of Theorem B, respectively, and complete the proof of Theorem B, and therefore that of the implication (iii) ⇒ (ii) of Theorem A.

The distinguishing feature of these three sections is that we are dealing with the non-symmetric pair (G, L), for which there is no known general theory on the double coset decomposition L\G/H of a compact Lie group G. We shall introduce a method of herringbone stitch (see Diagram 3.1) consisting of symmetric triples G_{2i+1} ⊂ G_{2i} ⊃ H_{i} and L_{i} ⊂ G_{2i+1} ⊃ G_{2i+2} (i = 0, 1, 2, …) such that the iteration of the double coset decomposition of G_{2i+1}\G_{2i}/H_{i} and L_{i}\G_{2i+1}/G_{2i+2} keeps on toward a finer structure of L\G/H = L_{0}\G_{0}/H_{0}.

This section treats the most interesting case for Theorem B, namely,

(G, L, H) = (U(n), U(n_{1}) × U(n_{2}) × U(n_{3}), U(p) × U(q))  (3.1)

where min(n_{1}, n_{2}, n_{3}) = 1. Theorem 3.1 below corresponds to Case I of Theorem B.

Without loss of generality, we may and do assume n_{1} = 1. Thus, L = U(1) × U(n_{2}) × U(n_{3}) (n_{2} + n_{3} = n − 1). We set

l := min(2p, 2q, 2n_{2} + 1, 2n_{3} + 1).  (3.2)

A simple computation shows [\frac{l}{2}] = min(p, q, n_{2}, n_{3}) and [\frac{l+1}{2}] = min(p, q, n_{2}+ 1, n_{3} + 1). We define two abelian subspaces:

𝔟′ := ∑_{i=1}^{[ \frac{l+1}{2} ]} ℝ(E_{i,n+1−i} − E_{n+1−i,i}),

𝔟′′ := ∑_{i=1}^{[ \frac{l}{2} ]} ℝ(E_{i+1,n+1−i} − E_{n+1−i,i+1}).

Then B′ := exp 𝔟′ and B′′ := exp 𝔟′′ are toral subgroups of dimension [\frac{l+1}{2}] and [\frac{l}{2}], respectively. We define a subset of O(n) by

B := B′′B′.  (3.3)

For the sake of simplicity, we shall write also B = 𝕋^{[ \frac{l}{2} ]} · 𝕋^{[ \frac{l+1}{2} ]}. We note that B is a compact manifold of dimension l = [\frac{l}{2}] + [\frac{l+1}{2}] because B is diffeomorphic to the homogeneous space (B′ × B′′)/(B′ ∩ B′′) and B′ ∩ B′′ is a finite subgroup.

We are ready to describe the double coset decomposition for L\G/H in the case (3.1).

Theorem 3.1 (generalized Cartan decomposition). G = LBH, where B ≃ 𝕋^{min(p,q,n_{2},n_{3})} · 𝕋^{min(p,q,n_{2}+1,n_{3}+1)}.

Proof. First, for i ≥ 1, we define one dimensional toral subgroups by

B_{i} := exp ℝ(E_{i,n+1−i} − E_{n+1−i,i}),
C_{i} := exp ℝ(E_{i+1,n+1−i} − E_{n+1−i,i+1}).

Then, B_{1}, B_{2}, ⋯, and B_{[ \frac{l+1}{2} ]} commute with each other, and we have

B′ = B_{1}B_{2} ⋯ B_{[ \frac{l+1}{2} ]} = B_{[ \frac{l+1}{2} ]} ⋯ B_{2}B_{1}.

Likewise, B′′ = C_{1}C_{2} ⋯ C_{[ \frac{l}{2} ]}.

For m ≥ 1, we define an embedding of a one dimensional torus into G by

ι_{m} : 𝕋 → G, a ↦ diag(\underbrace{a, …, a}_{[ \frac{m+1}{2} ]}, \underbrace{1, …, 1}_{n−m}, \underbrace{a, …, a}_{[ \frac{m}{2} ]}).

We write T_{m} for its image, and define a subgroup G_{m} of G by

G_{m} := T_{m} × U(n − m).

Here, we regard U(n − m) as a subgroup of G by identifying with \{I_{[ \frac{m+1}{2} ]}\} × U(n − m) × \{I_{[ \frac{m}{2} ]}\}. (I_{m} stands for the unit matrix of degree m.) It is convenient to set T_{0} = {e} and G_{0} = G. Then, we have a decreasing sequence of subgroups:

G = G_{0} ⊃ G_{1} ⊃ G_{2} ⊃ · · · .

The point of our definition of G_{m} is that G_{2i} commutes with all of B_{1}, ⋯, B_{i}, C_{1}, ⋯, C_{i−1}. G_{2i+1} commutes with all of B_{1}, ⋯, B_{i}, C_{1}, ⋯, C_{i}.

Next, for i ≥ 0, we define the following subgroups:

H_{i} := H ∩ G_{2i} ≃ T_{2i} × U(p − i) × U(q − i),
L_{i} := L ∩ G_{2i+1} ≃ T_{2i+1} × U(n_{2} − i) × U(n_{3} − i).

The following obvious properties play a crucial role in the inductive step below.

H = H_{0} ⊃ H_{1} ⊃ H_{2} ⊃ ⋯; H_{i} commutes with B_{1}, …, B_{i},  (3.4)
L = L_{0} ⊃ L_{1} ⊃ L_{2} ⊃ ⋯; L_{i} commutes with C_{1}, …, C_{i}.  (3.5)

With these preparations, let us proceed the proof of Theorem 3.1 along a herringbone stitch consisting of triples (G_{2i+1}, G_{2i}, H_{i}) and (L_{i}, G_{2i+1}, G_{2i+2}) (i = 0, 1, 2, ⋯):

\begin{aligned}& & L_{0} & & & & L_{1} & & & & L_{2} & & & & L_{3} & & & & & ⋯ \\ & & & ⊂ & & & & ⊂ & & & & ⊂ & & & & ⊂ \\ & & & & G_{1} & & & & G_{3} & & & & G_{5} & & & & G_{7} & & & ⋯ \\ & & & ⊃ & & ⊃ & & ⊃ & & ⊃ & & ⊃ & & ⊃ & & ⊃ & & ⊃ \\ & & G_{0} & & & & G_{2} & & & & G_{4} & & & & G_{6} & & & & G_{8} & ⋯ \\ & ⊂ & & & & ⊂ & & & & ⊂ & & & & ⊂ & & & & ⊂ \\ H_{0} & & & & H_{1} & & & & H_{2} & & & & H_{3} & & & & H_{4} & & & ⋯ \end{aligned} Diagram 3.1

We claim that each triple has the following decomposition formula:

G_{2i} = G_{2i+1}B_{i+1}H_{i},  (3.6)
G_{2i+1} = L_{i}C_{i+1}G_{2i+2}.  (3.7)

To see this, we first take away the trivial factor from G_{2i} and G_{2i+1}, respectively. Then, the following bijections hold for i = 0, 1, …,

G_{2i+1}\G_{2i}/H_{i} ≃ (U(1) × U(n − 2i − 1))\U(n − 2i)/(U(p − i) × U(q − i)),
L_{i}\G_{2i+1}/G_{2i+2} ≃ (U(n_{2} − i) × U(n_{3} − i))\U(n − 2i − 1)/(U(n − 2i − 2) × U(1)).

Since the right-hand side is the double coset space by symmetric subgroups, we can apply Theorem 2.2. Thus, (3.6) and (3.7) have been proved.

By using (3.6) and (3.7) iteratively, together with the commutating properties (3.4) and (3.5), we obtain

G = G_{0} = G_{1}B_{1}H_{0}
= (L_{0}C_{1}G_{2})B_{1}H
= LC_{1}(G_{3}B_{2}H_{1})B_{1}H
= LC_{1}G_{3}B_{2}B_{1}H
= ⋯
= LC_{1} ⋯ C_{i}G_{2i}B_{i} ⋯ B_{1}H  (3.8)
= LC_{1} ⋯ C_{i}G_{2i+1}B_{i+1} ⋯ B_{1}H.  (3.9)

If min(p, q) ≤ min(n_{2}, n_{3}), then this equation terminates with G_{2i} = H_{i} when i reaches min(p, q). Then, i = [\frac{l}{2}] = [\frac{l+1}{2}], and we have G = LB′′B′H_{i}H = LBH from (3.8). If min(p, q) > min(n_{2}, n_{3}), then this equation terminates with G_{2i+1} = L_{i} when i reaches min(n_{2}, n_{3}). Then, i = [\frac{l}{2}] and i + 1 = [\frac{l+1}{2}], and we have G = LL_{i}B′′B′H = LBH from (3.9). Hence, we have completed the proof of Theorem.

4 Non-symmetric case 2: min(n_{1}, n_{2}, n_{3}) ≥ 2

In this section we study the double coset space L\G/H in Case II of Theorem B, that is, the non-symmetric case:

(G, L, H) = (U(n), U(n_{1}) × U(n_{2}) × U(n_{3}), U(p) × U(q)),

where n = n_{1} + n_{2} + n_{3} = p + q, min(n_{1}, n_{2}, n_{3}) ≥ 2 and min(p, q) = 2. Without loss of generality, we may and do assume p = 2.

We define three abelian subspaces:

𝔞_{1} := ℝ(E_{1,n} − E_{n,1}) + ℝ(E_{2,n−1} − E_{n−1,2}),
𝔞′ := ℝ(E_{n_{1}+1,n−1} − E_{n−1,n_{1}+1}),
𝔞_{2} := ℝ(E_{n_{1}+1,n} − E_{n,n_{1}+1}) + ℝ(E_{n_{1}+2,n−1} − E_{n−1,n_{1}+2}),

and correspondingly three toral subgroups by A_{1} := exp 𝔞_{1}, A′ := exp 𝔞′, and A_{2} := exp 𝔞_{2}. We then set

B := A_{2}A′A_{1}.  (4.1)

Note that B is a five dimensional subset of O(n), but is no more a subgroup.

Here is a generalized Cartan decomposition for L\G/H in the non-symmetric setting min(n_{1}, n_{2}, n_{3}) ≥ 2 and min(p, q) = 2:

Theorem 4.1. G = LBH.

Proof. The proof again uses herringbone ‘stitch’, of which each stitch is a special case of Theorem 3.1 or Theorem 2.2, respectively.

Step 1. We define a subgroup G_{1} of G by

G_{1} := U(n_{1}) × U(n_{2} + n_{3}).

Since both (G, H) and (G, G_{1}) are symmetric pairs, we have

G = G_{1}A_{1}H  (4.2)

by Theorem 2.2. We observe from (2.2) that A_{1} is of dimension min(2, n − 2, n_{1}, n_{2} + n_{3}) = 2.

Step 2. We define a subgroup H_{1} of G by

H_{1} := Z_{H∩G_{1}}(A_{1}) ≃ Δ(𝕋^{2}) × U(n_{1} − 2) × U(n_{2} + n_{3} − 2).

Here, Δ(𝕋^{2}) := {diag(a, b, 1, ⋯, 1, b, a) : a, b ∈ 𝕋}. Then, taking away the first factor inclusion U(n_{1}), we have

L\G_{1}/H_{1} ≃ (U(n_{2}) × U(n_{3}))\U(n_{2} + n_{3})/(U(n_{2} + n_{3} − 2) × U(1) × U(1)).

Applying Theorem 3.1 to the right-hand side, we obtain

G_{1} = LA_{2}A′H_{1}.  (4.3)

We note that A_{2}A′ is of dimension min(2n_{2}, 2n_{3}, 2(n_{2} + n_{3} − 2) + 1, 3) = 3, as explained in Section 3.

Combining (4.2) and (4.3), we obtain

G = G_{1}A_{1}H
= (LA_{2}A′H_{1})A_{1}H
= LA_{2}A′A_{1}H
= LBH

because H_{1} commutes with A_{1}. Thus, we have shown Theorem 4.1.

5 Non-symmetric case 3: min(p, q) = 1

This section treats the double coset space L\G/H in Case III of Theorem B, that is, the non-symmetric case:

(G, L, H) = (U(n), U(n_{1}) × ⋯ × U(n_{k}), U(1) × U(n − 1))

for an arbitrary partition n = n_{1} + ⋯ + n_{k}. In this case, although the pair (G, L) is non-symmetric, the symmetric pair (G, H) gives a very simple homogeneous space, namely, G/H ≃ ℙ^{n−1}ℂ. Thus, it is much easier to find an explicit decomposition G = LBH than the previous cases in Sections 3 and 4.

For 1 ≤ i ≤ k − 1, we set

H_{i} := −E_{1,n_{1}+···+n_{i}+1} + E_{n_{1}+···+n_{i}+1,1}
B_{i} := exp(ℝH_{i}) (≃ 𝕋),

and define a (k − 1)-dimensional subset B in G′ = O(n) by

B := B_{1} ⋯ B_{k−1}.

We note that B is not a group if k ≥ 3. The subset B is contained in the subgroup O(k) of O(n) in an obvious sense.

Then, we have

Theorem 5.1. G = LBH, where B = \underbrace{𝕋 · · · · · 𝕋}_{k−1} (⊂ O(n)).

Proof. We shall work on the L-action on G/H ≃ ℙ^{n−1}ℂ. First, we observe that the map

U(n_{i}) × ℝ → ℂ^{n_{i}}, (h_{i}, a_{i}) ↦ h_{i} ^{t}(a_{i}, 0, …, 0)

is surjective for each i (1 ≤ i ≤ k). Therefore, the following map

U(n_{1}) × ⋯ × U(n_{k}) × ℙ^{k−1}ℝ → ℙ^{n−1}ℂ,  (5.1)

((h_{1}, …, h_{k}), [a_{1} : ⋯: a_{k}]) ↦ [h_{1} ^{t}(a_{1}, 0, …, 0) : ⋯: h_{k} ^{t}(a_{k}, 0, …, 0)]

is also surjective.

Next, an elementary matrix computation shows

exp(θ_{1}H_{1}) ⋯ exp(θ_{k−1}H_{k−1})^{t}(1, 0, …, 0)
= ^{t}(a_{1}, …, 0, a_{2}, …, 0, …, a_{k}, 0, …, 0),

where

( a_{1} ; a_{2} ; a_{3} ; ⋮ ; a_{k−1} ; a_{k}) = (\begin{matrix}cos θ_{1} cos θ_{2} & ⋯ & cos θ_{k−2} cos θ_{k−1} \\ sin θ_{1} cos θ_{2} & ⋯ & cos θ_{k−2} cos θ_{k−1} \\ sin θ_{2} & ⋯ & cos θ_{k−2} cos θ_{k−1} \\ & ... & ⋮ \\ & & sin θ_{k−2} cos θ_{k−1} \\ & & sin θ_{k−1} \end{matrix}) ∈ ℝ^{k}\\{0\}.

Hence, B · [1 : 0 : ⋯: 0] = ℙ^{k−1}ℝ.

Combining with the surjective map L × ℙ^{k−1}ℝ → G/H, we have shown that the multiplication map L × B × H → G is surjective.

Now, Theorems 2.2, 3.1, 4.1 and 5.1 show that we have completed the proof of Theorem B.

6 Proof of Theorem A

This section gives a proof of Theorem A.

Suppose we are in the setting of Theorem A. The equivalence (i) ⇔ (ii) (likewise, (i) ⇔ (ii)′ and (i) ⇔ (ii)′′) is clear from the following natural identifications:

G′/G′ ∩ H ≃ B_{m_{1},...,m_{l}}(ℝ^{n}), G/H ≃ B_{m_{1},...,m_{l}}(ℂ^{n}).

Further, Theorem B shows (iii) ⇒ (i).

Thus, the rest of this section is devoted to the proof of the implication (i) ⇒ (iii).

We begin with a question about when the multiplication map

L × G′ × H → G, (l, g′, h) ↦ lg′h  (6.1)

is surjective, in the general setting that G = U(n), G′ = O(n), and H is a Levi subgroup G. Our key machinery to find a necessary condition for the surjectivity of (6.1) is Lemma 6.3. Let us explain briefly the ideas of our strategy that manages the three non-commutative subgroups L, G′ and H:

L ⋯ finding L-invariants (invariant theory),
G′ ⋯ using the geometric property (G′ gives real points in G/H),
H ⋯ realizing H as the isotropy subgroup of the G-action.

For this, we take J ∈ M(n, ℝ) and consider the adjoint orbit:

G/G_{J} ≃ {Ad(g)J : g ∈ G} ⊂ M(n, ℂ),

where Ad(g)J := gJg^{−1} and

G_{J} := {g ∈ G : gJ = Jg}.

Later, we shall choose J such that H is conjugate to G_{J} by an element of G′. Here, we note that the surjectivity of the map (6.1) remains unchanged if we replace H with aHa^{−1} and L with bHb^{−1} (a, b ∈ G′). Then, the following observation:

G′G_{J}/G_{J} ≃ {Ad(g)J : g ∈ G′} ⊂ M(n, ℝ)

will be used in Lemma 6.1 (‘management’ of G′ ≃ O(n)), while an invariant theory will be used in Lemma 6.2 (‘management’ of L ≃ U(n_{1})×⋯×U(n_{k})).

Lemma 6.1. Let J ∈ M(n, ℝ), G′ = O(n), and L a subgroup of G = U(n). If there exists g ∈ G such that

Ad(L)(Ad(g)J) ∩ M(n, ℝ) = ∅,  (6.2)

then G ⫌ LG′G_{J}.

Proof. First we observe that

Ad(G′G_{J})J = Ad(G′)J ⊂ M(n, ℝ).

Then, the condition (6.2) implies Ad(Lg)J ∩ Ad(G′G_{J})J = ∅, whence Lg ∩ G′G_{J} = ∅. Therefore, g ∉ LG′G_{J}.

Next, we fix a partition n = n_{1} + ⋯ + n_{k}, and find a sufficient condition for (6.2) in the setting:

L = U(n_{1}) × ⋯ × U(n_{k}).

For this, we fix l ≥ 2 and take a loop i_{0} → i_{1} → ⋯ → i_{l} consisting of non-negative integers 1, …, k such that

i_{0} = i_{l}, i_{a−1} ≠ i_{a} (a = 1, 2, …, l).  (6.3)

Now, let us introduce a non-linear map:

A_{i_{0}···i_{l}} : M(n, ℂ) → M(n_{i_{0}}, ℂ)

as follows: Let P ∈ M(n, ℂ), and we write P as (P_{ij})_{1≤i,j≤k} in block matrix form such that the (i, j)-block P_{ij} ∈ M(n_{i}, n_{j}; ℂ). We set P̃_{ij} ∈ M(n_{i}, n_{j}; ℂ) by

P̃_{ij} := \begin{cases}P_{ij} & (i < j), \\ P_{ji}^{∗} & (i > j). \end{cases}

Then, A_{i_{0}···i_{l}}(P) is defined by

A_{i_{0}···i_{l}}(P) := P̃_{i_{0}i_{1}} P̃_{i_{1}i_{2}} ⋯ P̃_{i_{l−1}i_{l}}.  (6.4)

Lemma 6.2. If there exists a loop i_{0} → i_{1} → ⋯ → i_{l} (= i_{0}) such that at least one of the coefficients of the characteristic polynomial det(λI_{n_{i_{0}}} − A_{i_{0}···i_{l}}(P)) is not real, then

Ad(L)P ∩ M(n, ℝ) = ∅.

Later, we shall take P to be Ad(g)J and apply this lemma to the following loops:

1) 1 → 2 → 3 → 1,

A_{1231} = P_{12}P_{23}P_{13}^{∗}  (6.5)

2) 1 → 3 → 2 → 4 → 1,

A_{13241} = P_{13}P_{23}^{∗}P_{24}P_{14}^{∗}  (6.6)

3) 1 → 4 → 2 → 4 → 3 → 4 → 1,

A_{1424341} = P_{14}P_{24}^{∗}P_{24}P_{34}^{∗}P_{34}P_{14}^{∗}  (6.7)

Proof. For a block diagonal matrix l = (\begin{matrix}l_{1} \\ & l_{2} \\ & & . & . & . \\ & & & & & l_{k} \end{matrix}) ∈ L, the transform

P ↦ Ad(l)P induces that of the (i, j)-block matrix:

P_{ij} ↦ l_{i}P_{ij}l_{j}^{−1} (1 ≤ i, j ≤ k).

Then P_{ij}^{∗} is transformed as P_{ij}^{∗} ↦ (l_{i}P_{ij}l_{j}^{−1})^{∗} = l_{j}P_{ij}^{∗}l_{i}^{−1}. Hence, P̃_{ij} is transformed as

P̃_{ij} ↦ l_{i}P̃_{ij}l_{j}^{−1} (1 ≤ i, j ≤ k),

and then A_{i_{0}···i_{l}}(P) is transformed as

A_{i_{0}···i_{l}}(P) ↦ l_{i_{0}}A_{i_{0}···i_{l}}(P)l_{i_{0}}^{−1}.

Therefore, the characteristic polynomial of A_{i_{0}···i_{l}}(P) is invariant under the transformation P ↦ Ad(l)P. In particular, if Ad(L)P ∩ M(n, ℝ) ≠ ∅, then det(λI_{n_{i_{0}}} − A_{i_{0}···i_{l}}(P)) ∈ ℝ[λ]. By contraposition, Lemma 6.2 follows.

Here is a key machinery to show the implication (i) ⇒ (iii) in Theorem A:

Lemma 6.3. Let n = n_{1}+⋯+n_{k} be a partition, and L = U(n_{1})×⋯×U(n_{k}) be the natural subgroup of G = U(n). Suppose J is of a block diagonal matrix:

J := (\begin{matrix}J_{1} \\ & J_{2} \\ & & . & . & . \\ & & & & & J_{k} \end{matrix}) ∈ M(n, ℝ),  (6.8)

where J_{i} ∈ M(n_{i}, ℝ) (1 ≤ i ≤ k). If there exist a skew Hermitian matrix X ∈ 𝔲(n) and a loop i_{0} → i_{1} → ⋯ → i_{l} (= i_{0}) (see (6.3)) such that

det(λI_{n_{i_{0}}} − A_{i_{0}···i_{l}}([X, J])) ∉ ℝ[λ],

then the multiplication map L × G′ × G_{J} → G is not surjective. Here, we recall G′ = O(n).

Proof. We set P (ε) := Ad(exp(εX))J. In view of Lemmas 6.1 and 6.2, it is sufficient to show

det(λI_{n_{i_{0}}} − A_{i_{0}···i_{l}}(P (ε))) ∉ ℝ[λ]

for some ε > 0. We set Q := [X, J]. The matrix P (ε) depends real analytically on ε, and we have

P (ε) = J + εQ + O(ε^{2}),

as ε tends to 0. In particular, the (i, j)-block matrix P_{ij}(ε) (∈ M(n_{i}, n_{j}; ℂ)) satisfies

P_{ij}(ε) = εQ_{ij} + O(ε^{2}) (ε → 0)

for i ≠ j. Then, we have

det(λI_{n_{i_{0}}} − A_{i_{0}···i_{l}}(P (ε)))
= det(λI_{n_{i_{0}}} − ε^{l}Q̃_{i_{0}i_{1}} ⋯ Q̃_{i_{l−1}i_{l}} + O(ε^{l+1}))
= det(λI_{n_{i_{0}}} − ε^{l}A_{i_{0}···i_{l}}(Q) + O(ε^{l+1}))
= ∑_{r=0}^{n_{i_{0}}} λ^{n_{i_{0}}−r}ε^{rl}h_{r}(ε),  (6.9)

where h_{r}(ε) (0 ≤ r ≤ n_{i_{0}}) are real analytic functions of ε such that

det(λI_{n_{i_{0}}} − A_{i_{0}···i_{l}}(Q)) = ∑_{r=0}^{n_{i_{0}}} λ^{n_{i_{0}}−r}h_{r}(0).

From our assumption, this polynomial is not of real coefficients, namely, there exists r such that h_{r}(0) ∉ ℝ. It follows from (6.9) that det(λI_{n_{i_{0}}} − A_{i_{0}···i_{l}}(P (ε))) ∉ ℝ[λ] for any sufficiently small ε > 0. Hence, we have shown Lemma.

For the applications of Lemma 6.3 below, we shall take a specific choice of a skew Hermitian matrix X ∈ 𝔲(n). According to the partition n = n_{1} + ⋯ + n_{k}, we write X = (X_{ij})_{1≤i,j≤k} as a block form. We note that the (i, j) block of Q = [X, J] is given by

Q_{ij} = X_{ij}J_{j} − J_{i}X_{ij} .  (6.10)

We also note that if J ∈ M(n, ℝ) is a diagonal matrix whose entries consist of non-negative integers 1, 2, …, l such that

#{1 ≤ a ≤ n : J_{aa} = i} = m_{i} (1 ≤ i ≤ l),

then G_{J} is conjugate to U(m_{1}) × U(m_{2}) × ⋯ × U(m_{l}) by an element of G′ = O(n). Since the surjectivity of (6.1) remains unchanged if we take the conjugation of H by G′, we can apply Lemma 6.3 to study the surjectivity of (6.1) in the setting (1.1).

Now, we apply Lemma 6.3 to show the following four propositions:

Proposition 6.4. Let n = n_{1} + n_{2} + n_{3} = m_{1} + m_{2} + m_{3} be partitions of n by positive integers. We define (natural) subgroups L and H of G = U(n) by

L := U(n_{1}) × U(n_{2}) × U(n_{3}),
H := U(m_{1}) × U(m_{2}) × U(m_{3}),

and G′ := O(n). Then, G ⫌ LG′H.

In the following three propositions, we set

(G, L, H) = (U(n), U(n_{1}) × ⋯ × U(n_{k}), U(p) × U(q)) and G′ = O(n)

where n = n_{1} + n_{2} + ⋯ + n_{k} = p + q.

Proposition 6.5. G ⫌ LG′H if

min(p, q) ≥ 3, k = 3 and min(n_{1}, n_{2}, n_{3}) ≥ 2.  (6.11)

Proposition 6.6. G ⫌ LG′H if

min(p, q) = 2 and k ≥ 4.  (6.12)

Proposition 6.7. G ⫌ LG′H if

min(p, q) ≥ 3 and k ≥ 4.  (6.13)

Proof of Proposition 6.4. We consider the partition n = n_{1} + n_{2} + n_{3} and the loop 1 → 2 → 3 → 1. We take J = diag(j_{1}, …, j_{n}) ∈ M(n, ℝ) to be a diagonal matrix with the following two properties:

j_{1} = 1, j_{n_{1}} = 2, j_{n_{1}+n_{2}+1} = 3,
#{k : j_{k} = i} = m_{i} (1 ≤ i ≤ 3).

Then, the isotropy subgroup G_{J} is conjugate to H by an element of O(n). We fix z ∈ ℂ and define a skew Hermitian matrix X = (X_{ij})_{1≤i,j≤3} ∈ 𝔲(n) as follows:

X_{12} = ( 0^{1}) , X_{23} = ( 0^{1}) , X_{13} = ( 0^{z}) .

Then it follows from (6.10) that Q = [X, J] has the following block entries:

Q_{12} = ( 0^{1}) , Q_{23} = ( 0^{1}) , Q_{13}^{∗} = ( 0^{2\overline{z}}) ,

and therefore we have

A_{1231}(Q) = Q_{12}Q_{23}Q_{13}^{∗} = ( 0^{2\overline{z}}) .

Thus, det(λI_{n_{1}} − A_{1231}(Q)) = λ^{n_{1}} − 2\overline{z}λ^{n_{1}−1}. This does not have real coefficients if we take z ∉ ℝ. Hence, Proposition follows from Lemma 6.3.

Proof of Proposition 6.5. We shall apply Lemma 6.3 with k = 3 and the loop 1 → 2 → 3 → 1. In light of the assumption (6.11), an elementary consideration shows that there exist positive integers p_{i}, q_{i} (1 ≤ i ≤ 3) satisfying the following equations:

n = n_{1} + n_{2} + n_{3}
= = = =
p = p_{1} + p_{2} + p_{3}
+ + + +
q = q_{1} + q_{2} + q_{3}

We set

J_{i} := ( \begin{matrix}1 ... 1 & & p_{i} \\ 0 ... 0 & & q_{i} \end{matrix} ) ∈ M(n_{i}, ℝ) (1 ≤ i ≤ 3).

Then G_{J} ≃ U(p_{1} + p_{2} + p_{3}) × U(q_{1} + q_{2} + q_{3}) = H, where J is defined as in (6.8).

Now, let us take a specific choice of X ∈ 𝔲(n) as follows: for z ∈ ℂ,

X_{12} := ( 0_{1}^{1}) , X_{23} := ( 0_{1}^{1}) , X_{13} := ( 0_{z}^{1}) .

This is possible because n_{i} ≥ 2 (i = 1, 2, 3). Then the (i, j)-block of Q := [X, J] amounts to

Q_{12} = ( 0_{1}^{−1}) , Q_{23} = ( 0_{1}^{−1}) , Q_{13} = ( 0_{z}^{−1}) .

Associated to the loop 1 → 2 → 3 → 1, we have

A_{1231}(Q) = Q_{12}Q_{23}Q_{13}^{∗} = ( 0_{1}^{−\overline{z}}) .

Hence, det(λI_{n_{1}} − A_{1231}(Q)) = λ^{n_{1}} + λ^{n_{1}−2}\overline{z} ∉ ℝ[λ] if we take z ∉ ℝ. Now, Proposition 6.5 follows from Lemma 6.3.

Proof of Proposition 6.6. With the notation as in (6.8), we define J by setting k = 4 and

J_{1} = ( 0^{1}) , J_{2} = ( 0^{1}) , J_{3} = 0, J_{4} = 0.

Then the isotropy subgroup G_{J} is conjugate to H ≃ U(2) × U(n − 2) by an element of G′ = O(n). We consider the loop 1 → 3 → 2 → 4 → 1. Then it follows from (6.10) that

A_{13241}(Q) = Q_{13}Q_{23}^{∗}Q_{24}Q_{14}^{∗}
= (−J_{1}X_{13})(−J_{2}X_{23})^{∗}(−J_{2}X_{24})(−J_{1}X_{14})^{∗}

= (\begin{matrix}(a⃗, b⃗)(d⃗, c⃗) & 0 & ⋯ & 0 \\ 0 \\ ⋮ & & 0 \\ 0 \end{matrix}) ,

where a⃗, b⃗ ∈ ℂ^{n_{3}} denote the first row vectors of X_{13}, X_{23} and c⃗, d⃗ ∈ ℂ^{n_{4}} denote the first row vectors of X_{14}, X_{24}, respectively. Since n_{1}, n_{2}, n_{3} and n_{4} are positive integers, we can find X ∈ 𝔲(n) such that (a⃗, b⃗)(d⃗, c⃗) ∉ ℝ.

Then

det(λI_{n_{1}} − A_{13241}(Q)) = λ^{n_{1}−1}(λ − (a⃗, b⃗)(d⃗, c⃗)) ∉ ℝ[λ].

Thus Proposition 6.6 follows from Lemma 6.3.

Proof of Proposition 6.7. Without loss of generality, we may and do assume 1 ≤ n_{1} ≤ n_{2} ≤ ⋯ ≤ n_{k}. We give a proof according to the following three cases:

Case 1) k ≥ 5.

Case 2) k = 4 and n_{3} > 1.

Case 3) k = 4 and n_{3} = 1.

Case 1) Suppose k ≥ 5. Then, n_{5} + ⋯ + n_{k} > 1. In fact, if it were not the case, we would have k = 5 and n_{5} = 1, which would imply n_{1} = n_{2} = ⋯ = n_{5} = 1 and n = 5. But, this contradicts to the assumption n = p + q ≥ 3 + 3 = 6. Now, we apply Proposition 6.5 to L′ = U(n_{1} + n_{2}) × U(n_{3} + n_{4}) × U(n_{5} + ⋯ + n_{k}) (⊃ L), and conclude

G ⫌ L′G′H ⊃ LG′H.

Case 2) Suppose k = 4 and n_{3} > 1. Then, 2 ≤ n_{3} ≤ n_{4} and 2 ≤ n_{1} + n_{2}. Then apply Proposition 6.5 to L′ = U(n_{1} + n_{2}) × U(n_{3}) × U(n_{4}) and we conclude that

G ⫌ L′G′H ⊃ LG′H.

Case 3) Suppose k = 4 and n_{3} = 1. Then,

k = 4 and n_{1} = n_{2} = n_{3} = 1.  (6.14)

In the setting (6.8), we define J by setting

J_{1} = J_{2} = J_{3} := (1) ∈ M(1, ℝ) and J_{4} := ( \begin{matrix}1 ... 1 & & p − 3 \\ 0 ... 0 & & q \end{matrix} ) .

Then, G_{J} ≃ U(p) × U(q) = H.

Given Y = (v_{1}, …, v_{p}) ∈ M(q, p; ℂ) (v_{1}, …, v_{p} ∈ ℂ^{q}), we define X ∈ 𝔲(n) by

X := (\begin{matrix}0 & −Y^{∗} \\ Y & 0 \end{matrix}) .

Then, Q := [X, J] = (\begin{matrix}0 & Y^{∗} \\ Y & 0 \end{matrix}). Associated to the partition n = n_{1} + n_{2} +

n_{3} + n_{4} (= 1 + 1 + 1 + (n − 3)) and the loop 1 → 4 → 2 → 4 → 3 → 4 → 1,

A_{1424341}(Q) = Q_{14}Q_{24}^{∗}Q_{24}Q_{34}^{∗}Q_{34}Q_{14}^{∗}

= (\begin{matrix}0 \\ v_{1} \end{matrix})^{∗} (\begin{matrix}0 \\ v_{2} \end{matrix}) (\begin{matrix}0 \\ v_{2} \end{matrix})^{∗} (\begin{matrix}0 \\ v_{3} \end{matrix}) (\begin{matrix}0 \\ v_{3} \end{matrix})^{∗} (\begin{matrix}0 \\ v_{1} \end{matrix})

= (v_{2}, v_{1})(v_{3}, v_{2})(v_{1}, v_{3}),  (6.15)

where 0 denotes the zero vector in ℂ^{p−3}. Since p ≥ 3 and q ≥ 3 (≥ 2), we can take

Y = (v_{1}, v_{2}, …, v_{p}) = (\begin{matrix}\begin{matrix}1 & 1 & 1 \\ 1 & z & 0 \end{matrix} & 0 \\ 0 & 0 \end{matrix} ) ,

so that (v_{2}, v_{1})(v_{3}, v_{2})(v_{1}, v_{3}) = 1 + z. Thus,

det(λI_{1} − A_{1424341}([X, J])) = λ − 1 − z ∉ ℝ[λ]

if z ∉ ℝ. Hence Proposition follows from Lemma 6.3.

Hence, the proof of Theorem A is now completed.

7 Visible actions on generalized flag varieties

Suppose a Lie group L acts holomorphically on a connected complex manifold D with complex structure J.

Definition 7.1 (see [9, Definitions 2.3, 3.1.1 and 3.3.1]). The action is previsible if there exists a totally real submanifold S such that

D′ := L · S is open in D.  (7.1)

The previsible action is visible if

J_{x}(T_{x}S) ⊂ T_{x}(L · x) for generic x ∈ S (J-transversality),  (7.2)

and is strongly visible if there exists an anti-holomorphic diffeomorphism σ of D′ such that

σ|_{S} = id,  (7.3)
σ preserves each L-orbit on D′.  (7.4)

A strongly visible action is visible ([9, Theorem 14]). Furthermore, we have:

Lemma 7.2. Suppose there exists an automorphism σ̃ of L such that

σ(g · x) = σ̃(g) · σ(x) (g ∈ L, x ∈ L).  (7.5)

Then, a previsible action satisfying (7.3) is strongly visible.

Proof. Any L-orbit on D′ is of the form L · x for some x ∈ S. Then, by (7.3) and (7.5), we have σ(L · x) = σ̃(L) · σ(x) = L · x. Hence, the condition (7.4) is fulfilled.

Now, let us consider the setting of Theorem A.

Example 7.3. We set

σ̃(g) := \overline{g} for g ∈ G = U(n).

Then σ̃ stabilizes subgroups L and H in the setting (1.1) in particular, induces a Lie group automorphism, denoted by the same letter σ̃, of L, and an anti-holomorphic diffeomorphism, denoted by σ, of the homogeneous space

G/H ≃ B_{m_{1},...,m_{l}}(ℂ^{n}).

The L-action on G/H satisfies the compatibility condition (7.5). Now, we consider the totally real submanifold S := B_{m_{1},...,m_{l}}(ℝ^{n}) in G/H. Since σ̃ = id on G′ = O(n) and S ≃ G′/G′ ∩ H, we have σ|_{S} = id. Therefore, if S meets every L-orbit on G/H, then the L-action on G/H is not only previsible by definition but also is strongly visible by Lemma 7.2. Hence, the assertion in Remark 1.5.2 is proved.

8 Applications to representation theory

This section gives a flavor of some applications of Theorem A to multiplicityfree theorems in representation theory.

In [10] (see also [12] and [9, Theorem 2]), we proved that the multiplicityfree property propagates from fibers to spaces of holomorphic sections of equivariant holomorphic bundles under a certain geometric condition. The key assumption there is strongly visible actions (Definition 7.1) on base spaces.

First of all, we observe that one dimensional representations are obviously irreducible, and therefore is multiplicity-free. Then, by [10], this multiplicityfree property propagates to the multiplicity-free property of the representation on the space O(G/H, L_{λ}) of holomorphic sections as an L-module for any G-equivariant holomorphic line bundle L_{λ} → G/H if (G, L, H) satisfies (ii)′ (or any of the equivalent conditions) in Theorem A. Then, by a theorem of Vinberg–Kimelfeld[20], this implies that G/H ≃ B_{m_{1},··· ,m_{l}}(ℂ^{n}) is a spherical variety of L_{ℂ} ≃ GL(n_{1}, ℂ) × ⋯ × GL(n_{k}, ℂ), namely, a Borel subgroup of L_{ℂ} has an open orbit on B_{m_{1},...,m_{l}}(ℂ^{n}).

More generally, applying the propagation theorem of multiplicity-free property to higher dimensional fibers, we get from Theorem A a new geometric proof of a number of multiplicity-free results including:

• (Tensor product) The tensor product of two irreducible representations π_{λ} and π_{μ} of GL(n, ℂ) is multiplicity-free if the highest weight μ ∈ ℤ^{n} is of the form,

(\underbrace{a, …, a}_{p}, \underbrace{b, …, b}_{q}) (a ≥ b).  (8.1)

for some a, b ∈ ℤ (a ≥ b) and some p, q (p + q = n) and if one of the following conditions is satisfied:

1) min(p, q) = 1.

1)′ a − b = 1.

2) min(p, q) = 2 and λ is of the form

(\underbrace{x, …, x}_{n_{1}}, \underbrace{y, …, y}_{n_{2}}, \underbrace{z, …, z}_{n_{3}}) (x ≥ y ≥ z)  (8.2)

2)′ a − b = 2 and λ is of the form (8.2).

3) λ is of the form (8.2) satisfying

min(x − y, y − z, n_{1}, n_{2}, n_{3}) = 1.

Stembridge [19] gave a proof of this fact using a combinatorial method by a case-by-case argument. He proved also that this exhausts all multiplicity-free cases.

• (Restriction: GL_{n} ↓ GL_{p} × GL_{q}) The representation π_{λ} of GL(n, ℂ) is multiplicity-free when restricted to GL(p, ℂ) × GL(q, ℂ) if one of the three conditions (1), (2), and (3) is satisfied.

• (Restriction: GL_{n} ↓ GL_{n_{1}} × GL_{n_{2}} × GL_{n_{3}}) The irreducible representation π_{μ} of GL(n, ℂ) having highest weight μ of the form (8.1) is multiplicity-free when restricted to GL(n_{1}, ℂ)×GL(n_{2}, ℂ)×GL(n_{3}, ℂ) if min(a − b, p, q) ≤ 2.

It is noteworthy that the above three multiplicity-free results (“triunity”) are obtained from a single geometric result, Theorem A (see [8, Theorems 3.3, 3.4 and 3.6]). In particular, we get the equivalence (ii) ⇔ (iv) in Remark 1.5.3 by considering the (G × G)-equivariant line bundle L_{λ} ⊠ L_{μ} → G/L × G/H.

Besides, Theorem A gives a new geometric proof of yet more multiplicityfree theorems (see [9, Theorems 19, 20]):

• (GL_{p}–GL_{q} duality) The symmetric algebra S(M(p, q; ℂ)) ≃ S(ℂ^{pq}) is multiplicity-free as a representation of GL(p, ℂ) × GL(q, ℂ).

• (Kac [5]) S(ℂ^{pq}) is still multiplicity-free as a representation of GL(p − 1, ℂ) × GL(q, ℂ).

• (Panyushev [18]) Let N be a nilpotent orbit of GL(n, ℂ) corresponding to a partition (2^{p} 1^{n−2p}). Then the representation of GL(n, ℂ) on the space ℂ[N] of regular functions is multiplicity-free.

All of the examples discussed so far are finite dimensional. As we saw in [12], we can also expect from strongly visible actions yet more multiplicityfree theorems for infinite dimensional representations for both continuous and discrete spectra. Applications of Theorem A (and its non-compact version) to infinite dimensional representations will be discussed in a future paper.

References

[1] L. Biliotti and A. Gori, Coisotropic and polar actions on complex Grassmannians, Trans. Amer. Math. Soc. 357 (2004), 1731–1751.

[2] A. Borel and L. Ji, Compactification of symmetric and locally symmetric spaces, 69–137, In: Lie Theory, Unitary Representations and Compactifications of Symmetric Spaces (eds. J.-P. Anker and B. Ørsted), Progr. Math. 229 (2005).

[3] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. Math. 111 (1980), 253–311.

[4] B. Hoogenboom, Intertwining functions on compact Lie groups. CWI Tract, 5. Stichting Mathematisch Centrum, Centrum voor Wiskunde en Informatica, Amsterdam, 1984.

[5] V. Kac, Some remarks on nilpotent orbits, J. Algebra 64 (1980), 190– 213.

[6] T. Kobayashi, Invariant measures on homogeneous manifolds of reductive type, J. reine angew. Math. 490 (1997), 37–53.

[7] T. Kobayashi, Multiplicity-free branching laws for unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (ed. K. Mimachi) (1997), 9–17.

[8] T. Kobayashi, Geometry of multiplicity-free representations of GL(n), visible actions on flag varieties, and triunity, Acta Appl. Math., 81 (2004), 129–146.

[9] T. Kobayashi, Multiplicity-free representations and visible actions on complex manifolds, Publ. RIMS, 41 (2005), 497–549 (a special issue of Publications of RIMS commemorating the fortieth anniversary of the founding of the Research Institute for Mathematical Sciences).

[10] T. Kobayashi, Propagation of multiplicity-free property for holomorphic vector bundles, preprint

[11] T. Kobayashi, Visible actions on symmetric spaces, preprint

[12] T. Kobayashi, Multiplicity-free theorems of the restrictions of unitary highest weight modules with respect to reductive symmetric pairs, to appear in Progr. Math., Birkhäuser.

[13] P. Littelmann, On spherical double cones, J. Algebra, 166 (1994), 142– 157.

[14] T. Matsuki, Double coset decomposition of algebraic groups arising from two involutions. I., J. Algebra, 175 (1995), 865–925.

[15] T. Matsuki, Double coset decompositions of reductive Lie groups arising from two involutions, J. Algebra, 197 (1997), 49–91.

[16] T. Matsuki, Classification of two involutions on compact semisimple Lie groups and root systems, J. Lie Theory, 12 (2002), 41–68.

[17] T. Oshima and T. Matsuki, Orbits on affine symmetric spaces under the action of the isotropy subgroups, J. Math. Soc. Japan, 32 (1980), 399–414.

[18] D. Panyushev, Complexity and nilpotent orbits, Manuscripta Math., 83 (1994), 223–237.

[19] J. R. Stembridge, Multiplicity-free products of Schur functions, Ann. Comb., 5 (2001), 113–121.

[20] É. B. Vinberg and B. N. Kimelfeld, Homogeneous domains on flag manifolds and spherical subgroups of semisimple Lie groups, Funct. Anal. Appl., 12 (1978), 168–174