For a class of even dimensional asymptotically hyperbolic (AH) manifolds, we develop a generalized Birman-Krein theory to study scattering asymptotics and, when the curvature is constant, to analyze the Selberg zeta function. The main objects we construct for an AH manifold (X,g) are, on the one hand, a natural spectral function ξ for the Laplacian Δg, which replaces the counting function of the eigenvalues in this infinite volume case, and on the other hand the determinant of the scattering operator Sx (λ) of Δg on X. Both need to be defined through regularized functional: renormalized trace on the bulk X and regularized determinant on the conformal infinity (∂X, [ho]). We show that Sx (λ) is meromorphic in λ Є C, with divisors given by resonance multiplicities and dimensions of kernels of GJMS conformal Laplacians (Pk) k Є N of  (∂X, [ho]). Moreover ξ(z) is proved to be the phase of det Sx {n\2+iz) on the essential spectrum {z Є R+}. Applying this theory to convex co-compact quotients X= Г \ H n+1 of hyperbolic space Hn+1, we obtain the functional equation Z (λ)/Z(n- λ)= (detSHn+1(λ))χ(X)}\ det Sx(λ) for Selberg zeta function Z(λ) of X, where χ(X) is the Euler characteristic of X. This describes the poles and zeros of Z(λ), computes det Pk in term of Z(n\2-k)/Z({n\2}+k) and implies a sharp Weyl asymptotic for ξ(z).

We call the complement of a union of at least three disjoint (round) open balls in the unit sphere Sⁿ a Schottky set. We prove that every quasisymmetric homeomorphism of a Schottky set of spherical measure zero to another Schottky set is the restriction of a Möbius transformation on Sⁿ. In the other direction we show that every Schottky set in S² of positive measure admits nontrivial quasisymmetric maps to other Schottky sets. These results are applied to establish rigidity statements for convex subsets of hyperbolic space that have totally geodesic boundaries.