Relative Information Loss in the PCA
In this paper, the authors analyze Principle Component Analysis (PCA) as a deterministic input-output system. They show that the relative information loss induced by reducing the dimensionality of the data after performing the PCA is the same as in dimensionality reduction without PCA. Furthermore, they analyze the case where the PCA uses the sample covariance matrix to compute the rotation. If the rotation matrix is not available at the output, they show that an infinite amount of information is lost. The relative information loss is shown to decrease with increasing sample size.