Phase Transitions in Rate Distortion Theory and Deep Learning
share
Rate distortion theory is concerned with optimally encoding a given signal class 𝒮 using a budget of R bits, as R→∞. We say that 𝒮 can be compressed at rate s if we can achieve an error of 𝒪(R^-s) for encoding 𝒮; the supremal compression rate is denoted s^∗(𝒮). Given a fixed coding scheme, there usually are elements of 𝒮 that are compressed at a higher rate than s^∗(𝒮) by the given coding scheme; we study the size of this set of signals. We show that for certain "nice" signal classes 𝒮, a phase transition occurs: We construct a probability measure ℙ on 𝒮 such that for every coding scheme 𝒞 and any s >s^∗(𝒮), the set of signals encoded with error 𝒪(R^-s) by 𝒞 forms a ℙ-null-set. In particular our results apply to balls in Besov and Sobolev spaces that embed compactly into L^2(Ω) for a bounded Lipschitz domain Ω. As an application, we show that several existing sharpness results concerning function approximation using deep neural networks are generically sharp. We also provide quantitative and non-asymptotic bounds on the probability that a random f∈𝒮 can be encoded to within accuracy ε using R bits. This result is applied to the problem of approximately representing f∈𝒮 to within accuracy ε by a (quantized) neural network that is constrained to have at most W nonzero weights and is generated by an arbitrary "learning" procedure. We show that for any s >s^∗(𝒮) there are constants c,C such that, no matter how we choose the "learning" procedure, the probability of success is bounded from above by min{1,2^C· W⌈log_2(1+W)⌉^2 -c·ε^-1/s}.
READ FULL TEXT
