Compression and GenAI
By Raymond Tana (Published on July 6, 2024)
The main points one should take away from this project are as follows:
- An understanding of the theoretical connections between compression and induction.
- A sense that large language models likely do not (yet) qualify as general-compression algorithms; thus are “far” from performing universal induction.
- A series of jumping-off points for further research in this area, including the measuring of a model’s compression capabilities via latent representation sizes in a manner inspired by Shannon information theory.
Read the full piece here.