Essay: A Mathematical Theory of Communication

Overview
Claude Shannon’s 1948 essay establishes a quantitative framework for communication, defining how information can be measured, compressed, and transmitted reliably despite noise. It replaces technological particulars with abstract models and precise limits, turning communication into a branch of probability and mathematics. Two central achievements anchor the paper: a measure of information called entropy and the coding theorems that set ultimate limits on compression and error-free transmission.

A general model of communication
Shannon formalizes a communication system as five components: an information source, a transmitter (encoder), a channel, a receiver (decoder), and a destination, with a noise source perturbing the channel. Messages are represented as symbols drawn from an alphabet and often modeled by stochastic processes such as Markov sources. Channels are defined by transition probabilities between input and output symbols. This abstraction allows the same theory to cover telephony, telegraphy, radio, printed text, and even biological signaling.

Entropy and information
To quantify information, Shannon introduces entropy H = −∑ p log p, measured in bits when the logarithm is base 2. Entropy captures uncertainty: highly predictable sources have low entropy; uniform sources have maximal entropy. He derives H by requiring natural properties, continuity, monotonic increase with the number of equally likely outcomes, and additivity for compound events, showing that the logarithmic form is essentially unique. He extends the calculus to joint entropy, conditional entropy (equivocation), and mutual information I(X;Y) = H(X) − H(X|Y), which measures the information about an input revealed by the output. These quantities enable precise statements about what a channel can convey and what a code can compress.

Noiseless and noisy coding theorems
For noiseless sources, the source coding theorem states that the average length of a uniquely decodable code per source symbol cannot be less than the source entropy and can be made arbitrarily close to it using variable-length codes for long blocks. This sets the fundamental limit of lossless compression. For noisy channels, the noisy-channel coding theorem establishes a capacity C, and shows a phase transition: if a transmission rate R is less than C, there exist block codes of sufficient length that achieve arbitrarily small error probability; if R exceeds C, reliable communication is impossible. The proof uses random code ensembles and typicality arguments, demonstrating existence rather than constructing explicit optimal codes.

Channel capacity and continuous channels
Channel capacity is defined as the supremum over input distributions of the mutual information per symbol, C = max I(X;Y). For discrete memoryless channels this yields computable limits; for continuous-time, band-limited channels with additive white Gaussian noise, Shannon derives the celebrated formula C = W log2(1 + S/N) bits per second, where W is bandwidth and S/N the signal-to-noise ratio. He also treats constraints such as average power and discusses signaling with finite versus continuous alphabets.

Redundancy, language, and cryptography
Beyond engineering channels, the paper examines natural-language sources. English text is modeled as a high-order stochastic process with significant redundancy, implying that the entropy per letter is well below log2(26). Redundancy provides error resilience and compressibility and underpins cryptanalysis by making patterns exploitable. Shannon quantifies the tradeoff between redundancy added by codes to resist noise and efficiency needed to approach capacity.

Separation and design implications
The architecture implied by the theory separates source coding from channel coding: first compress to near the source entropy, then protect with a channel code to approach capacity. Under broad conditions this separation is optimal, guiding communication system design. The theory also clarifies the role of modulation and waveform design as means to realize symbolic codes in physical media subject to bandwidth and power limits.

Impact
The essay created information theory, providing tools, entropy, mutual information, capacity, and limits that still govern data compression, error-correcting codes, digital communication, data storage, cryptography, and even neuroscience and machine learning. Its blend of abstraction and operational meaning set a standard for engineering science, revealing that reliable communication is a matter of structure and probability rather than mere signal strength.