Information theory

Information theory is the mathematical study of the <a href="/facts/Quantification_(science)/5YoKbeTW">quantification</a>, <a href="/facts/Data_storage/uCvJ6nrq">storage</a>, and <a href="/facts/Telecommunications/vxoj1cxR">communication</a> of <a href="/facts/Information/3LdrzXtJ">information</a>. The field was established and formalized by <a href="/facts/Claude_Shannon/EremsCws">Claude Shannon</a> in the 1940s, though early contributions were made in the 1920s through the works of <a href="/facts/Harry_Nyquist/BIDzoc9I">Harry Nyquist</a> and <a href="/facts/Ralph_Hartley/oVKw8QUK">Ralph Hartley</a>. It is at the intersection of <a href="/facts/Electronic_engineering/WCSVulpu">electronic engineering</a>, <a href="/facts/Mathematics/pxTouaz4">mathematics</a>, <a href="/facts/Statistics/DNPsKGYU">statistics</a>, <a href="/facts/Computer_science/lN3pEyPz">computer science</a>, <a href="/facts/Neuroscience/o5RV8Cas">neurobiology</a>, <a href="/facts/Physics/a7aH2y08">physics</a>, and <a href="/facts/Electrical_engineering/YYXW1KLW">electrical engineering</a>.
A key measure in information theory is <a href="/facts/Information_entropy/NLg4NLvt">entropy</a>. Entropy quantifies the amount of uncertainty involved in the value of a <a href="/facts/Random_variable/TwTBXnLT">random variable</a> or the outcome of a <a href="/facts/Random_process/Ng1n1dnB">random process</a>. For example, identifying the outcome of a <a href="/facts/Fair_coin/vTBk29k7">fair</a> <a href="/facts/Coin_flip/uAJd6pAJ">coin flip</a> (which has two equally likely outcomes) provides less information (lower entropy, less uncertainty) than identifying the outcome from a roll of a <a href="/facts/Dice/qBnylzwJ">die</a> (which has six equally likely outcomes). Some other important measures in information theory are <a href="/facts/Mutual_information/HIUvsjvV">mutual information</a>, <a href="/facts/Channel_capacity/55wJYsS6">channel capacity</a>, <a href="/facts/Error_exponent/pyzBMNvK">error exponents</a>, and <a href="/facts/Relative_entropy/nh7SjlPE">relative entropy</a>. Important sub-fields of information theory include <a href="/facts/Source_coding/gMFuY4FT">source coding</a>, <a href="/facts/Algorithmic_complexity_theory/L9ezq8pY">algorithmic complexity theory</a>, <a href="/facts/Algorithmic_information_theory/Tltn3I0Y">algorithmic information theory</a> and <a href="/facts/Information-theoretic_security/DG2q6XeI">information-theoretic security</a>.
Applications of fundamental topics of information theory include source coding/<a href="/facts/Data_compression/gMFuY4FT">data compression</a> (e.g. for <a href="/facts/ZIP_(file_format)/qDrrYIB9">ZIP files</a>), and channel coding/<a href="/facts/Error_detection_and_correction/o2rH8e3n">error detection and correction</a> (e.g. for <a href="/facts/Digital_subscriber_line/3pY5osov">DSL</a>). Its impact has been crucial to the success of the <a href="/facts/Voyager_program/cPm6qnNx">Voyager</a> missions to deep space, the invention of the <a href="/facts/Compact_disc/g51GRloc">compact disc</a>, the feasibility of mobile phones and the development of the <a href="/facts/Internet/Xw1rVvJU">Internet</a> and <a href="/facts/Artificial_intelligence/lJGauQwX">artificial intelligence</a>. The theory has also found applications in other areas, including <a href="/facts/Statistical_inference/sJLjubxm">statistical inference</a>, <a href="/facts/Cryptography/e94PEVau">cryptography</a>, <a href="/facts/Neurobiology/o5RV8Cas">neurobiology</a>, <a href="/facts/Perception/xLK0n2q4">perception</a>, <a href="/facts/Signal_processing/Npaja6zb">signal processing</a>, <a href="/facts/Linguistics/oQuamvf2">linguistics</a>, the evolution and function of molecular codes (<a href="/facts/Bioinformatics/D5x2L8ee">bioinformatics</a>), <a href="/facts/Thermal_physics/CeLdAABy">thermal physics</a>, <a href="/facts/Molecular_dynamics/yR3DNt6U">molecular dynamics</a>, <a href="/facts/Black_hole/1r2lmkcF">black holes</a>, <a href="/facts/Quantum_computing/nbbcgmvq">quantum computing</a>, <a href="/facts/Information_retrieval/MFSmpeNw">information retrieval</a>, <a href="/facts/Intelligence_(Information_Gathering)/FsILyU4n">intelligence gathering</a>, <a href="/facts/Plagiarism_detection/3lXBIVGK">plagiarism detection</a>, <a href="/facts/Pattern_recognition/o1s38opj">pattern recognition</a>, <a href="/facts/Anomaly_detection/wjEGMGGz">anomaly detection</a>, the analysis of <a href="/facts/Music/hDVP0xsQ">music</a>, <a href="/facts/Art/ILbnQHHb">art creation</a>, <a href="/facts/Imaging_system/Lkw7V9M8">imaging system</a> design, study of <a href="/facts/Outer_space/R3LRKyFs">outer space</a>, the dimensionality of <a href="/facts/Space/tq6LUM3C">space</a>, and <a href="/facts/Epistemology/NsDwpcYH">epistemology</a>.

Information theory open-in-new

Information theory