Fundamentals of Multimedia Ze-Nian Li and Mark S. Drew School of Computing Science Simon Fraser University PEARSON Prentice. Iall Pearson Education International
ContentsPrefacexivMultimedia Authoring and Data Representations1Introduction to Multimedia31.1What is Multimedia?31.1.1Componentsof Multimedia31.1.2Multimedia Research Topics and Projects41.2Multimedia and Hypermedia41.2.1History of Multimedia51.2.2Hypermedia and Multimedia71.3World Wide Web81.3.1Historyof theWwW81.3.2HyperText Transfer Protocol (HTTP)91.3.3HyperText MarkupLanguage (HTML)101.3.4Extensible Markup Language (XML)111.3.5Synchronized Multimedia Integration Language (SMIL)121.4Overview of Multimedia Software Tools141.4.1Music Sequencing and Notation 141.4.2Digital Audio151.4.3Graphics and Image Editing151.4.4Video Editing151.4.5Animation161.4.6Muitimedia Authoring 171.5Further Exploration171.6Exercises181.719References2Multimedia Authoring and Tools202.1Multimedia Authoring 202.1.1Multimedia AuthoringMetaphors212.1.2Multimedia Production 232.1.3Multimedia Presentation 252.1.4Automatic Authoring 332.2Some Useful Editing and Authoring Tools372.2.1AdobePremiere372.2.2Macromedia Director402.2.3Macromedia Flash462.2.4Dreamweaver 512.3VRML512.3.1Overview512.3.2Animation and Interactions542.3.3VRMLSpecifies 542.4Further Exploration 552.5Exercises562.6References 59
vi603Graphics and Image Data RepresentationsGraphics/lmage Data Types603.11-Bit Images 613.1.13.1.2 8-Bit Gray-Level Images613.1.3ImageData Types 64643.1.424-Bit Color Images8-Bit Color Images 653.1.5673.1.6Color Lookup Tables (LUTs)3.2Popular File Formats713.2.1GIF71JPEG753.2.23.2.3PNG76TIFF773.2.4773.2.5EXIF773.2.6Graphics Animation Files783.2.7PSandPDFWindows WMF783.2.83.2.9Windows BMP783.2.10 MacintoshPAINTandPICT78793.2.11X Windows PPM3.3Further Exploration 79793.4Exercises3.581References82Color in Image and Video44.1Color Science 824.1.182Light and Spectra4.1.2Human Vision844.1.384Spectral Sensitivity of the Eye4.1.4Image Formation854.1.5Camera Systems86Gamma Correction 874.1.6894.1.7Color-Matching Functions914.1.8CIE Chromaticity Diagram4.1.9Color Monitor Specifications94Out-of-Gamut Colors 954.1.10White-Point Correction 964.1.114.1.12XYZtoRGBTransform 974.1.13Transform withGamma Corection974.1.14 L*a*b*(CIELAB)Color Model 984.1.15 More Color-Coordinate Schemes 1004.1.16 MunsellColorNaming System 100Color'Models in Images 1004.24.2.1RGB Color Model for CRT Displays100Subtractive Color:CMY Color Model 1014.2.2TransformationfromRGBtoCMY1014.2.3UndercolorRemoval:CMYK System 1024.2.44.2.5Printer Gamuts 1024.3Color Models in Video1041044.3.1Video Color TransformsYUVColorModel 1044.3.2
vii4.3.3YIQColorModel 1054.3.4YCbCrColor Model1074.4Further Exploration 1074.5Exercises1084.6References1115Fundamental Concepts in Video1125.1Types of Video Signals1125.1.1Component Video1125.1.2Composite Video1135.1.3S-Video1135.2Analog Video 1135.2.1NTSCVideo1165.2.2PALVideo1195.2.3SECAMVideo1195.3Digital Video 1195.3.1Chroma Subsampling 1205.3.2CCIR Standards for Digital Video1205.3.3HighDefinitionTV (HDTV)1225.4FurtherExploration1245.5124Exercises5.6References1256Basies of Digital Audio1266.1Digitization of Sound1266.1.1What Is Sound? 1266.1.2Digitization1276.1.3Nyquist Theorem1286.1.4Signal-to-Noise Ratio (SNR)1316.1.5Signal-to-Quantization-Noise Ratio (SQNR)1316.1.6Linear and Nonlinear Quantization 1336.1.7Audio Filtering1366.1.8Audio Qualityversus Data Rate 1366.1.9Synthetic Sounds1376.2MIDI:Musical Instrument Digital Interface1396.2.1MIDIOverview1396.2.2HardwareAspects of MIDI1426.2.3Structure of MIDIMessages1436.2.4General MIDI1476.2.5MIDI-to-WAV Conversion1476.3Quantization and Transmission ofAudio1476.3.1Coding ofAudio1476.3.2Pulse Code Modulation 1486.3.3Differential Coding of Audio1506.3.4Lossless Predictive Coding1516.3.5DPCM 1546.3.6DM1576.3.7ADPCM1586.4Further Exploration1596.5Exercises1606.6163References
vili165Multimedia Data Compression167Lossless Compression Algorithms77.1Introduction1677.2Basics of Information Theory1687.3Run-Length Coding 1717.4171Variable-Length Coding (VLC)Shannon-Fano Algorithm 1717.4.17.4.2HuffmanCoding1737.4.3176Adaptive Huffman Coding7.5Dictionary-Based Coding 1817.6Arithmetic Coding1871917.7Lossless Image Compression7.7.1191Differential Coding of ImagesLossless JPEG1937.7.27.8Further Exploration 1947.9Exercises1957.10197References1998Lossy Compression Algorithms8.1Introduction 1998.2Distortion Measures 1998.3The Rate-Distortion Theory2008.4Quantization 2008.4.1 Uniform ScalarQuantization 2018.4.2Nonuniform Scalar Quantization 2048.4.3Vector Quantization*2068.5207Transform Coding8.5.1Discrete Cosine Transform (DCT)2078.5.2Karhunen-Loeve Transform*2208.6Wavelet-Based Coding2228.6.1Introduction 222Continuous Wavelet Transform*2278.6.28.6.3Discrete WaveletTransform*2308.7Wavelet Packets 2408.8EmbeddedZerotree of Wavelet Coefficients241The ZerotreeData Structure2428.8.18.8.2Successive Approximation Quantization 2448.8.3EZWExample2448.9SetPartitioning inHierarchical Trees (SPIHT)2478.10FurtherExploration2488.11Exercises2498.12References2522539Image Compression Standards9.1The JPEG Standard2539.1.1Main Steps inJPEGImageCompression12539.1.2JPEGModes2629.1.3AGlance at theJPEGBitstream 2659.2TheJPEG2000Standard 2659.2.1Main Steps of JPEG2000 Image Compression*267
ix9.2.2 AdaptingEBCOTtoJPEG2000 2759.2.3Region-of-Interest Coding2759.2.4Comparison of JPEG and JPEG2000 Performance2779.3The JPEG-LS Standard 2779.3.1Prediction2809.3.2Context Determination2819.3.3Residual Coding2819.3.4Near-Lossless Mode 2819.4Bilevel Image Compression Standards2829.4.1TheJBIG Standard 2829.4.2The JBIG2 Standard2829.5Further Exploration 2849.6Exercises2859.7References28710 BasicVideo Compression Techniques28810.1Introduction to Video Compression28810.2VideoCompression Based on Motion Compensation 28810.3Search for Motion Vectors 29010.3.1 Sequential Search 29010.3.22DLogarithmic Search29110.3.3 Hierarchical Search 29310.4295H.26110.4.1 Intra-Frame (1-Frame)Coding29710.4.2 Inter-Frame (P-Frame) Predictive Coding29710.4.3 Quantization inH.261 29710.4.4H.261Encoder andDecoder29810.4.5 A Glance at the H.261 Video Bitstream Syntax 30110.5H.26330310.5.1 Motion Compensation inH.26330410.5.2Optional H.263Coding Modes30510.5.3H.263+andH.263++30710.6Further Exploration30810.7Exercises30910.8References310II MPEG Video Coding IMPEG-1 and 231211.1Overview 31211.2MPEG-131211.2.1Motion Compensation inMPEG-131311.2.2OtherMajorDifferences fron H.26131511.2.3MPEG-1VideoBitstream31811.3MPEG-231911.3.1Supporting InterlacedVideo 32011.3.2MPEG-2Scalabilities32311.3.3 OtherMajorDifferencesfromMPEG-132911.4FurtherExploration 33011.5Exercises33011.6References331
33212 MPEG Video Coding II MPEG-4, 7, and BeyondOverviewofMPEG-4 33212.1Object-Based Visual Coding in MPEG-433512.212.2.1 VOP-Based Coding vs.Frame-Based Coding 33512.2.2Motion Compensation 33712.2.3 Texture Coding 34112.2.4 Shape Coding 34334612.2.5Static Texture Coding12.2.6 Sprite Coding 34734812.2.7 Global Motion Compensation12.3Synthetic Object Coding in MPEG-434912.3.1 2D Mesh Object Coding 34912.3.2 3DModel-based Coding354MPEG-4 Object types, Profiles and Levels35612.4MPEG-4Part10/H.26435712.512.5.1 Core Features 35836012.5.2 Baseline ProfileFeatures12.5.3Main ProfileFeatures36012.5.4 Extended ProfileFeatures361MPEG-736112.612.6.1 Descriptor (D)36312.6.2 Description Scheme (DS) 36512.6.3 Description Definition Language (DDL)36812.7MPEG-2136937012.8Further Exploration12.9370Exercises37112.10References37413Basic Audio Compression Techniques37413.1ADPCM in Speech Coding13.1.1 ADPCM 37413.2G.726ADPCM376Vocoders37813.313.3.1 Phase Insensitivity 37837813.3.2 Channel Vocoder13.3.3 Formant Vocoder38038013.3.4 Linear Predictive Coding13.3.5CELP38313.3.6 Hybrid Excitation Vocoders*38913.4Further Exploration 39213.5Exercises39239313.6References39514 MPEG Audio CompressionPsychoacoustics 39514.139614.1.1 Equal-Loudness Relations14.1.2 Frequency Masking 39814.1.3Temporal Masking403MPEGAudio40514.214.2.1 MPEG Layers405
xi14.2.2 MPEGAudio Strategy 40614.2.3MPEGAudioCompressionAlgorithm 40714.2.4 MPEG-2AAC(AdvanicedAudio Coding) 41214.2.5MPEG-4Audio41414.3Other Commercial Audio Codecs 41514.4TheFuture: MPEG-7 and MPEG-2141514.5FurtherExploration41614.6Exercises41614.7References417IIIMultimedia Communication and Retrieval41915Computer and Multimedia Networks42115.1Basics ofComputer and Multimedia Networks.42115.1.1OSINetworkLayers42115.1.2TCP/PProtocols42215.2Multiplexing Technologies 42515.2.1Basics of Multiplexing42515.2.2 Integrated Services Digital Network (ISDN)42715.2.3Synchronous Optical NETwork (SONET)42815.2.4Asymmetric Digital Subscriber Line (ADSL) 42915.3LANandWAN 43015.3.1 Local Area Networks (LANs)43115.3.2Wide Area Networks (WANs)43415.3.3Asynchronous Transfer Mode (ATM)43515.3.4 Gigabit and10-Gigabit Ethermets 43815.4Access Networks43915.5Common Peripheral Interfaces44115.6FurtherExploration44115.7Exercises44215.8References44216 Multimedia Network Communications and Applications44316.1Quality of Multimedia Data Transmission44316.1.1 Quality of Service(QoS) 44316.1.2 QoS for IP Protocols44616.1.3 Prioritized Delivery 44716.2Multimedia over IP44716.2.1 IP-Multicast 44716.2.2 RTP (Real-time Transport Protocol) 44916.2.3 Real TimeControl Protocol (RTCP)45116.2.4 Resource ReSerVation Protocol (RSVP) 45116.2.5 Real-Time Streaming Protocol (RTSP)45316.2.6 Internet Telephony45516.3Multimedia over ATM Networks45916.3.1 Video Bitrates over ATM45916.3.2ATMAdaptation Layer (AAL)46016.3.3 MPEG-2 Convergence to ATM46116.3.4 Multicast over ATM462
xiiTransportofMPEG-4 46216.416.4.1 DMIFinMPEG-446216.4.2 MPEG-4over IP463Media-on-Demand (MOD)46416.546416.5.1 Interactive TV (ITV) and Set-Top Box (STB)16.5.2 Broadcast Schemes forVideo-on-Demand 46516.5.3 BufferManagement 472Further Exploration 47516.647616.7ExercisesReferences47716.847917 Wireless NetworksWireless Networks 47917.148017.1.1 Analog Wireless Networks48117.1.2 Digital Wireless Networks17.1.3 TDMAandGSM48117.1.4Spread Spectrumand CDMA48317.1.5Analysis of CDMA48617.1.6 3G Digital Wireless Networks48817.1.7 Wireless LAN (WLAN)49217.2Radio Propagation Models49317.2.1MultipathFading 49417.2.2 Path Loss 49617.3496Multimedia over Wireless Networks17.3.1 Synchronization Loss 49749917.3.2 Error Resilient Entropy Coding17.3.3ErrorConcealment50117.3.4 ForwardErrorCorrection (FEC)50317.3.5Trends in Wireless Interactive Multimedia50617.4Further Exploration 508Exercises50817.517.6References51051118 Content-Based Retrieval in Digital Libraries18.1How Should We Retrieve Images? 511C-BIRD-ACase Study 51318.218.2.1 C-BIRDGUI 51418.2.2 Color Histogram 51451618.2.3Color Density18.2.4 Color Layout51618.2.5 Texture Layout 51718.2.6SearchbyIlumination Invariance51918.2 SearchbyObjectModel 52018.3533SynopsisofCurrentImageSearchSystems18.3.1QBIC53518.3.2UC Santa Barbara SearchEngines53618.3.3 BerkeleyDigital LibraryProject 53618.3.4 Chabot 53618.3.5 Blobworld 53718.3.6Columbia University Image Seekers537
xili18.3.7Informedia53718.3.8MetaSEEk53718.3.9PhotobookandFourEyes53818.3.10MARS53818.3.11Virage53818.3.12 Viper 53818.3.13Visual RetrievalWare53818.4Relevanice Feedback53918.4.1MARS53918.4.2 iFind 54118.5Quantifying Results54118.6QueryingonVideos54218.7Querying on OtherFormats54418.8Outiook for Content-Based Retrieval54418.9Further Exploration 54518.10 Exercises54618.11 References547Index551