Multimedia Systems Development

Dr. Omar Masmali

Introduction
 What is Multimedia?
• There are a quite different, or even opposing, viewpoints.

• A consumer entertainment vendor: interactive TV with hundreds of digital channels available, or a cable

TV-like service delivered over a high-speed Internet connection; a smartphone.

• A Computer Science (CS) student: applications that use multiple modalities, including text, images,

drawings (graphics), animation, video, sound including speech, and interactivity.

• Graphics, visualization, artificial intelligence, computer vision, data compression, graph theory,

networking, database systems all have contributions to make in multimedia.

 What is Multimedia?
• Derived from the word “Multi” and “Media”.

• It’s the applications that use multiple modalities, including text, images, drawings

(graphics), animation, video, sound including speech, and interactivity.

• Multimedia means that computer information can be represented through text,

audio, graphics, images, video, and animation in addition to traditional media.

 History of Multimedia
• Newspaper: perhaps the first mass communication medium, uses text, graphics,
and images.

• Wireless radio transmission: Guglielmo Marconi, Italy, in 1895.

• Television: the new medium for the 20th century, established video as a
commonly available medium and has since changed the world of mass
communications.
 Multimedia in the New World
• 2010 - Netflix migrated its infrastructure to the Amazon’s cloud computing
platform. - Microsoft introduced Kinect, a horizontal bar with full-body 3D motion
capture, facial recognition and voice recognition capabilities, for its game console
Xbox 360.

• 2012 - HTML5 subsumes the previous version, HTML4. it is able to run on low
powered devices such as smartphones and tablets.
 Multimedia in the New World
• 2013 - Twitter offered Vine, a mobile app that enables its users to create and post short
video clips. - Sony released its PlayStation 4 a video game console, which is to be
integrated with Gaikai, a cloud-based gaming service that offers streaming video game
content. - 4K resolution TV started to be available in the markets.

• 2015 YouTube launched support for publishing and viewing 360- degree videos, with
playback on its website and its Android mobile apps. - AlphaGo, a computer program that
plays the board game Go, became the first program to beat a human professional player.
Its core technology Deep Learning attracted significant attention and have seen success
in multimedia content understanding and generation.
 Multimedia in the New World
• 2018 The world’s first 16K Ultra High Definition (UHD) short video film, Prairie
Wind, was created. 5G cellular systems started deployment, providing enhanced
mobile broadband and ultra low latency access. The WiFi 6 (802.11ax) standard
was released, offering theoretical maximum throughput of 1 Gbps.

• 2020 Due to the outbreak of corona virus (COVID-19) around the world,
work/study from home became a norm in early 2020. Multimedia-empowered
online meeting and teaching tools, e.g., Zoom, Google Class, and Microsoft
Teams, saw booming use during this period.
Multimedia Systems Development
Dr. Omar Masmali
Multimedia
 Characteristics of a Multimedia Systems

• Multimedia systems has four basic characteristics:

1. Multimedia systems must be computer controlled.

2. Multimedia systems are integrated.

3. The interface to the final presentation of media is usually interactive.

4. The information they handle must be represented digitally.

 Characteristics of a Multimedia Systems
• Computer Controlled
• Producing the content of the information – e.g. by using the authoring tools, image
editor, sound and video editor
• Storing the information – providing large and shared capacity for multimedia
information.
• Transmitting the information – through the network.
• Presenting the information to the end user – make direct use of computer peripheral
such as display device (monitor) or sound generator (speaker).
 Characteristics of a Multimedia Systems

• Integrated
• All multimedia components (audio, video, text, graphics) used in the system
must be somehow integrated.

• Every device, such as microphone and camera is connected to and controlled

by a single computer.

• A single type of digital storage is used for all media type.

 Characteristics of a Multimedia Systems

• Interactivity
• Level 1: Interactivity strictly on information delivery. Users select the time at which
the presentation starts, the order, the speed and the form of the presentation itself.

• Level 2: Users can modify or enrich the content of the information, and this
modification is recorded.

• Level 3: Actual processing of users input and the computer generate genuine result
based on the users input.
 Characteristics of a Multimedia Systems

• Digitally Represented

• Digitization: process involved in transforming an analog signal to digital signal.

 Challenges for Multimedia Systems

• The key issues multimedia systems need to deal with are:

• How to represent and store temporal information.

• How to strictly maintain the temporal relationships on play back/retrieval.

• What process are involved in the above.

 Desirable Features for a Multimedia System
• Very High Processing Power
• Multimedia Capable File System
• Data Representations/File Formats that support multimedia
• Efficient and High I/O
• Special Operating System
• Storage and Memory
• Network Support
• Software Tools
 Components of a Multimedia System
• Capture Devices
• Video Camera
• Video Recorder
• Audio Microphone
• Keyboards
• Graphics tablets
• 3D input devices
• Tactile sensors
• VR devices
• Digitizing/Sampling Hardware
 Components of a Multimedia System
• Storage Devices
• Hard disk drive
• Zip drive
• Compact Disc
• Digital Versatile Disc (DVD)
• Blu ray Disc
 Components of a Multimedia System
• Communication Networks
• Ethernet
• Token Ring
• Fiber Distributed Data Interface (FDDI)
• Asynchronous Transfer Mode
• Intranet
• Internet
 Components of a Multimedia System
• Computer Systems
• Desktop computer
• Processor
• RAM
• Display card
• Sound card
• Capture card
 Components of a Multimedia System
• Display Devices
• High resolution monitor
• High quality speakers
• Color printer
• Projector
 Multimedia Research Topics and Projects
• Multimedia processing and coding: multimedia content analysis, content-based
multimedia retrieval, multimedia security, audio/image/video processing, compression,
etc.

• Multimedia system support and networking: network protocols, Internet, operating

systems, servers and clients, quality of service (QoS), and databases.

• Multimedia tools, end-systems and applications: hypermedia systems, user interfaces,

authoring systems, multi-modal interaction and integration — web-everywhere devices,
multimedia education including Computer Supported Collaborative Learning, and design
and applications of virtual environments.
 HyperMedia

• HyperMedia: can include other media, e.g., text, graphics,

images, and especially the continuous media, sound and
video.
• The World Wide Web (WWW) — the best example of a
hypermedia application.
Multimedia Systems Development
Dr. Omar Masmali
Text
 Definition of Text
• Text is words and symbols in any form, spoken or written, are the most
common system of communication.
• Text is used in most Multimedia applications.
• With multimedia technology, text can be combined with other media in a
powerful and meaningful way to present information and express moods.
• Text is the easiest to manipulate
 Text Elements

• Text elements can be categories into:

• Alphabets characters: A – Z
• Number: 0 – 9
• Special characters: . , ; : ‘ “
• Symbols: @ # $ & *
 Text Usages
• Bullet / list

• Heading / Title

• Paragraph

• Navigation

• Text as graphics

• ….
 Text Usages
• Bullet / list

• Heading / Title

• Paragraph

• Navigation

• Text as graphics

• ….
 Text Usages
• Bullet / list

• Heading / Title

• Paragraph

• Navigation

• Text as graphics

• ….
 Font
• A design for a set of characters.

• A collection of characters of a single size and style belonging

to a particular typeface family.

• There is some basic consistency of look that makes the

individual characters, regardless of size and style variations,
part of the same family.
 Font Size
• The size of a font, typically represented in points (pt).

• The font size is the distance from the top of the “Ascender Height”
to the bottom of the “Descender Height" in letters.
 Font Size
Font Size Example

Text
8 point

12 point Text

24 point Text
48 point
Text
96 point

Text
 Font Style
• Refers to whether text is bold, italicized, underlined, or any combination of the
three

• The term font style refers to the particular style of textual characters.

• Styles are usually standard.

 Font Style
• Use bold fonts for emphasis, to highlight important points.
 Font Style
• It is useful to readers to have titles, proper names, or key terms in a manual
within a block of copy highlighted with bold for ease in scanning.
 Font Style
• Use italics to emphasize small amounts of text within a block of text.
• Avoid long passages in italics.
• It is harder to read than normal roman faces.
• Underlines are appropriate for section headings and some bibliographical
notations.
• In web browsers, default settings typically distinguish hyperlinks by
underlining them.
 Font Categories

• Fonts can be characterized as

• Serif
• Sans Serif
• Decorative
 Font Categories
• Used to decorate, embellish, and beautify a text.

• With the help of decorative fonts any informal passage can become more reader-
friendly: it will quickly capture attention of the readers and make a text easier to
perceive, unusual, and fascinating.
 Paragraph Alignment

• The arrangement of text relative to a margin.

• Four types of alignment:

• Flush left
• Flush right

• Centered

• Justified.
Multimedia Systems Development
Dr. Omar Masmali
Image
 Categories of Digital Graphics

• 1. Bitmap

• 2. Vector

• 3. Meta

• 4. Animated
 1- Bitmaps

• Bitmaps are maps of binary color information.

• They store this information in a grid of points, or pixels, which has a fixed width

and height.

• They can store various ranges of colors according to the image type.
 1-Bit Images
• Images consist of pixels (picture elements in digital images).
• A 1-bit image (also called binary image) consists of on and off bits only and thus is
the simplest type of image.
• Each pixel is stored as a single bit (0 or 1)
• It is also sometimes called a 1-bit monochrome (called Lena image by scientists)
image since it contains no color.
• 1-bit images can be satisfactory for pictures containing only simple graphics and
text.
• fax machines use 1-bit data, so in fact 1-bit images are still important.
 Monochrome 1-bit image

• A 640×480 monochrome image requires 38.4 kB of storage

 8-Bit Gray-Level Images
• 8-bit image is one for which each pixel has a gray value between 0 and 255.

• Each pixel is represented by a single byte.

• The entire image can be thought of as a two-dimensional array of pixel values

referred to as a bitmap.

• Image resolution refers to the number of pixels in a digital image (higher

resolution always yields better quality but increases size)
 Grayscale image of Lena

• 640×480 grayscale image requires 300kB of storage

 24-Bit Color Images
• In a color 24-bit image, each pixel is represented by three bytes, usually
representing RGB.
• Since each value is in the range 0–255, this format supports 256×256×256, or a
total of 16,777,216, possible combined colors; which increases storage size.
• a 640 × 480 24-bit color image would require 921.6 kB of storage. (without any
compression applied)
• Compression is used to decrease the image size by simply grouping pixels
effectively.
24-bit color image
 Higher Bit-Depth Images
• In some fields such as medicine (security cameras, satellite imaging) more
accurate images are required to see the patient’s liver, for example.

• To get such images, special cameras that view more than just 3 colors (RGB) are
used.

• Such images are called multispectral (more than three colors) or hyperspectral
(224 colors for satellite imaging).
 Graphics File Formats
• There are many standard formats for saving bitmaps in files.
• Bitmap (BMP)
• Graphics Interchange Format (GIF)
• Joint Photographic Experts Group (JPEG)
• Exchangeable Image File (Exif)
• Portable Network Graphics (PNG)
• Tag Image File Format (TIFF)
• .......
• https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types
 2- Vector images
• Vector images are completely computer generated.

• They are known as object-oriented graphics as they consist of objects such as

shapes.

• Vectors are used to create graphics such as interface elements (banners, buttons)
text, line art and detailed drawings (plans, maps).

• Effects can be added to vector graphics to add realism, however, they need to be
converted to bitmaps in order to do this.
 Raster graphics
• raster graphic is a mechanism that represents a two-dimensional image as a
rectangular matrix or grid of square pixels, viewable via a computer display, paper,
or other display medium.

• A raster is technically characterized by the width and height of the image in pixels
and by the number of bits per pixel.

• Raster images are stored in image files with varying dissemination, production,
generation, and acquisition formats.
Raster graphics
 3- Meta Graphics
• Meta graphics can be termed as hybrid graphics as they are a combination of
bitmap and vector graphics.

• They aren’t as widely used as bitmaps and vectors, and aren’t as widely supported.

• An example of a meta graphic would be a map consisting of a photo showing an

aerial view of a town, where the landmarks are highlighted using vector text and
graphics.
 4- Animated Graphics
• Animated graphics are ‘moving graphics’ that consist of at least more than one
graphic.

• Vector graphics are mainly the basis of animations.

• Think of cartoons such as the Simpsons and Family Guy.

• Effects generated by bitmaps can be added and bitmaps themselves can also be
animated.
 Three Dimensional (3D) Drawing and
Rendering
• 3 Dimensional refers to objects that are rendered visually on paper, film or on screen
in three planes representing width, height and depth (X, Y and Z).

• The 3D object that a user creates is called a model and it can be simple as well as
complex.
Multimedia Systems Development
Dr. Omar Masmali
Digital Image Processing (1)
 Digital Image

• An image defined as a two- dimensional function, f(x,y)

where x and y are spatial (plane) coordinates, and the
amplitude of f at any pair of coordinates (x, y) is called the
intensity or gray level of the image at that point.

• image is a projection of a 3- D scene into a 2D projection

plane.
 Digital Image

• A digital image is composed of a finite number of elements,

each of which has a particular location and value.

• These elements are called picture elements, image elements,

pels, and pixels.

• Pixel is the term used most widely to denote the elements of

a digital image.
Digital Image
Digital Image
 Digital Image Processing

• The field of digital image processing refers to processing

digital images by means of a digital computer
 Digital Image Processing
• Images based on radiation from the Electromagnetic (EM) spectrum are the most
familiar, especially images in the X-ray and visual bands of the spectrum.

• Electromagnetic waves can be conceptualized as propagating sinusoidal waves of

varying wavelengths, or they can be thought of as a stream of massless particles,
each traveling in a wavelike pattern and moving at the speed of light. Each
massless particle contains a certain amount (or bundle) of energy. Each bundle of
energy is called a photon
 Applications of Digital Image
• Remote sensing via satellites and other space crafts
 Applications of Digital Image
• Image transmission and storage for business applications
 Applications of Digital Image
• Medical processing
 Applications of Digital Image
• RADAR (Radio Detection and Ranging)
 Applications of Digital Image
• SONAR (Sound Navigation and Ranging)
 Applications of Digital Image
• Robotics and automated inspection of industrial parts
Steps of Digital Image Processing
 Steps of Digital Image Processing

• Image Acquisition:

• It could be as simple as being given an image that is already in digital form.

Generally, the image acquisition stage involves processing such scaling.

 Steps of Digital Image Processing

• Image Enhancement:

• It is among the simplest and most appealing areas of digital image processing.

The idea behind this is to bring out details that are obscured or simply to

highlight certain features of interest in image.

• Image enhancement is a very subjective area of image processing.

 Steps of Digital Image Processing

• Image Restoration:

• It deals with improving the appearance of an image.

• It is an objective approach, in the sense that restoration techniques tend to be

based on mathematical or probabilistic models of image processing.

• Enhancementis based on human subjective preferences regarding what

constitutes a “good” enhancement result.

 Steps of Digital Image Processing

• Color Image Processing:

• It is an area that is been gaining importance because of the use of digital images

over the internet.

• Color image processing deals with basically color models and their

implementation in image processing applications.

 Steps of Digital Image Processing

• Multiresolution Processing:

• These are the foundation for representing image in various degrees of resolution.
 Steps of Digital Image Processing

• Compression:

• It deals with techniques reducing the storage required to save an image, or the

bandwidth required to transmit it over the network.

• It has two major approaches

• 1. Lossless Compression

• 2. Lossy Compression
 Steps of Digital Image Processing

• Morphological Processing:

• It deals with tools for extracting image components that are useful in the

representation and description of shape and boundary of objects.

• It is majorly used in automated inspection applications.

 Steps of Digital Image Processing

• Representation and Description:

• It always follows the output of the segmentation step that is, raw pixel data,

constituting either the boundary of an image or points in the region itself.

 Steps of Digital Image Processing

• Recognition:

• It is the process that assigns label to an object based on its descriptors. It is the

last step of image processing which use artificial intelligence of software.

Multimedia Systems Development
Dr. Omar Masmali
Digital Image Processing (2)
 Color Fundamentals

• 6 to 7 million cones in the human eye can be divided into three principal sensing

categories, corresponding roughly to red, green, and blue.

• 65%: red

• 33%: green

• 2%: blue (blue cones are the most sensitive)

 Color Characteristics

• The characteristics generally used to distinguish one color from another are:

• Brightness: the achromatic notion of intensity.

• Hue: dominant wavelength in a mixture of light waves, represents dominant color as

perceived by an observer.

• Saturation: relative purity or the amount of white light mixed with its hue.
 Color Characteristics

• The characteristics generally used to distinguish one color from another are:

• Brightness: the achromatic notion of intensity.

 Color Characteristics

• The characteristics generally used to distinguish one color from another are:

• Hue: dominant wavelength in a mixture of light waves, represents dominant color as

perceived by an observer.
 Color Characteristics

• The characteristics generally used to distinguish one color from another are:

• Saturation: relative purity or the amount of white light mixed with its hue.
 Image Computerized Processes

• There are three types of computerized processes in the processing of image

• Low level Process

• Mid-level Image Processing

• High level Processing

 Image Computerized Processes

• Low level Process:

• These involve primitive operations such as image processing to reduce noise,

contrast enhancement and image sharpening. These kinds of processes are

characterized by fact the both inputs and output are images.

 Image Computerized Processes

• Mid-level Image Processing:

• It involves tasks like segmentation, description of those objects to reduce them to

a form suitable for computer processing, and classification of individual objects.

The inputs to the process are generally images but outputs are attributes

extracted from images.

 Image Computerized Processes

• High level Processing:

• It involves “making sense” of an ensemble of recognized objects, as in image

analysis, and performing the cognitive functions normally associated with vision.
Image Computerized Processes
 Sampling and Quantization

• To create a digital image, we need to convert the continuous sensed data into

digital from. This involves two processes – sampling and quantization.

• An image may be continuous with respect to the x and y coordinates and also in

amplitude. To convert it into digital form we have to sample the function in both

coordinates and in amplitudes.

 Sampling and Quantization

• Digitalizing the coordinate values is called Sampling.

• Digitalizing the amplitude values is called Quantization.

 Representing Digital Images

• The result of sampling and quantization is matrix of real numbers.

• Assume that an image f(x,y) is sampled so that the resulting digital image has M

rows and N Columns.

• The values of the coordinates (x,y) now become discrete quantities thus the value

of the coordinates at origin become (x,y) = (0,0)

Representing Digital Images
 Relationship between Pixels
• A pixel p at coordinates (x,y) has four horizontal and vertical
neighbors whose coordinates are given by:

(x+1,y), (x-1, y), (x, y+1), (x,y-1)

• This set of pixels, called the 4-neighbors or p, is denoted by N4(p)

Relationship between Pixels
 Relationship between Pixels
• The four diagonal neighbors of p have coordinates and are denoted
by ND(p).

(x+1, y+1), (x+1, y-1), (x-1, y+1), (x-1, y-1)

Relationship between Pixels
Relationship between Pixels
Digital Image Storage Required
 Digital Image Storage Required (Examples)

• Find bits required to store a 4*4 digital image if we are using 64 different gray

levels
 Digital Image Storage Required (Examples)
• Find bits required to store a 4*4 digital image if we are using 64 different gray

levels

• Resolution (number of pixels) = 4 * 4 = 16

• Number of bits on each pixel = 6 ( or )

• So, 4*4*6 = 96 bits

 Digital Image Storage Required (Examples)
• N = 32

• M = 32

• L = 23 = 8

• Resolution (number of pixels) = ??

• Number of bits on each pixel = ??

• Size = ??
 Digital Image Storage Required (Examples)
• N = 32

• M = 32

• L = 23 = 8

• Resolution (number of pixels) = 1024

• Number of bits on each pixel = 3

• Size = 1024 * 3 = 3072

 Image Degradation
• An image that has been degraded by using a prior knowledge of the
degradation phenomenon (Noise).
 Noise Sources
• The principal sources of noise in digital images arise during

• Image acquisition
• e.g., light levels, sensor temperature, etc.
• Transmission
• e.g., lightning or other atmospheric disturbance in wireless network
 Noise Models
• White noise
• The Fourier spectrum of noise is constant
 Noise Models
• Gaussian noise
• Electronic circuit noise, sensor noise due to poor illumination and/or
high temperature

• Rayleigh noise
• Range imaging

• Erlang (gamma) noise

• Laser imaging
 Noise Models
• (b) Gaussian noise
• (c) Rayleigh noise
• (d) Range imaging
Multimedia Systems Development
Dr. Omar Masmali
Audio
 What is Sound?

• Sound is a wave phenomenon like light, but is macroscopic

and involves molecules of air being compressed and
expanded under the action of some physical device.
 What is Sound?

• For example, a speaker in an audio system vibrates back and

forth and produces a longitudinal pressure wave that we
perceive as sound.

• Without air there is no sound—for example, in space.

 What is Sound?
• Sound comprises the spoken word, voices, music and even
noise.
• It is a complex relationship involving:
• a vibrating object (sound source)
• a transmission medium (usually air)
• a receiver (ear) and;
• a preceptor (brain).
 Why Sound is Important in Multimedia?

• To reinforce message or theme

• To set the mood

• To catch the interest of the audience

• To alert the audience

• To include narration: effective for training and educational

application.
How it’s Work
 The Power of Sound

• Sound is measured in dB (decibel)

• Sound waves are known as waveforms.

• A pleasant sound has a regular wave pattern. The pattern is

repeated over and over.

• But the waves of noise are irregular. They do not have a

repeated pattern.
 Characteristic of Sound Waves

• Sound is described in terms of two characteristics:

• Frequency (or pitch)

• Amplitude (or loudness)

 Frequency

• Frequency is a measure of how many vibrations occur in one second.

This is measured in Hertz (abbreviation Hz) and directly corresponds
to the pitch of a sound.

• People can hear from 20 Hz to 20,000 Hz (20 kHz)

• Sounds below 20 Hz are infrasonic
• Sounds above 20 kHz are ultrasonic.
Frequency
 Amplitude

• Amplitude is the maximum displacement of a wave from an equilibrium position.

• The louder a sound, the more energy it has.

• This means loud sounds have a large amplitude.

Amplitude
Characteristic of Sound Waves
 Analogue to Digital Audio
• Analogue audio
• The name for an electronic signal that carries its information of sound as
continuous fluctuating voltage value.
• non digital tape or audio tape recording of sound.

• Digitizing
• The process of converting an analog signal to a digital one.
 Digital Audio

• Digital audio data is the representation of sound, stored in the form of samples

point.

• Quality of digital recording depends on the sampling rate, that is, the number of

samples point taken per second (Hz)

• The higher the sampling rate, the more the measurements are taken (better quality).

• The lower the sampling rate, the lesser the measurements are taken (low quality).
Digital Audio
 Audio File Formats
• AUDIO DIGITAL
• WINDOWS *.WAV

• MAC *.AIFF

• UNIX *.AU

• REALAUDIO *.RA

• MPEG3 *.MP3
 Advantages & Disadvantages of Using Audio
• Advantages
• Ensure important information is noticed.
• Add interest.
• Can communicate more directly than other media.

• Disadvantages
• Easily overused.
• Requires special equipment for quality production.
• Not as memorable as visual media.
Multimedia Systems Development
Dr. Omar Masmali
Natural Language Processing
(NLP)
 What is NLP?

• Natural Language Processing, or NLP for short, is broadly

defined as the ability of a computer program to understand
human language as it is spoken and written

• It is the automatic manipulation of natural language, like

speech and text, by software.
 Motivation for NLP
• Understand language analysis & generation
• Communication
• Language is a window to the mind
• Data is in linguistic form
• Data can be in Structured (table form), Semi structured (XML
form), Unstructured (sentence form).
Types of NLP
 Natural Language Understanding (NLU)

• It helps the machine to understand the data.

• It is used to interpret data to understand the meaning of

data to be processed accordingly.

• It solves it by understanding the context, semantic, syntax,

intent, and sentiment of the text.
 Natural Language Understanding (NLU)

• There are three linguistic levels to understand language.

• Syntax: It understands sentences and phrases. It checks the
grammar and syntax of the text.
• Semantic: It checks the meaning of the text.
• Pragmatic: It understands context to know what the text is trying
to achieve.
 Natural Language Generation (NLG)

• NLG is a process to produce meaningful sentences in Natural

Language.

• It explains the structured data in a manner that is easy to

understand for humans with a high speed of thousands of
pages per second.
NLP Levels
 Applications of (NLP)
• Search Autocorrect and Autocomplete
 Applications of (NLP)
• Language Translator
 Applications of (NLP)
• Voice Assistants
 Applications of (NLP)
• Email Filters
 Applications of (NLP)
• Chatbots
Multimedia Systems Development
Dr. Omar Masmali
Video
 What is Video?

• Video is the technology of electronically capturing,

recording, processing, storing, transmitting, and
reconstructing a sequence of still images representing scenes
in motion.

• Video makes use of all of the elements of multimedia,

bringing your products and services alive, but at a high cost.
 Analogue and Digital Video

• Analogue video is essentially a product of the television

industry and therefore conforms to television standards.

• Digital video is a product of the computing industry and

therefore conforms to digital data standards.
 Advantages of Digital Video

• It can be easily edited.

• The video is stored as a standard computer file.

• Software motion video does not require specialized

hardware for playback.

• Long-lasting.
 Disadvantages of Video

• Requires large storage capacity devices.

• Copies can be made illegally.

• Need fast computer system for playback and capture.

• Video is very hardware-intensive (require the highest

performance demand on your computer).
 Some Video Cables
RCA Coaxial
(F-connector)

BNC RGB

HDMI Firewire
 Broadcast Video Standards

• National Television System Committee (NTSC)

• Phase Alternating Line (PAL)

• System Electronique Pour Couleur Avec Memoire (SECAM)

• Advanced Television Systems Committee (ATSC)

• Digital Television (DTV)

 File Size Considerations

• Digitized video files can be extremely large.

• A single second of high-quality color video that takes up only one-

quarter of a computer screen can be as large as 1 MB.

• Several elements determine the file size:

• Frame Size
• Frame Rate
• Color Depth
• The length of the video
Frame Size
 Frame Rate

• Frame rate (expressed in frames per second or FPS) is the frequency

(rate) at which consecutive images (frames) are captured or
displayed.
 Frame Rate

• Frame rate greatly impacts the style and viewing experience of a

video.
 Calculate Video File Size

• Video File Size = Frame size * Frame rate * Color depth * Duration

• Color depth:
• B&W video = 1 byte
• Color video = 3 bytes
 Calculate Video File Size

• Calculate the file size for a video with 320 x 240 pixels, color video, 30
fps, and length 15 seconds ?
 Calculate Video File Size

• Calculate the file size for a video with 320 x 240 pixels, color video, 30
fps, and length 15 seconds ?

• Video file size = Frame size * Frame rate * Color depth * Duration

• Video file size = 320 x 240 x 3 x 30 x 15 = 103,680,000 bytes

 Calculate Video File Size

• Calculate the file size for a video with 320 x 240 pixels, B&W video, 30
fps, and length 15 seconds ?
 Calculate Video File Size

• Calculate the file size for a video with 320 x 240 pixels, B&W video, 30
fps, and length 15 seconds ?

• Video file size = 320 x 240 x 30 x 15 = 34,560,000 bytes

 Video Compression

• Because of the large sizes associated with video files, video

compression/decompression programs, known as codecs, have been
developed.

• These programs can substantially reduce the size of video files, which
means that more video can fit on a single CD and that the speed of
transferring video from a CD to the computer can be increased.
Video Compression
 Digital Video File Formats

• Motion Pictures Expert Group (.mpg)

• Quicktime (.mov)

• Audio Video Interleaved(.avi)

• Windows Media Video (.wmv)

• Adobe Flash video (.flv).

Multimedia Systems Development
Dr. Omar Masmali
Computer Vision
 Computer Vision

• Computer vision is a field of artificial intelligence (AI) that enables

computers and systems to derive meaningful information from digital
images, videos and other visual inputs and take actions or make
recommendations based on that information.
 How does computer vision work

• Computer vision needs lots of data.

• It analyzes data over and over until it understands distinctions and

ultimately recognizes images.

• For example, to train a computer to recognize an aircraft, it needs to

be fed vast quantities of planes images and planes-related items to
learn the differences and recognize the airplane.
How does computer vision work
Computer Vision
 Computer Vision Tasks
• Object Classification: What broad category of object is in this photograph?

• Object Identification: Which type of a given object is in this photograph?

• Object Verification: Is the object in the photograph?

• Object Detection: Where are the objects in the photograph?

• Object Landmark Detection: What are the key points for the object in the photograph?

• Object Segmentation: What pixels belong to the object in the image?

• Object Recognition: What objects are in this photograph and where are they?
Computer Vision Tasks
Computer Vision Tasks
What makes face recognition hard?
Expression
What makes face recognition hard?

Lighting
What makes face recognition hard?
Occlusion
What makes face recognition hard?
Viewpoint
 Examples of CV

• Google
Examples of CV
 Examples of CV

• You only look once (YOLO)

Examples of CV
 Computer Vision in Healthcare
• Cancer Detection
•
Computer vision is being successfully applied for breast and skin
cancer detection.
• With image recognition, doctors can identify anomalies by comparing
cancerous and non-cancerous cells in images.
• With automated cancer detection, doctors can diagnose cancer faster
from an MRI scan.
 Computer Vision in Transportation 1
• Self-driving cars
• Computer vision is widely used in self-driving cars. It is used to detect
and classify objects (e.g., road signs or traffic lights), create 3D maps
or motion estimation, and plays a key role in making autonomous
vehicles a reality.
 Computer Vision in Transportation 2
• Road Condition Monitoring
• Computer vision has also been applied for monitoring the road
infrastructure condition by accessing the variations in concrete and
tar.
 Computer Vision in Manufacturing
• Defect Detection
• Total production control is usually impossible. With computer vision,
we can detect defects such as cracks in metals, paint defects, bad
prints, etc., in sizes smaller than 0.05mm.
 Computer Vision in Agriculture 1
• Crop Monitoring
• With computer vision systems, real-time crop monitoring and
identification of any crop variation due to any disease or deficiency of
nutrition can be made.
 Computer Vision in Agriculture 2
• Automatic Weeding
• An automatic weeding machine is an intelligent project enabled with
AI and computer vision that removes unwanted plants or weeds
around the crops. Traditionally weeding methods require human
labour, which is costly and inefficient compared to automatic weeding
systems.
 Computer Vision in Retail 1
• Self-checkout
• Self-checkout enables the customers to complete their transactions
from a retailer without the need for human staff, and this becomes
possible with computer vision. Self-checkouts are now helping
retailers in avoiding long queues and manage customers.
 Computer Vision in Retail 2
• Automatic replenishment
• Automatic replenishment with computer vision systems captures the
image data and performs a complete inventory scan to track the
shelves item at regular intervals.
Multimedia Systems Development
Dr. Omar Masmali
Multimedia Data
Compression
 Introduction

• Compression: the process of coding that will effectively

reduce the total number of bits needed to represent certain
information.
• Lossless - the compression and • Lossy compression eliminates
decompression processes those bytes which are considered
induce no information loss. as not-noticable.
Compression ratio
 Image Compression Standards
• Recent years have seen an explosion in the availability of digital
images, because of the increase in numbers of digital imaging devices
such as smart phones, webcams, digital cameras, and scanners.
• The need to efficiently process and store images in digital form has
motivated the development of many image compression standards
for various applications and needs.
• In general, standards have greater longevity than particular programs
or devices and therefore warrant careful study
 Image Compression Standards
• Some current standards are:
• JPEG
• JPEG2000 standard
• JPEG-LS Standard
• JBIG Standard
• JBIG2 Standard
 Video Compression
• Three types of pictures (or frames) are used in video compression
• I-frame (Intra-coded picture), a complete image, like a JPG or BMP image file

• P-frame (Predicted picture) holds only the changes in the image from the
previous frame.

• B-frame (Bidirectional predicted picture) saves even more space by using

differences between the current frame and both the preceding and following
frames.
Video Compression
Multimedia Systems Development
Dr. Omar Masmali
Social Media Sharing
 Social Media Services
• Social media, a group of Internet-based applications, allow the creation and
exchange of user generated contents.
• Key factors to the success of the new generation of social media:
- Collective intelligence
- Rich connections and activities

• Two important social media services:

- User-generated content sharing
- Online social networking.
 User-Generated Content Sharing

• UGC is used for a wide range of applications with different types of

media (e.g., text, music, picture, and video).

• Video data are arguably more difficult for content generation and
sharing, given their large size, high bandwidth, and long playback
duration.
 User-Generated Content Sharing
• YouTube: the most significant and successful video sharing website

• Fast growth (established in 2005):

• By 2020: – Over 2 billion unique users visit YouTube per month, and
• Over 70% views are on mobile

• Highly globalized:
• 80 languages
• 85% of YouTube traffic comes from outside of the US.
User-Generated Content Sharing
User-Generated Content Sharing
 Online Social Networking

• OSN provides an Internet-based platform to connect people with

social relations.

• Both Facebook and Twitter support the sharing and propagation of

such media objects as pictures, music, and video among friends.

 Online Social Networking

• Facebook: -

• 2.5 billion active users.

• 1.66 billion million of them log on

a daily basis.
 Online Social Networking

• Twitter: -

• A representative of microblog, recently has also begun offering

the Vine service, which enables mobile users to create and post

video clips up to six seconds long.

• 330 million monthly active users.

• 145 million daily active users..

Online Social Networking
 Mobile Video Clip Sharing

• New generation video sharing services that use smart mobile

terminals to instantly capture and share ultra-short video clips

(usually of several seconds).

• Representatives: Twitter’s Vine, Instagram, Snapchat, TikTok.

Multimedia Systems Development
Dr. Omar Masmali
Artificial Intelligence
 Artificial Intelligence

• It is the science and engineering of making intelligent machines,

especially intelligent computer programs.

• It is related to the similar task of using computers to understand human

intelligence.

• Artificial intelligence leverages computers and machines to mimic the

problem-solving and decision-making capabilities of the human mind.
 Agent and Environment

• An agent is anything that can perceive its environment through sensors

and acts upon that environment through actuators (effectors).
• A human agent has sensory organs such as eyes, ears, nose, tongue and skin
parallel to the sensors, and other organs such as hands, legs, mouth, for
actuators.

• A robotic agent replaces cameras and infrared range finders for the sensors,
and various motors and actuators for effectors.
 Types of Agents

• Simple Reflex Agents

• Model Based Reflex Agents.

• Goal Based Agents.

• Utility Based Agents.

• Learning Agents
Agent and Environment
 Simple Reflex Agents
• They choose actions only based
on the current percept.
• They are rational only if a
correct decision is made only on
the basis of current precept.
• Their environment is completely
observable.
 Model Based Reflex Agents
• They use a model of the world
to choose their actions.
• Model − knowledge about “how
the things happen in the world”.
• Updating the state requires the
information about −
• How the world evolves.
• How the agent’s actions affect the
world.
 Goal Based Agents
• They choose their actions in
order to achieve goals.

• Goal − It is the description of

desirable situations.
 Utility Based Agents
• An agent that acts based not
only on what the goal is, but the
best way to reach that goal
 Learning Agents
Learning Element:
Adds knowledge, makes improvement to system

Performance Element:
Performs task, selects external actions

Critic:
Monitors results of performance, provides feedback to
learning element

Problem Generator:
Actively suggests experiments, generates examples to test

Performance Standard:
Method / standard of measuring performance
 Types of Artificial Intelligence

• Reactive Machines perform basic operations. This level of A.I. is the simplest. These types react
to some input with some output. There is no learning that occurs.

• Limited Memory refer to an A.I.’s ability to store previous data and/or predictions, using that data
to make better predictions.

• Theory of Mind level AI will be able to better understand the entities it is interacting with by
discerning their needs, emotions, beliefs, and thought processes.

• Self-aware This is the final stage of AI development which currently exists only hypothetically.
Types of Artificial Intelligence
Multimedia Systems Development
Dr. Omar Masmali
Machine Learning
 Artificial Intelligence

• It is the science and engineering of making intelligent machines,

especially intelligent computer programs.

• It is related to the similar task of using computers to understand human

intelligence.

• Artificial intelligence leverages computers and machines to mimic the

problem-solving and decision-making capabilities of the human mind.
 Machine learning

• Machine learning (ML) is a type of artificial intelligence (AI)

• Allows software applications to become more accurate at predicting

outcomes without being explicitly programmed to do so.

• The study of computer algorithms that can improve automatically

through experience and by the use of data.
 Machine Learning Areas
• Supervised Learning: Data and corresponding labels are given (Features and Labels)

• Unsupervised Learning: Only data is given, no labels provided

• Semi-supervised Learning: Some (if not all) labels are present

• Reinforcement Learning: An agent interacting with the world makes observations,

takes actions, and is rewarded or punished; it should learn to choose actions in such
a way as to obtain a lot of reward

7
 Supervised Learning
• Learning a discrete function: Classification
• Boolean classification:
• Each example is classified as true(positive) or false(negative).
• Learning a continuous function: Regression

10
 Tid Attrib1 Attrib2 Attrib3 Class Learning
1 Yes Large 125K No
algorithm
2 No Medium 100K No

3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply
Tid Attrib1 Attrib2 Attrib3 Class Model
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ? Deduction

14 No Small 95K ?

15 No Large 67K ?
10

Test Set
11
• Example to show the Different between Classification and Regression
Some of ML Algorithms
 Deep Learning

• Deep Learning is a subfield of

machine learning concerned with

algorithms inspired by the structure

and function of the brain called

artificial neural networks.

 Artificial Neural Networks (ANN)

• A neural network is composed of many simple neurons (processing

units) which are hooked together to generate output based on inputs
Artificial Neural Networks (ANN)
Examples
 Data Science
• Data science is the domain of study that deals with

vast volumes of data using modern tools and

techniques to find unseen patterns, derive

meaningful information, and make business

decisions.

• Data science uses complex machine learning

algorithms to build predictive models.