,j..

\ '.'
,
HAROLD J. MCELHINNY (CA SBN 66781)
HMcElhinny@mofo.com
2 MICHAEL A. JACOBS (CA SBN 111664)
MJacobs@mofo.com
3 JENNIFER LEE TAYLOR (CA SBN 161368)
JTaylor@mofo.com
4 JASON R. BARTLETT (CA SBN 214530)
JasonBartlett@mofo.com
5 MORRISON & FOERSTER LLP
425 Market Street
6 San Francisco, California 94105-2482
Telephone: 415.268.7000
7 Facsimile: 415.268.7522
8 Attorneys for Plaintiff
APPLE INC.
.
')'
i ';_
"'""
9
10
II
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF CALIFORNIA
APPLE INC., a California corporation,
Plaintiff,
v.
SAMSUNG ELECTRONICS CO., LTD., a
Korean corporation; SAMSUNG
ELECTRONICS AMERICA, INC., a New
York corporation; SAMSUNG
TELECOMMUNICATIONS AMERICA,
LLC, a Delaware limited liability company.
Defendants.
APPLE !NC. 'S COMPLAINT
sf-2981926
~ ~ V c a s e 1.1
184 6
JURY TRIAL DEMAND
COMPLAINT FOR PATENT
INFRINGEMENT, FEDERAL FALSE
DESIGNATION OF ORIGIN AND
UNFAIR COMPETITION, FEDERAL
TRADEMARK INFRINGEMENT,
STATE UNFAIR COMPETITION,
COMMON LAW TRADEMARK
INFRINGEMENT, AND UNJUST
ENRICHMENT
.L
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

1

Plaintiff Apple Inc. (Apple) complains and alleges as follows against Defendants
Samsung Electronics Co., Ltd., Samsung Electronics America, Inc., and Samsung
Telecommunications America, LLC (collectively Samsung).
THE NATURE OF THE ACTION

1. Apple revolutionized the telecommunications industry in 2007 when it introduced
the wildly popular iPhone, a product that dramatically changed the way people view mobile
phones. Reviewers, analysts and consumers immediately recognized the iPhone as a game
changer. Before the iPhone, cell phones were utilitarian devices with key pads for dialing and
small, passive display screens that did not allow for touch control. The iPhone was radically
different. In one small and lightweight handheld device, it offered sophisticated mobile phone
functions, a multi-touch screen that allows users to control the phone with their fingers, music
storage and playback, a mobile computing platform for handheld applications, and full access to
the Internet. These features were combined in an elegantly designed product with a distinctive
user interface, icons, and eye-catching displays that gave the iPhone an unmistakable look.
2. Those design features were carried over to the iPod touch, another product that
Apple introduced in 2007. The iPod touch has a product configuration and physical appearance
that is virtually identical to the iPhone. Moreover, the iPod touch utilizes the same user interface
icons and screen layout as the iPhone, displaying the unmistakable iPhone appearance.
3. Apple introduced another revolutionary product, the iPad, in 2010. The iPad is an
elegantly designed computer tablet with a color touch screen, a user interface reminiscent of the
iPhones user interface, and robust functionality that spans both mobile computing and media
storage and playback. Because of its innovative technology and distinctive design, the iPad
achieved instant success.
4. Apples creative achievements have resulted in broad intellectual property
protection for Apples innovations, including utility and design patents, trademarks, and trade
dress protection. Nevertheless, Apples innovations have been the subject of widespread
emulation by its competitors, who have attempted to capitalize on Apples success by imitating
Apples innovative technology, distinctive user interfaces, and elegant and distinctive product
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

2

design. One of the principal imitators is Samsung, which recently introduced the Galaxy line of
mobile phones and Galaxy Tab computer tablet, all of which use the Google Android operating
system, to compete with the iPhone and iPad. Instead of pursuing independent product
development, Samsung has chosen to slavishly copy Apples innovative technology, distinctive
user interfaces, and elegant and distinctive product and packaging design, in violation of Apples
valuable intellectual property rights. As alleged below in detail, Samsung has made its Galaxy
phones and computer tablet work and look like Apples products through widespread patent and
trade dress infringement. Samsung has even misappropriated Apples distinctive product
packaging.
5. By this action, Apple seeks to put a stop to Samsungs illegal conduct and obtain
compensation for the violations that have occurred thus far.
THE PARTIES

6. Apple is a California corporation having its principal place of business at 1 Infinite
Loop, Cupertino, California 95014.
7. Samsung Electronics Co., Ltd. (referred to individually herein as SEC) is a
Korean corporation with its principal offices at 250, 2-ga, Taepyong-ro, Jung-gu, Seoul, 100-742,
South Korea. On information and belief, SEC is South Koreas largest company and one of
Asias largest electronics companies. SEC designs, manufactures, and provides to the U.S. and
world markets a wide range of products, including consumer electronics, computer components
and myriad mobile and entertainment products.
8. Samsung Electronics America, Inc. (referred to individually herein as SEA) is a
New York corporation with its principal place of business at 105 Challenger Road, Ridgefield
Park, New Jersey 07660. On information and belief, SEA was formed in 1977 as a subsidiary of
SEC, and markets, sells, or offers for sale a variety of consumer electronics, including TVs,
VCRs, DVD and MP3 players, and video cameras, as well as memory chips and computer
accessories, such as printers, monitors, hard disk drives, and DVD/CD-ROM drives. On
information and belief, SEA also manages the North American operations of Samsung
Telecommunications America, Samsung Electronics Canada, and Samsung Electronics Mexico.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

3

9. Samsung Telecommunications America, LLC (referred to individually herein as
STA) is a Delaware limited liability company with its principal place of business at 1301 East
Lookout Drive, Richardson, Texas 75081. On information and belief, STA was founded in 1996
as a subsidiary of SEC, and markets, sells, or offers for sale a variety of personal and business
communications devices in the United States, including cell phones.
JURISDICTION

10. This Court has subject matter jurisdiction under 15 U.S.C. 1121 (action arising
under the Lanham Act); 28 U.S.C. 1331 (federal question); 28 U.S.C. 1338(a) (any Act of
Congress relating to patents or trademarks); 28 U.S.C. 1338(b) (action asserting claim of unfair
competition joined with a substantial and related claim under the trademark laws); and 28 U.S.C.
1367 (supplemental jurisdiction).
11. This Court has personal jurisdiction over SEC, SEA and STA because each of
these Samsung entities has committed and continues to commit acts of infringement in violation
of 35 U.S.C. 271 and 15 U.S.C. 1114 and 1125, and places infringing products into the stream
of commerce, with the knowledge or understanding that such products are sold in the State of
California, including in this District. The acts by SEC, SEA and STA cause injury to Apple
within this District. Upon information and belief, SEC, SEA and STA derive substantial revenue
from the sale of infringing products within this District, expect their actions to have consequences
within this District, and derive substantial revenue from interstate and international commerce.
VENUE AND INTRADISTRICT ASSIGNMENT

12. Venue is proper within this District under 28 U.S.C. 1391(b) and (c) because
Samsung transacts business within this district and offers for sale in this district products that
infringe the Apple patents, trade dress, and trademarks. In addition, venue is proper because
Apples principal place of business is in this district and Apple suffered harm in this district.
Moreover, a substantial part of the events giving rise to the claim occurred in this district.
Pursuant to Local Rule 3-2(c), Intellectual Property Actions are assigned on a district-wide basis.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

4

BACKGROUND

APPLES INNOVATIONS

13. Apple is a leading designer and manufacturer of mobile communication devices,
personal computers, and portable digital media players. As a result of its significant investment
in research and development, Apple has developed innovative technologies that have changed the
face of the computer and telecommunications industries. One such pioneering technology is
Apples Multi-Touch user interface, which allows users to navigate their iPhone, iPod touch,
and iPad devices by tapping and swiping their fingers on the screen.
14. In 2007, Apple revolutionized the telecommunications industry when it introduced
the iPhone. The iPhone combined in one small and lightweight handheld device sophisticated
mobile phone functions, media storage and playback, a tactile user interface that allows users to
control the phone with their fingers, mobile computing power to run diverse pre-installed and
downloadable applications, and functionality to gain full access to the Internet. These features
were combined in an elegant glass and stainless steel case with a distinctive user interface that
gave the iPhone an immediately recognizable look.
15. As a direct result of its innovative and distinctive design and its cutting edge
technological features, the iPhone was an instant success, and it immediately became uniquely
associated with Apple as its source. Reviewers and analysts universally praised the iPhone for its
game changing features. Time Magazine listed the iPhone number one on its List of Top Ten
Gadgets for 2007, noting that [t]he iPhone changed the way we think about how mobile media
devices should look, feel and perform. The New York Times called it revolutionary. As of
March 2011, more than 108 million iPhones had been sold worldwide.
16. Also in 2007, Apple launched the iPod touch, a digital music player. The iPod
touch incorporated the distinct style of the iPhone and also became an immediate success. By
March 2011, Apple had sold over 60 million units.
17. After introducing the iPhone, Apple continued to innovate and achieve success
with a series of pioneering designsmore sophisticated, advanced versions of the iPhone, and
then, in 2010, the iPad. The iPad is a computer tablet with a color 9.7-inch touch screen that
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

5

allows users to manipulate icons and data with their fingers in the same fashion as the iPhone and
iPod touch screens. Reviewers and analysts immediately recognized the iPad as a revolutionary
product, describing it as a winner and a new category of device that would replace laptops
for many people.
18. No computer product that preceded the iPad looked like the iPad, but its design did
resemble other Apple productsnamely, the iPhone and the iPod touch, thereby extending the
unique and innovative Apple design and trade dress to a new producttablet computers. In its
first 80 days on the market, Apple sold 3 million iPad units. By March 2011, Apple sold over 19
million iPads.
19. Apples iPhone, iPad and iPod touch products have been extensively advertised
throughout the United States in virtually every media outlet, including network and syndicated
television, the Internet, billboards, magazines and newspaperswith the vast majority of the
advertisements featuring photographs of the distinctive design of these products. Apples
advertising expenditures for these products for fiscal years 2007-2010 were in excess of $2
billion.
20. In addition, Apples iPhone, iPod touch, and iPad products have received
unsolicited comment and attention in print and broadcast media throughout the world. Each new
generation of these Apple products is the subject of positive commentary and receives unsolicited
praise from independent media commentators. Frequently, these unsolicited commentaries are
accompanied by images of the iPhone, iPad, and iPod touch products, including their unique
packaging.
21. The Apple iPhone, iPod touch, and iPad product design has come to represent and
symbolize the superb quality of Apples products and enjoys substantial goodwill among
consumers. The iPad, iPhone, and iPod touch have garnered widespread acclaim for their unique
product design and outstanding performance. Time Magazine named the iPad one of the 50 Best
Inventions of the Year 2010, Popular Science heralded it as the Top Tablet in its Best of Whats
New 2010 feature, and the popular technology blog Engadget selected the device as both the 2010
Editors Choice Gadget of the Year and Tablet of the Year. In addition, the iPad received a 2010
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

6

Red Dot Award for Product Design and was nominated for the 2010 Peoples Design Award.
The iPhone has received several awards over the years, including a 2008 Design and Art
Direction (D&AD) Black Pencil award, a 2008 International Forum (iF) Product Design
Award, and the 2008 International Design Excellence Award (IDEA) Best in Show. More
recently, Engadget named the iPhone 4 the 2010 Editors Choice Phone of the Year, and the
device received the Best Mobile Device award at the Mobile World Congress in February 2011.
Engadget also included the iPhone in its feature on the 10 Gadgets That Defined the Decade. The
iPod touch won the 2008 D&AD Yellow Pencil award as well as the 2008 iF Product Design
Award.
APPLES INTELLECTUAL PROPERTY RIGHTS

Apples Utility Patents

22. Apple has protected its innovative designs and cutting-edge technologies through a
broad range of intellectual property rights. Among those rights are the utility patents listed
below. Apples utility patents cover many of the elements that the world has come to associate
with Apples mobile devices. These include patents covering fundamental features of the Multi-
Touch user interface that enable Apples devices to understand user gestures and to respond by
performing a wide variety of functions, such as selecting, scrolling, pinching, and zooming.
23. In addition, Apple has patented many of the individual features that together add
up to the high-quality experience that users have come to associate with Apple products. Apples
innovations ranging from the arrangement of text messages on the screen, to the way images and
documents appear to bounce back when the user scrolls too far, down to movement of the
buttons have been recognized by the United States Patent and Trademark Office as patent-worthy
contributions to the art.
24. Among the patents that Apple has been awarded are the patents listed below,
attached as Exhibits 1-7, to which Apple owns all rights, title, and interest.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

7

Patent Number Title
7,812,828 (the 828 patent) Ellipse Fitting For Multi-Touch Surfaces
7,669,134 (the 134 patent) Method and Apparatus For Displaying
Information During An Instant Messaging
Session
6,493,002 (the 002 patent) Method and Apparatus for Displaying and
Accessing Control and Status Information
in a Computer System
7,469,381 (the 381 patent) List Scrolling and Document Translation,
Scaling and Rotation on a Touch-Screen
Display
7,844,915 (the 915 patent) Application Programming Interfaces for
Scrolling Operations
7,853,891 (the 891 patent) Method and Apparatus for Displaying a
Window for a User Interface
7,863,533 (the 533 patent) Cantilevered Push Button Having Multiple
Contacts and Fulcrums
Apples Design Patents

25. Apple also has protected its innovative designs through design patents issued by
the United States Patent and Trademark Office. The Apple design patents cover the many famous
ornamental features of Apples devices, such as the flat black face, metallic bezel, and the
distinctive matrix of application icons. Apple owns all right, title, and interest in and to each of
the asserted design patents listed below, copies of which are attached as Exhibits 8-10.
Patent Number Title
D627,790 (the D790 patent) Graphical User Interface For a Display
Screen or Portion Thereof
D602,016 (the D016 patent) Electronic Device
D618,677 (the D677 patent) Electronic Device
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

8

Apples Trade Dress

26. Apple holds trade dress protection in the design and appearance of the iPhone, the
iPod touch, and the iPad, together with their distinctive user interfaces and product packaging.
iPhone Trade Dress
27. The iPhone is radically different from the devices that preceded it. It has a
distinctive shape and appearancea flat rectangular shape with rounded corners, a metallic edge,
a large display screen bordered at the top and bottom with substantial black segments, and a
selection of colorful square icons with rounded corners that mirror the rounded corners of the
iPhone itself, and which are the embodiment of Apples innovative iPhone user interface. As
shown below, the end result is an elegant product that is more accessible, easier to use, and much
less technically intimidating than previously available smart phones and PDAs. The iPhone
product design immediately became closely associated with Apple.
28. Each of these elements of the iPhone product configuration is distinctive and
serves to identify Apple as the source of the iPhone products. Moreover, none of these elements
is functional.
29. Extending its innovative style to the packaging, Apple created an equally elegant
and distinctive packaging for the iPhone products. The packaging features a compact black or
black-and-white box with eye-catching metallic silver lettering on a matte black surface, with the
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

9

sides of the top of the box extending down to cover the bottom portion of the box completely.
The outside of the box has a clean stylewith minimal wording and a simple, prominent, nearly
full-size photograph of the iPhone product itself. The style carries over within the boxwith the
iPhone cradled within a specially designed black display so that the iPhone, and nothing else, is
immediately visible when the box is opened. The accessories and instructional materials are
hidden from view underneath the iPhone trayemphasizing the accessible nature of the iPhone
itself. The design entices purchasers to pick up the iPhone and try it out, without worrying that it
is complicated.
30. As with the product configuration itself, each of these elements of the iPhone
packaging is distinctive and serves to identify Apple as the source of the iPhone products.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

10

Moreover, none of these elements is functionaland there are a plethora of alternative packaging
options available to Apples competitors.
iPod touch Trade Dress
31. The iPod touch has a product configuration and physical appearance that is
virtually identical to the iPhone. It has a flat rectangular shape with rounded corners, a silver
edge, a large display screen bordered at the top and bottom with substantial black segments, and a
selection of colorful square icons with rounded corners that mirror the rounded corners of the
iPod touch (and the iPhone), and which are the embodiment of Apples innovative iPod touch
user interface. As shown below, the end result is an elegant product that invites use. Like the
iPhone, the iPod touch immediately became closely associated with Apple.
32. Each of these elements of the iPod touch product configuration is distinctive and
serves to identify Apple as the source of the iPod touch products. Moreover, none of these
elements is functional.
iPad Trade Dress
33. Because it embodies the same trade dress elements as the iPhone, the iPad
resembles a grown-up iPhone. It has a flat rectangular shape with rounded corners, a silver
edge, a large display screen with a substantial black border, and a selection of Apples colorful
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

11

square iconsall with rounded corners that mirror the rounded corners of the iPad, iPhone and
iPod touch.
34. Each of these elements of the iPad product configuration is distinctive and serves
to identify Apple as the source of the iPad products. Moreover, none of these elements is
functional.
35. The packaging for the iPad is similarly innovative and, like the iPhone, utilizes a
box that, when opened, prominently displays the product so that it is immediately visible, with all
other accessories and materials layered beneath it. Also similar to the iPhone, the outside of the
iPad box has a clean stylewith minimal silver metallic wording and a simple, prominent, nearly
full-size photograph of the iPad product on a white background.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

12

36. Each of these elements of the iPad packaging is distinctive and serves to identify
Apple as the source of the products. Moreover, none of these elements is functional.
Trade Dress Registrations
37. Apple owns three registrations for the design and configuration of the iPhone.
38. U.S. Registration No. 3,470,983 is for the overall design of the product, including
the rectangular shape, the rounded corners, the silver edges, the black face, and the display of
sixteen colorful icons. Attached hereto as Exhibit 11 is a true and correct copy of U.S.
Registration No. 3,470,983.
39. U.S. Registration No. 3,457,218 is for the configuration of a rectangular handheld
mobile digital electronic device with rounded corners. Attached hereto as Exhibit 12 is a true and
correct copy of U.S. Registration No. 3,457,218.
40. U.S. Registration No. 3,475,327 is for a rectangular handheld mobile digital
electronic device with a gray rectangular portion in the center, a black band above and below the
gray rectangle and on the curved corners, and a silver outer border and side. Attached hereto as
Exhibit 13 is a true and correct copy of U.S. Registration No. 3,475,327.
Trade Dress at Issue
41. The following non-functional elements of Apples product designs comprise the
product configuration trade dress at issue in this case (the Apple Product Configuration Trade
Dress):
a rectangular product shape with all four corners uniformly rounded;
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

13

the front surface of the product dominated by a screen surface with black borders;
as to the iPhone and iPod touch products, substantial black borders above and
below the screen having roughly equal width and narrower black borders on either side of the
screen having roughly equal width;
as to the iPad product, substantial black borders on all sides being roughly equal in
width;
a metallic surround framing the perimeter of the top surface;
a display of a grid of colorful square icons with uniformly rounded corners; and
a bottom row of square icons (the Springboard) set off from the other icons and
that do not change as the other pages of the user interface are viewed.
42. The following non-functional elements of Apples packaging designs comprise the
product packaging trade dress at issue in this case (the Apple Product Packaging Trade Dress):
a rectangular box with minimal metallic silver lettering and a large front-view
picture of the product prominently on the top surface of the box;
a two-piece box wherein the bottom piece is completely nested in the top piece;
and
use of a tray that cradles products to make them immediately visible upon opening
the box.
43. Collectively, the Apple Product Configuration Trade Dress and the Apple Product
Packaging Trade Dress are referred to herein as the Apple Product Trade Dress.
Apples Trademarks

44. Apple has protectable trademark rights in various icons used in the user interface
in the iPhone, iPod touch, and iPad productsicons that are consistently used in the Apple family
of mobile products.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

14

45. For example, U.S. Registration No. 3,886,196 covers an icon that is green in color
with a white silhouette of a phone handset arranged at a 45 degree angle and centered on the icon
that represents the application for making telephone calls:
Attached hereto as Exhibit 14 is a true and correct copy of U.S. Registration No. 3,866,196.
46. U.S. Registration No. 3,889,642 covers an icon that is green in color with a white
silhouette of a speech bubble centered on the icon that represents the application for messaging:
Attached hereto as Exhibit 15 is a true and correct copy of U.S. Registration No. 3,889,642.
47. U.S. Registration No. 3,886,200 covers an icon featuring a yellow and green
sunflower against a light-blue background that represents the application for photos:
Attached hereto as Exhibit 16 is a true and correct copy of U.S. Registration No. 3,866,200.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

15

48. U.S. Registration No. 3,889,685 covers an icon that features gears against a gray
background that represents the application for settings:
INSERT
Attached hereto as Exhibit 17 is a true and correct copy of U.S. Registration No. 3,889,685.
49. U.S. Registration No. 3,886,169 covers an icon that features a yellow note pad that
represents the application for notes:
Attached hereto as Exhibit 18 is a true and correct copy of U.S. Registration No. 3,886,169.
50. U.S. Registration No. 3,886,197 is for the silhouette of a man on a spiral bound
address book that represents the icon for contacts:
Attached hereto as Exhibit 19 is a true and correct copy of U.S. Registration No. 3,886,197.
51. Collectively, the application icons displayed in Paragraphs 45-50 represent
Apples Registered Icon Trademarks. Apple uses these Registered Icon Trademarks in
connection with its iPhone, iPod touch, and iPad products.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

16

52. To represent the iTunes application, Apple uses an icon that is purple in color with
a white circular band and a silhouette of two eighth-notes superimposed on the white circular
band:
Pending U.S. Application Serial No. 85/041,463 covers this icon (the Purple iTunes Store
Trademark). Attached hereto as Exhibit 20 is a true and correct copy of the TARR status report
for U.S. Application Serial No. 85/041,463.
53. Moreover, Apple also owns a federal trademark registration for a logo for its
iTunes on-line music service, U.S. Registration No. 2,935,038, (the iTunes Eighth Note and CD
Design Trademark):
U.S. Registration No. 2,935,038 issued on March 22, 2005. Apple filed an Affidavit under
Section 15 of the Lanham Act on March 24, 2010, rendering the registration incontestable.
Attached hereto as Exhibit 21 is a true and correct copy of U.S. Registration No. 2,935,038.
SAMSUNGS INFRINGING PRODUCTS

54. Samsung has imported into or sold in the United States the following products,
each of which infringes one or more of Apples Intellectual Property Rights: the Samsung
Captivate, Continuum, Vibrant, Galaxy S 4G, Epic 4G, Indulge, Mesmerize, Showcase,
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

17

Fascinate, Nexus S, Gem, Transform, Intercept, and Acclaim smart phones and the Samsung
Galaxy Tab tablet.
55. Rather than innovate and develop its own technology and a unique Samsung style
for its smart phone products and computer tablets, Samsung chose to copy Apples technology,
user interface and innovative style in these infringing products.
56. Samsungs Galaxy family of mobile products, introduced in 2010, is exemplary.
The copying is so pervasive, that the Samsung Galaxy products appear to be actual Apple
productswith the same rectangular shape with rounded corners, silver edging, a flat surface face
with substantial top and bottom black borders, gently curving edges on the back, and a display of
colorful square icons with rounded corners. When a Samsung Galaxy phone is used in public,
there can be little doubt that it would be viewed as an Apple product based upon the design alone.
57. Samsung had many options in developing its smart phones. Indeed, earlier
versions of Samsung smart phones did not embody the same combination of elements of Apples
trade dress. Even the icons in earlier versions of the Samsung smart phones looked different
because they had a variety of shapesand did not appear as a field of square icons with rounded
corners.
58. Samsung it chose to infringe Apples patents, trade dress, and trademark rights
through the design, packaging and promotion of its Galaxy mobile phones and the Galaxy Tab
computer tablet, and similar products, and it did so willfully to trade upon the goodwill that Apple
has developed in connection with its Apple family of mobile products.
Infringement of Apples Patents
59. Samsungs infringement of the Apple utility patents identified in this Complaint
provides Samsung with unique functionality for its products that was the result of Apples
innovation, not Samsungs. Samsung has not obtained permission from Apple to use its
inventions in the identified utility patents.
60. Moreover, as the side-by-side comparisons shown below reveal, Samsung has
misappropriated Apples patented mobile phone design in the accused products, including the
Samsung Galaxy mobile phone depicted below.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

18

D677 Patent

D790 Patent

D016 Patent

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

19

Infringement of Apples Trade Dress
61. Samsung announced its Galaxy line of Android-based smart phones in March of
2010 in South Korea. The original model, Galaxy S i9000, is shown below side by side with an
iPhone 3GS.


62. Each of Samsungs Galaxy phones embodies a combination of several elements of
the Apple Product Configuration Trade Dress identified above, namely, a product configuration
with
Apple iPhone 3GS

Galaxy S i9000

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

20

a rectangular product shape with all four corners uniformly rounded;
the front surface of the product dominated by a screen surface with black borders;
substantial black borders above and below the screen having roughly equal width
and narrower black borders on either side of the screen having roughly equal width;
a metallic surround framing the perimeter of the top surface;
a display of a grid of colorful square icons with uniformly rounded corners; and
a bottom row of icons set off from the other icons and that do not change as the
other pages of the user interface are viewed.
63. Samsung also imitated Apples Product Packaging Trade Dress:
Samsungs packaging includes:
a rectangular box with metallic silver lettering and a large front-view picture of the
product prominently on the top surface of the box;
a two-piece box wherein the bottom piece is completely nested in the top piece;
and
use of a design that cradles products to make them immediately visible upon
opening the box.
64. As shown below, Samsungs Galaxy Tab computer tablet also slavishly copies a
combination of several elements of the Apple Product Configuration Trade Dress.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

21

Samsung Tab

Specifically, the Samsung computer tablet has:
rectangular product shape with all four corners uniformly rounded;
front surface of the product dominated by a screen surface with black borders;
substantial black borders on all sides being roughly equal in width; and
a display of a grid of colorful square icons with uniformly rounded corners.
65. Samsungs Galaxy Tab computer tablet packaging also features key elements of
the Apple Product Packaging Trade Dress for Apples iPad products.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

22

The following Apple Product Packaging Trade Dress is incorporated in the Samsung Galaxy Tab
tablet computer products:
a rectangular box with metallic silver lettering and a large front-view picture of the
product prominently on the top surface of the box;
a two-piece box wherein the bottom piece is completely nested in the top piece;
and
use of a design that cradles products to make them immediately visible upon
opening the box.
Infringement of Apples Trademarks
66. In addition to copying Apples Product Trade Dress, Samsung has also copied
numerous application icons in which Apple had valid trademark rights, as shown below:
Apple Icons

Samsung Icons

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

23

Apple Icons

Samsung Icons



1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

24

67. Moreover, the icon that Samsung uses for its music application is virtually
identical to the iTunes Eighth Note and CD logo that Apple has registered with the United
States Patent and Trademark Office:
68. Samsungs adoption of a trade dress that slavishly copies the Apple Product Trade
Dress and its use of various icons that infringe Apples trademark rights is likely to cause
confusion or mistake, or to deceive consumers, purchasers, and others into thinking that Samsung
products are Apple products, or that they are sponsored by or affiliated with Apple, when they are
not. The copying is particularly problematic because the Samsung Galaxy products are the type
of products that will be used in publicon the bus, in cafes, in stores, or at school, where third
parties, who were not present when the products were purchased, will associate them with Apple
because they have the unmistakable Apple look that is created from the various elements of the
Apple Trade Dress.
69. Of significant concern for Apple is that Apple devotes a tremendous amount of
resourcestechnical research and development and design resourcesto develop its cutting edge
products. Part of the cachet of Apple products is the very fact that they consistently stand-out
from all of the other products on the market. Apples goodwill among consumers is closely tied
to its position as an outlier in technology and communications products, which causes each
release of a new product to be highly anticipated among consumers who want to be among the
early adopters of the newest Apple product. Samsungs flagrant and relentless copying of
Apples intellectual property rights in its Galaxy family of products not only allows Samsung to
Apple Icon

Samsung Icon

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

25

reap benefits from Apples investment, it also threatens to diminish the very important goodwill
that Apple has cultivated with its products.
70. On information and belief Samsungs marketing has played up the similarities
between its Galaxy family of phones and the Apple iPhone. On information and belief,
Samsungs mobile phones were marketed as the phone that is the closest to the iPhonefor
consumers who wanted a product with the distinctive Apple look, but who did not want to pay for
the real product.
71. Apples efforts to address Samsung's pervasive copying of Apple's innovations and
intellectual property directly with Samsung have been unsuccessful. Apple is left with no choice
but to file this lawsuit in order to protect one of its most valuable assetsthe technology used in
and the designs of the iPhone, iPod touch, and the iPad.
FIRST CLAIM FOR RELIEF

(Trade Dress Infringement)
(Lanham Act Section 43(a), 15 U.S.C. 1125(a))
72. Apple incorporates and realleges paragraphs 1 through 71 of this Complaint.
73. Apple is the owner of all right and title to the distinctive iPhone, iPod touch, and
iPad trade dress. The Apple Product Configuration Trade Dress, as embodied in the Apple
iPhone, iPod touch, and iPad products, has acquired secondary meaning, and is not functional. In
addition, the Apple Product Packaging Trade Dress, embodied in the packaging for the Apple
iPhone and iPad devices, is inherently distinctive and not functional.
74. In addition, based on extensive and consistent advertising, promotion and sales
throughout the United States, the Apple Product Trade Dress has acquired distinctiveness and
enjoys secondary meaning among consumers, identifying Apple as the source of these products.
75. Apples extensive promotion of the distinctive Apple Product Trade Dress has
resulted in Apples acquisition of valuable, legally protected rights in the Apple Product Trade
Dress as well as considerable customer goodwill.
76. The Samsung Galaxy line of products has misappropriated the Apple Product
Trade Dress by mimicking a combination of several elements of that trade dress.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

26

77. Samsungs manufacture and distribution of the Samsung Galaxy products with
packaging, product design, and product user interface features that mimic a combination of
several elements of the Apple Product Trade Dress is likely to cause confusion, or to cause
mistake, or to deceive the consumer as to the affiliation, connection or association of Samsung
with Apple, or as to the origin, sponsorship, or approval by Apple of Samsungs goods, services
or commercial activities.
78. Samsungs manufacture and distribution of the Samsung Galaxy line of products
with packaging, product design and product user interface features that mimic a combination of
several elements of the Apple Product Trade Dress enables Samsung to benefit unfairly from
Apples reputation and success, thereby giving Samsungs infringing products sales and
commercial value they would not have otherwise.
79. Samsungs actions constitute unfair competition and false designation or origin in
violation of Section 43(a) of the Lanham Act, 15 U.S.C. 1125(a).
80. Samsung knew of Apples Product Trade Dress when it designed its Galaxy line of
products, and has refused to change its product or packaging design in response to Apples
repeated objections. Accordingly, Samsungs infringement has been and continues to be
intentional, willful and without regard to Apples Product Trade Dress.
81. Apple has been and will continue to be irreparably harmed and damaged by
Samsungs conduct, and Apple lacks an adequate remedy at law to compensate for this harm and
damage.
82. Apple is informed and believes, and on that basis alleges, that Samsung has gained
profits by virtue of its infringement of the Apple Product Trade Dress.
83. Apple also has sustained damages as a direct and proximate result of Samsungs
infringement of the Apple Product Trade Dress in an amount to be proven at trial.
84. Because Samsungs actions have been willful, Apple is entitled to treble its actual
damages or Samsungs profits, whichever is greater, and to an award of costs, and, this being an
exceptional case, reasonable attorneys fees pursuant to 15 U.S.C. 1117(a).
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

27

SECOND CLAIM FOR RELIEF

(Federal Trade Dress Infringement)
(15 U.S.C. 1114)
85. Apple incorporates and realleges paragraphs 1 through 84 of this Complaint.
86. Apple owns three registrations for the design and configuration of the iPhone.
87. U.S. Registration No. 3,470,983 is for the overall design of the product, including
the rectangular shape, the rounded corners, the silver edges, the black face, and the display of
sixteen colorful icons.
88. U.S. Registration No. 3,457,218 is for the configuration of a rectangular handheld
mobile digital electronic device with rounded corners.
89. U.S. Registration No. 3,475,327 is for a rectangular handheld mobile digital
electronic device with a gray rectangular portion in the center, a black band above and below the
gray rectangle and on the curved corners, and a silver outer border and side.
90. The Samsung Galaxy line of products copy and infringe these three trade dress
registrations (collectively, Apples Registered Trade Dress).
91. Samsungs manufacture and distribution of the Samsung Galaxy products with
packaging, product design, and product user interface features that copy a combination of several
elements of Apples Registered Trade Dress is likely to cause confusion, or to cause mistake, or
to deceive the consumer as to the affiliation, connection or association of Samsung with Apple, or
as to the origin, sponsorship, or approval by Apple of Samsungs goods, services or commercial
activities.
92. Samsungs manufacture and distribution of the Samsung Galaxy line of products
with packaging, product design and product user interface features that copy a combination of
several elements of Apples Registered Trade Dress enables Samsung to benefit unfairly from
Apples reputation and success, thereby giving Samsungs infringing products sales and
commercial value they would not have otherwise.
93. Prior to Samsungs first use of Apples Registered Trade Dress, Samsung was
aware of Apples business and had either actual notice and knowledge, or constructive notice of
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

28

Apples Registered Trade Dress, and has refused to change its product or packaging design in
response to Apples repeated objections.
94. Samsungs unauthorized use of a trade dress for its Galaxy product line that
infringes Apples Registered Trade Dress is likely, if not certain, to deceive or to cause confusion
or mistake among consumers as to the origin, sponsorship or approval of the Samsung Galaxy
line of products and/or to cause confusion or mistake as to any affiliation, connection or
association between Apple and Samsung, in violation of 15 U.S.C. 1114(a).
95. Apple is informed and believes, and on that basis alleges, that Samsungs
infringement of Apples Registered Trade Dress as described herein has been and continues to be
intentional, willful and without regard to Apples Registered Trade Dress.
96. Apple is informed and believes, and on that basis alleges, that Samsung has gained
profits by virtue of its infringement of Apples Registered Trade Dress.
97. Apple will suffer and is suffering irreparable harm from Samsungs infringement
of Apples Registered Trade Dress insofar as Apples invaluable goodwill is being eroded by
Samsungs continuing infringement. Apple has no adequate remedy at law to compensate it for
the loss of business reputation, customers, market position, confusion of potential customers and
good will flowing from the Samsungs infringing activities. Pursuant to 15 U.S.C. 1116, Apple
is entitled to an injunction against Samsungs continuing infringement of Apples Registered
Trade Dress. Unless enjoined, Samsung will continue its infringing conduct.
98. Because Samsungs actions have been committed with intent to damage Apple and
to confuse and deceive the public, Apple is entitled to treble its actual damages or Samsungs
profits, whichever is greater, and to an award of costs and, this being an exceptional case,
reasonable attorneys fees pursuant to 15 U.S.C. 1117(a) and 1117(b).
THIRD CLAIM FOR RELIEF

(Federal Trademark Infringement)
(15 U.S.C. 1114)
99. Apple incorporates and realleges paragraphs 1 through 98 of this Complaint.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

29

100. Apple owns seven federal trademark registrations for the distinctive and colorful
square application icons used in the user interface for the iPhone, iPod touch, and iPad Registered
Icon Trademarks, i.e., the Registered Icon Trademarks.
101. The Samsung Galaxy line of products has infringed the Registered Icon
Trademarks by using variations of those application icons in Samsungs products. The Samsung
Galaxy line of products have also infringed Apples registered iTunes Eighth Note and CD Logo.
102. Samsungs use of its infringing application icons is likely to cause confusion, or to
cause mistake, or to deceive the consumer as to the affiliation, connection or association of
Samsung with Apple, or as to the origin, sponsorship, or approval by Apple of Samsungs goods,
services or commercial activities.
103. Samsungs use of the infringing application icons enables Samsung to benefit
unfairly from Apples reputation and success, thereby giving Samsungs infringing products sales
and commercial value they would not have otherwise.
104. Prior to Samsungs first use of the infringing application icons, Samsung was
aware of Apples business and had either actual notice and knowledge, or constructive notice of,
Apples Registered Icon Trademarks.
105. Samsungs unauthorized use of the infringing application icons is likely, if not
certain, to deceive or to cause confusion or mistake among consumers as to the origin,
sponsorship or approval of the Samsung Galaxy line of products and/or to cause confusion or
mistake as to any affiliation, connection or association between Apple and Samsung, in violation
of 15 U.S.C. 1114(a).
106. Apple is informed and believes, and on that basis alleges, that Samsungs
infringement of Apples Registered Icon Trademarks as described herein has been and continues
to be intentional, willful and without regard to Apples rights.
107. Apple is informed and believes, and on that basis alleges, that Samsung has gained
profits by virtue of its infringement of Apples Registered Icon Trademarks.
108. Apple will suffer and is suffering irreparable harm from Samsungs infringement
of Registered Icon Trademarks insofar as Apples invaluable good will is being eroded by
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

30

Samsungs continuing infringement. Apple has no adequate remedy at law to compensate it for
the loss of business reputation, customers, market position, confusion of potential customers and
good will flowing from the Samsungs infringing activities. Pursuant to 15 U.S.C. 1116, Apple
is entitled to an injunction against Samsungs continuing infringement of Apples Registered Icon
Trademarks. Unless enjoined, Samsung will continue its infringing conduct.
109. Because Samsungs actions have been committed with intent to damage Apple and
to confuse and deceive the public, Apple is entitled to treble its actual damages or Samsungs
profits, whichever is greater, and to an award of costs and, this being an exceptional case,
reasonable attorneys fees pursuant to 15 U.S.C. 1117(a) and 1117(b).
FOURTH CLAIM FOR RELIEF

(Common Law Trademark Infringement)
110. Apple incorporates and realleges paragraphs 1 through 109 of this Complaint.
111. Apple has prior rights in Apples Registered Icon Trademarks and the Purple
iTunes Store Trademark.
112. The Samsung Galaxy line of products have infringed Apples Registered Icon
Trademarks and the Purple iTunes Store Trademark by using identical or similar application icons
in Samsungs products.
113. Samsungs use of its infringing application icons is likely to cause confusion, or to
cause mistake, or to deceive the consumer as to the affiliation, connection or association of
Samsung with Apple, or as to the origin, sponsorship, or approval by Apple of Samsungs goods,
services or commercial activities.
114. Samsungs use of the infringing application icons enables Samsung to benefit
unfairly from Apples reputation and success, thereby giving Samsungs infringing products sales
and commercial value they would not have otherwise.
115. Prior to Samsungs first use of the infringing application icons, Samsung was
aware of Apples business and had either actual notice and knowledge, or constructive notice of
Apples Registered Icon Trademarks and the Purple iTunes Store Trademark.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

31

116. Samsungs unauthorized use of the infringing application icons is likely, if not
certain, to deceive or to cause confusion or mistake among consumers as to the origin,
sponsorship or approval of the Samsung Galaxy line of products and/or to cause confusion or
mistake as to any affiliation, connection or association between Apple and Samsung, in violation
of 15 U.S.C. 1114(a).
117. Apple is informed and believes, and on that basis alleges, that Samsungs
infringement of Apples Registered Icon Trademarks and the Purple iTunes Store Trademark as
described herein has been and continues to be intentional, willful and without regard to Apples
rights in its Registered Icon Trademarks and the Purple iTunes Store Trademark.
118. Apple is informed and believes, and on that basis alleges, that Samsung has gained
profits by virtue of its infringement of Apples Registered Icon Trademarks and the Purple iTunes
Store Trademark.
119. Apple will suffer and is suffering irreparable harm from Samsungs infringement
of Apples Registered Icon Trademarks and the Purple iTunes Store Trademark insofar as
Apples invaluable good will is being eroded by Samsungs continuing infringement. Apple has
no adequate remedy at law to compensate it for the loss of business reputation, customers, market
position, confusion of potential customers and good will flowing from the Samsungs infringing
activities. Apple is entitled to an injunction against Samsungs continuing infringement of
Apples Registered Icon Trademarks and the Purple iTunes Store Trademark. Unless enjoined,
Samsung will continue its infringing conduct.
120. Because Samsungs actions have been committed with intent to damage Apple and
to confuse and deceive the public, Apple is entitled to treble its actual damages or Samsungs
profits, whichever is greater, and to an award of costs and, this being an exceptional case,
reasonable attorneys fees pursuant to 15 U.S.C. 1117(a) and 1117(b).
FIFTH CLAIM FOR RELIEF

(Unfair Business Practices California Business and Professions Code 17200, et seq.)
121. Apple incorporates and realleges paragraphs 1 through 120 of this Complaint.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

32

122. The acts of Samsung described above constitute fraudulent and unlawful business
practices as defined by California Business & Professions Code 17200, et seq.
123. Apple has valid and protectable prior rights in the Apple Product Trade Dress, the
Registered Trade Dress, the iTunes Eighth Note and CD Design Trademark, the Purple iTunes
Store Trademark, and the Registered Icon Trademarks. The Apple Product Trade Dress and the
Registered Trade Dress do not serve any function other than to identify Apple as the source of its
mobile products. The Apple Product Trade Dress and the Registered Trade Dress are inherently
distinctive, and, through Apples long use, have come to be associated solely with Apple as the
source of the products on which it is used.
124. Samsungs use of its infringing trade dress is likely to cause confusion as to the
source of Samsungs products and is likely to cause others to be confused or mistaken into
believing that there is a relationship between Samsung and Apple or that Samsungs products are
affiliated with or sponsored by Apple.
125. The above-described acts and practices by Samsung are likely to mislead or
deceive the general public and therefore constitute fraudulent business practices in violation of
California Business & Professions Code 17200, et seq.
126. The above-described acts constitute unfair competition under Section 43(a) of the
Lanham Act, 15 U.S.C. 1125(a) and trademark and trade dress infringement under Section 32 of
the Lanham Act, 15 U.S.C. 1114, and are therefore unlawful acts in violation of California
Business & Professions Code 17200, et seq.
127. Samsung acted willfully and intentionally in designing its infringing trade dress,
with full knowledge of Samsungs prior rights in the distinctive Apple Product Trade Dress,
Registered Trade Dress, the iTunes Eighth Note and CD Design Trademark, the Purple iTunes
Store Trademark, and the Registered Icon Trademarks and with an intent to cause confusion or
mistake or to deceive customers into believing that there is an affiliation between Samsung and
Apple or between Samsungs products and Apples products.
128. The unlawful and fraudulent business practices of Samsung described above
present a continuing threat to, and is meant to deceive members of, the public in that Samsung
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

33

continues to promote its products by wrongfully trading on the goodwill of the Apple Product
Trade Dress, Registered Trade Dress, the iTunes Eighth Note and CD Design Trademark, Purple
iTunes Store Trademark, and the Registered Icon Trademarks.
129. As a direct and proximate result of these acts, Samsung has received, and will
continue to profit from, the strength of the Apple Product Trade Dress, Registered Trade Dress,
the iTunes Eighth Note and CD Design Trademark, the Purple iTunes Store Trademark, and the
Registered Icon Trademarks.
130. As a direct and proximate result of Samsungs wrongful conduct, Apple has been
injured in fact and has lost money and profits, and such harm will continue unless Samsungs acts
are enjoined by the Court. Apple has no adequate remedy at law for Samsungs continuing
violation of Apples rights.
131. Samsung should be required to restore to Apple any and all profits earned as a
result of their unlawful and fraudulent actions, or provide apple with any other restitutionary
relief as the Court deems appropriate.
SIXTH CLAIM FOR RELIEF

(Unjust Enrichment)
132. Apple incorporates and realleges paragraphs 1 through 131 of this Complaint.
133. As a result of the conduct alleged herein, Samsung has been unjustly enriched to
Apples detriment. Apple seeks a worldwide accounting and disgorgement of all ill gotten gains
and profits resulting from Samsungs inequitable activities.
SEVENTH CLAIM FOR RELIEF

(Infringement of the 022 Patent)
134. Apple incorporates and realleges paragraphs 1 through 133 of this Complaint.
135. Samsung has infringed and continues to infringe one or more claims of the 022
Patent by using, selling and/or offering to sell, in the United States and/or importing into the
United States, one or more of the Samsung mobile communication devices identified in this
Complaint. Samsungs infringing activities violate 35 U.S.C. 271.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

34

EIGTH CLAIM FOR RELIEF

(Infringement of the 381 Patent)
136. Apple incorporates and realleges paragraphs 1 through 135 of this Complaint.
137. Samsung has infringed and continues to infringe one or more claims of the 381
Patent by using, selling and/or offering to sell, in the United States and/or importing into the
United States, one or more of the Samsung mobile communication devices identified in this
Complaint. Samsungs infringing activities violate 35 U.S.C. 271.
NINTH CLAIM FOR RELIEF

(Infringement of the 134 Patent)
138. Apple incorporates and realleges paragraphs 1 through 137 of this Complaint.
139. Samsung has infringed and continues to infringe one or more claims of the 134
Patent by using, selling and/or offering to sell, in the United States and/or importing into the
United States, one or more of the Samsung mobile communication devices identified in this
Complaint. Samsungs infringing activities violate 35 U.S.C. 271.
TENTH CLAIM FOR RELIEF

(Infringement of the 828 Patent)
140. Apple incorporates and realleges paragraphs 1 through 139 of this Complaint.
141. Samsung has infringed and continues to infringe one or more claims of the 828
Patent by using, selling and/or offering to sell, in the United States and/or importing into the
United States, one or more of the Samsung mobile communication devices identified in this
Complaint. Samsungs infringing activities violate 35 U.S.C. 271.
ELEVENTH CLAIM FOR RELIEF

(Infringement of the 915 Patent)
142. Apple incorporates and realleges paragraphs 1 through 141 of this Complaint.
143. Samsung has infringed and continues to infringe one or more claims of the 915
Patent by using, selling and/or offering to sell, in the United States and/or importing into the
United States, one or more of the Samsung mobile communication devices identified in this
Complaint. Samsungs infringing activities violate 35 U.S.C. 271.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

35

TWELFTH CLAIM FOR RELIEF

(Infringement of the 891 Patent)
144. Apple incorporates and realleges paragraphs 1 through 143 of this Complaint.
145. Samsung has infringed and continues to infringe one or more claims of the 891
Patent by using, selling and/or offering to sell, in the United States and/or importing into the
United States, one or more of the Samsung mobile communication devices identified in this
Complaint. Samsungs infringing activities violate 35 U.S.C. 271.
THIRTEENTH CLAIM FOR RELIEF

(Infringement of the 533 Patent)
146. Apple incorporates and realleges paragraphs 1 through 145 of this Complaint.
147. Samsung has infringed and continues to infringe one or more claims of the 533
Patent by using, selling and/or offering to sell, in the United States and/or importing into the
United States, one or more of the Samsung mobile communication devices identified in this
Complaint. Samsungs infringing activities violate 35 U.S.C. 271.
FOURTEENTH CLAIM FOR RELIEF

(Infringement of the D790 Patent)
148. Apple incorporates and realleges paragraphs 1 through 147 of this Complaint.
149. Samsung has infringed and continues to infringe the D790 Patent by using, selling
and/or offering to sell in the United States, and/or importing into the United States one or more of
the Samsung mobile communication devices identified in this Complaint, which embody the
design covered by the D790 design patent.
FIFTEENTH CLAIM FOR RELIEF

(Infringement of the D016 Patent)
150. Apple incorporates and realleges paragraphs 1 through 149 of this Complaint.
151. Samsung has infringed and continues to infringe the D016 Patent by using, selling
and/or offering to sell in the United States, and/or importing into the United States one or more of
the Samsung mobile communication devices identified in this Complaint, which embody the
design covered by the D016 design patent.
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

APPLE INC.S COMPLAINT
sf-2981926

36

SIXTEENTH CLAIM FOR RELIEF

(Infringement of the D677 Patent)
152. Apple incorporates and realleges paragraphs 1 through 151 of this Complaint.
153. Samsung has infringed and continues to infringe the D677 Patent by using, selling
and/or offering to sell in the United States, and/or importing into the United States one or more of
the Samsung mobile communication devices identified in this Complaint, which embody the
design covered by the D677 design patent.
PRAYER FOR RELIEF

WHEREFORE, Apple prays for relief, as follows:
1. A judgment that Samsung has infringed one of more claims of each of Apples
asserted patents;
2. An order and judgment preliminarily and permanently enjoining Samsung and its
officers, directors, agents, servants, employees, affiliates, attorneys, and all others acting in
privity or in concert with them, and their parents, subsidiaries, divisions, successors and assigns,
from further acts of infringement of Apples asserted patents;
3. A judgment awarding Apple all damages adequate to compensate for Samsungs
infringement of Apples asserted patents, and in no event less than a reasonable royalty for
Samsungs acts of infringement, including all pre-judgment and post-judgment interest at the
maximum rate permitted by law;
4. A judgment awarding Apple all damages, including treble damages, based on any
infringement found to be willful, pursuant to 35 U.S.C. 284, together with prejudgment interest
5. An order preliminarily and permanently enjoining Samsung and its officers,
directors, agents, servants, employees, affiliates, attorneys, and all others acting in privity or in
concert with them, and their parents, subsidiaries, divisions, successors and assigns, from directly
or indirectly infringing the Apple Product Trade Dress, Registered Icon Trademarks, Purple
iTunes Store Trademark, and iTunes Eighth Note and CD Design Trademark, or using any other
product or packaging design or designations similar to or likely to cause confusion with the Apple
Product Trade Dress, Registered Icon Trademarks, Purple iTunes Store Trademark, and iTunes
I Eighth Note and CD Design Trademark; from passing off Samsilllg's products as being associated
2 with and or sponsored or affiliated with Apple; from committing any other unfair business
3 practices directed toward obtaining for themselves the business and customers of Apple; and from
4 committing any other unfair business practices directed toward devaluing or diminishing the
5 brand or business of Apple.
6 6. Actual damages suffered by Apple as a result of Samsung' s unlawful conduct, in
7 an amount to be proven at trial, as well as prejudgment interest as authorized by law;
8
9
10
11
12
7.
8.
9.
10.
11.
Reasonable funds for future corrective advertising;
An accounting ofSamsung's profits pursuant to 15 U.S.C. 1117;
A judgment trebling any damages award pursuant to 15 U.S.C. 1117;
Punitive damages pursuant to California Civil Code 3294;
Restitutionary relief against Samsung and in favor of Apple, including
13 disgorgement of wrongfully obtained profits and any other appropriate relief;
14 Costs of suit and reasonable attorneys' fees; and
15
12.
13. Any other remedy to which Apple may be entitled, including all remedies provided
16 for in 15 U.S.C. 1117, Cal. Bus. & Prof Code 17200, et seq., 17500, et seq., and under any
17 other California law.
18
19
20
21
22
23
24
25
26
27
28
Dated: Apri115, 2011
APPLE INC.'S COMPLAINT
Case No.
sf-2981926
HAROLDJ.MCELHll{NY
MICHAEL A. JACOBS
JENNIFER LEE TAYLOR
JASON R. BARTLETT
MORRISON & FOERSTER LLP
By " " ' - - - ~ ~ 0 (L ~
MICHAEL A. JAC S
Attorneys for Plaintiff
APPLE INC.
37
1 DEMAND FOR JURY TRIAL
2 Pursuant to Rule 38(b) of the Federal Rules of Civil Procedure, Apple hereby demands
3 trial by jury on all issues raised by the Complaint.
4
5
6 Dated: April15, 2011
7
8
9
10
HAROLD J. MCELHINNY
MICHAEL A. JACOBS
JENNIFER LEE TAYLOR
JASON R. BARTLETT
MORRISON & FOERSTER LLP
By/' . - 'Q ~ ~ ~
MICHAEL A. JACO -
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
APPLE INC.'S COMPLAINT
Case No.
sf-2981926
Attorneys for Plaintiff
APPLE INC.
38












































Exhibit 1
(12) United States Patent
Westerman et al.
(54) ELLIPSE FITTI:'\G FOH.
SURE\(:ES
(75) !mentors: \Vestt>rman. San Francisco. CA
(US): .John G. Elias, 'lllwnscnd, ])];
(US)
Assignee: Apple Inc .. Cupc11ino. CA (US)
( * ) Notice: Subject to any disclaimer. the term of this
pntcnt is extended or adjusted under 35
U.S.C. l54(b) by 707 days.
(21) App!. No.: 111677,958
(22) Filed:
(65)
Feb. 22, 2007
Prior Publication I>ata
US 2(Xl7/0!39395 AI Jun. 2!. 2(XJ7
H.clatcd U.S. Application I>ata
(63) Continuation of application No. 111015.434. Hied on
Dec. 17. 2004. now Pat. No. 7.339.580. which is a
ctmlinuation of application No. 091236.513. !!led on
Jan. 25. 1999. now Pal. No. 6.3n.R46.
(60) Provisional application No. 60/072,500. tiled on Jan.
26. 199R.
(51) Int.Cl.
<ifJ6F 311141
(52) u.s. Cl.
(2006.01)
345/173:
l7R/!R.Ol: l?R/IR.m
(5R) Field ofCiaJ>J>ification Search 345/173 17R:
178/18.01.18.03.19.01.20.01: 715/863
Sec application Jile l(Jr complete search history.
(56) RcfcrcnccJ> Cited
U.S. P.\TI:NT I)OCUMI:N"J"S
A 7 1967 Gorski
3.54!.54t
'
"
t<)7() Fngdlmrl
3.ti62.!0."
'
5 t<)72 llursl d al
A 3 1974 llurst
4.D7.42l A 12
__ ,.
I "",:,_., f.-
111111 1111111111111111111111111111111111111111111111111111111111111
CA
US007812828B2
(I OJ Patent No.: US 7,812,828 B2
Oct. 12,2010 (45) Date of Patent:
4.24ti.452
'
l ch;mdkr
4.2titi.t44
'
"
Bristol
4.2<JO.Otit
'
9

'
9J<)g] Cutkrd
4 .. 'i50.221
'
10 ]<)g)
4 .. 'itil.002
'
12J<)g) Chiu
4.672.364
'
6 l.ucas
4.672 .
'
6 Fkchs al
(Continu<.'d)
I:ORIOICiN I'Al"ION"I"I)OC"UMI;N"l"S
12430%
(Continued)
Oriii:t{ PUBIJC..\TIONS
C-C et (.\ug I. "'A :-.:em-est
:-.:cighbm Pauem Rt."N>gnition f.t."llen.
(i2.'i-630
(Continu<.'d)
f'rimwT Hxaminer Amarc Meng.islu
Assislanl Ewminer Koosha Shari!i
(74) Allome_1, Agenl. or Firm Morrison & Foerster I J.P
(57) AHSTR\(T
.\ppmatus and methods arc disclosed Jix simultaneously
tracking multiple !Inger and palm contacls as hands approach,
touch, and slide across a proximity-sensing. multi-touch sur-
!Ucc. ldcnti!ication and classi!ication of intuitive hand con-
!ig.urmions and motions enables unprcc<.-dcntcd integration of
typing. resting. pointing. scrolling. 3D manipulation. and
handwriting into a versatile. ergonomic computer input
device.
Claims, 45 Drawing Shccts

4.(i92.iW9 A
4.(i9.'i.iQ7 A
4.733.222 _\
4.734.(iK'i A
4.746.770 A
4.77!.276 _\
4.7l>iUg,J _\
4.go6.iW> A
4.1NiC5-':;S _\
4.<J@.l>77 _\
.'i.Om5t9 A
A
.".171>.477 _\
S.Jg<J.403 _\
.'i.l94.U>2 A
.".224.1>61 _\
.".24!..101> _\
.'i.2.'i2.9.'i1 A
5.270.711 A
.".21<!.<)66 _\
.'i.30.'i.017 A
.'i.327.l61 A
." .. 14.".543 _\
." .. 169.221> _\
5.376.941> A
.'i.379.0.'i7 A
S.WlU!O _\
5.442.742 A
5.463.31>1> A
.".463.(><)6 _\
.".47!.571> _\
5.479521> A
S.41G.261 _\
.".41>1<.204 _\
.'i.49.'i.077 A
.'i5D.309 A
.".523.77." _\
.".530.4-'-' _\
5543590 A
.".543.591 _\
S.541Hi67 _\
.'i563.(i32 A
5563.996 A
S.56S.ti."l> _\
5579.036 A
.'i5l>l.276 A
S.51H.4g,J _\
S.51H.MH _\
.'i5iG.946 A
.'i51>9X'i6 A
.".590.219 _\
5592566 A
5594.469 A
.".594.1>!0 _\
S.596.ti94 _\
.'i.(i12.719 A
S.ti2.".71S _\
S.ti3U>OS _\
.'i.(iD.<J.'i.'i A
.'i.(i34.l02 A
S.ti36.101 _\
S.ti42.101> _\
.'i.(i44.(i.'i7 A
S.ti66.1 13 _\
S.ti6ti.502 _\
.'i.ti665.'i2 A
.'i.ti7.'i.361 A
S.ti7." .. 162 _\
S.ti77.710 _\
A
A
lkiningdal
lkiningdal
h;ms
3 Wmanabc
5
<) l'"rks
11 Bruere-Dmvsond "I
2 Kerber
2 1<)90 S<l!npson
11 1<)90 "1
3 1991 :\oiljean
5 1991
1 1<)9_1 G;unlmro
21<)9_1 Fr;m/d"l
3 1993 Ldwards
7 1<)9_1 Gbss d "1
1<)9_1 Young
10 1993 -L1nnenbaum et al
121993 Knappetal
1 1<)94 Walsh
4 1994 Gerphcide
7 1994 Logan et al
<) 1<)94 C.1pps d "1
11 1<)94 hmsl
12 1994 Robc11s
I 1995 Clough d al
.11<}95 Tch"'' d "1
1995 d al
101995 Boicetal
10 1<}95 Beernink d al
11 1<}95 \1omn el al
12 1995 Spo.:der
1 1<)<}(, Yasul"kc
1 1<)<}(, \1e"'ld"l
2 19% Vliller et al
419% Vlcierdal
ti 1<)<}(, C.1pps
ti 1<)<}(, Gillick el al
19% Gillespie d al
1<)<)(, Gillespie cl al
1<)<)(, Tu
10 19% Robc11s
10 19% Tdlao
10 1<)<}(, Gerpheide el al
II 19% Ymes.l\'
12 19% Cipolla d al
12 1<)<}(, Prince
12 1<)<}(, Tch"'' d "1
12 19% Gourdol
12 19% Stein et al
12 1<)<}(, Gourdol
I 1997 l'agallo et al
1997 Frwman et al
1<)97 Gourdol
1<)97 C.1pps
3 1997 lkemink d al
4 1<)97 TR'W cl al
5 1<}97 Bons"H
5 1997 Bo;-inovk d al
5 1997 Capps
ti 1<)97 Bons"H d "1
ti 1<)97 Gopherd"l
71997 Cappsetal
<) 1<}97 I og;m
<) 1<}97 C.1pps
9 1997 d al
10 1997 S<llllilli
10 1<)97 Clough d al
10 1<)97 Thompson-Rohrlich
II 1997 llargreavesdal
I 19n Capps et al
US 7,812,828 B2
Page 2
5.729.249 .\
5.72<}.250 -\
5.730.1(,5 .\
5.73ti.97ti .\
5.741.9<}0 -\
5.745.11(, -\
5.745.71(, .\
3 Yasutake
3 Bishopd "1
3 Philipp
4 Cheung
4 Dmies
4 Pisullm-_\mond
lchaodal
.\ .'i Ymake
-\ -' Shieh .......................... 14-' 17_1
.\ .'i llanisd al
5.7ti4.222 .\
5.7ti7.457 -\
5.7ti7.g42 -\
5.790.104 .\
5.7<)0.107 -\
5.go2.s1r; -\
. .'iti7 .\
.\
5.g2J.6<)0 -\
5.g21.9.10 -\
.\
5.g25.351 -\
6 Shieh
6 Ge1pheide d "1
6 Korlh
Shieh
g Kasserd"l
9 Shw,rlsd"l
9 McCloud
Ylomndal
10 \1arlms d al
10 llansen
10 Yl<ll'CliS et aJ
10 T<l!n
5.g25.352 -\ 10 Bisset cl al
.\ II Shieh
II kterv>ak
Frischdal
I 1999 Shieh
3 1999
3 19<)<) Gillespie el "I
3 19<)<) llo d ;!I
3 1999 :\aughton d al
_14-' 17.1
.\
5.g54.625 -\
.\
.\
-\
5.gK1.61<J -\
.\
.\ $
-\
5.917.47(, .\
5.920.309 .\
5.92.1,31<) -\
5.9.1.1. 134 -\
5.943.043 .\
5.94.1.044 -\
-\
5.995.10(, .\
.\
-\
-\
ti.03L'i24 .\
-\
ti.050.g25 -\
ti.052.339 .\
ti.Otil.177 .\
ti.072.4<)4 -\
.. 'i7(i .\
ti.097.371 .\
ti.107.9<)7 -\
-\
ti.131.299 .\
.\
-\
.\ $
ti.l54.209 .\
ti.1ti0 . ."51 -\
Bl
.. 'il5 Bl
B 1
ti.222.4ti5 B 1
ti.239.790 Bl
ti.243.071 Bl
ti.24ti.gti2 B 1
ti.24<J.60ti Bl
ti.255.604 Bl
B 1
Bl
Bl
ti.292.1n Bl
4 1999 Shieh ......................... 34.'i 173
4 19<)<) Small el al
6 1999 C;-ernic<:ki
7 1999 Bisset d al
7 19<)<) Bishopd "1
g 19<)<) Shieh
1999 lumhata et al
g 19<)<) \1arlindli d al
11 19<}<) L"re
II 1999 :\aughton d al
12 1999 Kasser
12 19<}<) Freem;m
2 2000 :\aughlon d al
2 2000 Kunen
3 2000 ]_e\)'
4 2000 :\ichol d "1
4 2000 1-renkd d al
.'i 2000 lujimoto
6 2000 :\guyen
7 2000 Leu et al
2000 Siddiqui d al
g 2000 L"re
10 2000 Smith el al
102000 Raabetal
10 2000 Ylikula-Ctutis d al
11 2000 Sch\\arls el al
II 2000 l<lleev ........................ 34.'i 173
II 2000
12 2000
2 2001
3 2001
3 2001
4 2001
.'i 2001
6 2001
6 2001
6 2001
7 2001
g 2001
9 2001
9 2001
9 2001
:\aughton d al
:\aughlon d al
Seely el al
Cole
Balh!re
Kumarel al
Yl<lltindli d al
ShwaJtsd al
err;,-," d al
Kimlydal
lokioka ct al
-\mro el al
Philipp
LaFleur
Bcmstein ct al
Bl
6 .. 123.1>49 Bl
6.344.U>l Bl
6.347.290 Bl
6 .. 177JJO<J Bl
6 .. 1l>O.<J3! Bl
Bl
6.4l4.(i7J Bl
6.42!.234 Bl
6.4.'i25l4 Bl
6.4.'i7.3.'i.'i Bl
6.466.(!36 Bl
6.5!S.ti6<J Bl
6525.749 Bl
6.53.''.200 B2
6.543.tig,J Bl
6543.947 B2
6570557 Bl
6.5<J3.<Jl6 Bl
6.ti!O.'H6 B2
6.(i24.iG3 Bl
6.ti39.577 B2
6.ti."0 .. 1l<J Bl
Bl
6.(i7Q.g94 B2
6.ti77.<J32 Bl
6.(>77.934 Bl
6.724.366 B2
6.7."7JJ02 Bl
6.1>03.<J06 Bl
6.g42.(i72 Bl
6X'i6.2.'i9 Bl
6.1>1>1>.536 B2
6.900.795 Bl
6.927.761 B2
6.<J42.57! Bl
6.<J6." .. 17S Bl
6.972.401 B2
6.<J77.ti66 Bl
6.<JK''.l>Ol Bl
6.992.(i.'i9 B2
7mt.221> B2
7.0g,J.l>-':;<J Bl
7.129.416 Bl
7.240.21>9 B2
R!A0.1."_1 F
7 .. 139.51>0 B2
2002 OJJgg4g At
2002 0140(>61> At
200_1 000(,<)74 _\ l
2003 007(>301 At
20o3 007(i3m At
200_1 007(>_106 _\ l
200_1 ()()<))()<)_'; _\ l
2003 0095096 A 1
20o3 oong:;:; At
200_1 0132<)-':;0 _\]
2003 0!64iQO At
2003 020(>202 At
200_1 02347@ _\ l
2004 014tiM<g _\!
2004 02634il4 At
2005 00!2723 _\]
2005 00-':;242-':; _\]
2005 0104U>7 At
2005 OJ to76g At
200(i 0022<)-':;-':; _\]
200(i 0022<)-':;6 _\]
200(i 002(>52 t At
200ti 002tiH' _\ l
200(i 002ti536 _\]
200(i 0032(>110 At
200(i 0033724 At
t I 200 I Westerman eta!
!1 2001 lle
2 2002 :\aughton et al
2 2002 Banktt
4 2002 Philipp
4 2002 d at
(i 2002 et al
7 2002 Gillespie ct a!
7 2002 Ricks ct al
<J 2002 Philipp
to 2002 Philipp
!0 2002 Philipp
2 20o.1 \1ohri
2 2003 Vloran eta!
_1 20o.1 Philipp
4 20o.1 ct ,]
4 2003 Lee
5 2003 Westerman eta!
7 20o.1 _\roy"n
11 20o.1 d at
<J 2003 Kumar ct al
!0 20o.1
!1 20o.1 llurst ct al
12 2003 Vlc\lliHan
12 2003 Vlehring
1 2004
I 2004 B!andmrd
4 2004 Crawford
(i 2004 {)ross ct ,]
!0 2004 \1orrison ct al
I 2005 Straub eta!
2 2005 Shmp
5 2005 ,]
5 2005 Knight. 111 d a!
11 2005 Badaye ct al
<) 2005 \k\llisler d at
!1 2005 at
12 2005 Akitt eta!
!2 2005 Hedrick
1 200(i Stmub ,]
I 200(i Gcttemy
4 200(i Born eta!
11200(i Pryor
!0 200(i Steinfdd d at
7 2007 :\aughton et al
_1 20011 ,]
_1 20011 ,]
11 2002 Karpenstein
to 2002 Crawford
1 20o.1 Clough ct al
4 2003 Tsuk eta!
4 2003 lluppi
4 20o.1 i'"'lesky d at
520o.1 Pihl"j"
5 2003 Robbin ct al
5 2003 Perski et al
7 200_1 Sumcu d at
<J 2003 Kent
t I 2003 Vloriya
!2 20o.1 Rekimolo ct ,]
7 2004 Treat
12 2004 Vlontysalo d a!
1 2005 p,Jtako!T
.12005 i'"'leski ct ,]
5 2005 Westerman eta!
5 2005 Vlmriou eta!
2 200(i
2 200(i I engeling
2 200(i lloteHing ct al
2 200(i llotelling ct ,]
2 200(i llotelling ct ,]
2 200(i Hias eta!
2 200(i Chaudhri ct al
US 7,812,828 B2
Page 3
2006 00533117 .\1
2006 OO(i(i-':;112 -\1
2006 00115757 .\1
200600'!79<)1 .\1
20060!<)775_1 -\1
2006 02.12-':;(,7 -\1
200602311.'il7 .\1
200602311.'il11 .\1
200602.111-':;1<) -\1
2006 02311520 .\1
2006 02311521 .\1
2006 02.111-':;22 -\1
2007 0070050 -\1
2007 0070051 .\1
2007 0070052 -\1
2007 0071191<) -\1
2007 0011172(> .\1
2007 022<J4(i4 .\1
2007 02_1(i4(i(i -\1
2007 024742<) -\1
2007 025711'!0 .\1
2007 02(,1127_1 -\1
2007 02ti11274 -\1
2007 02(>11275 .\1
20011 004163<) .\1
2oog 0042911(, -\1
20011 00429il7 .\1
20011 00429il11 .\1
2oog 0042911<) -\1
2oogoJ211!112 -\1
2009 0021411<) .\1
2009 0J(i011J(i .\1
20090244o.11 -\1
2009 0244032 .\1
2009 0244033 .\1
2009 024<)2_1(i -\1
200902514.15 -\1
2009 02514311 .\1
200902514_1<) -\1
3 200(i Ording
3 200(i I .yon ct ,]
4 200(i .\mired al
.'i 200(i llotdling eta!
9 200(i lloldling
10 200(i ct ,]
10 200(i King ct al
10 200(i \\'esterman ct al
10 200(i ct ,]
10 200(i \\'esterman ct al
10 200(i \\'esterman ct al
10 200(i ct ,]
3 2007 ct ,]
3 2007 \\'esterman ct al
3 2007 ct ,]
4 2007 ct ,]
4 2007 \\'esterman ct al
10 2007 llotdling eta!
10 2007 lloldling
10 2007
It 2007 llotdling eta!
1! 2007 ct ,]
1! 2007 ct ,]
I t 2007 \\'esterman ct al
2 20011 \\'esterman ct al
2 20011 ct ,]
2 20011 \\'esterman ct al
2 20011 \\'esterman ct al
2 20011 ct ,]
6 20011 ct ,]
t 200'! \\'esterman ct al
6 200'! \\'esterman ct al
10 200<) ct ,]
10 200'! \\'esterman ct al
10 200'! \\'esterman ct al
10 200<) ct ,]
10 200<) ct ,]
10 200'! \\'esterman ct al
10 200<) ct ,]
I:ORIOICiN I'AriON'I-IJOC'UMI;N"rS
DF
LP
FP
FP
LP
LP
FP
LP
LP
FP
FP
LP
GB
IP
Jl'
Jl'
IP
IP
Jl'
IP
IP
Jl'
wo
\\'0
\\'0
wo
wo
\\'0
\\'0
wo
wo
!02-':; 129ti _\ l
0 .'i4(i 704 A2
0 -':;4(i 704 _\3
02g11692
0 622 722 A2
0 622 722 A3
0 622 722 Bl
0664504
04649011
ogJ7000 _\]
ogJ7000 Bl
1014295
2330670 A
02-05ti0!7 _\
02-304613 A
04-23361'! A
06-0K1S2_1 _\
06-3.12620 _\
07-057103 A
07-!2<)3!2 _\
og-211r;g3o _\
2000-J(i3031 A
W0-92 02000
\\'0-% 07'W1 _\!
\\'0-97 111-':;47
W0-97 237311
W0-97 3(>225
\\'0-97 40744
\\'0-911 14&>_1
W0-9'! 31114'! At
\\'( >-03 01111 t 7(i
5 2004
(i 1993
(i 19<)_1
7 19<)_1
II 1994
II 1994
11 19<)4
19<)5
<) J9<)(i
1 19<)11
1 19<)11
I 2002
4 19<)<)
2 19<)0
12 19<)0
11 1992
.119<)4
12 19<)4
3 19<)5
5 19<)5
11 19<)(i
(i 2000
2 1992
_119<)(i
5 19<)7
7 19<)7
to 19<)7
11 19<)7
4 19<)11
7 19<)<)
to 2003
US 7,812,828 B2
Page 4
\\'() W0200(i OD569 2006
Crowky. J I (\hr 1. l <J<J7) "Vision](,-
Robotic.< am/ /wwumrwu.< \nlem.<.
Dm is, J cl al (\1"y 2. 19<)4) !land Gestures," Furo-
1"-'a/1 (f"!knme 011 (f>mpwer Vi1ion. Stockholm, SL l 331-340
Scmch Rcron Dec 12. 200g. !Or Fl'
:\o 0(i0J(iK'i.'i 6 fikd 2.'i. 1999. six pages
Eurorcan Scm-.:h Rq>orl mailed Dec n. 20mc for U' .\pplication
'\o OtiO!MG 1 7 lilcd Jan 2.", ]<)<)<)_ sc\cn pages
h1mpcan Scardllkp011 maikd Jan 9. 2009. for Fl' .\pplkmion '\o
06016K12-' likd .h!n 2.". ]<)<)<),four pages
!lc"P T cl ,] ((kl 14, ]<)<}6) '({man!s 3D l!;md Tracking l:sing"
Dcfonnablc Model." of 1/w 2'"
1
Imemarional Confi'r-
<'IIC<'. Killington. V J. l'SA. <let !4-!6. !9% . . 1utomaric race am/
Rt.'('(>gnilion. lf1F Compul. Soc .. pp !40-!4."
Rq>ot1 2005.!0r PC l
tion :--:o PC l l'S05 OD25. fik-.1 Jcb 2005. two
:--:irci. K cl a! ( -\ug 5. !996) "llmmmll;md Tmcking from Binocli-
h!r Image Sequences," Pr<xt.edingl f!(llie I'J'J6lf1F IFCO\' 22'"
1
Imemarioua/ Con(<'l'<'!l('<', latjNi. kzi,wl. Aug 5-10. 1996,/ndu.'-
rria/ Uei'/ronic.,, Comrol. and In.l/l'llll!<'illatiou !(5) 297-302
:--:on-Fim!l ()Jiicc -\clion m"ikd Scp 21.2007. J(lf I.'S -\ppl :--:o
ll !5. fikd Jul 3. 2006. seven
:--.:on-1-inal maik-.t Scp 24.2007. for l. S .\ppt :--:o
1! 421> . ."06. filed Jul .1. 2006. six pages
:--:on-Fim!l ()Jiicc -\clion m"ikd Scp 24.2007. J(lf l:S -\ppl :--:o
ll fikd Jul 3. 2006. six
:--:on-Fim!l On<cc .\ction m"ikd Jan 20mc ](,-!; S .\ppl :--:o
1! 421> . ."22. filed Jul .1. 2006. sc1cn pages
:--.:on-1-inal maik-.t Jcb 4. for l. S .\ppt :--:o
ll !5. fikd Jul 3. 2006. six
:--:on-Fim!l On<cc -\clion mailed Jut <J. 2001>. for 1: S .\ppl :--:o
1! 421> . ."21. filed Jul .1. 2006. ll P"gcs
:--.:on-1-inal .\dion Sep 2. 20m:. !Or L S .\ppt :--:o
1! 421> . ."22. filed Jul .1. 2006. six pages
:--:on-Fim!l ()Jiicc -\clion m"ikd Scp !5. 20mc J(lf l:S -\ppl :--:o
ll fikd Jul 3. 2006. eight
:--.:on-1-inal Ollkc Od 3!. 20m:. !Or l. S .\ppt :--:o
1!421> . ."!5. filed Jul .1. 2006. sc1cn pages
:--.:on-1-inal Ollkc Feb 17.2009. !Or l. S .\ppt :--:o
ll fikd Jul 3. 2006. six
:--:oticc of .\ll<m,bility m"ik-.1 Feb !1. 200<), for 1: S .\ppl :--:o
1! 421> . ."21. filed Jul .1. 2006. fi1c P"gcs
l. S Appl :--:o ll fikd Jan l 2006 which is a Reissue of
(>.677.932
Wcslcrman, W (Jan !. 19<J<J) "!land Tracking. Finger ldcntilic,tion.
Chordk Maniputmion V!ulti-loudl Dissct1ation.
l'nivcrsity of pp !-3D
Final On<cc .\ction mailed \1"r !9. 2009. ](,- 1; S .\ppl :--:o
1! 421> . ."06. filed Jul .1. 2006. sc1cn pages
:--:on-1-inal Adion VIM HL 2009. !Or{. S .\ppt :--:o
1!421> . ."!5. filed Jul .1. 2006. !2 P"gcs
:--:on-Fim!l OJiicc -\clion m"ikd -\pr 2. 200<). ](,-!: S -\ppl :--:o
ll l. fikd Jul 3. 2006. !! pages
:--.:on-1-inal maik-.t .\pr 2. 2009. for l' s .\ppt :--:o
1! 421> . ."0.1. filed Jul .1. 2006. !2 P"gcs
:--:oticc of .\llmv,bility mailed Jut !0. 200<), for 1; S -\ppl :--:o
ll fikd Jul 3. 2006. fom pages
Rubin<:. D (\hy 1<)92) "Combining Gestures ;md Direct \1;mipli-
h!tion." {']II '<)2, pp 659-660
l. S Appl :--:o !2 342.027. fik-.t 22. 2001>. by \\'cstcrman ct
ts Appt :--:o !2422.l97.fik-.l.\pr
1: S .\ppl :--:o !2 422.20.". Jikd -\pr 10. 200<). by \\'cslcrman cl a!
1: S .\ppl :--:o !2 422.2!2. Jikd -\pr 10. 200<). by \\'cslcrman cl a!
l. S Appt :--:o !2 422.222. fik-.t .\pr JO. 2009. by ct
l. S Appt :--:o !2 422.225. fik-.t .\pr JO. 2009. by ct
1: S -\ppl :--:o 12 434.43<), filed \1ay 1. 2009. by Wcslcnmm cl ,]
1: S -\ppl :--:o 12 47<),."7.1. filed Jun 5. 2009. by \\blcrman cl a!
l. S .\ppt :--:o !2 479.6!7. fikd Jun 5. 2009. by ct
l. S .\ppt :--:o !2 479.671>. fikd Jun 5. 2009. by ct
('ui. Y ct at (!996) Segmentation l'sing
Verification for I l;md Sign Recognition." Pmneding1
of rile 1996/UOJO Compuler Sociel) Cou{in'ni'<'OI! Computer l/.1/on
and l'allem R<'i'OJi!lilion. pp 1>1>-93
Fim!l On<cc .\ction m"ikd :--:m 19. 2009. for 1: S -\ppl :--:o
!1 421>.51 .". Jikd Jtil 3. 2006. !4 pages
Ollkc Aug 25.2009. for l. S Appt :--:o
ll 421>.522. fik-.t Jut 200(i. six pages
:--:on-Final On<cc -\clion m"ikd Scp 2. 2009. for 1; S .\ppl :--:o
!1 ."59,7.16. Jikd :--:m !4. 2006. 12 pages
Adion <let 5. 2009. !Or L S Appl :--:o
!1 ."59,7<)9. Jikd :--:m !4. 2006. 14 pages
:--:on-Final <ln<cc .\ction mailed {lei 6. 2009. for 1; s .\ppl :--:o
ll fik-.1 :--:ov !4. 2006. 24
Adion maik-.t <let 14. 2009. for l. s .\ppl :--:o
!1 421>.50!. Jikd Jtil 3. 2006. six P"gcs
:--:on-Final On<cc .\ction m"ikd Ocl 2<J. 2009. j(,-1: s -\ppl :--:o
ll 559.1>22. fik-.1 :--:ov !4. 2006. ll
Adion maik-.t <let 2009. for l. s .\ppl :--:o
!1 421>.503. Jikd Jtil 3. 2006. nine pages
:--:on-Final On<cc -\clion m"ikd 22. 2009. j()f l: s .\ppl :--:o
ll fik-.1 :--:ov !4. 2006. six
27. 20!0. !Or l. S .\ppt :--:o
!1 421>.522. Jikd Jtil 3. 2006. fi1c pages
:--:otkc of Allowability maik-.1 Scp 2. 2009. for l. S .\ppl :--:o
ll 421>.506. fik-.t Jut 200(i. five
D II (Dec l<J<Jl) "The -\ulomalic of Ges-
tures." C\ft:-CS-<Jl-202. Submitted in l'"rli;!l of the
Requirements of the Degree of Dod or of Philosophy in Computer
m ('arnegic V!dlon t:niversity. 21>5
1; S -\ppl :--:o 10 6."4.!01>. Jikd Scp 2. 200.1 cnlitkd _\mbidcxlrolis
\1olisc"
L S .\ppt :--:o JO 71>9.67(>. fik-.t Feb 27. 2004 cmitk-.t "Shape
Detecting lnplil Dc1icc"
"4-\\'irc Rcsisli1c Touchscreens" obt,incd Jfom hllp www
htmt generated Aug
5.2005
"5-\\'irc Rcsisli1c Touchscreens" obt,incd Jfom hllp www
lolichsnccns com inlro-lolichlypcs-rcsislilc him! gcncmlcd -\ug 5.
2005
"A Brief (lvervk'V>' of Gesture obtained from http
wwwdai cd ac uk C1<mlinc I()('_\ COPIES CO!I!O:--:
gesture ovenk-v. hun!. Apr 20. 2004
loudls<:rccns" from http
com inlro-lolichlypcs-c"p"cililc him! generated .\lig .". 200."
"(-ap,cili\c Position Sensing" obt,incd from hllp www symtplics
K-.:hnology generated Aug 5. 2005
"Comparing Jcdlnologics" Jlom http ""'""
lmtchsnccns com inlro-lmtchlypcs him! gcncmlcd { kl !0. 2004
"GiidcPoinLR" obtained from http """ cirque com technology
tedlnology gp htmt gcncmK'<l .\ug 5. 2005
"!low do monitors know where you're
obt,incd from http """ dcctronics howsllill'works com qucs-
tion7J(i htmt gcncmK'<l .\ug 5. 2005
"!low docs a work''" from http www
lmtchsnccns com inlr<Hm,lomy him! gcncmlcd .\lig .". 2005
"!nlfarcd obtained from hllp www lmtchsnccns
hun! generated Aug 5. 2005
"Mouse Fmutmion" Fingcr\Vorks obtained from http ""'""
fingcrworks com gesture guide molls<: him! gcncmlcd -\ug 30,
200."
"Mouse Gestures in < lpcra" from http w\\w opera wm
desktop mouse index dml generated Aug 2005
"\louse Gestures," (lplim O/, \hy 2!. 2004
"\ltilliTouch {h-c!Yic\\" Fingcr\\'orks obtained from hllp www
fmgcrworks wm multovcrvk'V>' htmt gcncraK'<l Aug 30. 2005
":--:em- J-idd obtained from http ""'""
lmtchsnccns com inlro-lmtchlypcs-nfi him! gcncmlcd .\lig .". 2005
"PcnTouch Cap,cili\c Touchscreens" obt,incd Jfom hllp www
htmt generated Aug
5.2005
US 7,812,828 B2
Page 5
Awustk \\'ave !lom http v.v.w
touchscreens com inlro-louchlypcs-smv html gcncmlcd _\ug -'
2005
"Symbol Commander'' obtained !lom http v,v./v.; scnsiva com
symbokmmmdcr. gcncmlcd _\ug 30. 200."
Tips for Typing" FingcrWorks hllp www fingcn\orb com
mini typing html .\ug 30.2005
"Touch Technologies ( >vcrvk'V>'" 200 l. 3V! Touch Systems. Y!assa-
chusdls
"Wawm Components Technology" !lom http v.v.w
wawm-componcms wm english tech asp gcncmK'<l on <kt to.
2004
watershed .\lgorilhm" hllp r;b inJ(, nih g<"- ij plugins w"lcrshcd
hun! generated Aug 5. 2005
r;ngcrWorks Gesture -\pplicalion Sv.ilching." obtained
from hllp www fingcn\orks com gesture guide apps htmL
emK'<l on Aug 27. 2004. 1-pg
''Finger\Vorks Gesture Guide Lditing."' !lom http
""" com guide htmL on
-\ug 27. 2004. l-pg
''Finger\Vorks Gesture Guide t-ile Opcmtions."' obtained !lom
http """ com guide files him!.
on -\ug 27. 2004. l-pg
''Finger\Vorks Gesture Guide !ext Manipulation."' !lom
http \\V.W fingetworks wm gesture guide text manip html.
on .\ug 27. 2004. 2-pg
''Finger\Vorks Gesture Guide Tips and Jrkks."' !lom
http \\V.W fingetworks wm gesture guide tips html. generated
-\ug 27. 2004. 2-pgs
Guide
fingetworkscom gesture guide
2004. 1-pg
from http """
webhtml. genemK'<l on .\ug 27.
Guide lo Hand {(,- t:SB Touchpads."
obtained from http v,v.w fingetworks com igesture uscrguide
hunt. genemk'<l Aug 27. 2004. 1-pg
Technical Details."' lfom http
""" com tech html. -\ug 27.
2004. 1-pg
{lnly Touchp"'ls with Ergonomic Full-lland
mul from hllp www fingerworks
com resting html. ('opyright 200 I. t -pg
''Finger\Vorks Tips tOr Typing on the Mini." obtained from http
""" com mini typing htmL on -\ug 27.
2004. 2-pgs
''iGesture Pad the MultiFinger l'SB louchPad with \Vholc-lland
Jfom hllp www fingerworks com igesture
hlml. .\ug 27. 2004. 2-pgs
Bier. et at . ''loolglassand Magic l.enses The see-through imer!Uce"
In James Kijiya. editol". Computer Gmphics (SIGGK\PII '93 Pro-
\ol 27. pp 73-1>0 .. \ug 1<)<)3
Buxton. W et al (Jul 22. !9l>.'i) ''Issues and Jcchniques in 'J(ntch-
Sensitive Jab let Input." Pmceediug< .1('Jf pp 2 t.'i-224
Culler.!. D al (-\pr 27. 1<)<)7) "hvo-!lan<kd Direct \l;mipula-
tion on Pmcealing< f!(the I'JY7 Sym-
po.<ium oulnterai"/h'<' 3/l (irap/iic.<. Providence. Rl. .\pr 27-30.
1997. pp !07-1 14
Dmis. J d al (Oct .11. 1<)<}4) Determining 3-D !land \lotion,"
\Diem.< am/('omplller.<. 19Y.J ('ou{innc<' Raord of"t/w J,\"
1
'
A.<ilomar Coufi'I"<'IIC<' 011 l'aci(u (imve. CA. Oct 3 l -:\ov 2. l 994.
!.os -\lamitos. { .\, pp 1262-1266
Doug his et at. The Ergonomics of Pointing ices
(!997)
European in FP 1 621 <)l><) ((a. 13<J"<"r
1/im<r & l"/wma1, !Il') \lar 27. 2006
F\"B Llektmnik '"TSOP623g IR Receiver Modules !Or Infrared
Remote Comml Systems" daK'<l Jan 2004 1-pg
Fisher al . \lotion Disonkrs The Design of { lptimal
Rest hJclors. 3."(2) 21>3-304 (.hm I <)<)3)
lukumoto. cl al . ''.\ctiveClick lactilc h'<..'<lback for Touch Panels."
!nC!l!2001 Summary.pp 121-122.2001
Fukumoto and Yoshinobu Ton<Hnum. "'Body Coupk-.1 Fingering
\Virekss \Veamblc CII/97. pp 147 -l.'i4 (Mar 1997)
llardy. ''lingetworks"' Mar 7. 2002. BBC World On Line
Hillier and Gerald J Lieberman. Immdui"lion 10 OpNatiom
R,.,.,.w-,h (l<Jl>ti)
Jacob et al . ''Integrality and Sepmability of Input Devices."' AC:\.1
fiwl.<aNiou.< 011 Complller-llumau Imeracliou. I 3-2ti (Mar 1994)
Kinkk]' et al . lnput in c!l! '<)<)
ings. pp 22.1-230. 1<)<)<)
Kl<l:\X ""K.'\Pl>4 Series Summmy Data copyright2005.dak'<l
<let 21.200.'i.4-pgs
I at."".\ \lulli-"!(ll!ch "!iJb-
lct."' in ("Ill g:; Procwdings. pp 121-121>. 2000
l.ee. "".\ last Multiplc-Jouch-Sensitive Input Device." Master's I he-
sis. of'l(m>nto (I <}1>4)
I S K et at (-\pr 191>5) "'_\ \luhi-"!(ntch
Touch-Sensitive Jabkt." l'nxeeding< of Cllf: .1CJ! Cou{innc<' on
1/uman Fa('for_,. in Compwing s_, l'lem_,._ pp 21-2.'i
\hlsllshila et at , "!lolo\\';J!] a 1 l;md, Body ;md
< >bject Sensitive \Vall." In of l-IST '97. < >ct 1997
Pavlovic.\' I d al (Jul l 997) "Visual Interpretation of !land Ges-
!(>r !ntemction -\ ff1F liYw\ac-
Jion_,. on Pall em Anal},.;,- and Mmliill<" !md!igmce 19(7) ti77-6<}5
QuiUllum Research Gmup "QT.'ilO Q\Vk-cl'" Touch Slider IC"
copyright 2004-2005, 14-pgs
Gestuml ff1F .lfu!Jirll<"dia,
3 36-47 (\\'inter 1996)
Radwin. ".\ctivation Force and Travel Effects on Overexenion in
Tapping." 1/uman Factor_,._ .1<)(1) 1.10-140 (\hr
1997)
Rckimoto "Smm1Skin .\n lnllasm1cture !Or Freehand Manipulation
on lntemcli1e Cl 11 2002, -\pr 20-2.'i. 2002
Rckimoto cl al , Erl'-'cti1e l:se of Physical \lanipll-
lation Vocabularies oflnput Devices." In Pmc of LIST 2000.2000
Rubine et al. "Pmgmmmablc l-inger-Tracking lnstmment Contml-
krs," Complll<"r .lfu,-;,- .lemma/. \ol 14. :\o 1 (Spring I <)<)0)
Rutk-.tge d al . "Force-To-Motion !unctions !Or Pointing." lluman-
Computer Interaction 1:\TLR.\C I (1990)
-\hmmL "_\ l:sabk .1D !land Tmcker,"
ings of the 2l>th .\silmmlf Con!'-'rence on Sigm!ls. ;md
Computers Pm1 2 (of2). vol 2 (Oct 1994)
lnstrumcnts "TSC2003 !2C "!(ntch Conlrolkr" Data
SB.\S 1ti2, <hik-.1 <let 2001. 20-pgs
Wellner. "I he Digital Desk Calculators L'lllgiblc Manipulation on a
Desk Top Display" In .\CM LIS I '9 l pp 27-34. :\ov
1<)<)1
Williams. "-\pplicalions J(>r a Switched-Capacitor lnslmm@lalion
Building Block" Linear kchnology Application :\ote 3. Jut 191>5.
pp 1-16
Yamada al . ".\ S\\ !(>r capacili\e
sure Sensors" ILLL Transactions on lnstnunentation and Measure-
ment. vol 41. :\o I. Feb 1992. pp gj.g(i
at. "Switched Capacitor Cirelli! !(>r
Transducers" !9K'i 11:1:10
/.hai et al . "Dual SlJ'eamlnput !Or Pointing and Scmlling." l'mceed-
ing, f!((Jff ''J7 FHended Aim ram (1997)
al. ".\pplying Electric Field Sensing lo !lummi-
Computer Interfaces." In Clll 'K'i pp 2g0-21>7. t 99.'i
lntemational SeiUch Repolt in conesponding PC I applica-
tion :\o PCT t:S2006 001>34<) <Jet ti, 2006
lntemational search repo11 !Or PCT LS99 01454. mailed May 14.
1999
Fl!roJl"an search J(>r hm>Jl"an Patent .\pplica-
lion :\o <}<)<)04221> ti. maikd lti. 200_'i
Suppkmentmy IClll'opean search repon !Orl S Appl :\o 9990422g
6. mailed Jut 1>, 2005
hm>Jl"an examination repm1 !(>r European patent -\pplication :\o
99904221> ti. maik-.t Apr 20. 2006
Lumpcan examination repo11 !Or Lumpcan Patent .\pplication :\o
<)<)<}04221> ti, maik-.1 \lar n. 2007
European search repm1 !(>r European -\pplicalion
:\o 21.2007
European search repm1 !(>r European -\pplicalion
:\o 060161>56 4. maikd \lar 14, 2001>
Lumpcan examination repo11 !Or Lumpcan Patent .\pplication :\o
or;o J6g3o 9. maik-.t Aug ti. 2oog
US 7,812,828 B2
Page 6
scardl n:p011 !Or European Patent .\pplkation :\o
06016K10 <) maikd 3. 2007
h1mpcan examination rcron for h1mpcan Pmcnt .\pplkmion :\o
Scp Hi. 20m:
of .\llow;mcc m"ikd \1ar n. 2010, j()f t: s .\ppl ""
11 421> . ."01, filed Jul .1. 2006. eight pages
h1mpcan Scardl Rq,ot1 maik'<l h:b 17.2010. for FP Applkation
'\o 0(i0J(iK'i7 2. six pages
Eurorcan Sc,rch Rcpm1 maikxl Feb 24, 20!0. J(lf EP .\pplication
'\o 0(i0JMG3 3. nine
Sun V!kmsystcms ( 1992) "J he Stm-7 I'D.\ l'mtotypc." located at
<hUp wwwyoulubc com w"lch'l,- .\hgl>< >BYixl 0. last 'isitcd
Jan 15. 2010. seven
Tomita. A Jr d al ('\ov 9. 1997) "An Image l'mccssing VIet hod !Or
l:sc in a GU for the Vi>tmlly Impaired." l'mceeding1 c4 i/u_/FCO.V
97: 23"
1
lwemaJiona! Cot!fi'nlln' 011 fnduwrial Fhctro11in. (fmtml
and fii.<II'WII<'illation. Orleans. L\. :\ov 9-14. 1997. pp 264
2(ii-:
* cited by examiner
U.S. Patent
24
Oct.12,2010 Sheet 1 of 45 US 7,812,828 B2
4
2
FINGER
SYNCHRONIZATION
DETECTOR
22
ELECTRODE
6
SCANNING
HARDWARE
CALIBRATION AND
8
PROXIMITY IMAGE
FORMATION
CONTACT 10
TRACKING AND
IDENTIFICATION
HAND
MOTION
COMPONENT
EXTRACTION
17
18
16;....----"'----,
HOST HOST
20
i+-___j COMPUTER ~ - 1 COMMUNICATION
L_ __ ..-J SYSTEM INTERFACE
FIG. 1
U.S. Patent Oct.12,2010 Sheet 2 of 45 US 7,812,828 B2
32
35
46
45
'
34
48
31 36
~ 3 7
FIG. 2
U.S. Patent Oct.12,2010 Sheet 3 of 45 US 7,812,828 B2
32
39
33
40
46 45
58
34
48
30 31
41
FIG. 3A
69
39
58
34
41
FIG. 38
U.S. Patent Oct.12,2010 Sheet 4 of 45 US 7,812,828 B2
32
FIG. 4A
5(
43
n n n
\,
44
n n n
\,
TIME
FIG. 48
U.S. Patent Oct.12,2010 Sheet 5 of 45 US 7,812,828 B2
46\ 44\
r
)3
\..
'-
v-43
46/
'24
W""
!'I 'i
47\
ds ~ 1
30
33
32
)5
\.
4(
FIG. 5A
46\ 44\
4 ~ 4 ~
47
4 ~
4 ~
4 6 ~
44-'
FIG. 58
j5
.5Q
35
-
5 5
r ~ : r 3 7
34
FIG. 6
~
'JJ

~
"" ;-
=
.....
0
n
,...
- p
"'
0
- 0
00
""
"
"
-
"' 0
~
..
'JO
~
"'
-...)
Oo
-N
~
00
s
U.S. Patent Oct.12,2010 Sheet 7 of 45 US 7,812,828 B2
58
66
70
M
63
8
61 60
K
69
68
FIG. 7A
71
s
58ah
67
65 66
70
M 8
K
69
68 64
FIG. 78
7 ~
76
H 47 H 47 t::1 47 t::147 t::1 47 t::147 D 47 0 47 ~ "
.,.,
-
0
0)
43 ../ '
'IT' 'IT' IT' IT' IT IT T' T'
l'l, . ~ .
34 ' 1.
(
~
'JJ

~
""
It
=
....
0
n
,...
- p
"'
0
- 0
00
""
"
"
-
"" 0
~
..
'JO
~
"'
__,
Oo
-N
Oo
N
00
Cll
N
U.S. Patent Oct.12,2010 Sheet 9 of 45 US 7,812,828 B2
78'\
78'\
76':::J
0 Q
77'\
80
75'\
'\
75'\
/
81/'
79
'-.-./
FIG. 9A
/'
44
7 ~ 7l
76
7(_
\.....,
7(
.1
7(_
FIG. 98
44/'
U.S. Patent Oct.12,2010 Sheet 10 of 45 US 7,812,828 B2
FIG. 10
U.S. Patent
85
90
Oct.12,2010 Sheet 11 of 45 US 7,812,828 B2
FIG. 11
U.S. Patent Oct.12,2010 Sheet 12 of 45 US 7,812,828 B2
FIG. 12
U.S. Patent Oct.12,2010 Sheet 13 of 45 US 7,812,828 B2
~ 4
E
202 I
u
(f)
~
X
<(
'
>
-
w
0
<(
lL
a:
::0
(f)
201
z
0
z
~
0
f-
(f)
0
a.
...J
0
<(
r ~ 0 .
f-
a:
w
>
207
FIG. 13
U.S. Patent Oct.12,2010 Sheet 14 of 45 US 7,812,828 B2
12
'[ 10
rJ)
~
c 8
w
0
i't
a:
:::>
Ul
z
0
z
0
f-
Ul
0
0..
...J 0
{3
tr
w -2
>
202 203 204 205
h ,-1--, ,.-1--, r-.C.,
201
0 2 4 6 8 10 12 14 16 18
HORIZONTAL POSITION ON SURFACE (X AXIS em)
FIG. 14
U.S. Patent Oct.l2,2010 Sheet 15 of 45 US 7,812,828 B2
14
12
~ 10
(fJ
202
,i-,
><
..:
,..
8
-
llJ
0
..:
6 u.
a:
:::J
(fJ
z
4
0
z
0
f- 2
203 204 205
~
( (
)
~
(fJ
0
a.
_J
0
..:
0
f-
a:
llJ -2
>
207
206
-4
- 6 ~ - - ~ - - - - L - - - ~ - - - L - - - - L - - - ~ - - ~ ~ ~ L - - - ~ - - -
0 2 4 6 8 10 12 14 16 18
HORIZONTAL POSITION ON SURFACE (X AXIS em)
FIG. 15
U.S. Patent Oct.12,2010 Sheet 16 of 45 US 7,812,828 B2
243
PATHS FROM
PREVIOUS IMAGES
247
CURRENT 240
PROXIMITY
IMAGE
PARAMETERIZED
LECTRODE GROUP
NEW PATHS &
UPDATED PATH
PARAMETERS
241
242
246
248
HAND FINGER & PALM
IDENTIFICATION IDENTIFICATION
~ ~ ~ ' - - - - - - '
IDENTIFIED
CONTACT PATHS
ESTIMATED HAND &
FINGER OFFSETS
FIG. 16
250
251
U.S. Patent Oct.12,2010 Sheet 17 of 45 US 7,812,828 B2
(START)
l
GET HAND'S r2so
IDENTIFIED PATHS
~
COMPUTE OFFSETS BETWEEN
EACH FINGER'S MEASURED AND
r25
4
DEFAULT POSITIONS
!
COMPUTE AVERAGE OF
r255
OFFSETS WEIGHTED BY
CONTACT PROXIMITY
~
ADJUST FILTER POLE TO
v-256
CURRENT IDENTIFICATION
CONFIDENCE
~
COMPUTE WEIGHTED
r257
AVERAGE OF HAND
CONTACT VELOCITIES
!
AUTOREGRESSIVELY UPDATE HAND
v-
OFFSET ESTIMATES FROM MEASURED
258
OFFSETS AND VELOCITIES
~
UPDATE FINGER r2s9
OFFSET ESTIMATES
~
CONVERT ESTIMATED OFFSETS r2
60
TO ABSOLUTE POSITIONS
~
( END
FIG. 17
U.S. Patent Oct.12,2010 Sheet 18 of 45 US 7,812,828 B2
CURRENT
PROXIMITY
IMAGE
DIFFUSE
CURRENT IMAGE
SMOOTHED
PROXIMITY
IMAGE
SEARCH FOR
SIGNIFICANT
LOCAL MAXIMA
LOCAL
MAXIMUM
PIXELS
240
262
263
267
FLATIENED
FINGERTIP
FEEDBACK
252
ESTIMATED HAND
POSITION OFFSETS
265
DEFINE SEGMENTATION
STRICTNESS REGIONS
268
CONSTRUCT ELECTRODE
GROUPS AROUND LOCAL
MAXIMUM PIXELS
COMBINE
OVERLAPPING
GROUPS
FIT ELLIPSES TO
COMBINED
GROUPS
PARAMETERIZED
ELECTRODE GROUPS
FIG. 18
270
272
266
242
U.S. Patent Oct.12,2010 Sheet 19 of 45 US 7,812,828 B2
274
279
FIG. 19
U.S. Patent Oct.12,2010 Sheet 20 of 45 US 7,812,828 B2
15
282
-
10
+ + +
+ 252
+ +


+
\
5
0 284
lr-285 28!r
286
-
-
+ + + +
-5
-20 -15 -10 -5 0 5 10 15
'E
-2-15
5
1-
(J) 10
a?
w
5
II
:::>
(J)
....J 0
()
I-
II -5
282
-

2F
/
284
[--- 285
-
+ +
FIG. 20A
252_./"+
+ +
+


286

-
'-t
252
-20 -15 -10 -5 0 5 10
FIG. 208
15
20
20

10
282
5
0
M 286

-20 -15 -10 -5 0 5 10 15 20
HORIZONTAL SURFACE POSITION (em)
FIG. 20C
U.S. Patent Oct.12,2010 Sheet 21 of 45 US 7,812,828 B2
300
1-------j GET NEXT ELECTRODE IN !<-----( A
DIRECTION OF SEARCH
RAW PROXIMITY >
BACKGROUND?
y
IN STRIC
SEGMENTATION
REGION?
294
SEARCHING
HORIZONTAL?
HORIZ.
DIST. TO LOCAL MAX
> 2cm?
298
TALL
HORIZONTAL
MINIMUM?
REACHED EDGE
BETWEEN PALM
HEELS
N
y
N
N
N
A
300
REACHED BACKGROUND
LEVEL EDGE f
304
A}<-----,
306
N
SEARCHING
N
VERTICAL
HORIZONTAL?
MINIMUM?
y 310
y
REACHED EDGE
312
BETWEEN
FINGERTIP AND
THUMB OR PALM
308
HORIZ.
y
OR DIAGONAL
MINIMUM?
N
REACHED EDGE
314
BETWEEN
FINGERS
FIG. 21
U.S. Patent Oct.12,2010 Sheet 22 of 45 US 7,812,828 B2
(START
+
PREDICT CURRENT POSITIONS
v--3
20
OF EXISTING PATHS
FOR EACH GROUP ,-322
FIND CLOSEST PATH
FOR EACH PATH, FIND
r-324
CLOSEST GROUP WITHIN
TRACKING RADIUS
FORM GROUPPATH PAIRS IF
,-3
GROUP & ACTIVE PATH ARE
26
CLOSEST TO ONE ANOTHER
ATIEMPT TO PAIR REMAINING
!,r-3
GROUPS WITH RECENTLY
34
DEACTIVATED PATHS
ALLOCATE NEW PATHS FOR ANY
v-
336
REMAINING UNPAIRED GROUPS
DEACTIVATE ANY REMAINING
,,.3 44
UNPAIRED PATHS
UPDATE PATH v--346
PARAMETERS
+
( END
FIG. 22
U.S. Patent Oct.12,2010 Sheet 23 of 45 US 7,812,828 B2
DEFINE IDENTITY
A TTRACTORS AT DEFAULT
CONTACT POSITIONS
TRANSLATEATTRACTOR
TEMPLATE BY ESTIMATED
HAND OFFSET
COMPUTE MATRIX OF
DISTANCES FROM EACH PATH
TO EACH A TTRACTOR
COMPUTE A TTRACTOR
WEIGHTING FACTORS FROM
FEATURES OF EACH PATH
FIND ASSIGNMENT BETWEEN
PATHS AND ATTRACTORS
WHICH MINIMIZES SUM OF
WEIGHTED DISTANCES
360
HAND
ASSIGNMENTS
ENTATIVE?
y
N
350
352
354
356
358
364
UPDATE FINGER
COUNTS AND
SUBSETS
# FINGE
ATTRACTORS
ASSIGNED> 1
OR<5?
362
y
f--N __ --IVERIFY THUMB
ASSIGNMENT
368
FIG. 23
U.S. Patent Oct.12,2010 Sheet 24 of 45 US 7,812,828 B2
12
10
380
8
E
u 372--><
-
z
6
0
X-375
,_
<f)
0 4
a.
w
(.)
..:
2 u.
a:
:::>
<f)
x- 371
-'
0
..:
(.)
;::
380
a:
w -2
>
-4
,(377
-6
0 2 4 6 8 10 12 14 16 18 20
HORIZONTAL SURFACE POSITION (em)
FIG. 24
U.S. Patent Oct.12,2010 Sheet 25 of 45 US 7,812,828 B2
z
0
J-1-a:
I<(O
(!)1-1-
-z()
a:w<(
-LL.
a: 0 0
0 50 100 150
CONTACT ORIENTATION (degrees)
FIG. 25A
CONTACT SIZE (NORMALIZED TOTAL PROXIMITY)
FIG. 258
...Ja:
wO
wt)
I<(
:z;u..
...Jw
ii:N 11-----
W
0 1 2 3 4 5 6
TOTAL PROXIMITY DIVIDED BY ECCENTRICITY
FIG. 25C

0 2 4 6 8
DISTANCE TO NEAREST NEIGHBOR CONTACT (em)
FIG. 250
U.S. Patent Oct.12,2010 Sheet 26 of 45 US 7,812,828 B2
COMPUTE INTERPATH
THUMB FACTORS
COMBINE WITH THUMB SIZE
& ORIENTATION FACTORS
OF INNERMOST AND NEXT
INNERMOST CONTACT
404
combined_thumb_fact >
is_thumb_thresh?
406
combined_ thumb _fact
not_thumb_thresh?
408
INNERMOS N
ASSIGNED TO
THUMB?
y
SHIFT INNERMOST
PATHS AWAY FROM
THUMB ATTRACTOR
N
400
402
410
412
INNERMOS
ASSIGNED TO
THUMB?
y
N
SHIFT INNERMOST
PATH TO THUMB
ATTRACT OR
414
EXISTING
ASSIGNMENTS
OK
413
FIG. 26
U.S. Patent Oct.12,2010
GET ALL PATHS ASSIGNED
TO THE GIVEN HAND
COMPUTE DISTANCES
FROM EACH PATH TO
OTHER PATHS
FIND SHORTEST RING
CONNECTING ALL PATHS
AND PASSING THROUGH
EACH ONCE
COMPUTE THUMB & PALM
WEIGHTING FACTORS FOR
EACH PATH
PICK INNERMOST 438
PATH IN RING
440
Sheet 27 of 45
430
432
434
436
INNERMOST
PATH THUMB?
N INNERMOS
y
PATHS ABOVE THIS
VERTICAL LEVEL ARE
FINGERTIPS, BELOW
ARE PALMS
444
PATH A PALM
HEEL?
PATHS AT THIS
VERTICAL LEVEL
ARE PALMS
FIG. 27
N
US 7,812,828 B2
448
PATHSATTHIS
VERTICAL LEVEL
ARE FINGERTIPS
U.S. Patent Oct.12,2010
485
INNER
& OUTER HAND
PARTS TOUCHING
?
y
N
Sheet 28 of 45
486
PENGRIP
PREVIOUSLY
DETECTED?
y
US 7,812,828 B2
N
END
GET ESTIMATED FINGER &
f----------1 PALM POSITIONS FOR
LIFTED HAND PARTS
487
GET MEASURED POSITIONS &
SIZES OF TOUCHING FINGERS
& PALM HEELS
COMPUTE KNUCKLE FACTOR
FOR OUTER FINGERS
COMPUTE INDEX JUTTING
FACTOR FOR INNER FINGERS
COMBINE FACTORS & FILTER
488
489
490
495
SEND PARAMETERS OF
INNER FINGER PATHS TO
HANDWRITING RECOGNIZER
WITH OLD COMBINED FACTORS 491
y
492 493
FILTERED y
FACTOR> PENGRIP >--->-<
THRESH?
INNER FINGERS
TOUCHING?
N
N
494
SEND STYLUS LIFT SIGNAL TO
HANDWRITING RECOGNIZER &
KNUCKLE/PALM MOTION TO CURSOR
FIG. 28
U.S. Patent
456
PICK FIRST
CONTOUR
PICK NEXT
CONTOUR
CHOOSE
LOWEST
COST HAND
PARTITION
Oct.12,2010 Sheet 29 of 45
y
DEFINE & TRANSLATE LEFT &
RIGHT ATIRACTOR TEMPLATES
GENERATE PARTITIONING
CONTOURS
TENTATIVELY DIVIDE HAND
IDENTITIES ACROSS CONTOUR
TENTATIVELY ASSIGN FINGER
IDENTITIES WITHIN EACH HAND
EVALUATE BIOMECHANICAL
COHERENCE OF PARTITION
470
472
473
474
464
y
US 7,812,828 B2
452
RETAIN PREVIOUS
IDENTIFICATIONS VIA
PATH EXTENSION
453
454
458
460
462
466
RECORD PARTITION
AS LOWEST COST
ASSIGN FINAL
CONTACT IDENTITIES
WITHIN EACH HAND I
FIG. 29
U.S. Patent Oct.12,2010 Sheet 30 of 45 US 7,812,828 B2
10
5
0
-20
'E
.2.
z
0
f- 10
(/)
0
<l.
w
:.;1 5
u..
a:
::>
(/)
...J
<(
0
0
f- -20
a:
w
>
10
5
0
-20
Q,Jl
-15

476
-15


477
0'---477
-10 -5 0 5
FIG. 30A

477
0'---476
lr'--475
-10 -5 0 5
FIG. 308
476-D
477
0----476
f----475
10 15
10 15
-15 -10 -5 0 5 10 15
HORIZONTAL SURFACE POSITION (em)
FIG. 30C
20
20
20
U.S. Patent
a:
f.2
<9:i
Zu.
I
()Z
1-0
::>-
_.JI-
u&']
Oct.12,2010 Sheet 31 of 45
e::: 1 r------------'
US 7,812,828 B2
0
-50 0 50
HORIZONTAL VELOCITY OF RIGHT HAND CLUSTER (mm/s)
(/) 1
(/)
wa:
zo
0}-
wo
0<{
zu.
<>:
I
FIG. 31A

-12 -10 -8 -6 -4 -2 0 2
VERTICAL POSITION OF OUTERMOST FINGER
RELATIVE TO NEXT OUTERMOST (em)
FIG. 318

0 5 10 15 20
HORIZONTAL SEPARATION BETWEEN PALM CONTACTS (em)
FIG. 31C
U.S. Patent Oct.12,2010 Sheet 32 of 45 US 7,812,828 B2
cr:
cr:o
Wi-
ZO
Z<(
-lL
1--w
I ....I
<!l<!J
cr:z
<(
1

-150 -100 -50 0 50 100 150
ANGLE BETWEEN INNERMOST AND
NEXT INNERMOST FINGER CONTACTS (degrees)
FIG. 32
cr:1
8
oo
z<(
<(lL
IZ
'O
cr:-
w!;{
1--cr:

a.
w
(/)

-6 -4 -2 0 2 4 6 8 10 12
ESTIMATED HORIZONTAL
SEPARATION BETWEEN THUMBS (em)
FIG. 33
U.S. Patent Oct.12,2010 Sheet 33 of 45 US 7,812,828 B2
( START )
+
GET HAND'S CURRENT
PATH PARAMETERS &
lr-5
00
I D'S
~
SUPPRESSIVE
FINGER VELOCITY
r5
02
FILTERING
~
MEASURE HAND'S
POLAR VELOCITY
r50
4
COMPONENTS
~
MEASURE HAND'S
v-
TRANSLATIONAL
506
VELOCITY COMPONENTS
~
MEASURE HAND'S
DIFFERENTIAL TILT
v-
508
PRESSURE
COMPONENTS
~
DOWNSCALE
ir510
WEAKER
COMPONENTS
~
DEAD-ZONE FILTER ALL
COMPONENTS
r
512
BY FRACTION OF
FASTEST COMPONENT
~
( END
FIG. 34
U.S. Patent Oct.12,2010 Sheet 34 of 45
203
~
202
~
I
I
I
I
I
I
I
'
I
\ \
I
'
I
I
-- -- ..
-
- ~
/
207
- ~

FIG. 35
I
I
I
204
@I
I
I
I
_,
I
-
/
206
fi
.
US 7,812,828 B2
205
;1!
"
/
/
/
/
U.S. Patent Oct.12,2010 Sheet 35 of 45 US 7,812,828 B2
AT LEAST
2 FINGERS
DOWN?
y
N
GET CURRENT AND PREVIOUS
POSITIONS OF INNERMOST AND
524
SET RADIAL AND
ANGULAR VELOCITY
TO ZERO
END
OUTERMOST TOUCHING FINGERS 526
COMPUTE RADIAL VELOCITY FROM
CHANGE IN SEPARATION BETWEEN
INNERMOST AND OUTERMOST
COMPUTE ROTATIONAL VELOCITY
FROM SEPARATION AND CHANGE IN
ANGLE BETWEEN INNERMOST AND
OUTERMOST
COMBINE WITH ROTATION AND
SCALING ABOUT A FIXED POINT
BETWEEN THUMB AND OTHER FINGERS
PROXIMITY
DROPPING?
y
532
N
FIG. 36
528
530
531
534
CHECK FOR
RADIAL OR
ROTATIONAL
DECELERATION
U.S. Patent Oct.12,2010
START
540
I NIT TRANSLATION
WEIGHTINGS TO FINGER
PROXIMITIES
544
DECREASE TRANSLATION
WEIGHTING OF
RELATIVELY SLOW
FINGERS
546
DECREASE TRANSLATION
WEIGHTING OF CENTRAL
FINGERS AS POLAR
COMPONENT SPEEDS
INCREASE
548
COMPUTE TRANSLATION
VELOCITY AS WEIGHTED
AVERAGE OF FINGER
VELOCITIES
AVERAGE
PROXIMITY
DROPPING?
y
550
N
Sheet 36 of 45 US 7,812,828 B2
551
ACCEL RATIO =
CURRENT SPEED/
PAST AVERAGE
ACCEL RATIO
> THRESH NEAR
ONE?
y
554
N
TRANSLATION
DIRECTION CLOSE TO
AST AVERAGE?
N
y
CLEAR
TRANSLATION
DECELERATION
FLAG
UPDATE MOVING
556
SET
TRANSLATION
DECELERATION
FLAG
558
f--------- WINDOW AVERAGE OF 560
TRANSLATION
VELOCITIES
FIG. 37
U.S. Patent Oct.12,2010 Sheet 37 of 45 US 7,812,828 B2
START
HAND
FLATTENED?
ALL PATH
PROXIMITIES
TABILIZED?
y
N
566
N
COMPUTE UNWEIGHTED
AVERAGE OF PATH POSITIONS
COMPUTE RATIOS OF CURRENT
PROXIMITY TO REFERENCE
PROXIMITY FOR EACH PATH
SET RATIOS LESS
THAN ONE TO ONE
574
564
SET TILT & ROLL
COMPONENTS
TO ZERO
568
STORE CURRENT PATH
PROXIMITIES AS
REFERENCE PROXIMITIES
570
572
COMPUTE AVERAGE OF
PATH POSITIONS
WEIGHTED BY PROXIMITY
RATIOS
576
COMPUTE TILT & ROLL
COMPONENTS FROM
DIFFERENCE VECTOR BETWEEN
WEIGHTED AND UNWEIGHTEO
AVERAGES
FIG. 38
578
U.S. Patent Oct.12,2010
GET HAND'S CURRENT
PATH PARAMETERS & !D'S
SEARCH FOR FINGER
SUBSETS PRESSED OR
RELEASED
SIMULTANEOUSLY
YN
602
MARKER
N
PENDING
?
#FINGER
PRESSES
YNCED > 2?
y
DELETE ASSOCIATED
KEY QUEUE ELEMS
612
N
608
Sheet 38 of 45
600 601
603
604
ANY
PRESSES
SYNCED
y
605
610
PAUSE SENDING OF
ASSOCIATED
KEYPRESS QUEUE
ELEMS
US 7,812,828 B2
DELETE
ASSOCIATED
KEYPRESS
QUEUE
ELEMENTS
N
#FINGER
RELEASES
y
YNCED > 2
624
CLEAR
SYNC
MARKER
c )----1
OUCHIN
OR HAL TED TOO
LONG?
DELETE
y ASSOCIATED KEYf---------1
QUEUE
614
ELEMENTS
622
616
N
y
618
FIG. 39A
U.S. Patent Oct.12,2010
8
DELETE ASSOCIATED
KEYPRESS QUEUE
ELEMENTS
626
Sheet 39 of 45 US 7,812,828 B2
620
N
628
SIGNIFICANT
TERAL MOTION ?
LOOKUP CHORD
FROM SYNCED
FINGERID'S
632
HORD HAS TA
EVENTS?
y
APPEND CHORD
TAP EVENTS TO
COMMQUEUE
FIG. 398
>---4( c
630
N
636
634
RESTING CHORD:
NO EVENTS
GENERATED
U.S. Patent Oct.12,2010 Sheet 40 of 45 US 7,812,828 B2
GET HAND'S
EXTRACTED MOTION &
IDENTIFIED PATHS
CHORD
SLIDE
ONGOING?
y
FINGER
SUBSET
LIFTED?
N
N
652
N
664
y
650
A
N
654
N
656
#FINGERS y
TOUCHING > 2 ? r:--+<
SYNCED
SUBSET OR
ALL FINGERS
LID lNG
660
DISABLE KEY &
CHORD TAPS FOR
THIS HAND
PRE-LIFTOF
DECELERATION
FLAG SET?
666
658
SELECT SLIDE CHORD
FROM SYNCED SUBSET
OR COMBINATION OF
FINGERS TOUCHING
y
668
LEAVE
CHORD
SLIDE MODE
667
SET CURRENT VELOCITY
COMPONENTS TO PRE
LIFTOFF AVERAGE
N
B
B }.-----i
SELECT NEW SLIDE
CHORD FROM NEW
SYNCED SUBSET
674
FIG. 40A
U.S. Patent Oct.12,2010
PICK FIRST SLICE
DEFINED FOR SELECTED
CHORD
APPLY SLICE'S VELOCITY
GAIN FUNCTION TO
MOTION COMPONENTS
PROJECT
VELOCITY COMPONENTS
INTO SLICE'S SPEED AND
DIRECTION RANGE
INTEGRATE PROJECTED
VELOCITY COMPONENTS
OVERTIME
#UNITS OF
MOTION
>= 1?
y
680
N
LOOKUP SLICE'S
KEY/MOUSE/3D
EVENTS
682
675
676
677
678
Sheet 41 of 45
PICK NEXT SLICE
N
LAST SLICE
FOR CHORD?
US 7,812,828 B2
694
692
690
DISABLE FURTHER
EVENTS FROM ONE
SHOT SLICE
RESET OTHER
SLICES'
INTEGRATORS
y
689
688
APPEND EVENTS W/ REMOVE INTEGER#
686
# MOTION UNITS f----+! MOTION UNITS FROM
TO COMM QUEUE INTEGRA TORS
FIG. 408
U.S. Patent Oct.12,2010
RETRIEVE KEY LAYOUT 700
REGIONS AND SYMBOLS
GET CURRENT IDENTIFIED 702
PATHS FOR BOTH HANDS
5
FINGERS
ON SAME HAND
PRESSED IN
SYNC?
N
NOTHING
TOUCHING
SURFACE FOR
AWHILE?
N
704
y
714
y
Sheet 42 of 45
706
TRANSLATE HAND'S
KEY REGIONS BY
MEASURED HAND
OFFSETS
FINGERS
PARTIALLY
CLOSED?
N
708
y
710
US 7,812,828 B2
716
r--:Rc::Ec::Sc::El'::T-cKc:'E'"'Y,----
LAYOUT
OFFSETS TO
ZERO
ADJUST REGIONS IN
EACH FINGER'S
COLUMN BY FINGER
OFFSETS
PROCESS
UPDATE DISPLA YEO
POSITIONS OF KEY
SYMBOLS
FINGER TAPS 718
ON MORPHED KEY
LAYOUT
FIG. 41
712
U.S. Patent Oct.12,2010
START
GET ANY PATH RECENTLY
CREATED BY HAND PART
TOUCHDOWN
PATH
752
N
PROXIMITY
JUST CROSSED
KEYPRESS
HRESH?
y
N
PATH
IDENTIFIED
754
AS FINGER NOT
PALM?
y
756
y PATH'S
N
HAND SLIDING/
WRITING?
766
Sheet 43 of 45
750
DEBOUNCE
OK?
N
764
757
y
APPEND KEYPRESS CREATE KEYPRESS
N
762
US 7,812,828 B2
760
FIND CLOSEST
KEY REGION
L
--j QUEUE ELEMENT TO "----1 QUEUE ELEMENT
TAIL OF FIFO , ~ CONTAINING PATH ID,i+---..J
KEYPRESS QUEUE CLOSEST KEY &
PRESS TIMESTAMP
FIG. 42
U.S. Patent Oct.12,2010 Sheet 44 of 45 US 7,812,828 B2
DELETE CURRENT
~ - - - - - - - - - L - - - - - 4 ELEMENT FROM ~ - - . - - 4 8
PICK ELEMENT AT HEAD
OF KEYPRESS QUEUE
1------{ c
LEMEN
PATH STILL
IDENTIFIED AS
FINGER?
N
772
770
KEYPRESS QUEUE
778
y
PATHINA N
SYNCHRONIZED
SUBSET?
774
776
y
FINGER SLID
TOO FAR?
N
782 780
N TIME
B 1 < - ~ - - . . . . L - - - - - - - - - - - < SINCE PRESS<
TAP TIMEOUT
:>+---y-< FINGER
LIFTED?
784
APPEND PRECEDING
MODIFIERS&
ELEMENT'S KEY
REGION SYMBOL TO
HOST COMM QUEUE
792
y
y
786
788
SKIP TO NEXT
ELEMENT IN ~ - <
QUEUE
PATH
PROXIMITY
PROFILE
IMPULSIVE
?
N
790
y
N
KEY REGION
A MODIFIER?
?
N
D)+------.L_ ____ _J
FIG. 43A
U.S. Patent Oct.12,2010 Sheet 45 of 45 US 7,812,828 B2
D
798
TYPEMATIC
N
STARTED FOR
ELEMENT?
y
B
805
ANOTHER
y
ASYNCHRONOUS
TAP?
N
COMPUTE REPEAT
INTERVAL FROM
CURRENT
FINGER PROXIMITY
808
TIME
SINCE LAST y
SEND> REPEAT
INTERVAL
?
N
794 796
FINGER
TOUCHDOWN
N
TIME
SINCE
>-->..( FINGER PRESS >
TAP TIMEOUT.
?
MOST
RECENT?
y
800
FINGER
802
N
1 - ~ A
N
DOWN > .5s AND
y
HAND'S
OTHER FINGERS
LIFTED > .Ss ?
806
810
812
< 1s?
804
N
INITIALIZE
TYPEMATIC
MODE FOR
ELEMENT
APPEND PRECEDING
MODIFIERS & ELEMENT'S
KEY REGION SYMBOL TO
HOST COMM QUEUE
y
1------------1 UPDATE LAST TYPEMATIC
SEND TIMESTAMP
A
FIG. 438
us 7,812,828 82
1
ELLIPSE FITIT\(; FOR \WITI-TOUCII

2
will always he a need in the art ll1r multi-li.mction manual
input devices which supplement voice input.
..\generic manual input device which combines the typing..
pointing.. scrolling.. nnd handwriting. capabilities of the stnn- CROSS-RITJ:RJ:NCJ: TO R!] . ..\T!])
API'! !CATIONS
"111is npplicntion is a continuation of II 1015.434. entitled
"Method nnd .\ppmatus for lntcg.rnting. Mnnunllnput." tiled
Ike. 17. 2(XJ4 now U.S. Pal. No. 7YW.5RO. which is a con-
tinuation of091n6.Sn (now Pal. No. 0.323.846) Jilcd Jan.
25. 1900 which claims the bcndit of prm isional application
601072.509. H!cd Jan. 26. 1998. cnch of which is hereby
incorporntcd by reference in its entirety. This applicmion is
also rdatcd to Application Scr. No. l !142X.50L entitled
"Capacitiu: Sensing Arrangement." I !/428.503. entitled
"Touch Sur!Ucc." 11/428.506. entitled "User Interface Ges-
tures." 111428.515. entitled "User lntcr!Ucc Gestures".
111428.522. entitled "ldcntil)ing. Contncts on a Touch Sur-
liKe." II 1428.521. entitled "IdentiFying Contacts on a 'llmch
SurliJCc". 11/559.730. entitled "Multi-'llmch Contact Track-
ing. ..\lg.oritlun". 11/559.763. "Multi-Touch Contact Motion
Fxtrnction." 11/559.799. entitled "Multi-Touch Contnct
Motion I Oxtraction." II /559.iQ2. entitled "Multi-Touch Con-
tact Motion l;xtraction." 111559.833. entitled Multi-'llmch
I land Position OllSet Computation. each of which is hcrchy
incorporntcd by reference in its entirety
5 dard input de\ icc collection must ha\ecrgonomic. economic.
and productivity advantages which outweigh the unavoidable
sacrifices of abnndoning device spccinlitntion. The g.cncric
device must tightly intcg.ratc yet clearly distinguish the dif-
1\:rent types of input. It should therdlJrc appear modeless to
lCJ the user in the sense that the user should not ne\.:d to provide
explicit mode switch signals such as huttonprcsscs. arm relo-
cmions. or stylus pickups before switching from one input
nctivity to another. Fpidcmiolog.ical studies sug.gcst thm rep-
etition and l(lrcc multiply in causing repetitive strain injuries.
15 Awkward postures. device activation ll1rce. waskd motion.
nnd repetition should be minimized to improve C!'6onomics.
Furthcnnorc. the work!ond should be spread evenly over all
nvni!able muscle g.roups to nvoid repetitive strain.
Repetition can he minimi/L'tl by allocating to several
"" graphical manipulation channels those tasks which require
complex mouse pointer motion sequences. Common g.raphi-
ca! user interface operations such as Ending. and mnnipu !ating
a scroll bar or slider control are much less cllicicnt than
specia!i/L'tl linger motions which cause scrolling directly.
ISACK(jROUND OF Till: IN\"I;NTION
"5 without the step of repositioning the cursor over an on-screen
control. l'rc!Crab!y the grnphical mnnipu!ntion chmmds
should be distributL'd nmongst many !ing.cr and hand motion
combinations to spread the workload. 'llmchpads and mice
with auxilliary scrolling controls such as the ('irquc f\ '"
A Field of the Invention
'll1c present invention rclntcs g.cncra!ly to methods and
apparatus ll1r data input. and. more p;u1icubr!y. to a method
and apparatus ll1r integrating manual input.
30 Smnrtcnt touchp.1d with edge scrolling.. the IBMR
1
-" Scro!l-
l'ointT"1 mouse with embedded pointing stick. nnd the Roller
Mouse dcscribL'd in U.S. I'm. No. 5.530.455 to Gi!!ick ct nl.
represent small improvements in this area. but still do not
IS. Description of the Related Art
Mnny methods for manual inplll of d.1w and conunands to 35
compmcrs nrc in usc todny. bm each is most cllicicnt nnd easy
to usc ll1r particular types of data input. For example. drawing
tah!ets with pens or pucks excel at dralling. sketching. and
quick cmnmand gestures. llandwriting with a stylus is con-
venient Jix tilling. olll fonns which require sig.nnturcs. spccin! 40
symbols. or smnll amounts of text. but handwriting. is slow
compared to typing and voice input ll1r long documents.
Mice. !ingcr-sticks and touchpads excel at cursor pointing
and g.raphicn! object mnnipu!ations such as drug nnd drop.
Ro!lcrs. thumbwhccls and trackballs excel m panning. and 45
scrolling. The diversity of tusks that many computer users
encounter in a single day call ll1r a!! of these techniques. hut
l(:w users wi!l pay ll1r a multitude of input dL'\ ices. and the
scparmc devices arc olicn incompmib!c in a usnbility and nn
C!'60nomic sense. For instance. drawing tnb!cts nrc n must lix '0
graphics prol\:ssionals. hut switching between drawing and
typing is incomenient because the pen must he put down or
held awkwardly between the lingers while typing. rhus. there
is a !ong.-JC!t ncL'd in the art for n mnnun! inplll device which
is cheap yet offers convenient intcgrmion of common mnnun! '5
input IL'chniqucs.
Speech recognition is an exciting new technology which
promises to relie\e some of the input burden on user hands.
llowcvcr. voice is not npproprimc Jix inputting. n!l types of
dnta either. Currently. voice input is best-suited Jixdictmion 60
of long text documents. Until natura! language recognition
matures sullicient!y that \cry high level voice commands can
be understood by the computer. voice wi!! hnve little ndvan-
wg.c over keyboard hot-keys and mouse menus for conunand
and controL Furthcnnorc. precise pointing.. drawing. and 65
manipulation ofgraphica! objects is dillicult with \oicecom-
mands. no matter how well spel.:ch is understood. 'llms. there
prmide enough direct manipulation channels to eliminate
many o lien-used cursor motion sequences. Furthermore. as S.
/hni ct nl. found in "Dun! Stream Input for Pointing. nnd
Scrolling.." Procccding.s of Clll '97 Extended ..\bstrncts
( 1997). manipulation of more than two degrees of li-cL'tlom at
a time is very dillicult with these devices. pre\enting simu!-
tmKous p.1m1ing. tooming and rotming..
Another common method for reducing excess motion nnd
repetition is to automatically continue pointing or scrolling
mmement signals once the user has stoppL'tl moving or !ills
the linger. RelatL'tl ;u1 methods can be distinguishL'tl by the
conditions under which such motion continuation is cnnb!cd.
In U.S.I'm. No. 4.734.685. Watnnnbc continues imag.c p.1n-
ning when the distance and velocity of pointing de\ icc mme-
ment exceed thresholds. Automatic panning is. stopped by
moving the pointing device buck in the opposite direction. so
stopping. R'<.juircs ndditiona! precise movements. In U.S. l'nt.
No. 5.543.591 to Gi!!cspic ct al.. motion continuation occurs
when the linger enters an edge border region around a small
touchpad. Continued motion speL'tl is lixL'tl and the direction
corresponds to the direction from the ccntcrofthc touchpnd to
the !ingcr nt the edge. Continuation mode ends when the
linger lea\es the border region or !ills off the pad. I )isad\;m-
tagcously. users sometimes pause at the edge of the pad
without intending ll1r cursor motion to continue. and the
unexpected motion continuntion becomes annoying.. U.S.
I'm. No. 5327.161 to Lognn ct al. describes motion continu-
ation when the !Inger enters a border area as welL hut in an
altemativc trackball emulation mode. motion continuation
can be ali.mction solely of Intern! !ingcr velocity and direction
m lilio!I Motion continuntion decays due to aliiction factor
or can be stopped by a subSL'<.jUCnt touchdown on the surface.
Disad\antagcous!y. touch velocity at !illol'l' is not a reliable
indicator of the user's desire ll1r motion continuation since
us 7,812,828 82
3
when approaching a large large! on a display at high spc<._-ds
1hc user may not stop 1hc pointer completely before !ilio!I
Thus it would be an advance in the nrt to provide a motion
continuation method which docs not b<..><::omc activmcd uncx-
pec!cdly when the user really intended to slop pointer move-
ment a! a large! but happens lobe on a honk:rm happens to be
moving m sig.ni!icant spc<..'d during !ilio!I
Many attempts have been made to embed pointing. devices
in a keyboard so the hands do no! have lo k:melyping position
lo access the pointing device. These include the inlegratcd
pointing h:y described in U.S. Pal. No. S.IWJAtn to Fran/el
al.. the integrated pointing stick disclosed by J. Rutledge and
T. Sdkcr in "Force-to-Motion Functions for Pointing."
4
An ergonomic typing syslcm should require minimal key
tapping force. easily dis1inguish ling.cr wps lfom resting
hands. and cushion 1hc ling.crs from 1hc jarring. force of sur-
lUcc impact. Mechanical and membrane keyboards rely on
5 the spring !l1rcc in the k'-')switchcs lo prevent activation when
the hands arc resting on the k9s. This causes an iJTcnmci!-
ahlc lradco!f hc!wccn the ergonomic desires lo reduce the
lU1iguc from key ac1ivming force and 10 relax 1hc fu!! wcig.ht
of1hc hands omo the keys during. rest periods. Force minimi-
1" zmion on 1ouch surlUccs is possible wi1h capacitive or active
optical sensing. which do no! rdy on linger pressure, rather
than rcsisli\c-mcmhranc or surliJcc-acouslic-wa\c sensing
techniques. The related an touch devices discuss<.-d below
will become conli.1scd if a whole hand including its li.1ur
15
lingc11ips a thumb and possibly palm hcds, rests on the sur-
bee. Thus. there exists a long !dl need in the arl ll1r a multi-
touch sur!iJCc typing syslcm has<.-d on zcro-!(Jrcc capacili\c
sensing which can 1olcratc resting hands and a surlUcc cush-
I !uman-Compulcr Interaction IN"rl ORACI- '90. pp. 701-00
(1990), and the position sensing keys dcscrihcd in U.S. Pal.
No. 5.675.361 to S.1ntilli. Ncvenhdcss. the limited move-
ment runge and rcsolmion of these devices. leads to poorer
pointing speed and accuracy than a mouse. and they add
mechanical complexity lo keyboard conslnKlion. Thus there
exists a need in the arl !l1r pointing methods with higher ""
rcso!mion. !arg.cr movemcm range. and more dcg.r<.-cs of lf<.-c-
dom yet which arc easily accessible lfom typing. hand posi-
ion.
.-\.n crg.onomic 1yping. sys1cm should also adap1 to indi-
vidual hand si/cs tolerate \ariations in typing style. and sup-
purl a range of healthy hand postures. nwugh many ergo-
nomic keyboards have been proposed. mechanical
lions.
'!(mch screens and louchpads olkn distinguish pointing
motions !i-om emulated bun on clicks or kcyprcsscs by assum-
ing. VC!)' little Intern! lingcrlip motion will occur during. taps
on the touch surlUcc which arc imcnd<.-d as clicks. Inherent in
these mc!hods is the assumption !hal lapping wi!! usua!!y be
straight down limnlhc suspended linger position. minimizing
1hosc componcms of ling.cr motion tang.cntia! 10 1hc surlUcc.
This is a valid assumption if1hc surlUcc is not !indy divided
into dis1inc1 key areas or if 1hc user docs a slow. "hum and
peek" visual search ll1rcach k<.y bdl1rc striking.l.-mcxamp!c.
in U.S. Pal. No. 5.543,591 lo (ii!lespic cl aL alouchpad sends
all Intern! motions to the host compmcr as cursor movemcms.
llowcvcr. if 1hc linger is lilkd soon cnoug.h alicr touchdown
10 coum as a tap and ifthcaccumu!atcd latera! mo1ions arc no1
excessive. any scnl motions are undone and a mouse bunon
click is scnl instead. This method only works ll1r mouse
commands such as poiming. which can safely be undone. no1
for drag.ging or o1hcr manipulations. In U.S. PaL No. 5.666.
113 to I ogan, laps with less than about
1
/c-" la!cra! motion
activalc k9s on a small keypad while la!cra! motion in excess
of
1
/c-" activates cursor control mode. In both palcnls cursor
mode is invoked by ddUuh when a linger stays on 1hc surlUcc
a long lime.
I low ever. fasllouch typing on a stubccdividcd into a large
array ofkcy regions lends lo produce mmclangcnlial motions
along. the surlUcc 1han related an lihcring tcclmiqucs can
1oleratc. Such an array contains keys in muhiple rows and
columns which may not be directly under the lingers. so the
user must reach with the hand or !lex or cx!cnd lingers lo
!ouch many of the k<.y regions. Quick reaching and cx!cnding
imparts sig.nilicmu Intern! linger motion while the linger is in
1hc air which may sti!! be present when 1hc ling.crcontacts the
surli1cc. (i!ancing laps with as much as
1
/4'' la!cra! motion
measured allhc smbcc can easily rcsul1. Ancmpling lo Iiller
or suppress this much motion would make the cursor seem
sluggish and unresponsive. Furthermore. i1 may be desirable
10 enter a typcmatic or mnomatic key repeat mode ins1cad of
pointing mode when the linger is hdd in one place on the
surli1cc. Any la!cra! shilling by the lingc11ip during a pro-
!ong<.-d ling.cr press would also be picked up as cursor jiHcr
wi1holl1 heavy li!tcring.. Thus. there is a JK'Cd in 1hc arl for a
mc1hod to distinguish keying from poiming. on the same
surli1cc \ia more robust hand conliguralion cues than la!cra!
motion of a single linger.
"
5
kcyswi1Chcs can only be repositioned at great cost. For
example. 1hc keyboard with concave kcywclls described by
llargrcmcs cl a!. in U.S. Pal. No. 5.6Wl.253 lils mosl hands
well but also lends lo lock the arms in a single position. A
touch surlUcc key layom could easily be morphcd. 1rnnslat<.-d.
30
or arbi1rari!y r<.>conligurcd as long. as the chang.cs did not
con!l1sc the user. llowcvcr, touch surli1ccs may no! provide as
much la!cra!ly orienting lacti!c l(:edhack as the <.-dgcs of
mechanical kcyswilchcs. Thus. there exists a need in the arl
li.1r a surlUcc 1yping r<.>cog.nizcr which can ad.1pt a key layout
35
to lit individual hand posmrcs and which can sustain typing
accuracy if the hands drill due lo limited lacti!c k<.-dback.
I landwriling on smooth !ouch surli1ccs using a stylus is
well-known in the arl. but ittypica!!y docs nul integrate wdl
wi1h 1yping. and poiming. because 1hc stylus must be pll1 down
40 somewhere or held awkwardly during o1hcr input activities.
Also, it may he dillicuh lo distinguish the handwriting a eli\-
ity of the stylus Jiom pointing motions of a lingc11ip. Thus
there exists a !K'\.:d in the arl ll1r a method lo capture coarse
handwri1ing. gcslllrcs witholl1 a s1y!us and wi1hout conli.1sing
45
them with poiming. motions.
Many of1hc inpll1 dillCrcntiation needs cited above could
be mel with a !ouch sensing technology which distinguishes a
variety of hand conliguralions and motions such as sliding
linger chords and g.rips. Many mechanical chord keyboards
-;o have been dcsig.ncd 10 dc1cc1 simultan<.ousdownward activi1y
lfom muhiple ling.crs. but they do no1 detect Intern! ling.cr
motion over a large range. Rdatcd art shows s<.'\cra! examples
ofcapacili\c louchpads which emulate a mouse or k9board
by tracking. a sing.le linger. These typically measure the
'5 capacitance of or bc1wccn elongated wires which arc !aid out
in rows and columns. A thin dielectric is interposed hc!wccn
the row and column !ayers. Presence of a linger pcrtmbs the
sdf or mutua! capacitance !(H nearby electrodes. Since mosl
of these 1<->clmo!og.ics usc projcc1ivc row and column sensors
60 which imcg.ratc on one ck>clrodc 1hc proximity of a!! objcc1s
in a particular row or column. they cannot uniquely dctcJminc
the positions of two or moreoh_j<.>cls as discussed inS. I ce. "A
Fas1 Multiple-Touch-Scnsi1ivc Input Device." University of
Toronto Masters Thesis ( 1984). The best they can do is count
65 lingcnips which happen 10 lie in a straig.ht row. and even that
wi!lli1il if a thumb or palm is inlrmluccd in the same column
as a lingcrlip.
us 7,812,828 82
5 6
those rows. and then timing the dischargeofsekctcd columns
to ground through n discharge resistor. I .cc's design required
only two diodes per electrode. but the principal disadvnntngc
of Lee's design is thnt the colunm diode reverse bins capnci-
In U.S.I'al. Nos. 5.565.65R and 5.305.017. (ierpheidect a!.
measure the mutual capacitance between row and column
electrodes by driving one set of electrodes m some dock
frequency and sensing how much of that frequency is coupled
onto a second ek:clrmk: scl. Such synchronous measurements
arc \cry prone to noise at the driving Jicqucncy. solo increase
signal-to-noise rmio they fonn virtual electrodes comprised
5 lances allowed interJCrcncc between electrodes in the same
column.
.-\1! oft he related cnpncitnncc scnsingnrt citcdnbove utilize
intcrpolmion between electrodes to nchicve lligh pointing
resolution with L><::omJmica! ck><::trodc ISoicct a!.
and Cii!!cspic ct a!. discuss cmnpultattion of a centroid from
a!! row and column electrode readings. !JowL'\ cr. l(Jr mu!tipk
!ingcr dct<.><::tion. centroid ca!cu!ntion must be carcli.llly lim-
ited around locn! maximn to include only one !ingcr m a time.
I L'C utilizes a bisccti\e search technique to lind local maxima
of multiple rows or multiple columns. instead of a single row
and column. and scan through dcclrmk: combinations until
the \arious mutual capacitances arc nulled or balanced. The lCJ
coupled signal increases with the produc! of the rows and
columns in each vimwl electrodes. but the noise only
increases with the sum. giving a net gain in signal-to-noise
ratio l(Jr virtual elcctrmk:s consisting of more than two rows
and two columns. I lowe\ cr. to uniquely distinguish multiple
objects. virtunl electrode sizes would hnvc to be reduced so
the intersection of the row nnd column virtual electrodes
would be no larger than a !ingcr tip. i.e .. about two rows and
two columns. which will degrade the signal-to-noise ratio.
Also. the signal-to-noise ratio drops as row and column ""
lengths increase to cover n large urea.
15 and then interpolates only on the eight nearest neighbor elec-
trodes of each local maximum electrode. This mny work !inc
In U.S. I'm. Nos. 5.543.59!. 5.543.590. and 5.495.077.
(ji!!espic ct a! measure the ekctrmk:-linger
l(Jr row and column electrodes independently. "I(Jta! electrode
capacitance is estimated by measuring the electrode voltage
chnngc caused by injecting or removing a known nmount of
chnrgc in n known time . .-\1! electrodes cnn be mcnsurcd
simultaneously if each electrode has its own drive/sense cir-
cuit. The centroid calculated Jium all row and column elec-
trode signals cswb!ishcs an intcrpo!ntcd verticnl nnd horiton-
wl position for a single object. Tills method may in gcncrn!
hnve higher signn!-to-noisc ratio than synchronous methods.
hut the signal-to-noise ratio is still degraded as row and col-
umn lengths increase. Signal-to-noise ratio is especially
important for accurmely !ocming objects which nrc !loming a
few millimctersnbove the p.1d. Though tills method can dct<.><::t
such objects. it tends to report their position ns being ncar the
middle of the pad. or simply does not detect Jloating objects
ncar the edges.
"llllls there exists n need in the art for a capncitnncc-scnsing
apparmus which docs not suffer li"om poor signal-to-noise
ratio and the multiple linger indistinguishability problems of
touchpads with long row and column ck><::trodcs.
lix smn!! fingertips. but thumb nnd pnlm contacts mny cover
more than tlinc electrodes. Thus there exists n nL-cd in the art
!(Jr imprmed means to group exactly those electrodes which
arecmercd by each distinguishable hand contact and to com-
pute n centroid li"om such potcntinl!y irrcgulnr groups.
To tnkc mnximum advnnwgc of multi-touch surlUcc sens-
ing. complex proximity image processing is necessary to
track and identify the parts oft he hand contacting the surbcc
"5 at any one time. Compared to passi\e opticaL images. prox-
imity images provide clear indicmions of where the body
contncts the surfncc. uncluttered by luminosity varimion nnd
extraneous objects in the background. Thus proximity image
liltcring and segmentation stages can be simpler and more
30 relinble than in computer vision npproachcs to li"cc-sp.1cc
hand tracking such asS . .-\limnd. ".\ Usnble Rcnl-Timc 3])
!land Tracker." ProcL-cdings of the 28'" .-\si!omnr Conference
on Signals. Systems. and Computers l'at1 2. vo!. 2. II ;I ;I:
(1994) mY. ('ui and .1. Wang. "!land Segmentation Using
35 Lcaming-BnsL'<i Prediction and \Cri!ication for lland Sign
Recognition." ProcL-cdings of the 19% 11-'1:1-' Computer Soci-
ety Conference on Computer Vision and l'nttcrn Recognition.
pp. XX-93 ( 1996). llowcvcr. pat1s of the hand such as inter-
mediate linger joints and the center of the palms do not show
40 up in cnpncitive proximity imngcs m all if the hnnd is not
Hmtcncd on the surlUcc. Without these intcnncdiatc linkngcs
betwL-cn lingertips and palms the overall hand stmcture can
only he guessed at. making hand contact identilication very
dillicult. I knee the optical Jlow and contour tracking tcch- U.S. Pat. No. 5.463.3XX to Boic ct a!. has a capacitive
sensing system npplicnble to either kcybonrd or mouse input.
but docs not consider the problem of integrating both types of
input simultaneously. Though t!u:y mention independent
dctL><::tion of arrayed unit-eel! ck><::trodcs. their capacitance
trnnsduction circuitry appears too complex to be cconomi-
cnl!y reproduced at cnch electrode. Thus the long !cad wires '0
connL><::ting ck><::trodcs to remote signnl conditiotling circuitry
can pickup noise and will ha\e signilicant capacitance com-
pared to the linger-electrode again limiting
signal-to-noise rntio. Also. they do not rccogtlizc the impor-
wncc of independent electrodes for multiple !inger trucking. '5
or mention how to track multiple lingers on an independent
electrode array.
45 niqucs which hnve been npplicd to li"cc-spacc hand sign lnn-
guagc recognition as in !' Quck. "Unencumbered Gestural
Interaction." 11;1:1; Multimedia. \ol. 3. pp. 36-47 (1996). do
not address the special challenges of proximity image track-
ing.
Synnptics Corp. hns succcssli.llly fabricntcd their electrode
nrray on Hcxibk mylar Him rmhcrthnn still" circuit board. This
is suitabk l(Jr conl(Hming to the contours of special products.
but does not prm ide signilicant linger cushioning !(Jr large
surlUccs. Even ifn cusllion wns placed under the lilm. the luck
of stretchability in the lilm. lends. and ck><::trodcs would limit
the compliance am1rded by the compressible material. ISoic
ct a! suggests that placing comressihle insulators on top of the
electrode array cushions linger impact. llowcvcr. an insulator
more than about one millimeter thick would seriously nttcnu-
!.ec built an early multi-touch electrode array. with 7 mm
by 4 nun mew! electrodes arranged in 32 rows nnd 64 col-
umns. The "Fnst Multiple-Touch-Sensitive Input Device
(FMTSID)" total acti\e area measurL-d 12" by 16". with a
0.075 mm Mylar dick><::tric to insulate lingers li"om ekc-
trodcs. Fnch electrode had one diode comlcctcd to n row
60 me the mcnsurcd !ingcr-ck><::trodc cnpaciwnccs. Thus there
exists a need in the art l(Jr a method to transl'-:r linger capaci-
tance inlluences through an arbitrarily thick cushion.
chnrging line nnd n second diode com1cct<.-d to n column
dischnl'oling line. Flcctrodc cap.1citnncc chnngcs were men- 65
surL'<i singly or in rectangular groups by raising the voltage on
one or more row lines. selectively charging the electrodes in
SUMMARY OF Till-' INVJ:NTION
It is a primary object of the present imention to provide a
system and method !(Jr integrating difJCrcnt types of manual
us 7,812,828 82
7
input such as typing. multiple manipula-
tion. and handwriting on a multi-touch sur!Ucc.
It is also an object of the present invention to provide a
system and method for distinguishing dillCrcnt types of
manual input such as typing. multiple
manipulation. and handwriting on a multi-touch surli1cc. via
dillCrcnt hand conHgurmions which arc easy Jix the user to
learn and easy for the system to rccognitc.
II is a Ji.n1hcr object of the present inn::nlion lo provide an
improved capacitancc-lransducing apparatus that is cheaply
implemented ncar each dcclrmk: so that two-dimensional
sensor arrays of nrbitrm)' site and resolution can be built
withom dcg.radmion in sig.nal to noise.
II is a Ji.n1her object of the present invention lo provide an
electronic system which minimizes the number of sensing
electrodes necessm)' to obtnin proximity imnges with such
resollllion thnt n vnricty of hnnd conHgurntions cnn be distin-
guished.
Yet another object of the present invention is to provide a
multi-touch smbce apparatus which is cmnpliant and con-
toured to be comlixtnble nnd ergonomic under extended usc.
Yet nnothcr object of the present invention is to provide
tactile key or hand position Ji.:cdback without impeding hand
n::sling on the stubce or smooth. accuralc sliding across the
surli1ce.
It is n li.Jrthcr object of the present invemion to provide nn
electronic system which cnn provide imngcs ofHcsh proxim-
ity to an <I!Tay of sensors with such resolution that a variety of
hand conliguralions can be distinguished.
It is nnothcr object of 1he present invention to provide nn
improved me1hod for invoking cursor motion continumion
only when 1he user wmus it by not invoking it when signiH-
canl deceleration is dclected.
8
means conn\..-cted in parallel across the integrating capacitor
to deplete its rcsidunl charge: nnd n vohage-to-voltnge 1rnns-
lmion device connected to the output node of the scrics-
comJccted switching means which produces a voltngc rcprc-
5 senting the magnitude of the sell:capacitance of the sensing
device. Altemali\ely. the sensing device comprises: two ek-c-
tricnl switching means com1ected 1oge1her in series hnving n
conunon node. nn input node. and nn olllplll node: a diclec-
tric-cmered sensing electrode ctmn\.."CI\..-d lo the common
lCJ node between the two switching means: a power supply pro-
viding an approximately constant \ullage connL-cted to the
input node of the serics-CO!l!K"C!cd switching menns: and an
integrating currem-to-voltagc translation device CO!l!K"C!cd to
the output node of the series connected switching means. the
15 current-to-voltagelranslation device producing a \oltage rep-
resenting the magnitude oft he scJJ:cnpncitnnceofthc sensing
device.
'!i.1 li.mher achieve the objects. the present invemion com-
prises a multi-touch smbce apparatus ll1r detecting a spatial
"" anangemenl of nmltiple touch devices on or near the surbce
of the multi-touch apparntus. comprising. one of a rigid or
Hcxiblc sur!Ucc: n plurality of two-dimensionnl arrays of one
of the sensing devices (rL-citL-d in the previous paragraph)
<llT<mg\..-d on the surbce in groups wherein the sensing d\..'\ ices
"5 within a group ha\e their output nodes ctmJK"CIL-d together
nnd share 1he same intcgrming cnpacitor. charge depletion
switch. nnd voltngc-to-vohagc translation circuit!)': control
circuitry ll1r enabling a single sensor device Jium each two-
dimensional anay: means ll1r selecting the sensor \oltage
30 datn from each two-dimensional nrrny: voltage mensurcmcnt
circuit!)' to convert sensor voltnge dma ton digitnl code: nnd
circuit!)' li.1r conununicnting 1he digitnl code to an01her clcc-
tnmic de\ ice. '1-he sensor voltage data sek-cting means com-
Another object of the present invention is to identify dif-
ferent hnnd puns as they contact the sur!Ucc so that a vnricty 35
ofhnnd con!igurntions can be recognized and used to distin-
guish dill"crelll kinds of input nctivity.
prises one of a multiplexing circuitry and a plurality of volt-
ngc measurement circuits.
']i.J still li.Jrthcr nchicvc the objects. 1he present invention
comprises n multi-touch surfacenpparntus Jixsensing diverse
conliguralions and activities of touch de\ ices and generating Yet another object of the present imenlion is lo reliably
extract rotation and scaling as we!! as translation degrees of
freedom from the motion of two or more hand contacts to nid 40
in navigmion nnd manipulation of two-dimensionnl eke-
Ironic documents.
inlcgrak-d manual input lo one of an electronic or electrmne-
chanicnl device. the nppnratus comprising. nn array of one of
the proximity sensing devices described nbovc: n dielectric
cmer ha\ing symbols printL-d thereon that represent action-
to-bc-laken when engagL-d by the !ouch d\..'\ ices: scanning
means ll1r l(mJJing digital proximity images li'llm the array of
II is a Ji.u1her object of the present imenlion to reliably
extract tilt and ro!! degrees of Ji-e\..-dmn Jimn hand pressure
dill"crcnccs to aid in nnvigntion and manipulation of thR'C-
dimcnsional cnviromncnts.
Additional objects and ad\;mtages of the invention will be
set ll1rth in part in the description which l(J!!ows. and in part
will be obvious from 1he description. or may be lcamcd by
prncticc of the invention. The objects nnd ndvantages of the
invention will be rcnliz\..'d nnd allnin\..'d by mcnns of 1he cle-
ments and combinations particularly point\..-d out in the
append\..-d claims.
45 sensing devices: cnlibrming menns Ji.1r removing bnckground
ollScts from the proximity imngcs: recognition means for
inlcrpreting the conliguralions and activities of the !ouch
devices that make up the proximity images: processing means
li.1r generating inplll signnls in response to pnnicular touch
'0 device conHgurmions and motions: nnd conununication
mcnns for sending 1he input signals to the ck"Clron.ic or clec-
tromL-chanical device.
']i.J achieve 1he objects and in accordance with 1he purpose
of the invention. ns cmbodk-d and broadly described herein. '5
the imenlion comprises a sensing de\ ice that is sensili\e to
changes in scJJ:capacilance brought about by changes in
proximity ofatouch de\ ice to the sensing d\..'\ ice. the sensing
device comprising: two electrical switching mcnns com1ected
together in series having n common node. nn input node. and 60
an output node: a dielectric-cmer\..-d sensing electrode con-
ll\.."CI\..-d to the common node bclween the two switching
means: n power supply providing an npproximatcly constmll
voltage conneck'd to the inplll node of the scries-com1ected
switching means: nn integrating cnpacitor to accumulntc 65
charge transl"crrl..-d during multiple conseculi\e switchings of
the series conneck-d switching means: another switching
'Ill even li.1rther achie\e the objects. the present invention
comprises n multi-touch surfacenpparntus li.1rsensing diverse
conHgurations nnd activities of lingers and palms of one or
more hands near the sur liKe and generating integra!L-d manual
input lo one of an ek-clronic or ek-clrom\..-chanical de\ ice. the
apparatus comprising: an array of proximity sensing means
embedded in 1he sur!Ucc: scanning means for forming digital
proximity imngcs from the proximitics measured by 1he sens-
ing means: image segmentation means ll1r co!!ecting into
groups those proximity image pixels inlensiliL-d by contact of
the same distinguishable p.1rt of a hand: contnct trucking
mcnns li.1r pnramcterizing hand contact l"cmures and trnjccto-
rics ns the contncts move across successive proximity imnges.
contact identilicalion means ll1r determining which hand and
which p;u1 of the hand is causing each surli1ce conlacl: syn-
us 7,812,828 82
9
dch.x:lion means !l1r identifying suhsc!s of idcn-
tiHcd contacts which touchdown or liftoff the sur!Ucc m
approximmcly the same time. and for generating conummd
signals in response to synchronous wps of multiple lingers on
the surliJCc: typing recognition means ll1r generating intended 'i
key symbols from asynchronous linger laps: motion compo-
nent extraction means for compressing multiple degrees of
freedom of multiple lingers into degrees of li"< .. -cdom common
in two and three dimensional graphical manipulation: chord
motion recognition means ll1r generating one of command lCJ
and cursor manipulation signals in response lo motion in one
10
enabling users to instantaneously switch between the input
nctivities by plncing their hnnds in dillCrent con!igurmions
comprising. distinguishnble combinmions of relmive hnnd
contnct timing.. proximity. shape. site. position. motion nnd/
or identity across a succession ofsurli1cc proximity images,
the method comprising the steps of: tracking each touching
hand part across successive proximity images: measuring the
times when each hand pnrt touches down and lilis olr the
surlUcc: detecting when hand parts touch down or Jill olr
simultnncously: producing discrete key symbols when the
user asynchronously taps. holds, or slides a linger on key
regions ddincd on the surbcc: producing discrete mouse
button click commands. k<..y comm;mds, or no signals when
the user synchronously wps two or more lingers lfom the
same hand on the surlUcc: producing gesture commands or
multiple dcgrcc-of-licedom manipulation signals when the
user slides two or more lingers across the surbce: and sending
the produced symbols. commands and manipulation signals
or more cxtract<..-d degrees of freedom by a selected combina-
tion of !ing.crs: pen grip detection means for recognizing
contact arrangements which resemble the conliguration of the
hand when gripping a pen. generating inking signals lium l'i
motions of the inner !ing.crs. nnd gcncrnting. cursor mnnipu-
!ation signals from motions of the palms while the inner
lingers nrc lillcd: and communicmion mcnns for sending. the
sensed conligurations and acti\ it ies o flinger and palms to one
"" ns inplll to an electronic or an elcctro-mcchanicnl device.
of the electronic and electromechanical de\ ice.
'li.1 li.1rthcr nchicve the obj< .. ><::ts. the present invention com-
prises a method lix trucking nnd idcntil)ing. hand contncts in
a sequence of proximity images in order to suppm1 interpre-
tation of hand conligurations and activities related to typing.
multiple manipulation via chords. and "-'
hnndwriting. the method comprising the steps of: seg.menting
cnch proximity imngc into groups of electrodes which indi-
cate signilicant proximity. each group representing proximity
of a distinguishable hand part or other touch device: extract-
ing. totnl proximity. position. shnpe. size. nnd oricntmion 30
pnramcters lfom cnch g.roup of electrodes: trucking g.roup
pnths through successive proximity imngcs including dct<..'<::-
tion of path endpoints at contact touchdown and liliolf: com-
puting \elocity and liltcred position vectors along each path:
assigning. a hand and !inger identity to ench contact pnth by 35
incorpornting. relmive pmh positions nnd velocities. indi-
vidunl contnct femures. nnd previous estimntcs of hnnd and
linger positions: and maintaining estimatesofhand and linger
positions Jfom trajectories of paths currently assigned to the
lingers. wherein the cstimmcs provide hig.h level fcedbnck to 40
bins scgmcntmions nnd identi!icntions in li.Jture imag.es.
'](J still Jlu1hcr achieve the objects. the present invention
comprises a method ll1r integrally extracting multiple degrees
of l].._,:dmn of hand motion Jium sliding motions of two or
more !ing.ers of a hnndncross n multi-touch surface. oneofthc 45
lingers prefcrnbly being the opposnb!c thumb. the method
comprising the steps of: tracking across successive scans of
the proximity sensor array the trajectories of indi\ idual hand
pnrts on the surlUcc: !inding. an inncnnost and nn omcnnost
!ingcr contnct lfom contncts identified as !ingers on the g.iven '0
hnnd: computing. a scaling velocity component from n chang.c
in a distance between the innermost and outennost linger
contacts: computing a rotational velocity component Jfom a
chnng.e in a V<..><::tor ang.!c between the itmermost nnd outer-
most !ing.cr contncts: compllling n translation weighting. lix '5
each contacting linger: cmnput ing translational \ elocity com-
ponents in two dimensions Jium a translation weighted aver-
age of the linger velocities tangential to stubce: supprcs-
sively !iltering. components whose speeds nrc consistently
lower than the lUstest components: transmitting the filtered 60
\elocity components as control signals to an electronic or
electromechanical de\ ice.
'li.1 even further nchieve the objects. the present invention
comprises n manual input intcgrmion method for supporting
diverse hand input activities such as resting. the hnnds. typing. 65
multiple dcgree-ol:j].._,:dmn manipulation, command gestur-
ing and handwriting on a multi-touch surbce. the method
'li.1 still even li.1rthcr nchieve the objects. the present inven-
tion comprises n method lix choosing whm kinds of input
signals will he generated and sent to an electronic or electro-
mechanical device in response to tapping or sliding oflingcrs
on n multi-touch surface. the method comprising the follow-
ing steps: identil)'ing cnch contnct on the surface as either n
thumb. lingcrtip or palm: measuring the times when each
hand p;u1touches down and !ills olfthc surli1cc: Jlmning a set
oftlmse lingers which touch down Jfom the all linger Jloating
swte before any one of the !ingers lilis back oll' the surlUcc:
choosing. the kinds of input sig.nnls to be g.cnerntcd by Ji.1rthcr
distinctive motion of the lingers Jfom the combination of
linger identities in the set: generating input signals of this kind
when lim her distinctive motions of the !ing.crsoccur: lixming
n subset nny two or more !ing.ers which touch down synchro-
nously nlicr m !cnst one !inger has lifted buck olfthe surlUcc:
choosing a new kinds of input signals to he generated by
llirlherdistinctivc motion of the lingers Jfom the combination
of !inger identities in the subset: gencrming. input sig.nals of
this new kind when Ji.1rthcr distinctive motions of the !ing.crs
occur: and continuing to llmnJK'W subsets. choose and gen-
erate new kinds of input signals in response to liliolf and
synchronous touchdowns until all lingers Jill off the surli1cc.
'](J Jlu1hcr achie\e the objects, the present invention com-
prises n method for continuing. gencrmion of cursor move-
ment or scrolling signnls from a tang.cntial motion of n touch
device mer a touch-sensitive input d<..'\ ice surli1ce alkr touch
device liliolf Jiom the surli1cc if the touch device operator
indicmes thm cursor movement continuntion is desired by
nccelcrnting. or failing. to decelerate the tnng.ential motion of
the touch device before the touch device is lifted. the method
comprising the lllllowing steps: measuring. storing and trans-
mitting to a computing device two or more rcprcsentati\c
tnng.ential velocities during touch device manipulmion: com-
puting nnd storing. a liliolr velocity from touch device posi-
tions immediately prim to the touch d<..'\ ice lilloff: comparing
the lillolr velocity with the rcprcsentati\e tangential \eloci-
ties. and entering a mode ll1r continmmsly mm ing the cursor
if n tnng.entinllilloff direction approximmely equnls the rep-
resenwtive tnng.ential directions nnd n tnng.entinllilloff speed
is greater than a predctennincd Jiactional multiple of repre-
sentati\e tangential spe'--ds: continuously transmitting cursor
movement sig.nals aller liliolrto n computing. device such that
the cursor movement velocity corresponds to one of the rep-
resenwtive wngcntial velocities: and censing. trnnsmission of
the cursor movement signals when the touch de\ icc engages
the surbce again, if comparing means detects signilicant
us 7,812,828 82
II
tkx:ek:ration b d ( m ~ lilloll or if the computing de\ ice replies
1hnt 1hc cursor can move no !Unhcr or a window can scroll no
farther.
It is to be understood thm both the foregoing gcncrnl
description and the lllllowing detailed description arc cxcm- 'i
pbry and explanatory only and arc no! n::slricli\ co f the im en-
tion as claimed.
IW!I;I: J)J;SCRWI-!ON 01'-r!!l; J)RAWINGS
"111c accompanying dmwing.s. which arc incorporntcd in
and constitute a part of tills spcciHcation. i!lustrntc severn!
embodiments of the invention and together with the descrip-
tion. serve lo explain the principles of the inn::nlion. In the
drawings: l'i
!-'IG. 1 is a block diagram of the integrated manual inplll
apparmus:
I'Ki. 2 is a schematic drawing of the proximity sensor with
\ohagc ampli!icr:
I'Ki. 3 is a schematic drawing of the proximity sensor with ""
integrating. current amplifier:
FIG. 4 is a schematic drawing of the proximity sensor
implemented with lidd dkct transistors:
I'Ki. 5 is a schematic drawing of the proximity sensor as
used to implcmem 21) arrays of proximity sensors: "-'
FIG. 6 is a block diagram showing. a typical architeclllre lix
a 2D array of proximity sensors where all sensors share the
same amplilier:
!'Ki. 7 is a block diagram of circuitry used lo convert
proximity sensor output to a dig.iwl code: 30
FIG. Sis a block diagram showing. a typical architeclllre lix
a 2]) array of proximity sensors where sensors within a row
share the same amplilier:
!'Ki. 9 is a schematic of a circuit usdill !(Jr enabling the
omput gates of all proximity sensors within a g.roup (arranged 35
in columns):
!'Ki. 10 is a side \iL"\\" of a 2]) proximity sensor array that
is sensili\e lo the pressure exerted by non-nmducting touch
objects:
FIG. 11 is a. side view of a 21) proximity sensor array that 40
provides a compliant surface without loss of spatial sensitiv-
ity:
!'Ki. I 2 is a side\ iLw of a 2]) proximity sensor array that
is sensitive to both the proximity of conducting. touch objects
and to the pressure exerted by non-conducting touch objects: 45
FIG. B is an example proximity imageofa hand Hattened
onto the surbce with lingers oulslrclchcd:
I'Ki. 14 is an example proximity image of a hand partially
closed with !ingertips normal to surlUce:
FIG. 15 is an example proximity image of a hand in the pen '0
grip ctmliguralion with thumb and index lingers pinched:
!'Ki. 16 is a data !low diagram of the hand tracking and
contact idcnlilicalion system:
FIG. 17 is a !low chart of hand position estimation:
FIG. 18 is a data How diag.ram of proximity image seg.men- '5
lalion:
!'Ki. 19 is a diagram of the boundary search pallcm during
construction of an ekctrode group:
FIG. 20.\ is a diagram of the segmentation strictness
reg.ions with both hands in their neutral. ddUult position on 60
surli1cc:
!'Ki. 20B is a diagram of the segmentation strictness
reg.ions when the hands arc in asynunclric positions on sur-
face:
I2
I'!G. 2 I is a !low chart of segmentation edge testing:
FIG. 22 is a !low chan of persistent path tracking.
FIG. 23 is a !low chart of the hand pmt idemi!ication
algorithm:
I'!G. 24 is a\ oronoi cdl diagram conslnJcted around hand
parl allractor points:
FIG. 25.-\ is a plot of oriemation weighting factor lix rig.ht
thumb. right im1er palm. and ldi outer palm versus contact
orientation:
FIG. 25B is a plot of thumb size bctonersus contact si/e:
FIG. 25C is a plot of palm si/e liJClor versus ratio of lola!
contact proximity to comact eccemricity:
FIG. 25D is a plot of palm separation factor versus distance
between a contact and it nearest neighbor contact:
FIG. 26 is a !low cha11 of the thumb presence vcrilicalion
algorithm:
FIG. 27 is a How chart of an alternative hand part identifi-
cation alg.orithm:
FIG. 28 is a !low chart of the pen grip detection process:
I'!G. 29 is a !low chart of the hand idenli!icalion algorithm:
FIGS. 30.-\-C show three dillCrem hand partition hypoth-
eses lix a !ixL'<i arrangemem of surface contacts:
FIG. 31A is a plot of the hand clutching direction liJClor
versus horizontal hand velocity:
FIG. 31 IS is a plot of the handedness liJClor \ersus vertical
position of outermost !inger relative to next omermost:
FIG. 31C is a plot of the p.1lm cohesion lUctor versus
maximum hori/onlal separation between palm contacts
within a hand:
FIG. 32 is a plot of the inner !inger angle factor versus the
angle between the innermost and next innermost !ing.er con-
tacts:
FIG. 33 is a plot oft he inter-hand separation factor \ersus
the estimated distance between the right thumb and !ell
thumb:
FIG. 34 is a !low chart of hand motion componem extrac-
tion:
FIG. 35 is a diagram of typical linger trajectories when
hand is nmlracting:
FIG. 36 is a !low chart of radial and angular hand velocity
extraction:
FIG. 37 is a Jlow chart showing extraction of translational
hand velocity components:
I'!G. 38 is a Jlow cha11 ofdill"ercnlial hand pressure extrac-
tion:
FIG. 39.\ is a How chart of the !ing.er synchronization
detection loop:
!'!G. 391S is a Jlow chart of chord lap detection:
FIG. 40.-\ is a !low chart of the chord motion rccog.nition
loop:
FIG. 40B is a How chart of chord motion event g.eneration:
FIG. 4I is a !low chart of key layout morphing:
FIG. 42 is a !low chart of the keypress detection loop:
FIG. 43.\ is a !low chan of the keypress acceptance and
transmission loop: and
FIG. 43B is a Jlow chart oflypcmalic emulation.
DJ:SCRII'TION 01-' Till: I'RITJ:RRJ:t)
J:tvlBOJ)JMI'NTS
RdCrcncc will now he made in detail to the present prc-
JCJTcd embodiments of the invention. examples of which arc
illustrated in the accompanying drawing.s. Wherever possible
the same rdCrence numbers will be used throug.holll the
FIG. 20C is a diagram of the segmentation stricllless
regions when the right hand crosses lo the ldi half of the
surli1cc and the Jell hand is off the surli1ce:
65 drawings to rdCr to the same or like p.1rts.
!'!G. I is a system block diagram oflhccnlirc. integrated
manual input apparatus. Sensor embedded in the multi-touch
us 7,812,828 82
13
surli1cc 2 detccl proximity of entire !lallencd hands 4. linger-
tips thumbs. palms. and other conductive touch devices to the
surface 2. In a prdCrr<..'d embodiment. the sur!Ucc is larg.c
enough to comlixtably accommodate b01h hands 4 and is
arched to reduce !(JrcamJ pronation.
In a!tcrnatin:: cmhmlimcnls the muhi-touch surli1cc 2 may
be lnrgccnough to acconunodmc motion of one hand. bm may
be Hcxiblc so it can be Httcd to an annrcst or cl01hing.
I ]cclronic scanning hardware 6 controls and reads li'llm
each proximity sensor of a sensor array. A calibration module
8 conslmcls a raw proximity image lium a complc!c scan of
the sensor array and subtrncts olr any background sensor
ollScts. The background sensor offsets can simply be a prox-
imity image taken when nothing is touching the surbcc.
'!11c o!1Sct-cmTL-ctcd proximity image is then passed on to
the contact trucking and idemiHcation module 10. which seg-
ments the image into distinguishable hand-surface comacts.
tracks and identities them as they move through successive
1m ages.
'!11c paths of identi!icd contacts arc passed on to a typing
rL-cognizer module 12. linger synchronization detection mod-
ule 14. motion component extraction module 16. and pen grip
dctL-ction module 17. which contain sollwarc algorithms to
distinguish hand conligurations and respond to dctL-ctcd hand
motions.
14
vary widely depending on the Jimction and processing power
of the host compmcr. In a prcJCrred embodiment. the com-
munication would take place over complllcr cables via indus-
try standard protocols such as .\pple Desktop Bus. PS/2 key-
-' board and mouse protocol ll1r I'Cs, or Uni\ersa! Serial Bus
(USB). In ahcmative embodiments the soli ware processing
of modules 10-18 would be pcrfonncd within the host com-
pUler 22. The multi-touch surlUcc apparatus would only con-
tain enough han.lware to scan the proximity sensor array 6,
lCJ Jlmn proximity images 8. and compress and send them to the
host computer mer a wireless network. '!11c host communi-
cation intcrlUcc 20 would then play the role of device driver
on the host complller. conveying results of the proximity
image recognition process as input to other applications
l'i residing on the host computer system 22.
In a prcJCm.>d embodimem the host computer system olll-
pllls to a visual display device 24 so that the hands and Hngers
4 can manipulate graphical obj<.-cts on the display screen.
llowL"\cr, in ahernati\e embodiments the host computer
"" might output to an audio display or control a machine such as
a robot.
The term "proximity" will only be used in rc!Crcnce to the
distance or pressure betwL-cn a touch device such as a !Inger
and the surbcc 2, not in rdCrcncc to the distance between
"-' adjacent !ingcrs. "llmizontal" and "\ertical" rdCr to x andy
dirL-ctional axes within the surface plane. Proximity measure-
ments arc then interprctL>d as pressure in a zaxis nonnal to the
smbce. The dirL-ction "inner" means toward the thumb of a
'll1e typing recognizer module 12 responds to quick presses
and releases of lingers which arc largely asynchronous with
respect to the acti\ ity of other lingers on the same hand. It
allempts to lind the kLy region nearest to the location of each
linger tap and forwards the key symbols or commands asso- 30
ciatcd with the nearest key region to the communication inter-
face module 20.
given hand, and the dirL-ction "outer" means towards the
pinky Hngerof a given hand. For the purposes oftllis descrip-
tion. the thumb is considered a linger unless otherwise noted.
bm it docs not coum as a Hngenip. "Contact" is used as a
gencraltcm1 ll1r a hand p;u1 when it touches the surli1cc and
appears in the current proximity image, and ll1r the group and
'!11c linger synchroni:t11tion dctL-clor I4 checks the linger
activity within a hand ll1r simultaneous presses or releases of
a subset of lingers. When such simultanL'OUS activity is
detected it signals the typing rccogni:tcr to ignore or cancel
keystroke processing for lingers contained in the synchro-
nous subset. ll also passes on the cmnbination oflinger iden-
tities in the synchronous subset to the chord motion recog-
nizer 18.
'll1e motion component extraction module 16 computes
nmhip!e degrees of Ji-eedom of control lium individual linger
motions during easily perllmnab!c hand manipulations on the
surli1cc 2, such as hand translations. hand rotation about the
wrist. hand scaling by grasping with the Hngcrs. and dillCr-
cntial hand tilting.
'll1c chord motion recognizer produces chord tap or motion
e\ents dependent upon both the synchronized !Inger subset
identiHcd by the synchronization delL-e tor 14 and on the dirL-c-
tion and spcL>d of motion extracted in 16. These events arc
then posted to the host conununication intcrlUcc 20.
'!11c pen grip delL-ction module I 7 checks ll1r spccilic
arrangements of idcntilied hand contacts which indicate the
hand isconHgurcdas if gripping a pen. If such anarrangemem
is. detected. it forwards the movcmems of the gripping lingers
as inking events to the host cmnmunication interbcc 20.
These inking L"\ents can either lay digital ink on the host
computer display ll1r drawing or signature capture purposes.
35 path data structures which represent it.
I-'IG. 2 is a schematic diagram of a device that olllpllls a
voltage 58 dependent on the proximity of a touch device J8 to
a conductive sense ek-ctrode JJ. The proximity sensing
device includes two ck-ctrica! switching means JO and JI
40 comJccted together in series having a common node 48. an
input node 46. and an output node 45 .. \thin dick-ctric mate-
ria! J2 covers the sensing ek-ctrode JJ that is dectrica!!y
nmnL-ctcd to the common node 48. A power supply J4 pro-
viding an approximately constant \ohagc is connected
45 between rc!Crence ground and the input node 46. The two
electrical switches JO and J 1 gate the How of charge from the
power supply J4 to an integrating capacitor J7. The \ohage
across the integrating capacitor J7 is translated to another
voltage 58 by a lligh-impL>dancc voltage mnplilier J5. The
'0 plates of the imcgrating capacitor J7 can be dischal'olCd by
closing electrical switch 36 until the voltage across the hue-
grating capacitor J7 is ncar /cro. The electrical switches JO
and J I are opened and dosed in sequence but arc nL"\er dosed
at the same time. although they may be opened at the same
'-' time as shown in 1-'lG. 2. l:t<.-ctrical switch 30 is rc!Crrcd to as
the input switch: electrical switch JI is rckrred to as the
output switch: and. electrical switch J6 is re!Crrcd to as the
shorting switch.
or they can be further interprctL>d by handwriting recognition
soliwarc which is well known in the art. The dctai!L>d steps 60
within each of the above modules will be lin1her described
The proximity sensing device shown in FIG. 2 is operated
by dosing and opening the electrical switches JO. 31. and J6
in a p;u1icu!ar sequence alier which the \ohage output li-om
the ampli!icr 58, which is dependent on the proximity of a later
'll1e host conununication intcrlUcc keeps events lfom both
the typing recognizer 12 and chord motion recognizer 18 in a
single temporally ordered queue and dispatches them to the
host computer system 22. The method of communication
betwL-cn the interbcc 20 and host computer system 22 can
touch device 38. is fL'Cordcd. Sensor operation begins with all
switches in the open state as shown in FIG. 2. 'll1e shorting
65 switch 36 is then dosed for a sullicicntly long time to reduce
the charge residing on the integrating capacitor J7 to a !ow
leveL rhc shm1ing switch J7 is then opened. 'll1c input switch
us 7,812,828 82
15
J() is then dosed thus allowing charge to !low between the
power supply and the conunon node 48 until the volwg.c
across the input switch 30 becomes zero. Cha!'6C Q will
accumtllatc on the sensing electrode 33 according to
Q I \<'''-11 D
'"
where \" is the voltage of the power supply 34. c is the
16
the desired \ullage across integrating capacitor 37. In typical
npplications the number is between one and several hundred
pulsc-pnirs.
I'IG. 5 shows the proximity sensor circuitry npproprimc for
5 use in a system comprising an <I!Tay of proximity sensors 47
as in a multi-touch surbce system. The proximity sensor 47
consists of the input trnnsistor 30. the output trnnsistor 31. the
sensing electrode 33. the dielectric cover 32 for the sensing
pcrmilli\ ity of the dielectric sensing ck:clrmk: cover 32 and
the air gap he! ween thccmcr and the !ouch de\ icc 38. I) is the 1"
thickness of this dielectric region. and A is the overlap area of
the touch device 38 and the sensing electrode 33. Therefore
dec!rode 33. and comluc!i\e !races 43. 44. 45. and 46. The
conduc!ive !races are arranged so as lo allow the proximity
sensors 47 comprising a 21) <I!Tay lo be closely packed and lo
share the same conductive traces. thus reducing the number of
wires needed in a system. I'IG. 6 shows an example of such a
system where the input nodes 46 of all proximity sensors are
the amount of clw!'olC accumulming on the sensing electrode
.B will depend. among other things. on the area ofmcr!ap of
the touch device 38 and the sensing dcc!mdc 33 and the
distance hc!wccn the !ouch device 38 and the sensing elec-
trode 33. The inpm switch 30 is open<.'d nlier the volwge
ncross it hns become zero. or nenrly so. Soon nlier inpm
switch 30 is opened the output switch 31 is closed until the
\oltage across it is nearly 1ero. ("losing the output switch 31
allows charge lo Jlow between the sensing elec!rode 33 and
the integrating capacitor 37 resulting in a voltage change
ncross the integ.rnting cnpncitor 37 nccording to:
15 nmnL><::!ed together and connec!ed loa power supply 34. The
ompm nodes 45 of all proximity sensors are connected
together and connected ton single imegrming cnpacitor 37. n
single shorting trnnsistor 36. nnd n single voltagc-to-voltngc
amplilier 35. In this implementation. a single proximity sen-
"" sm47 is enabled a! a time by applying a logic I signallirsllo
its input gate 43 and then to its ompm gme 44. This gntingof
n single proximity sensor47 oncm a time is done by input gme
controller 50 and output gale controller 51. I'm example. lo
enable the proximity sensor 47 in the lower right comer the
ddtaf' lf'-f'clii+C'!J'"'Ii
121 "5 input gate controller 50 would output a logic one pulse on
conductive truce 43a. "ll1is is Ji.1llowed by a logic one pulse on
conductive trace 44h produced by ompm gme controller 51.
Repetition or this pulse as shown in I'!Ci. 41S would cause
charge lo build up on integrating capacitor 37 and a cmre-
where Vc is the volwge ncross the integrnting cnpacitor 37
before the output switch 31 wns dosed. C is the capacitance
or the integrating capacitor 37. and A and I) are equal to their
\alues when input switch 30 was dosed as shown in I ;qualion
I Multiple switchings or the input 30 and output 31 switches
as described nbove produce n voltage on the imegrming
cnpacitor 37 thnt rcllects the proximity ofn touch device 38 to
the sensing dec!rode 33.
I'Ki. 3A is a schematic diagram or the proximity sensor in
which the shorting transistor 36 and the voltage-to-voltage
translation device 35 are replaced by a resistor 40 and a
currem-to-voltagc trnnslmion device 41. respectively. The
integ.rnting function of capacitor 37 shown in FIG. 2 is. in this
\ariation or the proximity sensor. carried out by the capacitor
39 shown in I'!Ci. 3:\. Those skilled in the ;u1 will see that this
\ariation or the proximity sensor produces a more linear
omput 58 from multiple switchings of the input nnd ompm
switches. depending on the rdmive vnluc of the resistor 40.
Ahemali\dy. the resistor 40 can he replaced by a shm1ing
switch 69 (cr I'!Ci. 31S) lo imprme linearity. Although. the
circuits shown in 1:](;. 3 prm ide a more linear oulpullhan the
circuit shown in I'IG. 2 the circuits ofi'IG. 3 generally r<:quire
dual power supplies while the circuit of FIG. 2 requires only
one.
"l11e dec!rical switches shown in. I'!Ci. 2 can be imple-
mented with various transistor !edmologies: discrete. inle-
grmed. thin !ilm. thick !ilm. polymer. optical. etc. One such
implememation is shown in FIG. 4.\ where !icld efl<..><::ttran-
sislms (1'1 ;1s)are used as the input 30. output 3 I. and shm1ing
36 switches. The I'FI"s are switched on and olfhy \oltages
applied lo their gale terminals (43. 44. and 55). For the pur-
pose of this description we will assume the FIT is switched
30 sponding voltngc to nppear nt the oll1pll1of the amplilier 58.
The entire arrny of proximity sensors 47 is thus scmmed by
enabling n single sensor nt n time nnd recording its output.
I'J(j. 7A is a schematic of typical circuitry useli.illllr con-
veJ1ing the proximity sensor output 58 loa digital code appro-
35 printe for processing by compmcr. The proximity sensor om-
pm 58 is typically non-zero even when there is no touch
device (e.g .. ref no. 38 in I'IG. 2) nearby. Tills non-tero signal
is due lo parasitic or stray capacitance present all he common
node 48 oft he proximity sensor and is or relatively constant
40 vnlue. 11 is dcsirnble to remove this non-zero bnckground
signnl before convening the sensorompm 58 ton digiwl code.
This is done by using a dif!Crenlial amplilier 64 lo subtracl a
stored rL><::on.l or the background signal 68 lium the sensor
output 58. rhe resulting dilkrence signal65 is then comerted
45 to a digitnl code by nn :\DC (nnnlog to digitnl convener) 60
producing n K-bit code 66. The stored background signnl is
!irs! recorded by sampling the array or proximity sensors 47
(FKi. 6) with no !ouch devices nearby and storing a digital
code specilic for each proximity sensor 47 in n memory
'0 device 63. "ll1c particular code corresponding to the buck-
ground signnl ofench proximity sensor is selected by nn M-bit
address input 70 lo the memory device 63 and applied 69 loa
DAC (digital to analog comerter) 61.
The 2]) array of proximity sensors 47 shown in FIG. 6 can
'5 be conncct<.-d in groups so us to improve the rme nt which the
entire array is scanned. n1is is illustrated in l'l(i. 8 where the
on when its gme voltage is logic 1 and switch<.'d oll"whcn its 60
gale \ullage is logic 0. :\controller 42 is used lo apply gale
\oltages as a li.mclion of time as shown in I'Ki. 41S. In this
exmnple. n sequence of thr<.-c pairs of pulses (43 nnd 44) nrc
applied to the input nnd ompmtrnnsistor gmes. Each pair of
pulses 43 nnd 44 produces a voltage change across the intc- 65
grating capacitor 37 as shown in !:qual ion 2. The number of
pulse pairs applied lo input43 and oulpu144 gales depends on
groups are arranged as columns of proximity sensors. In this
approach. the input nodes of the proximity sensors are con-
nected together nnd connected to a power supply 34. as in
FIG. 6. The output gmes 44 nrc also conn<.><::ted in the smnc
way. llowe\er. the input gates 43 are now all cmmec!ed
together and the output nodes 45 are connec!ed lo only those
proximity sensors 47 within a row and ton dcdicmed voltnge
nmplilicr 35. With this conn<.><::tion method. nllofthe proxim-
ity sensors inn colunm arccnnbledm a time. thus reducing the
time lo scan the array by a li!c!m N. where N is the number or
proximity sensors in a group. The outputs 58a-h could con-
us 7,812,828 82
17
m:cllo dedicated com crier circuitry as shown in FIG. ?A or
alternmivcly each olllplll 58a-h could be converted one m a
time using the circuit!)' shown in !-'IG. 7B. In tills Hgurc. the
olllput sig.nals from each group 58a-h arc selected one at a
time by nmltip!cxcr 62 and applied lo the posit in:: input of the
di!krcnlial ampli!icr 64. With this la!cr approach. it is
assumed that the AI)(' 60 comcrsion time is much lilslcr than
the sensor enable time. thus providing the suggested spc<..'d up
in sensor arrny scanning.
!-'IG. 9 shows a typical circuit uscfullix the control of the
proximity sensor's output gale 44. II consists of!lm::c input
signals 75. 76. 78 and two output signals 44. 77. nu:: output
gate sig.na144 is logic I when both inpllls to.\Nl) gate 79 arc
log.ic I. The AND inplll sig.nal 77 becomes logic I if inplll
signal76 is logic I when input signal78 transitions !i-om logic
0 lo logic I. olhen.vise it remains logic 0.:\ linear array of
these circuits 8I can be conneetL"t.! end-to-end lo enable the
omput g.mcs of a sing.le group of proximity sensors m a time
as shown in FIG. 8.
I'IG. 10 shows a cover for the multi-touch surlUcc 89 thm
permits the system lo he sensili\e lo pressure exertL"t.! by
non-eondueting !ouch ohjecls (e.g .. gloved lingers) conlaet-
ing.thc multi-touch surlUce. Tills cover comprises a deform-
able diekctric touch lnyer 85. a deformnblc conducting. layer
86. nnd a complimu dielectric lnyer 87. The touch surfnce 85
would ha\e a symbol set prin!L"t.! on it appropriate ll1r a spe-
cilic application. and this surliJCe could he removed and
replaced with another one hnving. a different symbol set. The
conducting laycr86 is electrically connected 88 to the refer-
ence ground of the proximity sensor's power supply 34.
When a !ouch objecl presses on !he lop surliJCe85 it causes the
condueting surface 86 under the !ouch dL'\ ice lo mme closer
18
"Ill illuslrale typical properties of hand conlacls as they
nppcar in proximity imnges. FIGS. D-15 contnin sample
imag.es cnptured by a prototype arrny of parnllelogrnm-
shaped electrodes. Shading ofcnch electrode darkens to indi-
5 calc heightened proximity signals as llesh gels closer lo the
smbce. compresses against the surliJCe due lo hand pressure.
nnd overlnps the parnllelogrnm more completely. Note that
the rcsolmion of these imng.es is in no way intcndL'<i to limit
the scope of the invention. since certain applications such as
lCJ handwriting rL>cognition will clearly require liner eleetrode
anays than indica!ed by the deetrode size in these sample
imag.es. In the discussion thm li.1llows. the proximity dmn
meusurcd m one el<.>ctrode during. a particular scnn cycle
conslilules one "pixel" of the proximity image caplurL"t.! in
15 !hal scan cycle.
I'IG. D shows a right hnnd llallencd ng.ainstthe surlUcc
with ling.crs olllslretched. At the far ]eli is the oblong. thumb
201 which tends to pointoll"ntabout 120-degrces. The colum-
nar blobs ananged in an arc across the lop of the image are the
"" index !Inger 202. middle !Inger 203. ring !Inger 204 and pinky
linger 205. Flesh from the proximnlling.cr joint. or proximal
phalnng.cs 209. will nppear below cnch lingcnip if the ling.ers
are !i.llly ex!ended. The inner 207 and outer 206 palm heels
cause the pair of \cry large conlaets across the boll om oft he
"5 image. Forepalm calluses 2 I 3 are\ isihle a! the cenler oft he
hand if the palm is li.llly !lnllencd. Tills imag.e shows thm all
the hnnd contncts arc roug.h.ly ovnl-shnpL'<i. but they dill"er in
pressure. size. orienlalion. eccentricity and spacing relative lo
one another. "!.his image includes all of the hand paris which
30 can touch the surlUce from the bon om of one hnnd bm in many
to the sensing electrode 33 of the proximity sensor. This
results in n chang.e in the amount of chnrge stored on the
35
sensing deetrode 33 and thus the presence of the !ouch ohjecl
can he deteeted. The amount of charge stored will depend on
the pressure cxcrtL'<i by the touch object. More pressure
results in more chnrge stored ns indicated in Equntion I.
instnnccs only n l"ew of these parts will be touching. the sur-
lUce. nnd the ling.ertips may ronm widely in relntion to the
palms as !ingers are !lexed and extended.
FIG. I4 shows another extreme in which the hand is par-
tially dosL'<i. The thumb 201 is ndductcd toward the ling.ertips
202-208 nnd the lingers nrc !lexL'<i so the !ing.ertips come
down normnl instead of tnng.cntialto the surlUce. The heig.ht
and intensity of !ingertip conlaets is lessened som<.'.hal
bL>cause the bon<.y tip rather than !leshy pulp pad is aclually
touching. the surlUce. but ling.cnip width rcmnins the snmc.
.\djncent lingcnips 202-205 nnd thumb 201 nrc so dose
"]i.J obtnin a soller touch sur!Ucc on the multi-touch device 40
a thicker and more. cmnplianl dieleetric cover could be usL"t.!.
I lowe\ cr. as the didec!ric thickness increases thedket of the
!ouch de\ ice on the sensing deetrodes 33 spreads out thus
lowering. spatial resolution .. \ complimll mllsotropicnlly-con-
ducting mmerinl can be used to counter this ncg.mive efk>ct
while also providing a soli !ouch surface. I'J(i. I I shows a
cover in which a compliant anisolropically-conduc!ing male-
rial90 is sci between a thin dieiL>cl!ic cover85 and the sensing
electrodes 33. If the conductivity of the compliant mmeria190
is oricmL'<i mostly in the verticnl direction. the imag.c li.xmcd
by a !ouch dL'\ ice on the surli1ce 85 will he lranslated without
signilicanl spreading lo the sensing eleetrodes 33. thus pre-
serving spntial resolution willie providing. n compliant touch
surface.
I'IG. 12 shows a cross section ofn multi-touch surlUce thm
senses both the proximity and pressure of a !ouch de\ ice. The
!ouch layer 85 is a thin dieleetric that separates !ouch devices
from the sensing eiL>ctrodes 33. Proximity sensing is relati\e
to this surfnce. The electrodes 33 nnd associmL'<i switches and
conductors nrc lUbricated on n compliant mmerial 89 which is
allached loa rigid metal base 92. The me!al base 92 is eiL>c-
lrically conneeted 88 lo the rd"erence ground of the proximity
sensor's power supply 34. When a touch device presses on the
touch surlUce 85 it causes the sensing electrodes 33 directly
below to movccloscrto the rig.id mew! base 92. The distance
movL"t.! depends on the pressure applied and thus the pressure
exertL"t.! by a !ouch dL'\ ice can bedetL>c!ed as describL"t.! b d l m ~ .
together as lo he distinguishable only by slight proximity
valleys 210 between them. The proximal phalange !Inger
joints are suspended well above the smbce and do no! appear
45 in the imng.e. nor do the li.xepnlm cnlluses. The palm heels
206. 207 nrc som<.wlwt shorter since only the rcnrofthc pnlm
can !ouch the sur liKe when lingers are !lexed. hut the separa-
tion between them is unchangL"t.!. Notice that the proximity
imag.es nrc undullerL'<i by bnckg.round objects. Unlike optical
'0 imag.es. only conductive obj<.'CIS within a l"ew millimeters of
the surface show up m all.
l'l<i. I5 is a proximity image of a right hand in a pen grip
conliguralion. "l11e thumb 20I and index !ingertip 202 are
pinched together us if they were holding a pen but in this cnsc
'5 they arc touclllng the surlUce instend .. \cnmlly the thumb nnd
index !Inger appear the same here as in I'Ki. I4. llowever. the
middle 203. ring 204. and pinky 205 !ingers are curled under
as if making a !is!. so the knuckles from the lop of the lingers
nctunlly touch the surfnce instcnd of the !ing.er tips. The curl-
60 ing under of the knuckles nctunlly places them behind the
pinched thumb 20I and index linge11ip 202 \cry dose lo the
palm heels 206. 207. "!.he knuckles also appear larger than the
curled lingcnips ofi'IG. 14 but the same size us the !lnnened
lingcnips in I'IG. D. These difl"erenccs in size and arrang.e-
65 men! will be measured by the pen grip detector 17 to distin-
guish this pen grip con!iguralion !i-mnlhedosed and !lallenL"t.!
hand conliguralions.
us 7,812,828 82
19
!'Ki. 16 represents the data !low within theconlacllracking
and identi!icmion module 10. The image seg.menwtion pro-
cess 241 takes the most recently scmmcd proximity image
data 240 and segments it into groups of electrodes 242 cor-
responding to the distinguishable hand parts of I:J(j. 13. '1-he 'i
!ihering and scgmcnlation rules applied in p;u1icu!ar regions
of the image arc partially dctcrmin<..'d by !Ccdback of the
cstimmcd hand offset dma 252. The image scgmcntmion pro-
cess 241 outputs a scl ofdcctmdc group data slruclurcs 242
which arc paramclcriA:d by !iuing an ellipse to the positions lCJ
and proximity measurements of the dcc!rodcs within each
group.
20
It is also comenicnt to maintain ll1r each hand a set of
bitlidd datn registers for which each bit represents touch-
down. cominued contnct or liliolr of n particulnr linger. Bit
positions within cnch bit Held correspond to the hnnd part
indices above. Such registers can quickly he tested with a hit
mask to detcm1inc whether a pat1icular subset of lingers has
touched down .. \ltcrnntivc!y. they cnn be fed into a lookup
tnblc to lind the inplll events associated with n particular
!ingcr chord (combination of lingers). Such linger identity
bitlields arc needed primarily by the synchronitation detector
14 and chord motion r\.><::ogniter 18.
'!-he last process within the tracking and idcntilication sub-
system is the hand position estimator 251. which as described
nbovc provides binsing fe\.'dbnck to the idcmi!ication nnd
segmentation processes. The hand position estimator is
intended to provide a conservati\e guess 252 of lateral hand
position under all conditions including when the hnnd is
Hoating above the surface witholll touching. In this cnsc the
estimate represents n best guess of where the hand will touch
"" down again. When parts of a hand arc touching the surliJCc.
the estimate combines the current position measurements of
currently identi!k-d hand pnrts with past estimates which may
have been mnde from more or less reliable idcnti!icntions.
"111c path trucking process 245 matches up the parmnctcr-
izcd dcc!rodc groups 242 with the predicted continuations of
contact path data stmcturcs 243 cxtrach.:d li'llm previous l'i
images. Such path tracking ensures cominuity of comact
representation across proximity images. "ll1is makes it pos-
sible to measure the velocity of individual hand contacts and
determine when a hand part !ilis olfthc stu bee. disappearing
from li.1turc images. The path tracking process 245 updates
the path positions. velocities. and contact gcomet!)' features
from the parameters of the current groups 242 and pmscs
them on to the contact identilication processes 247 and 248.
!'or notational purposes. groups and unidentilicd paths will he
rcJ'errcd to hy data stmcturc names of the llmn (ji and Pi "-'
respectively. where the indices i arc arbitral)' except for the
null group GO and null path PO. Particular group and path
parameters will he denoted hy subscripts to these structure
names and image scan cycles will he denoted by bracketed
indices. so thnt. for example. P2,. In I represcms the horiton- 30
tal position of path 2 in the current proximity image. nnd P2,.
The simplest but inlCrior method of obtaining a hnnd posi-
tion measurement would he to a\cragc the positions ofallthc
hand's contacts regardless of identity. If hand parts 201-207
were all touching the surbcc as in J'](j_ 13 the resulting
centroid would ben decem estimate. lying somewhere under
the center of the pnlm since the !ingers nnd palm heels typi-
cally llm11 a ring around the center of the palm. llowe\er.
consider when only one hand contact is available ll1r the
nveragc. The estimate would nssumc the hand cemer is at the
position of this lone contact. bm if the contnct is from the right
thumb the hand center would nctunlly be 4-8 em to the right.
or if the contact is lfom a palm heel the hand center is actually
4-0 em higher. or if the lone contact is lfom the middle linger
the hnnd ccmer should actually be nctunlly 4-6 em lower.
1-'IG. 17 shows the detailed steps within the hnnd position
r n-11 represents the position in the previous proximity imnge.
'!-he contact idcntilication system is hierarchically split into a
hand identilication process 247 and within-hand linger and
pnlm idemi!ication process 248. Given n hnnd identification 35
for each contnct. the linger nnd palm identification process
248 utilizes combinntorin! optimizntion nnd Ji.1zzy pnncrn
n.:cognition techniques to identify the part of the hand causing
each surbce contact. !'el.-dback of the cstimak-d hand ollSet
helps identi!)' hnnd contacts when so lCw comacts appear in
the imnge that the overall hnnd stmcturc is not nppnrent.
40
estimator 251. The steps must be repented lix ench hnnd
separately. In a prclCtT\.-d embodiment. the process utilizes the
within-hand contact idcntilications (250) to compute (step
254) for ench contnct an ollSet between the measured contact
position (1-'i,[ n].Fi,.]n ll and the default position of the particu-
'l11c hand idcntilication process 247 utilizes a separate
combinatorial optimi121tion algorithm to lind the assignment
of !ell or right hand identity to sur bee contacts which results
in the most biomechanically consistent within-hnnd idcntili-
cntions. It also receives lCedback of the cstimak'd hand and
linger oflSets 252. primarily ll1r the purpose of temporarily
storing the last measured hand position alkr lingers in a hand
45 lnr linger or pnlm heel with hand part idemity i.
The delimit positions prcJCrably correspond to linger and
palm positions when the hand is in a neutral posture with
!ingcrs partially closed. us when resting on home row of a
kcybonrd. Step 255 avernges the individun! contnct ollSets to
!ill olfthc surface. Then if the lingers soon touch buck down
in the same region they will more likely R><::eive their previous
hnnd idcntilicntions.
,
0
obtnin a measured hnnd ollSet. (II,", In 1.11,.,,-1 n]):
I: 1-i,.,,JII ]{1-i, I II I - 1-i""" I
II,.. ,]11] =
2.: Fi,.._.]ll]
"
L:li,.,,_]Hj{/'i,ln]-li,
1
<:
1
, I
11,.'"]11] = "'--,------
'"
'l11c output of the idcntilication processes 247 and 248 is
the set of contact paths with non-tcro hand and linger indices
attached. For notntionnl purposes identi!k-d paths will be
rcJCrrcd to as FO for the utlidcnti!ied or null linger. Fl lix the '5
thumb 201. 1'2 ll1r the index linger 202. 1'3 ll1r the middle
linger 203. F4 ll1r the ring linger 204. 1'5 ll1r the pinky linger
205. F6 the outer palm heel 206. 1'7 ll1r the inner palm heel
207. and F8 for the forcpnlm calluses 208. To denote a par-
ticu!nr hnnd idemity this notntion can be prc!ix\.'d with an I. 60
ll1r !ell hand or R ll1r right hand. so that. ll1r example. RF2
denotes the right index linger path. When rcJCrring to a par-
ticu!nr hnnd as n whole. ! J I denotes the !eli hand nnd Rll
denotes the right hnnd. In the nctun! nlgorithms !eli hand
identity is represcnt\.'d by n -I nnd right hnnd by +I. so it is
easy to R'\crse the handedness of measurements taken across
Preferably the weighting 1-'i""'" Jn I of ench !ingcr and pnlm
65 heel is approximately its measuR'<i total proximity. i.e .. Fi,.,.,
[ nj l'i .[ nJ. '1-his ensures that !i!led lingers. whose proximity is
tero. have no inlluence on the merage. and that contacts with the vct1ical axis of symmetry.
us 7,812,828 82
21
!ower than nmma! proximity. whose measured positions and
identities are less accurate. have !ow in!lucncc. Furthcmmrc.
if palm h<..-cls arc touching. 1hcir !arg.c tow! proximitics will
dominmc 1hc average. This is bcnc!icial because 1hc palm
h<..-cls. being immobile relative to the hand center compared to
5
the highly Jlcxib!c lingers. supply a more reliable imlicalion
of overall hand position. When a hand is no! touching the
surli1cc. i.e .. when all pmximitics arc /cro. the measured
o!1Sets arc sello /em. This will cause the !ihcn.:d hand posi-
1
,
tion cstimmc below to decay toward the dc!Uult hand position.
L 1-i,., .. ]n]
'
22
{S I
{61
The current contact \docities. (Fi.,[n[.F,.[nj). arc
As long as the contact idcntiHcations nrc correct. this hand
position measurement method eliminates the large errors
caus<..-d by assuming. lone contncts orig.inntc from the center of
the hnnd. Flexing of Engcrs from their delimit positions will
not perturb the measured centroid more than a couple centi-
meters. llowe\er. this scheme is susceptible to contact misi-
dcntilication. which can cause centroid measurement ctTms
15 retrieved Jium the path tracking process 245. which measures
them independent of linger identity. Step 258 updates the
estimated hand oll"scts (!lc"'fn].!lc"'[n]) using. the complete
H!tcr cqumions:
of up to X em if only one hand pari is touching. "!.herclllrc. the ""
current meusurcd offsets nrc not used directly. but nrc aver-
ag.cd with previous oll"sct estimates (!!emf n-! ].! !"",.[ n-! ])
using a simple Erst-order autorcg.ressive H!tcr. forming. cur-
I I" ]Iii ( 1 ]+If,.,""
/11/IVl
I /If,., .!" ]Iii( .. 1 ]+If,.,.
/11/IVl
"'
'"
Fina!!y. to provide a similarly ctmseJYali\e estimate of the
rent offset cstimntes (ll,.",Jn ].II,..,,.] n]).
"" positions of particular lingers step 259 computes indi\idua!
Step 256 adjusts the !i!tcr pole II""[ n] according. to conE-
dcncc in the cutTen! contact idcntilications. Since linger idcn-
tilications accumulate reliability as more parts of the hand
contact the surfiJCc one simple measure of identilication con-
fidence: is the number of lingers which hnvc touched down 30
from the hand since the hnnd lust !ell the surfnce. Contncts
with ln1'6C totnl proximities n!so improve idcntiEcntion rcli-
abi !ity because t!u:y ha\ e strong disambiguating !Cat urcs such
Enger oll"scts (Fi,.,,[n].Fi,.,,..[n]) Ji"om the distnnce between
identified contncts and their corresponding default !ing.er
positions less the estimated hand o!lSets. 1:oreach identi!iable
contact i. the o!lSets are computed as:
n. "' /11/ I (",/11]111 . .,,"/"f+!-i ,/11! -1\1." 1+11-/ (",!11]1
l+h,.jniAt)
('))
as size and orientation. Thcrel(Jre I!,,,[ nj is set roughly pro-
pm1iona! to the maximum linger count plus the sumofcontact 35
proximitics for the hnnd.ll""f n] must oJ"coursc be normn!izcd
n,._./1!/ .
1+1-1,.,./niAt)
These !ing.er oll"sets reHcct deviations of Enger !lcxion nnd
extension from the neutral posture. If the user plnccs the
Engcrs in an extreme config.urmion such us the !lnnened hnnd
conliguration. the co!!ectivc magnitudes of these !inger on:
sets can he used as an indication of user hand site and !inger
to be between zero nnd one or the H!ter will be unstnblc. Thus
when conlidcncc in contact identilications is high. i.e .. when
many parts of the hand Jim1!y touch the surface. the autorc-
grcssi\c Ji!tcr liJ\ors the current o!l"sct measurements. I low-
ever. when only one or two contncts hnvc renppcnred since
hnnd lilioll". the H!tcr cmphnsites previous oll"set cstimmcs in
the hope thnt they were bused upon more relinb!e identi!icn-
lions.
"l11c Jilten.:d o!lSets must also maintain a conservative esti-
mate of hand position while the hand is Jloating above the
surli1cc llll optima! segmentation and idcntilication as the
hnnd touches buck down. If a hand lifts oll" the surface in the
middle of n complex sequence of opermions and must.
quickly touch down ngain. it will probably touch down dose
to where it lined oll". I low ever. if the operation sequence has
ended. the hand is likely to e\entua!!y rctum to the neutral
posture. or delimit position. to rest. rhercl(Jre. while a hand is
not touching the surface. II".,[ n] is made small cnoug.h thm the
estimmed oll"sets g.radua!!y decny to zero at about the same
rnte as a hnnd !nzi!y rctums to default position.
40 lcng.th compnred to the averngc ndult.
The pnrametcrs (ll"",[n].ll"",.[n]) nnd (Fi"",[n].!'icoofn])
llll each hand and linger constitute the estimated hand and
linger ofl"sct data 252. which is !Cd hack to the segmentation
and identi!ication processes during analysis of the next prox-
45 imity imnge. If the other processes ne<.-d the estimate in abso-
lme coordinntes. they can simply add (step 260 ) the supplied
oll"sets to the delimit linger positions. but in many cases the
relative oll"set representation is actua!!y more convenient.
It should be dear to those skilled in the arl that many
'0 improvements cnn be made to the nbovc hnnd position esti-
mation procedure which remain well within the scope ofthis
invention. espccia!!y in the mannerofguessing the position of
lilkd hands. One improvement is to make the estimat<.-d hand
oll"sets dccny townrd zero m n constnnt speed when n hnnd is
'5 lilk-d rmhcr than decny cxponentia!!y. Also. the ofl"sct com-
putations ll1r each hand have been independent as describ<.-d
so br. It is actua!!y advantageous to impose a minimum
horimnta! separation between the estimated !eli hand posi-
tion and estimmed right hnnd position such thnt when n hnnd
When !l.,"[n] is made smnl! due to low identi!icmion con-
Jidence. the Ji!tcr tracking delay becomes large enough to lag
behind a pair of quickly moving lingers by several centime-
ters. The purpose oft he H!tcr is to rcnct slowly to questionable
chnng.cs in contnct identity. not to smooth contact motion.
This motion tracking. ddny can be safely diminmed by add-
ing the contact motion measured between images to the old 65
o!lSet estimate. Step 257 obtains motion from the a\erage.
60 such as the right hnnd slides to the opposite side of the bonrd
while the other hand is !ilk-d. the estimatL-d position of the
other hand is displacL-d. In this case the estimatL-d position of
the tilled !ell hnnd would be forced from deJUult to the JUr !eft
of the surJUcc. possibly off the surJUce completely. If the rig.ht
hand is tilled and the !eli is not. an equation like the Ji.1llowing
can be app!iL-d to l(Jrce theestimatL-d right hand position out of
(I 1.,, ,[ nj.l 1,...,.[ nJ) of the current contact \docities: the way:
us 7,812,828 82
23
Rh, c"l n]: min!R/ f. c"(11].i U"1,,.
1
; -RFI "''' i+Ut, ",{11]+
min hard ><:pi
24
the strict segmentation region 282. where proximity saddle
points must serve as contact boundaries. As n preferred
embodiment the sloppy regions arc r<.><:Wng.ulnr. their inner
boundaries 285 nrc placed just inside of the columns where
where (!.l:t,.,
1
,-RFl,.
4
J is the debuh separation between
!eli and rig.ht thumbs. is the minimum horizontal sepmation to
be impos<..'d. and 1.1 !emf nl is the current estimntedoffsct of the
!eli hand.
5 the index lingers 202 areexpL>cted to lie. and the upper bound-
aries 287 are placed at the estimatL-d \erticallevds of their
respective thumbs 201. The olller and lower boundnries of the
sloppy regions nrc dctennincd by the omside edg.es of the
!'Ki. 18 represents the data Jlow within the proximity
image scgmcnlation process 241 Step 262 makes a spatially
smoothed copy 263 of the current proximity image 240 by
1
,
pnssing a two-dimensional dilrusion opcrntor or Gaussian
kernel over it. Step 264 scnrchcs the smoothed image 26J Jix
local maximum pixds 265 whose Jihcrcd proximity cxcL-.._'<fs
a signilicancc threshold and cxcL-.._'<fs the !ihcn.:d proximitics
ofncnrcst neighbor pixds. The smoothing reduces the chance
15
!hnt an isolated noise spike on a sing.!c electrode will result in
a local maximum-which exceeds the sig.niHcancc threshold.
and consolidates local maxima to about one per distinguish-
able !leshy contact.
Process 268 then constmcts a group of ekctrodes or pixels ""
which reg.istcr sig.niHcant proximity nround cnch local maxi-
mum pixel by scnrching. outward from each local maximum
l(Jr contact edges. I Oach electrode cncountcn.:d bei(Jrc reach-
ing a contact boundary is added to the local maximum's
group. 1-'lG. 19 shows the busic bound<!!)' electrode search "
5
pnttem for an exmnple contnct boundary 274. In this ding.rnm.
an electrode or image pixel lies at the tip of each arrow. The
search starts at the local maximum pixel 276. proceeds to the
!eli pixels 277 until the boundary 274 is detected. The last
pixel before the bound.11)' 278 is mnrk<.'d us nn <.'dg.c pixel. and
30
the search resumes to the rig.ht 279 of the local maximum
pixel 276. Once the !eli and right edges of the local maxi-
mum's row ha\e been l(mnd. the search n.:curses to the rows
above nnd below. nlways stnrting. 281 in the column of the
pixel in the previous row which had the g.rcntest proximity .\s 35
the exnmple illustrates. the resulting. set of pixels or ekc-
trodes is connected in the mathematical sense hut need not be
n.:ctangubr. This allows groups to closely lit the typical oval-
shnpc of !lcsh contacts without lenving. electrodes olll or
including. those from ndjncent contacts.
If contacts were small and nlways well separnted. edges
could simply be established where\er proximity readings ldl
sur bee. I )ue to thedL>cay in estimated handollSets alkr hands
lea\e the surl'iJCe. the sloppy segmentation regions return to
the positions shown allerthe hands havestayedoffthe surbce
n few seconds. regardless ofhnnd position m liftoff. I-'IG. 20B
shows how the sloppy rcg.ions follow the estimated hnnd
positions 252 as the right hand mmes toward the upper Jell
and the Jell hand mmes toward the lower Jell. This ensures
that the palms and only the pnlms lUll in the sloppy regions as
long. us the hnnd position estimmcs nrc correct.
1-'lG. 20C shows thnt the ]eli sloppy region 284 is moved
Jell olrthe surbce entirely when the !eli hand is lilkd olfthe
sur bee and the right hand slides to the Jell sideofthe surl'i1ce.
This prevents the Hng.crs of one hand li"om entering. the sloppy
segmcntmion region of the opposite hnnd. This efl<..><:t is
implemented by imposing a minimum horizontal separation
between the sloppy regions and. should the regions get too
dose to one another. letting the hand with the most surbce
contncts override the estimnted position of the hnnd with
lCwer contncts. ]-'](j 21 is n detniled How chnrt of the edge
tests which are applied at each searched electrode depending
on whether the electrode is in a strict or sloppy segmentation
rcg.ion. D<.>cision diamond 290 checks whether the
unsmooth<.-d proximity of the ek>ctrode is greater than the
background proximity levels. If not. the electrode is labeled
an edge electrode in step 304 regardless of the segmentation
region or search direction. and in step 305 the search returns
to the row mnximum to recursc in nnother direction. If the
unsmooth<.'d proximity is sig.niHcnnt !Urther tests arc npplied
to the smoothed proximity of neig.hboring. electrodes depend-
ing on whether decision diamond 292 dL><:ides the search
electrode is in a sloppy or strict region.
If a strict reg. ion search is advnncing. horizontally within n
row. d<.>cision diamond 306 passes to d<.>cision diamond 308
which tests whether the electrode lies in a hori/Ontal or diago-
nal partial minimum with respL>ct to its nearest neighbor eiL>c-
trodes. If so. a proximity valley between adjacent lingers has
to the background level. But sometimes lingertips are only
scparnted by a slig.ht vnllcy or shnllow snddlc point 210. To
segment ndjncent Hng.crtips the partial minima of these vnl-
leys must be detected nnd used us group boundaries. Lnrg.e
palm heel contacts. on the other hand. may exhibit p;u1ial
minima due to minor nommil\m11ities in llesh proximity
across the contact. I fall electrodes under the contnct arc to be
collected in a sing.le group. such pnrtinl minimn must be
ignored. (iiven a hand position estimate the segmentation
system c;m apply strict edge detection mles in regions of the
image where lingertips and thumb are expected to appear but
apply sloppy edge detection rules in regions of the imag.e
where palms arc expected to. nppcnr. This ensures thnt adjn-
cent lingertips are not joined into a single group and that each
palm heel is not broken into multiple groups.
45 probably b<.-cn detected. the electrode is label<.'d as nn edge
314 nnd senrch resumes in other dir<.><:tions 305. If not. the
search continues on the next ek>ctrode in the row 302. If a
strict region search is ad\ancing \erlically to the next row.
decision diamond 306 pmscs to d<.>cision dinmond 310 which
Step 266 ofl:l(;. IS ddines the positions of these segmen-
wtion rcg.ions using. the hnnd position estimates 252 derived
from ann lyses of previous imng.cs. 1-'lG. 20..\ shows the extent
-;o tests whether the ek>ctrode lies in a vertical partial minimum
with respect to the smoothed proximity ofits nearest neighbor
electrodes. If so. a proximity valley between a linger and the
thumb has probably been detectL'<i. the electrode is labeled as
nn edg.e 312and senrch resumes in other directions 305.1fnot.
'5 the scnrch continues into the next row 302. If decision dia-
mond 294 determines that a sloppy region search is ad\ancing
hori/Ontally within a row. stringent horizontal minimum tests
are peri(Hmed to chL>ck l(Jr the crease or proximity valley
between the inner nnd olller pnlm heels. To qunlil). the elcc-
60 trodc must be more thnn about 2 em horizontnl diswnce from
of the strict and sloppy segmentation regions while the hands
are in their delimit positions. making estimated oi1Sets I(Jr
both hands zero. Plus signs in the dingrmn 252 indicmc the
estimmed position of ench !ing.cr nnd palm heel in ench hnnd.
Rcctnng.ular outlines in the lower corners represent the ]eli 65
284 and right 286 sloppy segmentation regions where partial
minima are largely ignored. '1-he r-shaped region remaining is
the originating local maximum. as chL>cked by dL>cision dia-
mond 296. Also the ek>ctrode must he part of a tall \alley or
partial horizontnl minimum which extends to the rows nbovc
nnd below nnd the next-nearest neighbors within the row. as
checked by d<.>cision diamond 298. If so. the electrode is
lahek-d as an edge 300 and search recurses in other directions
305. All other partial minima within the sloppy regions are
us 7,812,828 82
25
igmm::d, so the search continues 302 until a background k:vd
edge is reached on an upcoming dcclrmk:.
In sloppy scgmcnlalion regions it is possible l(Jr groups to
overlap sig.n.iHcantly because partial minima between local
5
maxima do not act as boundmics. Typically when this hap-
pens the overlapping groups arc pan of a lal'olC Hcshy contact
such us a p.1lm which. even after smoothing. has multiple
local maxima. Two groups arc ddincd to he mcrlapping if the
search originating local maximum ck:clrmk: of one group is lCJ
(;_
26
I
{111
- ,,,_
{13)
{14)
also an dement of the other group. In the in !crest of pn::scnl-
ing. only one group per distinguishable !lcshy contact to the
rest oft he system. step 270 of FIG. 18 combines overlapping
groups into single supcl'olroups before parameter extraction.
Those skilled in the nn will rcnlizc thm fccdbnck from high-
!e\el analysis of previous images can be applied in various
ahernali\ e ways lo imprm e the segmentation process and sli!!
No!e that since the Iota! group proximity (j integrales
proximity over each pixel in the g.roup. it depends upon both
of the size ofn hand pari. since !arg.c hand puns tend to cnusc
15
groups with more pixels. and of the proximity to or pressure
on the stu bee of a hand p;u1.
""
lie well within the scope of this invention. For example.
additional image smoothing in sloppy segmentation regions
could consolidate cnch palm heel comact into n sing.le !ocn!
mnximum which would pms strict scg.mcntmion region
boundal)' tests. Care must be tnken with this nppronch how-
ever. because too much smoothing cnn cnuse Enger pairs
which unexpec!ed!y enter sloppy palm regions lobe joined
15
into one group. Once a !Inger pair is joined the !Inger idenli-
!icalion process 248 has no way lo tell that the !ingerlips are
actually not a sing.le palm heeL so the Enger identi!icmion
process will be unnble to correct the hand position estimme or
30
adjust the sloppy regions Jix proper segmentation of li.nure
images.
More detnikd forms of fe<.'dbnck thnn the hnnd position
eslimale can be utilized as welL For example. the proxima!
pha!anges(209 in FJ(j_ 13) are aclua!ly pan of the !Inger but
35
tend to be scg.mcmed into scpnratc groups thnn the Hngcnips
by the vertical minimum test 310. The venicn! minimum test
is nccessal)' to sepmatc the thumb g.roup from index !ing.ertip
group in the p;u1ia!!y dosed FJ(j_ 14 and pen grip FJ(j_ 15
"
hand nmliguralions. llowL"\er. the proximal phalanges of
!lallened lingers can he distinguished Ji-om a thumb behind a
curled !ingerlip by the !Uct thnt it is very dillicuh to !Inn en one
long Hng.er without Hnncning. the other long. lingers. '!!.1 tnke.
advnntnge of this constraint. n Hnncned Enger Hag. 267 is set
45
whenever two or more of the conlacls identilied as index
Since most groups are convex. their shape is well approxi-
mated by ellipse pnrametcrs. The ellipse Jilting procedure
requires n unital)' transformntion of the group covnrinncc
matrix (jco of second mmnenls Q,,. Q,.. Ci,....
U, = l'
'<-r.
U, = U" = I: -<', IIG,- <".)
, .. ,,,_
u" =I,. 1U, _,.,J'
'<-r.
The eigenvalues f ... nnd f..
1
of the covnrinncc matrix
de!ermine the ellipse axis lengths and orientation (j
0
:
where G
0
is uniquely wrnpped into the runge (0.180).
(15)
(16)
(17)
() '!)
ICII)
through pinky in previous images are larger than norma!.
reliably indicating that linge11ips are llallening. rhen decision
dinmond 310 is modiHed during processing of the currem
image to ignore the !irst verlicnl minimum encoumcred dur-
ing. senrch of rows below the orig.inming.locn! minimum 276.
This allows the proximal phalanges lo he included in the
lingerlip group but prevents lingerlip groups Ji-um merging
with thumbs or The !as! step 272 of the segmen-
1-'or convenience while distinguishing !ingerlips from
,
0
palms a! higher system !e\els. the major and minor axis
lengths are comerled \ ia their ratio into an eccenlricity Ci,
(;, = (;,_.,,0,
(;,,.,,,0,
tat ion process is to cxtrnct shnpc. size. nnd position pnram- '
5
etcrs from each electrode g.roup. (jroup position rc!lects hand
No!e that since the major axis length is always greater than
or equal lo the minor axis length. the eccenlricity will always
be g.renter thnn or L'<.jtml to one. Finally. the total group prox-
60 imity is cmpiricn!!y renormnlizcd so thm the typicnl curled
linge11ip will have a total proximity around one:
comnct position and is JK>cessnl)' to determine !ing.er velocity
The total group proximity. eccenlricity. and orientation are
used by higher k-,.el modules lo help distinguish linger. palm.
and thumb conlacls.
Provid<.'d (jr: is the set of ek>ctrodes in g.roup G. c __ is the
unsmoothed proximity of an electrode or pixel c. nndc, nnd c,.
arc the coordinates on the sur!Ucc of the electrode center in
cenlimelers. lo gi\e a basic indicator of group position. the 65
proximity-weigh!ed center. or centroid. is compu!L'd liom
positions and proximities oft he group's dec!rodes:
(; __
c. ICJJ
u;,t]'>\ smn p,tmou m't]l J;o_'>\O] s;o!l!lU!XOJd Jl'IOI l]ll'>\ sdmuTI
JOJ p;)\a>.;u:l;Jp ;)]Od J;)WJ ;)l]J -d .\Jtm!xmd
[lllO) l[ll.\\ .([qBJ;:Jp!SU<Xl UO!JISOd
JO .\J![!qm[;:JJ ;:Jt[l JJU!S (' ',]' \J) JO)J;:JA .\Jpop,\ Jtjl tuOJJ ""d
;:Jjiltm pur, C""'bd JPill!Uillltu Jtjl Ol[llj;)Sil m]r l[
Jl[l ptm "J;oqJmm ;,uo 01 J;J_JJJ ,,,_,._,_...,l'ld ptm
"';)"! "J;)lj)OUl' .')UO OJ )S;)SOp ;)m .\aq1 _II J.')lj)llUI' ;)UO ljll'>\
pJJWd .\juo ;:JJB ld t[Wd pur, dnOJil ;:JA!)JB tm "9Zf dJ!S U[
IOU
\'d ld lfllll C lS1)>1)lS.'S0[.> IJ
((>j>) IV I]J-!1/"d-/u/"dl 'd
I JVill-11/\J-/il/\!1 [11/ \/
ILt>l IViiJ-1</\t-/!1/'dl
19!>1 ill-11/ dli"D-!1+ ':)'':) /11/\1
i>t>l i][-!IJ",J)("<)-)1+
0
</'<) /!IJ"d
lt-t>l ill -!1/"
1
"'' "
1
dli "<)-) )+" <)'') d
l[j>l I[ !-il/'
1
""
1
dli"D-ll+ ':/'0) /II/\/
l<:t>l ilt-"r'"""dli";-) )+ '/'') 1"r"
:u
JUH) 01][-U Jd l[lL'd ;:lAIPBJO UO!WilU!lUOJ ll !) dmuil Jl JS];)
Ill> I IV, .D /11/ \/
lnt>l n juj '.!
l(>tl " ju]"d
\) juj \1
ILf\
0
!) ju]"d
19U 'l [uj d
l>fl
!) juj d
lt-U
'<) ju]\1
PJ/!]r,[J!U!
JJB "U.\\llp Ulld pUBl[ ll "';)'!
u dJJS ;:Jtulllll !) dnOJil ,\q Slltj d l[llld rj[
(()<:\ \ d ld ild'1') l<: l'!l 1') OJ!'" ;l Ll' ')I'"'' ld (>
09 :1! 01 ;,t]l ptm dmuTI J.\!Pl'
Jt[l "ld tjlBd ;:JA!)JB tpBJ UJt[l PZf dJJS
/
1
""
1
/d- 'D1+,1""'"
1
M-'DI 1/,f'YO)I,o
:Jupm
pJJndmoJ .\]!9'!.') tm 1 p.').1m1hs ;,tp J.J;ot]'>\
,,
([<:\ !l1'l ild'1'J)<: I' \'d ld """ <: l'!l'd'''""l"
19<:1 U.U.U.i.
:11 OJ J:JlmJS!P Jl[l spJmJJ ptm l]Wd J.\!Pl'
;,t]l "]!) dnOJ'i! tpt'J ZZf "!) ;oq J'i!tml!
JUJJJilJ ;:Jt]l m sdno.ril );:JS Jt[l lJ] pur,
'\'d Jillltu! SllO!A;:JJd Jl[l Ul JAIPB JO ;:ll[l ilU!U;:J']
1,:<:)
ll-"r'"'v+/J-!1/'\t '"'''""'"
I!>CI II-11/''dJV+/I-U/\/ /11/'''""d
.UJl[l
Oi' Sl UOI]lmU!JUO:l l(ll'd p.')JJI(h')Jd-.(J!:lO(;o\ Jl(l .(J!:lO(;o\ li.Wlll"]
;otpj 1 -u ]" 'd ']]-U l"d ptm ]-U JlU!l tulU.J d l(ll'd .JO
;:ltJl Jq l!-ul\rl!-ul\1 ilmnJJ uopr-J;:J[JJJB
f!U!Jilp pr,q ,\[[B!HWJOd ;:Jtj) t[!l!J.\\)!10 UO!)J!pJJd
pJSBq-.\J!JO[J,\ JO S)!J;:lUJq Jl[l UOI)BJJ[;:JJJB OJ;:JZ Ulll[l
,,- Jll:J:Jll S)U;o\.') UOI)l'JJ(:lJJl' t]TI!l( J:JUIS 'UOI):l.')J!p
- SJ'i!tmtp JO JO SUI'IS .\jUJppns JJ'i!U!J l' u;,tj'>\ )d;oJXJ [[(l!l
Jl[l SJMlJdtu! UO!)BJO] ;:Jtj) 01 .(JpO];),\ SilO
-[,\JJd flm,\jdd\" 'SJilBUH Ul<.UJ S;:li)IJO(;:lA
ptm lj)l'd 'i!UIJS!XJ 'i!UOJl' SlJI'JUO:l
OC .JO SUO!ll,JO]JUJ.llll:l ;,tp SP!P.')Jd ()Zf JS.I!J JllJ.
Jq p]llll.\\ SJJI]!J
JS]nduu JJ!U!J Jo tmmp')l Jnq dmuTI TimpuodsJJ
-Jm tuOJJ SJJptm!md (ju] 'd ju]'\!) Jdm.JS ptm (lul"\rlu]',!) __
.\Jpop,\ -ou ld]u J\rlu 1\J) tprd ;:ltJl ;)JL>pdn 01 SJJl "-
-m J[dUHS .([ddr, .\\ll]Jq U.\\Ol]S SUO!)B!lbJ Jl[.L
.(J!lUIXOJd ptm p.');,ds 'i!UI)SIXJ UO )U;opu;odJp
UBJ -Y"""d SlllpBJ iluppBJl S, lj)lld l(Jll;:l J;:Jf!U!J
l[ilp:[)ll Stj)lldJOf!UI'jllJJq J;:JdOJdtu! lUJAJJd Oj_ "SillpBJ fltH'jJBJl
S)l U!l]ll.\\ dno.ril ll lllOl[ll.\\ l[llld ;:lA!)Jll liB ;),\ll;:ljlJI.\\jjO\JH pur,
"Sl[ll'd 'i!UIJS!XJ .JO ttpt'J TIU!"]Jl!IJ ;ot]l .mJddi' OJ dnOJ'i!
_\\.')U 1' .\jpmsn U.'>\optplllll Wl[l PIOJ ;ot]l .\q PJJ:l.')pp
,\[!SBJ JJB .[jOIJI[ pur, U.\\Opl[Jilll) SJ)BJ UBJS ;:J[qB
Tiu!JJI]!J l'!\ l[ll'd OJU! dmuTI
l[:ll'J _)0 S.IJ)JlUIO.md p.')JJIOJ]XJ Jl[l SJ]l!!Od.HDU! 9J'f d;,JS
o<:
Jt[l tuOJJ _[IOIJIJ )JBd pum[ st dmuil
-UOSBJJ JOJ SlllpBJ ilU!"]JBJ) Jl[l Ulltjl J;:J)ll;:JJ<) U;)..'),\\)Jq
[[(l!JIOJl,dJS jlOJ;,lq Jl':l!d.(] Jl(l J:JUIS J;otj]lllm JUO _)0 1\u!
"]Jl,J) ;,tjl UWlU"] JJlmJS!P l' U!lJI!-'>\;oJJ' .\al(] p.')tpll'lU
;,q JllU pj110l(S t]Wd pm' dmuTI 1' sJTitmu U:l.')'>\Plj
r, t[ll.\\ pJJ!L>d os Jq wuur-J tp!t].\\ t]lBd J,\I)JB .\tm ppf
Ul ;otjl O)UO JJ'i!U!J '>\;oU I' JO UWlptpllll) 1\U!JU;oSJJdJJ
"t]Wd . .,.;,u 1' p.')Jl,:lO[[l' t]Wd p.')Jl,.\IPl'JP K[JUJ:l.')J Jo ;o\JPI'
m' ljll'>\ pJJmd :ll[ ]llU ]]liS Sl"J'P!l]'>\ dno.Rl .\tm "9ff Ul
p,WJI UBJ JJf!U!J ll Jllj .\\Olj J!lU![ UO!)BJJ[;:JJJB pur, .()IJO[;:!A
JJilU!J [BJ;:J)Bj UO S)U!BJ)SUOJ jBJ!Ullt[JJlUOiq "OS]\" "Slj)Bd pur,
sdmu'i!J;ot]JO OJtmt]l J;ot]Jmm ]]!'>\ Pl'JUOJ;oums
,_-
1
;,t]l tull.l_) 'ilms!Jl' t]Wd ptm dmuTI 1' "J(ll.l(l!JJu;oTI 1' Pl'l
-uoJ ilm!S!XJ tp!t[.\\ t]ll.\\ Jq sdmuil HWJJilJ
l[Jil[.\\ JP!:l.')p ISiltu f!tH'jJBJI ;:ll[l .(J!ltHXOJd
;:ll[l pUBl[ l[Jll;:l ;:JJJl[.\\ JU!UlJ;:J)Jp OJ.
'.[IOIJI]
J;:JilU!J 01 Jllp OS JO pUOJ;:JS )Sll[ ;:ll[l U!l])!-\\ pJ)\l,\I)JBJp JJ;)_\\
tptt]'>\ ptm ;o\oqt' pJJ!t'dun l,P] an'>\ tptl(.'>\ .\jUO
n:tZZf sdJJS Tiu!ll';odJJ .\q Pff S:l'l OLdSfl 1'!.\
U! PJt[S!jdtuOJJB S! SllLJ. 'UOI)BJO[ ;)UJBS ;:l[jl J;)i\0 U;);:l_\\);:lq f)j
l]llld JO .()!nU!lUOJ 0) Jq .(Btu
'Pl'J[[(JJ ptmq ;ot]l OJ tp!l(-'>\ SJ'i!tml!
.\]ilUIXlUd lUlU_) sdmu'i! JSlll[l J;otjj;,1\0]
l[Jil[.\\ f!tH'jJBJl l]llld ;:Jtj)JO Jtjl ZZ
l! "JJilU!J JlUBS ;:Jtj) JO J,\I)IJ;:Jd;:JJ JO UOIPJ);:Jp Ul pm OJ.
(\_f) '"''""'[d O-'"'f")"""l
0
[d
ICU


IIU

:pjOl[)Siltu SUO!J!pUOJ
;,tp _10 llV smpt'J 'ilmpJoJJ ;ot]l tmqJ tuJl[l u;,a_'>\pq
8<:
S!Xll JOU!Ul pur,
Jofr,m JO PBlUOJ JO ,\Jrmud Jt[l sr,
Sl !) .\J!lU!XO.Id dmuTI Jl'IOJ ptm "SJlljlO\ JIJl(l lml(J
,_- JJljll'J l]l110PP OJ ps JJl' SlJl'JUO:l .\JD!.IJU;oJJ.')
pur, UO!)ll)U;:JUO Jl[l 'UO!lll[OSJJ .\\llj J,\llt[ .(J!ltHXOJd
J! "JJO,PJ;:Jt]J_ JSd![p p;:!ll!J ;:Jtj) Ulltjl J;:Jf!U!J
Sll [[;:!.\\ ;:JZ!S IJlllUOJ JO JOIBJ!PUI ;:JjqB![JJ ;:JJOtu ll .\l!lU!
-xmd dmuTI [l'IOI ;,tp s.\1o.cm JplUJ:l.')(J mltlllJOSaJ .'>\O( uo
a
Z8 8Z8'Z!8'L sn
us 7,812,828 82
29
signals arc weak. the system relies heavily on the pre\ iously
established path velocity. bllt when the Enger Hrmly touches
the sur!Ucc causing a strong. reliable sig.nal. the system relics
entirely on the current g.roup ccmroid measurement.
'!lu:: next process within the !racking module is conlacl 'i
idcntilication. On surliJCcs large enough l(Jr multiple hands.
the contacts of each hand tend to form a circular duster. and
the clusters tend to remain separate b<..><::ausc users like to avoid
entangling the !ingcrs of opposilc hands. lkx:ausc the
arrangement of!ingcrs within a hand duster is independent of lCJ
the location ofaml arrangement within the other hand's dus-
ter. the contact idcnti!icmion system is hierarchically split.
The hand idcnti!icntion process 247 !irst d<..>cidcs to which
cluster each contact belongs. '!lu::n a within-cluster identili-
cation process 248 ana!y tes ll1r each hand the ammgcmcnt of l'i
contacts within the hand's duster. independent of the other
hand's cluster. Because within-cluster or !ingcr identification
works the same lix each hand regardless of how many hands
can lit on thesurli1cc. it will he described lirst. t-he description
below is ll1r idcntilication within the right hand. Mirror sym- ""
met!)' must be applied to some parameters before identifying
!eli hand contacts.
30
hand's estimated position ollSet. The linal attractor positions
(.\j,ln ] .. \j,.J n]) arc therefore given by:
1511
In alternative embodiments the attractor ring can also be
rotat<..-d or scakd by estimates of hand rotation and size such
as the estimat<..-d linger oflScts. but care must be taken that
wrong !ing.cr ollSct estimates and idcnti!ication errors do not
reinforce one another by severely warping. the attractor ring..
Once the amactor tcmplmc is in place. step 354 constructs
a square matrix Jd,,J of the distances in the sur liKe plane from
each active contact path Pi to each attractor point A_i. If there
arc lCwcr surlUcc contacts than attractors. the null path PO.
which has tcro distance to each attractor. takes place of the
missing contacts. '!-hough any distance metric can be used. the
squared I ;uc!idean distance.
is prcJCrrcd because it sp<..>cially favors assignments wherein
the ang.!c between any pair of contacts is dose to the ang.!c
between the pair ofattractors assigned to those contacts. This
corresponds to the hiomechanica! constraint that lingertips
!'Ki. 23 shows the prekrred embodiment of the linger
idcntilication process 248. For the contacts assigned to each
hand this embodiment attempts to match contacts to a tem-
plntc of hand part attractor points. each attractor point having
an identity which corresponds to a panicu!ar !ingcr or palm
heeL This matching between contact paths and attractors
should he basically one to one but in the case that some hand
parts arc not touching the surlUcc. some attractors will be !eli
un!i!lcd. i.e .. assigned to the null path or dummy paths.
"" avoid crossing over one another. especially while touching. a
surlUcc.
In step 356. the distances from each contact to selected
attractors are weighted according to whether the geometrical
1\::atures of the given contact match those expected from the
Step 350 initialitcs the locmions of the attractor points to
the approximate positions of the cmTesponding lingers and
palms when the hand is in a neutral posture with lingers
partially curkd. Preferably these arc the same dcJUu!t !ing.cr
locations employed in hand ollSct estimation.
Setting. the distances and ang.!cs between attractorpoints from
30 hand part thm the attractor represents. Since the thumb and
palm heels exhibit the most disting.uishing g.comctrica! JCa-
tures. weighting functions are computed ll1r the thumb and
palm heel attractors. and distances to lingertip attractors are
unchang<..-d. In a prell::JT<..xl embodiment. each weighting func-
a ha!J:closed hand posture allows the matching algorithm to
perllm11 we!! ll1r a wide variety of linger Jlexions and ex ten-
35 lion is composed of several lUctorvcrsus !Caturc rclntionships
such as those plotted approximately in FIG. 25. J:ach lUctor is
design<..xl to take on a delimit value of I when its feature
measurement prm ides no distinguishing inllmnation. take on
larger values if the measured contact JCaturc uniquely
sions. 40 resembles the g.ivcn thumb or palm hand pan. and take on
smaller values if the measured !Cnturc is inconsistent with the
"111c resulting attractor points tend to lie in a ring as dis-
played by the crosses in FJ(i. 24. The identities of attractor
points 371-377 correspond to the identities of hand parts
201-207. If the gi\en hand is a Jell hand. the attractor ring
must be mirrored about the vertical axis from thnt shown. 45
given attractor's hand part. The liJCtor relationships can he
variously stor<..xl and computed as lookup tables. piL>c<..wise
linear Ji.mctions. polynomials. trigonometric functions. ratio-
nal li.mctions. or any combination of these. Since assig.nmcnt
between a contact and an attractor whose JCaturcs match is
bvor<..xl as the weight<..xl distance between becomes smaller.
the distances are actually weight<..xl (multipli<..xl) with the
reciprocals of the factor relationships shown.
]-'](j. 25.-\ shows the rig.ht thumb and right inner palm heel
orientation li1ctor \ersus orientation of a contact's litt<..xl
ellipse. Orientation of these hand parts tends to he about 120".
whereas lingertip and outer palm heel contacts are usually
vel)' dose to vertical (90). and orientation of the ]eli thumb
1-'lG. 24 also includes !inc segments 380 forming the Voronoi
eel! around each attractor point. I 0\ery point within an at trac-
tor's \(mmoi eel! is closer to that attractor than any other
attractor. When there is only one contact in the duster and its
features arc not distinguishing. the assig.nmcnt algoritlun '0
clk>ctivcly assigns the contact to the attractor point of the
\lmmoi eel! which the contact lies within. When there are
nmltip!e surbcecontacts in a hand duster. they could a!! lie in
the same \ixonoi cell. so the assigmncnt alg.orithm must
perform a global optimization which takes into account all of
the contact positions at once.
-;5 and !ell inner p.1!m heel avcrag.cs 60 The right orientation
bctor therd(m:: approaches a maximum at 120" It
approaches the delimit \alue of I at 0". 90". and I XO" where
orientation is inconclusive of identity. and reaches a mini-
Altemati\e emhmliments c;m include additional attractors
l\1r other hand part or altemati\e attractor arrangements ll1r
atypical hand con!igurations. I-' or example. attractors for fore-
palm contacts can be placed at the center of the ring.. but since 60
the ll1repalms typically do not touch thesurli1ce unless the rest
of the hand is Jlattened onto the surli1ce as we!!. ll1repalm
attractors should be wcig.ht<..-d such that contacts arc assigned
to them only when no regular attractors arc !eli unassig.ncd.
1-'or optima! matching. accuracy the ring should be kept 65
roughly centered on the hand duster. Therdl1re step 352
translates a!! of the attractor points l\1r a gi\en hand by the
mum at 60. the lUvorcd oricntmion ofthc opposite thumb or
palm heel. The corresponding relationship lix the !eft thumb
and inner palm heel orientation li1ctor is llipped about 00"
l'l<i. approximately plots the thumb site li1ctor Since
thumb size as indicated by total proximity tends to peak at two
or three times the size of the typical curled fingertip. the
thumb size factor peaks at these sizes. Unlike palm heels.
thumb contacts can not he much larger than two or three times
the delimit lingertip site. so the thumb bctordrops back down
us 7,812,828 82
31
l(Jr larger sizes. Since any hand p;u1 can appear small when
touching the sur!Ucc very lightly or just starting to touch-
down. small size is not distinguishing. so the site factor
defaults to 1 for very small contacts.
I'Ki. 25(' approximately plots the palm heel bet or. As
more pressure is applied lo the palms. the palm heel conlacls
can grow quite !arg.c. remaining !Uirly round as they do so.
Thus the p.1lm heel size factor is much like the thumb size
factor except the palm !Uctor is free to incrcnsc inddinitdy
32
ited cmnhinaloria! search. the llungarian method, or network
How solvers. Those ski!!cd in the art wi!! recognize that tills
type ofcombinmorial optimizmion problem has a mmhcmmi-
ca!!y cquivnlent dual representation in which the optimizntion
5 is rd(mJm!ated as a maximitalion of a sum of dual param-
eters. Such rd(mnulation of the ahme hand pari identilica-
tion method as the dual of nllrnctor-contnct distance minimi-
zmion remains well within the scope of this invention.
I lowe\ cr. Jingcrtip conlacls can grow by becoming taller as lCJ
the !ingcrsarc since !Inger width is constant. the
eccentricity of an ellipse !ittcd to a growing !ingcnip contact
increases in proportion to the height. To prevent Hattcncd
!ingcrs lium ha\ ing a large palm bel or, has !ink: dfecl !(Jr
palms. whose eccenlricily remains near I. hut cancels the 15
high proximities ofllallem:d !ingerlips. rhoughdin.x:t!y using
!ittcd ellipse width would be less accurmc l\1r !ow rcsollllion
electrode arrays. the nbove rmio bnsicn!!y cnpturcs comnct
width.
'](J nvoid unncccssa!)' complllation. decision dimnond J60
ends the linger identi!icalion process a! this stage if the hand
assignment of the given conlacl duster is only a lenlali\e
hypothesis being cvn!untcd by the hand idcntilicmion module
247. llowcver. if the given hand assignments nrc the !ina]
pre!CJT<.-d hypothesis, li1rther processes veriFy !Inger identi-
ties and compile identity slalistics such as !Inger counts.
The identilicalions produc<.-d by this anraclor assignment
method arc highly rdinble when all live lingers arc touching
the surface or when thumb and p.1!m lCmurcs arc unmnbigu-
ous. ChL><::king !hal the horimn!a! coordinates l(Jr identiliL-d
Another impm1anl distinguishing kalureofthe palm heels
is that wrist anatomy keeps the centroids of their conlacls
scparmcd lfom one other and from the lingers by severn!
centimeters. This is not true of the thumb and lingertips.
which can be mmed within a centimeter of one another via
Jlexih!c joints. The inter-palm separation kalure is measured
by scnrehing l\1r the nearest neighbor com net of a given con-
wet and measuring the distnncc to the neighbor .. \s plollcd
approximmdy in 1-'lG. 25D. the pn!m scparmion factor
quickly decreases as the separation between the conlacl and
its nearest neighbor b!!s below a JL-v.- centimeters. indicating
thm the given contact (and its nearest neighbor) nrc not palm
IK-c!s. Unlike the site nnd oricntmion factors which only
b<.><::omc reliable ns the weight of the hands li.llly compresses
the palms, the palm separation bclor is especially helpful in
distinguishing the palm heels Ji-om pairs ofadjacenl linge11ips
b<.><::ausc it applies cqun!!y we!! to light. small contncts.
Once the thumb nnd pnlm weightings hnve been npplicd to
the distnncc matrix. step J58 linds the one-to-one assignment
between allraclors and conlacls which minimites the sum of
weighted: distances between each allraclor and it's assigned
contnct. For notmiona! purposes. leta new mmrix fc,) hold
the weighted distances:
r/U''""'"''
'o"f'lo'< 1 1 'o" I
il j =I (531
,,,
il 1 "'j" s
'"
d J / {!'! ""' '"
' ' "'""'"' "'
il j = 6 ,,
'
d J / {!'! ""' '"
' ' "'""'"' "'
,,
'
il j = 7
Mmhcmnticnl!y the optimization can then be stm<.'d ns linding
the pemmlalion { :r,. . . . :r
7
} of integer hand pari identities
{ L , 7} which minimites:
i.'i41
"" linge11ip conlacls are in increasing order easily \eri!ies !hal
linge11ip identities are no! erroneously swappL-d. llowe\er,
when-only two 10 four lingers arc touching. yet no linger
strongly exhibits thumb size or oricntmion !Cmurcs. the
assignment of the innennosl !ingerconlacl may wrongly indi-
15 cale whether the conlacl is the thumb. In this case. decision
diamond J62 employs a thumb vcrilicmion process J68 to
tnkc li.Jr!hcr measurements between the inncnnos1 !ingcrcon-
tnct and the other lingers. If these limhcr mcnsurcmcms
slnmg!y suggest the innermost linger conlacl identity is
30 wrong. the thumb verilicalion process changes the assign-
ment of the innermost !ingcr contnc1. Once the linger nssign-
mcnts nrc verilicd. step J64 compiles stmistics about the
nssignmcnts within cnch hand such ns the number of touching
linge11ips and hit!ields of touching linger identities. These
35 slalistics prmide convenient summaries of identilicalion
results for other modules.
1-'lG. 26 shows the steps within the thumb verification
module. The !irs! 400 is to compute severn] velocity. scpnm-
tion. and angle faclors l(Jr the innennosl conlacl identi!iL-d as
40
a !Inger relative to the other conlacls identiliL-d as !ingers.
45
Since these intcr-pnth mcnsurcmcnts presuppose a contact
identity ordering. they could not hnve easily b<.-cn included as
anraclor distance weightings bL><::ause conlacl identities are
no! known until the anraclor distance minimi1111ion is com-
plete. For the li1c1or descriptions below. let Fl be the inner-
most linger contact. FN be the next inncnnostlingcr contact.
FO be the outermost !ingcr contact.
The separation between thumb and index !Inger is ollen
larger than the separations between !ingerlips. hut all separa-
'0 lions lend to grow as the !ingers are outstretched. rherel(Jrean
inner scparmion fnctor im1er_scp.1mtion_fact is ddincd as the
rmio of the distnncc between the inncnnostand next inner-
most linger conlacls lo the average of the distances between
other adjacent !ingerlip conlacls. avg_separalion: 12
'
5
inncrscparntionfact min
{

iwli'n''P"""i<mJc"'' mi I. '--'-;;cC;;c;;cc;c;--'---
vv mji.>l'fHIIYIIiOII
155)
where c,
1
is the weighted distance limn con lac! i lo allraclor _i.
and contact i and nnrnctor j arc considered nssigncd to one
another when :r,"'j. This combinatorin! optimitntion problem.
known more sp<.><::ilica!!y in mmhcmmics as nn assignment 65
problem. can he dlicient!y sohed by a \ariety of well-known
malhemalica! techniques, such as branch and hound, local-
The lUctor is dipp<-'<i to be grcmcr thnn one since nn inner-
most scparntion less thnn the average can occur regardless of
whether thumb or index linger is the inncnnos1 touching
linger. In case there are only two !Inger conlacls. a ddimh
average separation of 2-3 em is used. The bclor lends lo
ptm J:mJmD ;,t]J IJl' 'ilmp;,umn ;,p.b
;otp u;,t]J UO!JIU!UliJdO .LOtJ '>jJO.\'\.PU JO .(lo;otp ;ot]J
lUOJJ U.\\OlDj[p_\\ Ull]l]JO'i!je l[Wd e 't'ff:"
-UO::I J;:Jtj)O ;:ll]l OJ )Jll)UO::I lUOJJ S;:J::IU!J)S]p ;:ll[l JO X]J)BUJ
[tmo!su;,unp-o-\\J Zft d:JJS ott pm"l u;o.\!TI 1'
ll!l]Jl\\ SPI'JUO:l !UJl'd !UO.lJ :lljJ .1;op.10 OJ_ ":lWjd
-m:JJ Jopt!IJll' tm ;,pnpm J<.m s:Jop tpllj.1. JU;om!poqm;, utl!J
:m;,t]J u;oJ.\\Plj Jop;-o\
;ot]J OJ :l.\O!U :ll]J S!l]J ll! Jlllj
'SUO!)J;:l.!]p ;:l)!SOtftfo U] ;:lAOUl J;:Ji!U!J)SOUlJ;:l)llO pue
;:ll]l S!J _\\O(;:llj UO]ltJ!lb..") JOPBJ UO]l!JlOJ ;:lJlllU]XOJddr,
;otJ_L ptm qmm[J ;,t]J u;oJ.\\Plj JlH!d .\.mu!Titmn tm
JllOlJI' UO!JI,JO.l Jll.\10] JSll!U UO!JI,JO.l :lljJ ;o\!J
;omm :llj OJ_ .(HmO!JI,JS PI'JUO:l :lljJ :lJ!l]\\
-ll:l!J!lU;:lp! J;:J'i!U!J ;:lA!)lllU;:lJ[!J Ull SUJBJ"ik]p LZ 'J;:l!j]llWpl
)Jll)UO::I puet[ e OlU! pue S;:JJn)lJ;:lj ;:JS;:ll]l 'i!U]WJ
-mbo:Ju! .!OJ poqpm tm JO :J]dmt,X:J J;otpmm
Jlm!'il_mm pp1' .{JUO
Ill-\\ l]J]<[.\\ "Slj)'i!U;:J] J;:J'i!U!J [Bllp]A]pU] OJ JOJ::I!JJIW
09 p.!JJ.\\U.\\llp ilU]W[SU!JJl J;:Ji!U!J lSOlUJ;:JUU! ;:ll[l U;:ll(.\\
tpns UO]JOlU J;:Ji!U!J Ul OJ Ul .\\lUil ,\]SilO
p]nO-\\ Jl 'JJ'i!ug U:J.')\\Plj
;o('iltm :lljJ ll! S:l'i!tmtp OJ (I'UO!pm!tud JOPI'J UO!J
-lJ)lU ;:ll[l J[ ,\J;:JA ;:llj mjll)SlllU JOPilJ UO!W)OJ ;:lllJ.
JO UO!W)dr,pe Ulll]l )Ull)JOdtu] ;:JJOUJ (;:l;:llj Ul]lld
pue J;:JilU!J e OJ Ul[Bd pue l]Ul!U]l
:lljlJO 'SUOS.J:Jd SSOJ:ll' :l/!S ptllllj Ul
:lljJllllljJ .l:JJB,"U'i! JO Jl,a!TI am l]Jllj.1. UO!SU:JJX;o ptm
UO!X;:l!J J;:JilU!J OJ ;:JHp J;:JilU!J ;:JjilU]S ll U]
-r,pe,\ S;:J)BJ;:J[Ol ,\pe;:JJ[!J pOl[l;:llU UO!)ll::I!J!HWpl 0>
:ll]l'J:J.\;o_LOJJ p.')tt:Jlll'[l ptmq :ll]J ;o[!lj.1. U:l"!l'J ;o/ls ptmq J;osn
[tmp! \!Pll!JO OJ .\]Jl'!U!Jdo ;,q tm:l
'tfillt] ;:JJB S)JlllUOJ l[lOlj JO
U;:ll(.\\ .(]UO JOPBJ UO]StmdX;:J ll ilU]JilpOJd ,\q .\)!,\l)J;:l(;:JS
spm ptm :lt]JJO tm;,m
tmqJ ;,t]J Tiu!Jndum_) xtml :ll]J
.\q lU:lZ OJ l]J!l]\\ S:lllJl'\ JOPl'J :l.\!W'il;-,u
UO!P;:JJ]p ;:ll[l Ul
jlD!JP!Ull.')!) "P.')Jlmllm[;, :llj .{tml S.IOPIOJ ptm SJOPI'.llll'
SJ! pue UO]l!J:l!J!)U;:Jp] Ul[Bd Ul[Bd ;:1,\l)]SU;:JSU! 1
111]'01-1
11
1'11)
1111'0.-1 IIII'H
rep.oe = 111110.:1 "I 1111.:1!
-tpllOJ ;:Jpllpm tp!t[.\\ SllJBJedde ;:J::IBJJilS tpnoJ-]l[ntU
:ll]l JO :l\!Wtu;,J(I' U] "UO!JU;o\Ul :lljJ JO
]]J\\ umma1 p.\ s.{l,\\ u! p.')U!lJ
-lUOJ pur, ;:llj U!JJ UO]l!J::I!J!)U;:Jp! pm OJ ;:JJ;:llj p;:J::>np
-lU)U] pue S;:JJilWq) PJJ)UO::I puet[ ;:lll.L
IL,l
;,t]J u;oJ.\\Plj :lj'i!tm ;otp ([ u 10:1 1u 11:1)7 aJ;ot]\\
:[Ol'J '"'!"""'Jx.
ou fit Ot
:lljll]J!-1. 'i!U!JS!X:l ;ot]J
J! .10 p]m]s;-oJl]J -\\O] ptm qTI!lJ ;,tp ua.')_Lpq S! PIOJ-lJUllll]J-P.')U!lJ
19>1 I I I ufo.-rfuf f.-!17 -luf "
1
'0.1 I"''"
xi(lufo_.fl'l uf f_.fl7 -1 "
1
'! .flsooxl u 1'''"'\tlxlu 1'""''
1
' f_./ /l
-lUOJ ;:ll[l J[ 'JO)JBJUB l]Ullll[l ;:ll]l !HJU!l OJ'i!UU JOJ::IBJl)ll ;:ll[l UO
p.!e_\\HlO SJ::I!JlUOJ J;:J'i!U!J JU;:JJJJfpr, .(tm pur, )U;:JU!Uil!SS!J )J!J)UO::I
:ll]J Ol t dJJS "JOPI'Jlll' qmmp ;,tp OJ p.')uTI!SSI'
pt,JmD ;,t]J 80t pumtm!p Jl qumt]J
-".i""!'"''Jx.;
:UWl[l U;:J;:l_\\);:llj JOJJ;:l,\ ;:ll[l ll) j;)[]!JJBd 'SUO!IJ;:l.!]p ;:l)!S
-oddo Ul pue ;:ll[l ,\[;:lJllUl]XOJddr, )ll Sl::I!JlUOJ
ptm ;,tp JOPI]
;ot]J ;oJI,!U!XtUddl' ;otp .\q p.')U!PP u;,t]ifi :ll]l JllU .{pJ!U!PP S! Pl'JUO:l JSOU!J:lUU! :ll]J
.\\0] e U!Jl[l S] JOJ::IllJ ljlU!U]l fi<")U]ljlUOJ ;:lljl)!Jl[l
90P puommp J! 'JOPBJUB qtunt]l ;:Jtjl OJ
'i!uu JOPI'.lJJI' :lljJ UO p.m.LU! PI'JUO:l ;,t]J tJt
"Z{ t l]Ullll(J ;ot]J OJ .{pl,:l.lJl' JOU PI'J
-UO::I ;:Jt[l Zit' puommp qmnt]l e .([;:JJ]U
-]pp Sl PJJJUO::I J;:JilU!J JSOlUJ;:JUU! ;:ll[l t'Ot' puoum]p
e JOJ::IllJ qmmp ;:Jt[l Jl
-aJ:lllJ. ;o'i!Jq OJ ;,t]J ;osmD ][!\\ pt,JUll:l ;,t]J
m1 S:l.ll1Jl'q) ;o'i![qmm]J "J;o\;o_LOJI .m;,u uumn1 OJ SO!Jm
;:ll[l ')SOlUJ;:JUU! )X;:JU ;:ll[l OJ J!Jl!Ul]S ;:llj p[HOl]S
-ll;:lJ d]JJ;:Ji!U!J e OS]!J S!J::IlllUOJ )SOlUJ;:JUU! ;:ll]lJ[ tftU;:Ji!U!J
e OJ U.\\OU'j .(pe;:JJ]!J l]J]lj.\\ ')Jll)UO::I JX;:JU ;:ll[IJO
JUal.lll:l OJ PI'JUO:l JSOU!J;oUU! ;ot]J JO ;ot]J
!UO:l S!l]JJO JOPIOJ :l.ll1Jl'q) ;otJ_L
1191
I'"''' ,-, '1"""''.\'.1
,<o"r "-" ''"'"''IJIXI'o"r '"'"'"_\_/
/"'i ""'"'f.-JIX(Pt] \H)I[t')\\J+P\] UOISli\:<)X,'+Pt]
<)lliOU[l I''UlLJUI\\0
:utJOJ :ll]J S;J"!I'J PIOJ l]Ullll]J p.')U!lJ
-lUOJ ;:ll]l .!OJ UO!SS;:JJdX;:J ::1!'!!0[ .(ZZilJ S]l]l "HWUHpOljlU;:l
ll U[ "S)JlllUO::I J;:Ji!U!J )SOlUJ;:JUU! )X;:JU pur, JSOlUJ;:JUU!
;:ll[l ;)Jil)Bq) ljlUlll[l ;:ll]l l]l!.\\ )Jll)UO::I-J;)JU!
:lS:lljJ S:lll!lJ!UlD( Z()t UO!SS:lJdX:l:l!'i!Oj ,\/ I'
lj!Ulll]J JO JOJlD!pU! Sl JOPIOJ
UO!)BJOJ ;:ll]l 0_\\J U;:J;:l_\\J;:llj UUOjJ;:Jd OJ l[llJHJ!p Jill]
J;:JilU!J J;:Jl[IOUB pur, l]UHU]l ;:l]q!JSOddo ;:ll]l U;:l.."),\\);:llj UUOJJ;:ld OJ
.{SI,:lam JOPl'J UO!JI,JlU :lZ!Ul!XI'!U l]J!l]\\ SUO!JO!U :l:JU!S
ln9l
I
(IOIJ :[Ol'J IJ(l!II')Ol
1 1 I ufo.-t(ll/ t.-!17 -luf "
1
"0.1 l"'s
x(( lu/o .f'f uf 1 .f) 7 -1 '''t .Jiu "xlu I'""'''' o-11 u

I /l
- - . priuOqqo.r
t
sd!p:l'i!U!J ptm lj!UilljJ ;otp U:J.')_\'\.)Jlj JU!Od I' UO fl.').l;oJ
-U;:JJ Jll J;:Ji!U!J ::IU);:JlUUJ,\S ;:JJB
0[ tpll(.\\ Ulll)lllOJ pur, UO!StmdX;:J l[ll.\\ p;:1,}1() .\zz111 ;:JJB
SJOIJilJ ;:Jjiltm pur, J;:JUU] ;:llj)";:l)llStWdtuO::IOJ. ".\\;:lJ::IS ll
'i!U!ll.lllJ.J! :lWJOJ ;otp U!:lJ:llj\\ UO!JI,JlU ptllll(
JO pu;, Jo 'ilmum'il;-,q ;,t]JJI' JOPIOJ :l('i!tm J:lum :ll(J .\q
;:ll[l mjll Ull::l ;:Jji!tm PpU]-l]Uilll[l ;:ll[l ,\pB]
,:c -!lU]S ;:ll[lJll UeJS ;:ll[l W ]]!JlUS .([;:!A]W[;:JJ ;:llj ll) JllPBJ
J;oum :ll(J SlllJ. ;ouo UllU.J ;op!(S
d!p;oTiug ptm qmmp ;,tp u;,t]J Jill] ;,IPP!m Jo x;,pu!
;ot]J .{tml lj!Uill]J :ll[l amJs;-,1! tpmd-!Jllll
U!J ilU]Jilp ';:JJUBISU] Jll:] UO!)ll)OJ Jll i!U!l!J::IS puet[
o<: i!U]UUlljJ;:Jd tWlj.\\ ;:J[i!Ue pur, Ull]WJBd;:JS ;:lAOljll
;,t]J p:l.'):JX:l u;,J.JO SJ:lSil Jill] s;-oJilJSOd qutnt]J JO Tiu!JI'u
!lll!.IJS!P .{[l('illl( am ptm J:lUU! :lllJ.
mpt'j .U;o\ 'iluDilptud oOt U:l.')\\Plj
,\[;:1:-jl[]]l.\\ PlllUOJ )SOlUJ;:JUU! JX;:JU pur, )SlllUJ;:lllU! U;:l.."),\\);:llj
,:] "" . ' "" ' - '
;:J[otm ;:Jt[l d])J;:JoU!J tm A[[BilPB Sl l::IJJJUOJ
J;:JilU!J JSOtuJ;:JtJU! ;:Jt[IJI "
0
()(: [ pue
0
() p.!B.\\0) () S;:Jtpetuddr, pur,
,()') Jl' l]Jll(.1. "Zf !)]:] ll! JOPI'J :l('i!tm J;ouu! ;ot]J.JO JO[d
;oJI,mlxtuddl' :ll(J m p.')P;J[l:l.l S! SlllJ. mp'd ;,t]J J;opull
01
qmnt]l ;:Jt[l se
0
()(:[ 01 pue p.!e_\\JOJ qmnt]l ;:Jt[l () 01
S;:Ji!UeJ)! tji!HOl[l '
0
()') U;:llJO Sl J;:JilU!J X;:JpU] ;:ll]l OJ ljlUlll[l
;ot]J JOp:l.\ :ll(JJO ;ot]J ptmq Jt]'i!!J :ll(J JO_.] "l]Ullll(J :ll(J
Sl Pl'JUlD :lljJ J:ll(Pl(-1. :lJlD!pU! dpl( llll:l Pl'J
UO:l JX:lU ptm ;ot]J u;oJ.\\Plj :lj'i!tm
_ ;:ll]l ll O)Ul fi<")]Jil::l ;:JJ!J ;:lljl U;:Jtj.\\ Jd;:JJX;:J
' ;:ll]l p.!B.\;.;OJ J;:Jl[UilJ ,\pJBJ l]UHU]l ;:ll[l ;:l.::IU]S .
'd])J;:Ji!U!J
ll S] IJ!J)UO::I )SllUJJ;)llU] ;:ll]l Jl ;:lUll JB;:JU )Ill] ljlUlll[l
;ot]J PI'JUO:l JSO!UJ:lUU! ;ol]lJ! :lUO J;o1l:Jq ;outO:l.')lj

Z8 8Z8'Z!8'L sn
:ljlj!SU:JS :lj!OJ:JU:J'i! OJ .{1,'>\ JU:l!:l![P JSOUl :ll(.!_ "P:JJt,dtutD
ptm p:lJl!l:JU:J'i! :llj jSl1Ul S:JS;JljJOd.{l( jU:JtuU'i!!SS1'.JO j;)S 1' jX:JN
JJO lJ!J .\jptuodm;,J u;,t].\\ u;,,\;,
.(J]UHXOJd
01 ilu]d[;Jt[ puet] JU;JJ;JJ ;,[quws;oJ tptt(.\\
l(JOd.{tj JU;JlUU'i!!SS1' :l.\!WpJ :ll[l JJ'i!
-ug put' pm''l Jt[lt]l!.,. Tiu!J JOP10JJll' :ll(J 'ilmJsnfpv
"JOPeJJW SJ! pue )JBJUOJ l[JB;J U;J;J,\\J;!lj S;JJUeJ 09
-s]p p;,n]'il];,.\\ JO mns ;Jt]J uo ilu]pu;,d;,p ;Jt]J "J;Jt]JO ;Jt]J
'i!uu :lUll U! S.lOj:l10JJll' OJ SJ:ll'JUO:J'i!um'i!!SS1'JO JljJ
an,dmo:J ][!'>\ Ull[JUO'i!(l' U0!11':l!JIJU:lp! ptml( :ll(J .{][lD!Sqj
SUO!JISOd ptmtj UWlU)j JSl'J JljJ UWJ;)J .\j!.l10.lOdlU:JJ SJJ1'lU!JSJ
J:l!i_[IO put"] :ll]J :JJU!S ':lJPP!Ul JljJ.m:JU .{]JU:l:l:JJ JSOlU Sl,.1. ptmq
l[J!l[.\\ OJ'i!U!pJOJJe Jj;)[ JO Hj'ilp ;Jt[l p.!e.\\OJ "J]UWU.(p
;,q ;JJO.];)J;Jt]J p]nOt[S ;Jt[l JO Jt[!lu pue Jp]
u;,;,_\\J;oq ;,tl.L puet[Jt[!lu ;Jt]J pue ;,[ppm!
Jt]J OJ Tiu!.\om put"] Jj;)] :ll(J mOJJ Jtm u.wlp )o[J1'lJ 'ilmtpmlJ
ptmq Jt]'il!J :ll(J tulUJ :llj .\p)o[!l JSOtu II!"" :llpptm :Jt]J u! J1,Jddt'
l[J!l[.\\ SJ:llJlUOJ JX;JU ;JtjJ "JJO pue ;Jt[l JO ;![ppm!
;Jt]J OJ ;opts Jt[!lu ;Jt]J mtUJ s;,Aom ptmt] nj'il].! ;Jt[IJ! ;,[dumx;, JO;]
'lll.Jdptj S;J.\OJd OSJl' .{q.m;,u JJO p:llJ!I .\jJUJ:l:JJ l(Jil(.1. SP1'JUO:l
JO .\J!JU:lpt Jt]J Jnq uo SJ.mJs JJJsnp:Jt]J p.moq :ll(JJO :JP!S tp!t]'>\
UO .\j!\1,Jlj pu:Jd:Jp ]]!JS jSl1Ul UO!JlD!J!JU:lp! JJjSllj:l JS1'J Sll(J
1>91 "'''!JY+tur '(tY+tur 'mt fuJ'Mt
lt-91 ,m
1[91 "'
1
'!1"l+fuj"''(n+fuj ,n fuJ'fvJ
'
1
'
1
'!J"f+fur"l ..fl+f11f'" ''II fuf'lv I
U[ puet[ \Pl e mtUJ .\pe;,u sJe;odde
J;JJS!l[J J:llJlUOJ puetj Hj'il].! r, ";!:l!J:J.!IlS ;Jt]J JJO p;,\J!I ljlUill[l
Jl]JJ] 'SPl'JUO:l ptmt]Jj;)]JO JJJSll(J l' UlO.!J spt,JUOJ put"] Jl(TIU
J 0 J:lJSll(J :JUOJl' lJS! n'i!U!JS!P l(Jil(.1. [!(l!J!SOd JJJSllj:l J 0 jU:lpU:Jd
-;,pm )JlJlUOJ .([UO ;)ljl ;JJe JU;JUW:Jejd ljlUill]l ;)A!Je(;)J
1<:91
O> ;Jt]J pur, J:JP-JUOJ m]L>d J;oum pue qmm]J ;,ttL
:;oJe JopeJue 11 puet[ nj'il].! pur, 1 puet[Jp[
:ll[l JOJ (lu ]"fvlu ]'!\:) SUO!J!SOd JOj:l10JJll' [l'U!J :ll(J JJO.fJ.!:lliJ.
spSJJO JJ'i!U!J p:JWUl!JS:J Timpuodsawn J!Jt]J .{q p:lW]Sut,JJ
;,q .\pres tmJ snnod JopeJue [enptA!pU[ 'HOZ 'DLl m spurt]
;Jt[l .\\ll]]OJ SUO]f.;JJ ;Jt[l ;J'j][JSilf puetj S)!
.10 ml!J!sod p.")J1,mtJsJ Jtp ..... o
11
o.1 oJ s:Jw
1
sut,JJ Tiu!J 1p1" 1 voz
"!)]:] ll! U.'>\O!IS SUO!J!SOd P1'JUO:l ptmt]Jl(TIU ptm \PI
:ll[l OJ puods:JJJOJ PlllllljS J:ll(JOlW :JUO OJ :l\IW[:JJ S'i!U!J JljJ
JO JU;JlU;JJejd Jjll!lj;)p ;)ll.L ';JJUO Je p;oZ!]!Jll ;oq .\\OU JS!llU
Jt[ilU pur, Jj;)[ l[JOlj Jet[J Jd;JJX;J "UO]W:l!J!JU;Jp] puetj-U!l]l!.\\ Ot-
U! p.")Sll (ZS d:JJS 'p(': "!)J:J) s;,Jqdm:JJ JOPl'Jlll' :ll[l S1' Jums Jl]J
ii:JJl':l!Sl'lJ :llj PlllllljS :JS:l!IJ. "S:lJl'ldUl:Jj JOj:l10JJll' ptml( Jl]'il!J ptm
Jj;)[ pur, ;JU!J;op OJ S! fSP d;oJS JS.!!J ;Jt[J ;,'i!em] JU;JJJHJ
;Jt[l .!OJ p;o'jO,\U! ;oq lUl]JUOilje UO]Je:l!J!lU;!p] puetj ;Jt]J p[llOlJS
s;,ileun mtUJ p.")md
-mo:J SUO!l1':l!J!JU:lp! Pl'JUOJ :ll(J (ZSP d:JJS) pu:JjX:l put' umpJ
OJ JU:l!:l!lJllS S! SPZ SS:l:Jtud 'i!unp10JJ l]Wd :ll(J .{q S1'
UO!)lJllU]lUOJ l[ltJd "JtJ;Jdde SJ:Je)UOJ .\\;)U OU pur, fi<")Z][!l]P-JS ;)AlJlj
S;J!J]UI]XOJd l[J]lj.\\ U] JO:[";JSlJ;JJJU!
OJ S;JHU!JUO:l .\\;)U .(ur, JO .(J]UI]XOJd jeJOJ ;Jt]Jl[J!l[.\\ 0[
U! SJ'i!1'lU! .\J!lU!XOJd JU:Jllfl:JSljllS .H:J ptm SJt,:Jdd1' P1'JUOJ .\'\.")U
'J;JtjJO ;Jt]J OJ ;Jt[l
JO:lP!' ;,uo mOJJ S:JP!l' put"] I' S1,SUO!JlD!J!JUJpt 'ilmJS!X:lU!l'J:lJ
.\llJl'I(:J.l utD :llnpmu 'i!U!)o[J10JJ l(l1'd :ll[l "P.")!J!lll:JP! .\[[l'!llll!
JJ1' spt,JUOJJJ10Jll1S ]Jl' JJU() .{J!JU:lp! Jums :ll[l p'i! ptmq Jums
;Jt]J lUOJJ ;JJnStW OJ p;,][dde ;oq puet[
JO S;J!)J;Jdtud 'i!U].!;JJS!l[J ;JtjJJU!lOJJe OJU] ;J)jr,J l[J!lj.\\ SpOl[l;!lU
p:lJlD!JS!t]dos :JJom amp.!Jt].L ptmt] Jt]'il!J Jt]J OJ S.l:ll(JO ptm
ptml( Jj;)j :ll(J OJ p:lU'i!!SSl' dn pu:J Plllll.1. Sj:l1,JUOJ s,put"] :ll[l.JO
Jtuos "S:l.\ll"J Jt]'i!!J put' \PI Jl[l TIU!JPPl'JJS ":ljppnu :ll]J U! u.Lop
p;,tpllOJ puet[ r, J! "J;)A;!.\\0[[ ;JtjJ JO Jjel] Jtj'i!p JO Jj;)]
;Jt]J U! U.\\Op p;,tpllOJ.([[P-]J!U] )JlJlUOJ ;Jt[l J;)l[l;!l[.\\ OJ f.U]pJO:l:llJ
P1'JUO:l l(J1'J OJ .{J!JU:lP! ptmq u'i!!SSl' OJ :llj Plll0.1. Ull!JlD!J!l
-U:lp! ptml(JO pm1pm JS:lldUHS :lllJ. :lljJJO J[l'l( J:ll[ll:l
OJ ;,[ppm! ;Jt]J .\j;J..")JJ meoJ ueJ puet[ e JP-t]J tpns p;,mof
;J:JlJjJilS ;Jt]JJO Hj'il].! pue \P] ;Jt[l ;J,\!Jt[ l[J]<[.\\ pue
-eJ]IllU]S spurt] t]Joq ;,wpomo::>:Je OJ tj'ilnotw ;oilJe] ;oJe tp]t].\\
tpnoJ-!Jinm .H:J P:JP."):JU st ml!JlD!J!JUJpt ptm]J
"SSJ:JDJd U0!11':l!J!JU:lp! JJ'i!U!J
Jt]J OJU! ptmq p:JWUHJS:l:ll(J ptm spt,JUll:l U:J..").LJ:llj S:ljTitm
p;,p;,dx;, S;JJeJOdJO:JU] ,\pur,il;o[;J ;JJOlU JU;!lU]pOlJUW JO)JP-JUP-
;Jt]J JU;!UI]pOlJUW ;),\]leUJ;JJje J;)i\0 p;oJJ;Jj;JJd
st JU:Jtmpoqm;, JJqdtuJJ Jopt!lJll' :JILL ps_uo ptmq p.")J1'm
!JSJ :ll[l OJ :l\!Jq:J.J ptm 'i!U!J JljJ U! SJ:ll'JUOJ J:lljJO OJ J\!Jl'(:J.l
1' l(J!l(.1. Ul J'i!1,lU! .\J!Ul!XOJd l(:l1':l .H:J Ullll!JO'i![l' UO!jlD!J!JU:lp!
puet[ ;Jt[J S;!JilJ;JX;J ()St' puommp ;JJOj;JJ;Jt].L peJUO:l
;o::>e:pns [P-]l!U] J;Jlje spuo:J..")S]II!UI u;,zop .\\;Jj e ;Jt]J
JSUm'i!1,SJSS:J.JdlUOJ ljS:ltJ JljJj!JUll :ljqetpmn :l.ll'
UO!jr,JU:l!JO ptm :lZ!S P1'JUO:l :lllJ. "J:J'i!UOJJS JUlO:l:llj Sj:l1,JUlD
'i!U!JS!X:l JO 'i!U!Jlm'il!lJlUl"IP JO U Wlp S:ll(JllllJ lJl'd
puet[ .\\;)U r, tWlj.\\ p;,JenjeA;J;)J ;oq .(j<!O fi<");JU S;J!J]HWp! J;J'i!U!J
pue ;JljJJO .\J!]!l]P-JS ;JZ]UI]Xem OJ. p;,zmq
-l,JS :J.\1'l( S:JIJ!UHXOJd qwd Jll' J:Jt]J:Jl]'>\ 'ilunp:ll(:l .{q p:lJll:J:JX:J
:llj OJ sp.h")U .{jpmp1' lUl]J!.!O'i!Jl' UO!J1':l!JIJU:JP! ptmt] :ll[l J:ll(J:ll(.1.
JS.!!J ()St' puommp "LPZ ;Jjllpom UO!)
puet[ ;,t[J U]t]J].\\ s]!m;,p 6Z 'DLl
- P1'JUO:l JljJJO UO!JISOd jlD!p:J.\ :ll[l.JO SJUJUl:l.lllSl,:llU ;opnpu!
" OJ fi<");JU .(em peJUOJ tpe;, .!OJ SUO]l:ltH1J UO]le:l!J]J;!A ,\zZilJ
;Jt]J ;,,\!)P-pJ uo SlUWJJSUO:J ;osodtm OJ ;JJP-]dm;,J JOJ
-J10JJJ1' tm :Jpnpm Jtm s;oop mt]J!JO'i!Jl' :J\!Wtu:lJil' S!l]J JJU!S
ml!J!Sod llDtpJ.\ t]TI!l( .\p\tWpJ J!:ll]J ptm Tiu!J Jt]J u! 'iluu;opJo
flC :l\!p;,dsaJ .l!:ll]J .\q SPl'JUll:l dtp:J'i!U!J 'i!u!umm:JJ S:l!JIJU:lP!
8PP d:JJS "P.")!J!lll:JP! :JJl' SPl'JUll:l Ulll'd ptm ljlUl1lll .\tm J:JU()
pullOJ ;JJe Uljed JO l]UIIll[l J;JtjJO ,\ue [!lUll JSOUU;JUU]
;Jt]J OJ ilup ;Jt[l U] JlWJefpe ;Jt]J .!OJ p;,w;,d;oJ U;Jl[l
'OS op
pjllO!IS UO!Jtmmd p.m.L)o[.W' Tiu!SlllD .\q ;opts :lJ!Soddo ,:1
:ll[l OJ put"] I' Timp!JS lUO.l.J SJJSll J'i!10Jl10:JS!p OSJl' UlD :lJPP!Ul
;Jt[l ;J:JejJilS ;Jt[l f.U]l]:U\" .(j!JU;!p] ;)lUI JO
,\j!JU;!p] SJ[et[ ;Jt[l U;J,\]il ;oq .([;JjeS tW:lJ] ";!:l!J:JJHS ;Jt]J
JO JJl"J :lJ!SOddo :ll(J OJU! [[:l.1. UWlp S:llj:lllllj .\jU:lppns JJJSllj:l
ptml( l' U:llj'>\ ":J.H:JJ.!:lliJ. ptmtj :lJ!SOddo JljJ JO S[!(l!J10JJdO fll
JlldU! pmtp :ll(J SS:l:l:l1' OJ JJ10J.ll1S :ll(JJOJJl"J :lJ!SOddo :ll[l OJUO
JP-j pUet[ ;JtjJ ;JJe[d ,\[[P-UO]JU;!JU! .(em ;JJBjJilS ;Jt]J ;JSil OJ ;J;!JJ
puet[ ;,uo .\juo t[ll.\\ ;,]dmex;, JO;] m
:ll[JIOOJ OJl(SI'>\ .\Jil'l1Pl' .\1'm J:JSll JljJ 'SUll!JlDOj p:lp:JdXJUll
U! ptml( l' JO SJUJUlJ:Jqd U:lppns l]J!.1. Ul:Jjs.\s [!(l!jlD!J!JU:lp! ,:
puet[ ;JtjJ]OOJ tWJ J;)Sil ;Jt[ll[J!lj.\\ U] UWUI;JJ l[illlOll.L
p.!P-l] p;,wmps;, ;Jt]J
OJ JO)JP-JUP- puet]Jtj'i!U pur, puet[ \P] JO ;Jt]J
'i!m.{j .\q p:lJU:llU:lldUH S! UO!SI \!P :l!lUlm.\p l' l(Jl1S ].JOlJII .l:llJl'
9
;oJe m[ed pue qmnt]J ;,tl.L smwd ;,q p[nOt]S [;J,\;Jj[eJ]JJ;JA
JUWS :lljJ W Sl[l1'd :ljll(.1. SJJ'i!U!J :llj jSlllU JSOlUJJUU! :ll(J J\Ol]l'
.\ptm:l!J!U'i!!S Sl]Wd Jl]J s;opnpuo:J gpp d:JJS ym] Ul]t'd 1' S! l[l1'd
JSOlUJ:JUU! Jl]J (ZPP )J I "SUlJl'd .\p)o[!]JSOUl JJ1' l]Ull1l]J Jl]J Wl[Jlj
'i!UU ;)ljJ U] SJ:llJlUOJ pur, ,\[;!'j!l ;)JP, ;)AOljlJ SJ:lr,J
-UOJ Jel[l s;,pnpUO:l ppp d;oJS "l]UHU]J r, tped JSOUU;JUU] ;Jt[l
J1 JO)JBJUP- ";)l]l pur, SS;J:JtUd UO]le:l!J]J;!A ljlUill[l
JljJ ll! fAVI]!Jll :JSOljJ OJ sno'i!Oilml' SUO!SS;JJdXJ :l!'i!Oj .\/ /ll.) ptm
pt,JUOJ 'ilmsn :Juop ;,q tmJ stt].L (zpp pmmm!p ums
-D;op) Ulll'd JO (()pp puoum!p UO!S!:l.")p) l]Ull1ljJ1' S! 1! J:lljJ:llj'>\
pur, JO JSOlUJ;JUU! ;Jt[l l[JllS ilup ;Jt[l U]
;)lU;)JJX;J tm JP- J:llJlUOJ e 'j:J!d OJ S! 8ft d;oJS JX;JU ;!lJ.L
'lU;JtjJ .!OJ ilU].!;JpJO e
)U;JJe[pr, )J;JUUO:l OJ pU;JJ ]]!.\\ ;op.(:J t[deJf! S]l]J fluu
l' m :lll OJ pUJJ SPl'JUll:l ptmq JJU!S 'PP :JJUO l(Jl!.") l('illllUl(J
>
Z8 8Z8'Z!8'L sn
us 7,812,828 82
37
hypotheses is lode !inc a set of roughly vertical contour lines.
one between each horizontally adjacent contact. Step 454
docs this by ordering all surface contacts by their horizontal
coordinates and establishing a vcnical contour hall'way
between each pair of adjacent hmimntal coordinates. !'!(iS. 'i
JOA-C show examples of three di!krcnl contours 475 and
their associmcd assignment hyp01hcscs for a !ix<..'d set of
contacts. !'ach contour corresponds to a scparmc hyp01hcsis.
known also as a partition, in which a!! conlacls lo the lcll 476
of the contour arc li'llm the ldi hand. and a!! conlacls to the lCJ
38
Users olkn per!\m11 clutching motions in which the right
hand. forcxmnplc. !ills off from a slide at the right side oft he
surlUcc. touches backdown in the middle of the sur!Ucc. nnd
resumes sliding townrd the right. Therefore when a hnnd is
delL-ctL"t.! touching down in the middle of the surbce and
sliding toward one side. it probably came limnthe at side. A
hand velocity lUctor. plotted npproximntdy in FIG. J 1.-\. cup-
lures tills phenomenon by slightly incrcnsing in value when n
hand cluster's contacts are moving toward the cluster's
assignL"t.! side of the board. thus decreasing the basic cost of
the hand. The bctor is a !i.mction of the merage of the con-
tncts' horizontal velocities the side of the surlUcc the given
duster is assigned. Since high speeds do not nL-ccssarily give
a stronger indication of user intent the bctor saturates at
right 477 of the contour arc li'llm the right hand. Contours arc
also necessary m the lcll and right ends of the surfncc to
handle the hypotheses thm all contacts on the surface arc from
the same hand. Contours which hypothesize more contacts on
a gi\cn hand than can be caused by a single hand are immc-
dimely eliminmcd.
15 moderate spL-cds.
Gcncrnting pnrtitions vin vertical contours nvoids nll
hypotheses in which contncts of one hand horitonwlly over-
Though the thumb orientation lUctors help idcntil) which
hand n thumb is from when the thumb lies in the mnbiguous
middle region of the surlUcc. the vertical position of the
thumb relati\e to other !ingers in the same hand also gi\cs a lap or cross over contacts of the opposite hand. Considering
that each hand c;m cause se\en or more distinct contacts. this
r<:duccs the numbcrofhand identity pcrmutmions to cxmninc
from thousands tom most n dozen. With lCwcr hypotheses to
examine. the evaluation of each p;u1ition can be much more
sophisticated. and if m.:cessary. computationally costly.
"" strong indication ofhandL"t.!ncss. The thumb tends to be posi-
tioned much lower than the fingertips. bllt the pinky tends to
be only slightly lower than the other fingertips. The hnndcd-
ncss bctor plotlL"t.! approximately in ]."](j. 31 !S. takes ad\<lll-
tage of this constraint by boosting the hand cost when the
'!11c optimitation search loop l(Jilows. Its goal is to deter-
mine which of the contours divides the contacts into n parti-
tion of two contnct dusters such that the duster positions and
arrangement of contacts within each duster best satisfy
known anatomical and hiomechanica! constraints. The opti-
mizntion begins by picking (step 456) a !irst contour divider
such as the leftmost nnd tcntntively assigning (step 458) nny
contncts to the !eli of the contour to the !eli hnnd and the rest
"" contact identiliL"t.! as the outemmst lingertip is more than a
couple centimeters lower than the next outermost fingertip
contnct. In such cases the tentative hand assignment for all
contacts in the duster is probably wrong. Since this causes the
within-hand identi!ication algorithm to lit the contacts to the
to the right hand. Step 460 invokes the !Inger identi!ication
algorithm of F!(i. 23. which attempts to assign linger and
pnlm identities to contacts within cnch hnnd. Decision din-
mond J6() avoids the computmional expense of thumb vcri-
!icmion J68 and swtistics gathering J64 for tills tcntmivc
assignment hypothesis.
30 wrong nttrnctor ring. !ingcr identities b<.-comc rcvers<."t.! such
that the supposedly lowered pinky is tnlly n lowered thumb of
the opposite hnnd. Unfortunmcly. limited confidence can be
placed in the handL"t.!ness bctor. '!-hough the pinky should not
appear lowered as much as the thumb the outer palm heel can.
35 crcnting nn nmbiguity in which the thumb and fingertips of
one hnnd hnve the smnc contnct arrnngcmcnt as the fingertips
nnd omcr palm heel oft he opposite hand. This ambiguity can
cause the handL"t.!ness bctor to be erroneously low !(Jr an
Returning to F!(i. 29. step 462 computes a cost !(Jr the
pnrtition. This cost is mcnnt to cvnluatc how well the tcntn- 40
tively identified contacts !it their nssigncd nttrnctor ring and
how well the p;u1ition meets between-hand separation con-
straints. '!11is is done by computing !(Jr each hand the sum of
weighted distances from each tentati\dy identilied contact to
accurately identiliL"t.! hand duster. so the hand<."t.!ness liJCtor is
only used on clusters in the middle of the sur!Ucc where hnnd
position is nmbiguous.
! )istinguishing contact clusters is chal!enging because a
duster can h<.-come quite sparse and large when the lingers
outstretch<."t.!. with the pinky and thumb of the same hand
spamllng up to 20 em. However. the palm cnn stretch very
little in comparison. plncing uscli.ll constrnints on how !Ur
apart palm heel contacts and ! ( m ~ p a ! m s from the same hand
can he. The entire palm region of an outstretchL"t.! adult hand
is a boll! 10 em squnrc. so palm contact centroids should not be
scattcr<."t.! over a region larger than nbom 8 em. When n pnrti-
tion wrongly includes lingers from the opposite hand in a
duster. the within-duster identi!ication algorithm tends to
assign the extra !ingers li'llm the opposite hand to palm heel
nnd forcpnlm nttractors. This usun!!y cnuscs the contncts
its assigned nttrnctor point ns in Fquntion 54 of !ingcr idcnti- 45
!icmion. including size nnd oricntntion !Caturc lUctors lix
thumb and palm attractors. This sum represents the basic
template lining cost !(Jr a hand. I ;ach hand cost is then
weighted us n whole with the reciprocals of its dutdllng
velocity. handedness. nnd palm cohesion lUctors. These lUc- '0
tors. to be described below. represent ndditional constraints
which are underemphasitL"t.! by the weightL"t.! attractor dis-
tances. Finally. the weighted !ell and right hand costs are
added together nnd sen led by the rcciprocn! of n hnnd scp.1-
rntion lUctor to obtnin a total cost for the pnrtition. '5 nssigncd to the duster's palm nnrnctors to be scnncrcd across
the surbcewiderthan is plausible l(Jr true palm contacts from
a single hand. '!(J punish such partitions. the palm cohesion
bctor quickly drops below one !(Jr a tentati\e hand cluster in
If decision diamond 464 deteJmines this total cost is lower
than the total costs of the partitions evaluated so liu 464. step
466 records the partition cost as the lowest and records the
dividing contour. Decision diamond 472 repents this process
for cnch contour470 until the costs of nil partitions have been 60
e\aluatL"t.!. Step 473 chooses the partition which has the low-
est cost overall as the actual hand partitioning 473. and the
hnnd identities of nll contact paths nrc updmL"t.! nccording!y
Then step 474 reinvokcs the within-hand !ingcr contact iden-
tification process so thm the thumb vcri!icmion nnd swtistics 65
gathering processes are per!(JrnlL"t.! using the actual hand
assignments.
which the supposed pnlm contncts nrc scnncrcd over a region
!urger thnn 8 em. Therefore its reciprocal will greatly incrcnsc
the hand's basic cost. FIG. 31C shows the value of the palm
cohesion bctor versus horitonta! separation between palm
contncts. The horizontnl spread cnn be cllicicntly mcnsurcd
by !inding the mnximum and minimum horitonwl coordi-
nmcs ofn!! contacts identified as pnlm IK-cls or forcp.1lms nnd
taking the di!krence hetw<.-cn the maximum and minimum.
rhe measurement and bctor value lookup are repealL"t.! l(Jr the
us 7,812,828 82
39
\erlical separation. and the hori/Ontal and vcJ1ical liJCtors arc
mu!tiplicmivdy combined to obtain the Hnal palm cohesion
factor.
40
was mmm!ed on a p<..-des!al!o allow access limn all sides, the
hand idcnliEcalion module would also hyp01hcsizc various
row! ions of each allraclor ring.
The allraclor-bascd !ingcr idcnti!icmion system 248 will
5 successlidly identify the individual hand conlacls which
!-'I G. J3 is an approximmc plot oft he inter-hand scparmion
liJCtor. '!11is liJCtor increases the total costs of partitions in
which the estimated or aclual hori/Ontal positions of the
thumbs from each hand approach or overlap. II is measured by
finding the minimum of the horizontal olrscts of rig.ht hand
conlacls with rcspccl lo their cmn::spomling default linger
positions. Similarly the maximum of the horizontal ollSets of lCJ
the Jell hand conlacls with n::spccl lo their cmn::spomling
default !ingcr positions is found. If the dillCrcncc between
these hand offset extremes is small enough to suggest the
thumbs arc mer!apping the same columnar region of the
surliJCc while either touching the surliJCc or Jloaling above it.
1hc separation lUc!or b<..>comcs very small. Such overlap cor-
responds to a ncgmivc thumb scparmion in 1hc plo1. To
encourage assigmncnl of conlac!s which arc within a couple
ccnlimelcrs of one another lo the same dusler, the separation
li1c1or gradually hegins lo drop starling with positive scpara- ""
lions of a few ccnlimclcrs or less. "111c inter-hand scparmion
factor is no! applicable 10 partilions in which all surface
conlacls arc assigned lo the same hand, and lakes on the
ddimh value of one in this case.
comprise the pen grip hand conliguralion (FJ(i. 15). !low-
ever, additional sleps are n<..-...-d<..-d lo distinguish the unique
Enger arrangement within the pen grip lfom the nonnal
arrangcmcm wi1hin the closed hand conEgurmion (1-'lG. 14).
In this pen grip arrangement the outer lingers cur! under
toward the palms so their knuckles !ouch the surliJCe and the
index. Enger jms out ahead of !hem. The pen grip dc!cclion
module 17 employs a Jivzy paltcrn recognition process simi-
lar to the thumb vcriEcalion process to dc1cc1 this unique
15 anangemenl
An additional problem with handwriting r<..>cognition via
the pen grip hand conliguralion is that the inner gripping
Engcrs and sometimes 1hc whole hand will be up
bc1wccn strokes. causing 1hc dis1inguishing Enger arrange-
men! to lemporarily disappear. Therdl1re the pen grip recog-
nition process must have hysteresis lo slay in handwriting
Ahemali\e embodiments of this hand identilicalion pro-
cess can include additional conslraim lUc10rs and remain well
wi1hin 1hc scope of this invemion. For example. a velocity
coherence li1c1or could he computed lo li1vor partitions in
which all lingers within a cluster slide a! approximately the
same spc<..-d and direction. !hough each clus1Cr as a whole has
a dillCrcm average speed and direction.
mode belween gripping linger !ills. In the prekrr<..-d embodi-
ment. hysteresis is obwincd by temporal !i!tcring of the com-
bind fut/)' d<..>cision lUc10rs and by using 1hc cstimmcd !ingcr
"
5
positions in measurements of linger arrangement while the
aclua! lingers are lilkd off the surli1ce. rhe eslima!L-d linger
positions provide cllCc1ivc hysteresis because they tempo-
rarily retain the uniqucjul1ing arrangcmcm before d<..>caying
back 10ward 1hc norma! arched fingertip posilions a lCw scc-
30 onds aller lillo!I
FIG. 28 shows the sleps within the pen grip de!L>clion
module 17. D<..>cision diamond 485 de!eJmines whe!her all
pen grip hand parts arc touching 1hc surface. If no1 decision
diamond 486 causes the cstimmcd !ingcr and palm posi1ions
lo he relriev<..-d ll1r any !illL-d paris in slep 487 only if pen grip
or handwriting mode is already aclive. Othernise the process
cxi1s for lack of enough surlUcc con1ac1s. Thus the estimated
Enger positions canno1 be used to start handwri1ing mode. but
Sometimes irreversible decisions made by the chord
motion r<..>cognizer or typing recogni1ed on the basis of exist-
ing hand identilicalions pre\ en! !ale changes in the identili-
calions of hand contacts even when n<..'W proximity image 35
information suggests existing idcmiEcalions arc wrong. This
might be the case lix a chord slide which generales inplll
e\enls that can no! he undone, ye! well into the slide nL'\\"
image inllmnalion indicales some lingers in the chord should
have b<..'Cn mtributcd to the opposite hand. In !his case the user
can be warned to slOp 1hc slide and check for possible inplll
errors hut in the meantime it is best lo relain the existing
identilicalions even if wrong, rather than switch lo coJT<..'CI
assignments which could ha\e li1r1her unpredic!ah!c dkc!s
when added to 1hc crron<..'OUS inplll cvcms. Thcrclixc once a 45
chord slide has generated input cvcms. 1hc idcmiEcalions of
their existing paths may he locked so the hand identilicalion
algorithm can only swap identilicalions of subsequent nL'\\"
comac1s.
40
they can continue it. Step 488 retrieves the mcasur<..-d posi-
tions and si1es of lingers and palm heels which are touching
the surliJCe.
Step 489 computes a knuckle liJCior from the outer linger
sizes and their vertical distance from the palm heels which
peaks as 1hc omcr !ingcr contacts become larger !han nonnal
linge11ips and dose lo the palm heels. Step 490 computes a
julling bclor Ji-om the dilkrence belween the ve11ica! coor-
dinates of the inner and ouler lingers which peaks as the index
Engcrtip juts limhcr om in lfont of the knuckles. Step 491
,
0
combines the knuckle and jul1ing lUc1ors in a fut/)' logic
expression and a\erages the result with pre\ ious results\ ia an
auloregressi\e or moving average liher. D<..>cision diamond
492 continues or starts pen grip mode if the !il!cr<..-d expression
rcsuh is above a 1hrcshold which may itself be variable 10
"111is hand idcnliEcmion process can be modiEcd lix
fcrcnlly configured mul!i-1ouch surfaces and remain well
within the scope of this imenlion. !:or surbces which are so
narrow that thumbs invade one another's space or so !all that
one hand can lie above another. 1hc comours nc<..-d no1 be
slraig.!H vertical lines .. \ddilional contours could weave
around candidate merlapping thumbs. or t!u:y could be per-
pendicular lo the VL>clor belween the eslima!L'<i hand posi-
tions. I fthe surli1ce was large enough ll1r more than one user.
additional altraclor rings would have to be provided for each
additional hand. and muhiple partitioning contours would be 60
n<..>cessary per hypothesis lo p;u1ition the surli1ce into more
than two portions. On a surli1ce large enough ll1r only one
hand it migh1 still be ncccssm)' 10 delcrminc which hand was
1ouching the surface. Then ins1cad of hypothesizing dillCrcm
comours. 1hc hand idcnti!icmion module would cva!umc the 65
hypotheses that either the !ell hand allraclor ring or the right
hand allraclor ring was centered on the surliJCe. If the surface
,
5
provide addi1ional hys1ercsis. While in pen grip mode. typing
12 and chord motion r<..>cognition 18 are disab!L-d ll1r the pen
gripping hand.
In pen grip mode, decision diamond 493 de!ermines
whether 1hc inner gripping lingers arc acmally 1ouching the
surlUcc. If so. step 495 generates inking cvems from the pmh
paramelers of the inner lingers and appends them lo the out-
going even! queue of the host commtmicalion interli1ce.
These inking events can ci1hcr cause "digiwl ink" to be laved
on 1hc display 24 for drawing or signmurc capture purposes.
or they can be imcrccptcd by a handwri1ing recognition sys-
tem and inlerpre!ed as gestures or language symbols. !land-
writing r<..>cognition syslems are well known in the art.
.!:lljliO.! paxg )SU'>\ :lljlll' p.<l:J)U:l:l Sl Ull!li'JO.!
atpjt Wt]l Jtbaxa Tiu!.1np .1naao 1'uam<.maqd
.!BJ!m!S Jll::l!l.!;)A );)U ll 0) dn f!U!]BJS ptmt]
ll ilU!Jilp S;)!JpOja,\ Jaf.U!J [ll.!llWJ JO tuns at[l pur,
qmmp pur, .\)[U!d at[l ua.."').\\)aq a:ma;a.mp at[l UBt[l ssa[ .\[[rnsn
JaTiug J:ll]JO .\tm ptm 1' Jll mlt]om uaa.Lpq
a:maJ:l.J.JlP atp amp.!:ll!J. JlO.!JU:J:J :lljl JO ilmxau
atpmama[dtuoa 01 ilu![mJ ummaJ tu[Bd at[l
1nq 1mod WJJuaa r p.!B.\\lll aAom os1r toz-zoz WJJuaa
:ll!J. "1\U!Jl':lS ]aU a1\.1q I' SUll!JOlU .!l:lljl ll! :J:JU:J
.!:l_JJ!P :lljllllq ]aU lU:JZ .H:J S[:l:llm:J SUll!JOlU .!l:lljl
Jll ums :lljlll"[l os p."')ads :lljl .\[qTimu w suo!P:l.l!P
-oddo .(pr,au U] [a,wJt soz ,\.'imd pur, JOZ quHU]l atu.
purt[ a,\]PBJ)UOJ r, ilupnp Jailuu tpra JO sauopafBJl
Sf'!)!:! "O.!:JZ lll :lljlJll S:lll!:lll[:l\ Jl"lO!WJO.! ptm ]Btpm
atp tZS da]s :lljl Tiu!l(Jlllll o.LJ
)Bt]l zzs puommp Jl qmm[J at]llHOt]l!.\\
)[JO.\\ ][!.\\ potj)atu at]l mq qmmp a]qrsoddo
at[l S] JO auo J! m.!OJ.!ad 01 Jasn atp Ja!sra
ptm sTIU!Jl':lS :JUOJO:l.!l' .\:Jtjl S.J:Jf!U!J
J:ltjlll :Jt]lJO ]uapuadapu! a.1om tpmu UO!JOm qmnt]l aau!s
puBtj at[JJO .\)!::10];),\ .!lljlliltm .!0 ]mpB.! ll ;))lldtuOJ 0)
-l,llU ]J..'"Ofqo.Jll Jl':l!lJdl!!TI :lljl lll .\p\!JIUTIO:l
.\:Jt]l qmnt]l a[q1"oddo :Jt]l apnpu! .\p\ll!lllU!
tp!t[.\\ azm1n ,\atp uuo_pad 01 suo]wm
pum[ asat[l aJe .\[UO JON mu r, uat[.\\ se sJailuu at[l
uaa.\\)aq Jo pq Jde uat[.\\ lS].!.\\ atp)noqe Jatpw
UO!ll'ltu pm"l ptm s.1ailug :Jt]l ilmpua]xa .10 Tiu!X:l[J .\[tti.!(:J
lUll .\q f!U!Jl':lS ptmlj "Ull!JqSutOJ) ptmlj :ljOlj.\'\. ;,q UlD SUO!JOlU
09 at[l papaau aJe mopaq] JO moJ .(J<!O uat]/1:\
amn auo w mop."')a.!J JO aaJilap Jo S!XB
I' 1JUOJl' aq OJ UI"Jl .!:llj)l!l SUO!JOlU
Jl"lllTII'!P WlJII' .\:Jt]l asmDaq "-'"'P
111dm ptm ":J[qtw,das tmqJ J:Jt]Jl!l(l!JTia]m ;,q 01 P!l"
,\at[l atu]l at[l w aq tmamopaa.!JJO saaJilap a]d]l
-[nm uat]/1:\ ;at]lOtm auo t])l.\\ iluuaJJaJm mot[ll.\\ amp ames
.\(!SI':J pamn:p;,\.1 ;,q UlD lJ:l!l]'>\ SUll!JOlU
lUO.!J Ulllfhh<l.J Jll ')j7 :JS:lljl Pl!l]X:l ll)
JO]Jll.!)Xa )lWUOdtuOJUO!)OUl at[.L [eJ]).!;)A pur, [eJUOZ!JOl[
0> at]) punOJB ajqel!e,\B aJe tu0p."')a.!J JO Sa.."')Jilap [eUO!W)OJ [BUO]l
-!ppe 0.\\) 'SUO]SUaUHp aaJtj) U[ .!0 f.U]UlOOZ pur, f.u]
ll'lll.! "f!U!utmd Jl':l!U:l \ ptm UlllUlUO:l )SOlU
:lljl 0.\'\.] U] :h'"O.!ljl ptm 0.\'\.] Ul [[(l!l
;)Jll SJaf!U!J f.U]PlllUOJ W UO])ll]U;)UO qttHU]l
-e[ndmr,m [BJ!t[deJil uommoa .!OJ mopa.."')JJ JO t[ilnoua
mopa.."')JJ JO sa.."').ri!ap 91 at]l mOJJ JJeJtxa 01 s1 91
:l(llpom Ull!Pl!l]X:J )U:JUOdtull:J Ulll)lllU :lljl JO :JSOtf.md :lllJ. lUO.!J p."')JJIOJ]X:l UO!ll'llU "UO!)OlU ptm
(f!U!(lDS) (I'!PIOJ UllUJ S)U:JUOdtutD .\]Dllp\ Jqod :lljl
"9f '!)!:! ll! U.\'\.OljS :l.!l' (h"')[mPp :JSOlj.\'\. "f:"()S SS:l:lllld
.\[JUapuad
-apm SS:l:l:ll' ll) .!:JSll :J1JIOJ:JW :lljl lUO.!J .{]IJ:l]X:lp ptm f!U!JS! '>\)
a[qruosea;un aJ]nbaJ p[nO.\\ pue pawm1 a,wt[ mop ")Se[ U.\\Op
ilmtpnOJ Jo JJO ilmlJ![ dp.wiluu atp JO )[Jeq at[l 01 anp
Jl]TII:llj Pl'llllD m aTitmtp:ltjl OJ Jl"lO!].llldOJd S!.J.JlllJI( J:Jf!U!J .\q
.\]DO(:l\ Ul Ull!ll,qJn]Jad :lljl Wljl
'lt[ilWt] Jaemo::> Jailuu U] ailurtp atp 01
[BUO]Uodwd )mwmr, ur, paads Jailuu jeJaW[ at]l tuO.!J ilmpBJl
,\q pammqo aq UBJ asp,"')Jd aJotu .([aA!WU
ot -a.."')JJ JO JO .\tmm ;a,\a.\\0[] uo]W!Uauo qmm[t
JO lUO(h'"O:l.!J Jll :J.'"O.f.lap:JUll ":J:ll,)JllS :lljl OJ .{]llUIXlUd JO:J.!llS
dtp:JTiug Jl'llPL\tpm m mop.h'"O.!J.Jll saa.1ilap aat,pns
at]) OJjeJaW[ UO!)OUl U] tuopa.."')Jj JO lWl p[aj.( ;)Jllj.!llS
r, uo SJJBJUOJ Jailuu JO S)UatuaJnsr,aiii,J .\jnwpuadapm
-.!:ll]V Tiuoum .\J!Ul!XOJd m JU:JJJad
:Jt]l 01 m auop .\J!:lllP\ Jailug
JO ilmwasU.\\Op [r-UO]llppe JJO Jailuu auo
01 ptwJ purtJ e U]t]ll-\\ Jatpo aau1s .\Jlm]xtud
atp m 01 uop.wdoJd m .(JpopA jeJaW[
at]lJO IW aMltu Oll[HJHJIP 11 a)[Btu SUO]Wl]UHJ ailB)[tH] uoptwl
jlDI]:JIOJd I' llllJ paU!lJlUlD S)Ulllf Ul )U:llU
-a.\om .1 o .1 o tz: Sl"[ .\]Jl':l!utpa L
at]l .\q aq .\J![Bnpr, m ][!.\\ aatlJ.!a)U] uo1wam
-nmmo::> )SOt] at]l 01 tped d1uailuu xapu] at[l )et[l )daaxa
oc sn[.\Js e t[ll.\\ at]l uat].\\ .(B.\\ ames atp .\[[mmassa U]
tm:J Jopapp duil uad :Jt]l sananu"] :ll(JJO (:l\:J]
Jl':l!lJ:l.\ :lljl :J.\OlJI' JO Ull!ll':l!JI]U:lp!
d]uailuu xapu] at]l JO a::>e[d at]l sa)[m .\[[e]wassa
at]lJO d]l atLL m]ed Jamo at[l 01 papna spr,wo::> an::>nU)j
I' f!Ul(lDSU.\'\.Op .\jut!llldtu:J] .\q p."')JU:l\."')J\.1 J)lll.J!J J:Jf!U!J
fluunp Ullll!SOd (h"')J:l:l(:lS I' dumfOJJOS.lll:l:lljl :JSlllD
,\atp J! Jasn at]l .(omm UBJ .\atp mq "[OJJUOJ ilu]uop!sod
auu .!OJ d]uailuu at]l ilm[[OJ uat].\\ asn pooil 01 md aq
utD SUll!JqSut!l] ]Uapuadap-.\]ilUIXlUd l(JllS pttu]U:l:l J:Jf!U!J
:ll]l ll! jlO.!:JJq p.!1,'>\dl1 .!0 ]aU
dtpaTiug :Jt]l JO :Jt]l .J.Jll ptm a.mpq tt.wlp
uauo d]uailuu atp JO pr,d JUtUJ at]l at]l uo
U.\\Op tpnoJ pur, JJ0\1![ Jailuu iluunp asne::>aq s1
.\[p!diOJ flu!fltmtp S! .{J!UHX<.Ud J:JTIU!J U:llj'>\ PI')Ull:l flC
(l!l:JW] :Jt]l 01 Jlmll!liPPI' zos da]s ]xau :Jt].L
,:c :JTI.1q ]]!'>\ .\"(md ptm Tiuu :Jt]l .\"{md :ll(l
01 p;,dsaJ l]ll'>\ (l'lUJ<.m :J![.\1'm (:h'"Ot] mp'd
Ja]no :ll(lt]Tinmn '(:h'"Ot] mp'd J:JJllll :Jt]l ptm J:JTiug Tiu!.! .\"{md
at]lJO atp uo purt] atp a]!l].\\ atp_uo ,\e)s
pazJsetjduwap aq ur,a )Uauodmoa UO]JOm Je[
-na!prd apr,;ilap p]nO.\\ tp!t].\\ S]a.."')t[ ttqed Jo U]epaaJo
uonom Wt]l m papaau a;r, at[l pue
Ulll)lllU :lljl Ul aq OJ S:li]IUHXlUd ptm S:li]DO(:l\ jlO.!:JJq ,:1
:ll]l :JS:ll!J. ptmlj U:l.\11\ :lljl ()Sl Pl'llllD
pamwap] at[l satp1q1 lSJH nos da1s pf "!)1:1 m u.\\Ot]s aJe
91 JO]Jll.!)Xa )lWUOdtuOJ UO!lOUl at[l U]lj)l.\\ atlJ.
"J:ll]Jotm auo 01
.!!-'"''11 uo ilmpu;,d;,p .\JDO(:J-\ .\j.m:Jm[ ol
-uou ]saq (:l:lutD asotj'>\ s.1ailug :Jt]l
,\[[mJUa.!ap.!d JOJJeJtXa UO!lOUl at]) 'SUO!W)OJ pur,
pauuo_pad .([Pqpadtm tu<.UJ papuanmm ltWAaJd
ll) p.{ (1!11\:J]UI JO :lljl Wljl OS
]l1lU!S aq 01 .LOJJl' OJ. flu!Jl':ls .10 m111 ,.-
-mw at[l 01 UO]llppe U] uop::>a.!!p amm m
)au eo1 dn UBJ pur, .\p::>a_pad [a::>ueJ wu
pue SUO]WltU puet] iluunp atp 11e JO
(lmO!JqSUIO.!] :lljlll"(l Sl .\[UO Jl:ll!J. 'illii/!S ptm Ulll)
ZP
,\em pat] m]ed Jaum at[l e ilu]p[Ot] .\urnpr, uat].\\ a; om
:Jll"l!dns 01 spua1 mJI'--'H:J :Jt]l :J:lU!S Tiu!.1np aat,pns
:lljl flllll(Jllll) lml(J :l:ll,J.!llS :lljl :J\Oql' lUO.!J :ll(l
ilu]JJ.."').!]p aq m.\\ sJailuu ilmdduil JaUU] atp .\[ll[il![s
ailtmtp m.\\ uope;niluuoJ puet[ atp 11 t[ll.\\ aJU.\\ 01 1dmaur
pur, a,\!)Jnpuoa r, dn )[J]d .([[emae atp p]nOt[S
pa.1pum[
.\\.'"0.) I' fA'"OlJI( U:J:Jq :JWl( J:JUU! :lljl ]!lUll pa.!tlllfl!
aq p[HOl]S Ul[Bd J;)JilO pur, m]L>d at[l
JO apnpm OSje ][!.\\ SUO]JOm f!tHJ]J.\\ [BUUOU ;)JU!S
-"]:lntl"{ J:JJllll ptm Sj:l.'"Olj :ll(JJO Tiu!ml!l
JosJn:J Tiu!JIOJ:Juail .\q p.--.poddns aq tma S!lJ.L
-um d!.!TI u;,d :Jt]l Tiu!.\l':lJ Jlllll]l!-L
01 a[qr;ap.!d )! p;otp Jailuu e ilu]pqs pue uope;niluuoJ
duil uad at[l ilmABa[ ,\q JOSJilJ atp aAom p]llOJ e aaBJ.!IlS
tpmlJ-!l[nm uo t]ilnmu. .1osm:J ilmum1!sod .\q
:lljl Ull aq ll) :J.!I' p.'"Op.!dl:J)UI JO .. "]U! Jl'llfltp,
at]l aJat[.\\ awa1pm 01 p."')au .(r,m at]l suoJwaqddr, U[
'S]Oqtu.(S ua,."'),\\)aq )[eaJq eJO UO!l]<!f.OJ;)J f!Ul)!J_\\pUet[
at]) 01 ;)Jll_paHH UO!WJ!UilmtuOJ )SOl] at[l 0) S)lWAa
pas!IO.! t6t da]S "fA'"OlJ!( am Kl:JTIU!J .!:lUll! :lljl J]
lP
Z8 8Z8'Z!8'L sn
us 7,812,828 82
43
centered at the a net horimnta! translation will
appear in 1hc sum of motions li"om any combinmion of !in-
gcrs.
Since 1hc differences in !ing.cr motion arc usually grcmcst
between thumb and pinky. step 526 only rctric\cs the ctuTcnl 'i
and pre\ ious positions of the inncnnosl and oulennosltouch-
ing. !ing.crs for the hand scaling and rotmion mcnsurcmcnts.
Step 528 then computes the hand scaling velocity! I,, from
the change in distance between the innermost linger!'! and
oulennosl !Inger FOwith approximately the l(J!!owingcqua- lCJ
lion:
(66)
44
simplest way lo compule a hand lranslalion \elocily would he
to simply averngc the lmcral velocities of each !ingcr. How-
ever. the user expects the motion or control to display gain to
be constnm rcg.nrdless of how many !ing.crs nrc being. moved.
even ifsmne arc resting stationary. l'urthcrmm-e. if the user is
simultaneously scaling or rotating the hand. a simple meragc
is sensitive lo spurious netlranslalions caused: by unc;mcck-d
central !ingcr motions.
Therefore. in n preferred embodiment the trnnslmional
component cxtrnctor cardi.J!ly assigns wcig.htings for cnch
!ingcr before compming. the nverag.c translation. Step 540
initiali/cs the lranslalion weighting 1\, of each linger lo its
lola! conlacl proximity. i.e .. 1\., [ n] Fi .[n ]. This ensures thai
where d(F![ni.FOinJ) is the squared
between the lingers:
l;udidcan distance
15
!ingcrs not touching the surface do not dilute the nverag.c with
their zero velocities nnd thm !ingcrs which only touch lightly
hme less in!luence since their position and velocity mcasUJ-e-
menls may he less 1diahle. The ncxl slcp 544 deCJ-eases the
weightings of lingers which ;u-e Jdati\ely stationary so thai
dtn fn
1
, FOfnl vc,c,c,
1
c,c
1
c,o,c,,c
1
,c,
1
c,,c,c,c,c,
1
c,c
1
c,c,c,c
1
,c,
1
c,,
1671
If one of the innermost or oulermosl !ingcrs was not touch-
ing. during. the previous proximity image. the chang.c in scp.1-
ration is assumed to be zero. Similarly. step 5JO computes the
hand rotational velocity II,, lium the change in angle between
the innermost and oulcmmsl linger with approximately the
I(J!Iowing equation:
"" the comrol to displny g.ain of imcntionally moving. !ing.crs is
not dilut<.-d. This can be done by !inding the lUstcst moving
linger. recording its speed as a maximum linger speed and
scaling each linger's lranslation weighting in proportion lo its
sped divid<.-d by the maximum of the !ing.cr spc<.-ds. us shown
"" npproximately in the formula below:
(
. {/" 1[ 11 [. HJ[n[l-: {1-/[n- I[. HJ[n- l [I, ( HJ[n]l i6Si
30
1/,[n] .!.J "'
"ll1c change in angle is multiplied by the current separation
(
f"i,."'""J[II[ ]'"'"
f"i.,.[11[: =f"i.,.[II[X " ..
..'"'l"l
16'!1
where the power ptw adjusts the strength of the speed depen-
dence. Nolelhal slcp 544 c;m he skipped ll1r applications such
as compuler-aidcd -design in which users desire both a nonnal to convert it to the same units as the translmion and scnling
components. These cqumions capture nny rowtion nnd scnl-
ing. components of hand motion even if the hand is also
translating as a whole. thus making the rotation and scaling
degrees of liccdom integral with lranslation .
35 cursor motion gnin modcnndn low g.nin modc.l.owcr cursor
motion gain is usdi.J! for !inc. short rang.c positioning.. nnd
would be accessed by moving. only one or two !ingcrs while
keeping the resl stationary.
.-\.J1othcr rcnson the compuwtions above arc to the
thumb and pinky or inncnnost nnd outermost !ing.crs is thm 40
users may wanllo make line lranslaling manipulations with
the central lingers. i.e .. index. middle. and ring. while the
thumb and pinky remain stationary. lfchangcs in distances or
ang.lcs between the ccntrnl !ing.crs nnd the thumb were avcr-
ag.cd with Equations 66-68. this would not be possible 45
because ccnlral linger translations would cause the appear-
ance of rotation or scaling with 1-especl lo the stationary
thumb or pinky. !lowcver. Equmions 56-60 applied in the
thumb verification process nrc only sensitive to synunctric
rotmion and scaling about n !ix<.-d point between the !ing.crs. '0
"!.hey approach zero if any signilicanl whole hand translation
is occurring or the linger motions ;u-e nol complementary. In
cnsc the user fails to properly move the outermost !ingcr
during. a rotmion or scaling g.csturc. step 531 uses equations of
the approximmc fonn ofl:quations 56-60 to compute rotmion '5
and scaling velocities between the thumb and any touching
lingers other than the outermost. The resulting velocities are
prefcrnbly combined with the results of Equmions 66-68 via
a mnximum opcrmion rmhcr than an nveragc in cnsc trnnsln-
tional motion causes the lixL'<i point rotations or seatings lo he 60
zero. Finally. decision diamond 5J2 orders a check ll1r radial
Step 546 decreases the lranslation wcighlings ll1r the ccn-
lral lingers during hand scalings and rotations. though it does
not prevent the central !ing.crs li"om mnking. !inc trnnslmional
manipulmions while the thumb and pinky nrc stationnry. The
l(mnulas below accomplish this scamlessly by downscaling
the cenlral!ranslalion weightings as the magnitudes of the
rowtion nnd scaling velocities become sig.n.i!icant compnrcd
f"i, .. [n[X
Fi., +111"1"11
.,,,,.,,
[II[= +111"1"11 + [l/,.[11][
17111
1711
where these equations ;u-e applied only lo the central lingers
whose identities i are he! ween the inncmmsl and oulennosl.
Note that since hand scaling docs no! cause much horizontal
translation bias. the horizontaltrnnslmion weighting. Fi,, ,[n I
need not bcnffcctcd by hand scnling. velocity ll,,[n]. ns indi-
ca!<--d by the lack of a hand scaling term in l;qualion 70. The
lranslalion weighlings of the inncmmsl and oulcmmsl lingers
nrc unchanged by the polar component sp<.-cds. i.e .. !'I,,.,[ n]
Fl,,Jn] !'l,.,[n] nnd !'O,,[n] !'O,,,[nl !'O,,[n]. Step
or rolalional deceleration 5J4 during motions prior lo linger
Jill off The method fordct<..>cting. rndinl or rowtionnl dccelcrn-
tion is the snmc ns thnt dcwilcd in the description of trnnsln-
tion exlraclion.
I'Ki. J7 shows the details of hand lranslalional \elocily
measurements referred to in process 506 of FIG. 34. The
65 548 !innlly computes the hnnd trnnslntion velocity vector
(ll,.,[nJ.II,.,.[nJ) limn the weigh!<--d average of the linger
velocities:
us 7,812,828 82
45
17"1
1731
"111c last part ofthc translmion cn!culmions is to test for the
lateral dccdcralion of the !ingcrs hdlm:: lilloJL which reliably
indicates whether the user wishes cursor motion lo slop a!
li!loff !fdccdcralion is not de!cclcd prior lo !i!lofL the user
mny intend cursor motion to continue nlicr liftoff. or the user
mny intend a special "one-shot" conummd to be invoked.
Decision dimnond 550 only invokes the dccdcration tests
while !Inger proximitics arc no! dropping too quickly. to
prcn::nl the pcJ1urhations in linger centroids which can
accompany linger li!iolf li'llm in!erlCring with the dccdcra-
tion measurements. Step 551 computes the pcrccntngc ncccl-
crmion or ratio ofcurrcnttrnnslmion speed I I I,,[ nl.! I, ,.J nl)lto
a pas! mcragc translation speed prckrably computed by a
moving window average or autoregressive !!her. Decision
diamond 552 causes the translation deceleration flag lobe sci
556 if 1hc accelcrmion rmio is less than a !hrcshold. If 1his
1hrcshold is sci g.rcmcr !han one. the user will have to be
accelerating the lingers jus! prior lo li!ioff ll1r cursor motion
lo continue. If the threshold is sci jus! below one. cursor
mo1ion will reliably be continued as long as 1hc user main-
wins a conswm lateral sp<..-cd prior to lilioJI but if 1hc user
begins to slow 1hc cursor on approach to a large! area of the
display the deceleration flag will he sci. Decision diamond
554 can also cause thcdccelcralion flag lo he sci if the ctuTcnl
1ranslmion direction is substamially different Jh.1m an average
of pas! directions. Such change in dir<..><::lion indicmcs 1hc hand
motion lrajcclory is cuning. in which case cursor motion
should no! be continued alkr lillolfhL><::ausc accurately dc!cr-
mining the dirL><::Iion lo the user's intended large! becomes
vet)' dillicuh. If nci1hcr dccelcrmion nor curved 1rajcctorics
arc dc!cc!cd. s1cp 558 clears the 1ranslmion d<..><::clcralion Hag.
This will enable cursor motion nmlinualion should the !in-
gcrs subsequently begin lilloll No!c that decision diamond
550 prevents the slate of the translation deceleration !lags
from changing during lilloffso 1hat1hcdccision allcr lilioll"!o
cominuc cursor mo1ion depends on 1hc stmc of1hc decelera-
tion flag Iillo!"!" began. The !ina! step 560 updates the
autoregressive or mm ing window average oft he hand lrans-
lalion velocily V<..><::lor. which can b<..><::omc !he veloci1y of con-
1inucd cursor mo1ion after lilloff .. \clual gcncrmion of the
cominucd cursor mo1ion signals occurs in the chord mo1ion
rL><::ognizcr 18 as will he discussed with l'Ki. 40.
'
"'
46
Pal. No. which if enabled ll1r low linger speeds
will undesirably continue mo1ion when a user dccelcrmcs on
approach to a large target but JUils 10 slop complc!Cly bclixc
liliing off. l'crccnlagc accclcrmion also captures user intent
more clearly than positionofa linger in a border area. Position
of a linger in a hon.krarca as usL-d in U.S. Pal. No.
to Gillespie c1 a!. is ambiguous because 1hc cursor can reach
its desired target on 1hc display jus! as the linger cmcrs 1hc
border. yc! the touchpad de\ icc will continue cursor motion
past the large! hL><::ausc it thinks the linger has nm out of space
to move. In the present imcnlion. on the other hand. the
acceleration rmio will remain ncar one if1hc lingers can slide
oil" 1hc edge of the sensing array witholl1 hilling a physical
barrier. sensibly invoking motion ctmlinualion. But if the
15 lingers dL><::clcralc hcl(Jrc crossing or slop on the edge of the
sensing array. 1hc cursor will slop as desired.
The dcwils of the dilkrcntial hand pressure extraction
process 508 arc shown in FIG. 38. Fingertip proximity.
quickly saluralcs when pressure is appliL-d through the bony
"" tip nonnal!o a hard surliJCc. Unless the surli1cc itself is highly
compliam. 1hc bcs1 dynamic range of lingcnip pressure is
obwincd with the lingers outs1rc1Chcd and hand nearly Hm-
lcncd so thai the compressible soll pulp underneath the Jin-
gcrlips rests on the surface. DL><::ision diamond 562 thcrd(Jrc
"" causes the tilt and roll hand pressure components to he scllo
zero in s1cp 564 and pressure cxtrac1ion to abort unless the
hand is nearly !lal1cncd. lnhcrcm in 1hc !cst!(Jr hand !lal1cning
562 is a linger counllo ensure that most of the Jive lingers and
both palm heels arc touching the surli1cc lo maximize the
30 precision ofthc hand pressure measurements. though tcchn.i-
cally only 1hrcc non-collinear hand contacts arranged like a
tripod arc necessary 10 establish till and roll pressures. Deci-
sion diamond 562 can also require the user to explicitly
enable three-dimensional manipulation with an intuitive gcs-
35 lure such as placing all live lingers on the surJUcc bric!ly
tapping 1hc palm heels on 1hc sur!Ucc. and linally rcs1ing the
palm heels on the sur!Ucc. J)ccision diamond 566 causes step
568 to capture and store rdl:rcncc proximitics l(H each con-
lac! path when the proximity of all conlacls have stabilitcd a!
40 the end of1his ini1imion S<..'<.JUCncc. The 1ih and roll pressure
components arc again tcrocd 564 for the sensor array scan
cycle during which this calibration is pcrllmnL-d.
I lowe\ cr. during subsL'<.jUCnl scan cycles the user can tilt
the hand J(m.vard applying more pressure to the Jingcrlips or
45 backward applying more pressure to the palm heels or the
user can roll the hand outward omo the pinky and outer palm
heel or inward applying more pressure to the thumb. index
linger and inner palm heel. Step 5170 will procL'L""<i lo calcu-
lmc an unwcigh1<..-d average of the current comac1 positions.
'0 S1cp 572 computes Jix each hand pari s!illtouching 1hc sur-
JUcc the rmio of currcm proximity 10 1hc rcll:rcncc proximi1y
pre\ iously slorL-d. To make these ratios less scnsili\c lo acci-
dcn!allilling of hand parts. step 574 dips them lobe greater
or equal to one so only increases in proximity and pressure
'5 register in. the till and roll measurements . .-\.Jlo!hcr average
conlacl path position is compu!cd in step 576. hut this one is
wcigh!cd by the abmc computed proximity ratios l(H each
path. rhc dilkrcncc he! ween these weighted and unwcigh!L'<i
contact position averages wkcn in s1cp 578 produces a vcc1or
Note that this cursor motion continuation nwthod has sev-
eral ad\ anlages over motion ctmlinualion mc!hods in related
art. Since the decision to continue mo1ion depends on a pcr-
ccnwgc accelcrmion which inherently normalizes to any
speed range. the user can intentionally invoke motion con-
tinuation Jium a wide range of speeds including very low
speeds. "llllls the user can directly invoke slow motion con-
1inualion 10 mno scroll a document a! readable speeds. This is
no11ruc of\Vmanabc's mc1hod in U.S.l'a1. No. 4.734.685. 65
""hich only continues motion when the motion cxccL-ds
60 whose dircc1ion can indicmc 1hc dir<..><::lion of roll or till and
whose magnitudcc;m control the ralcofroll or tilt about x and
y axes.
Since 1hc weighted and unwcightcd posilion averages arc
only in!lucnccd by positions of currently conwc1ing lingers
and increases in contac1 pressure or proximity. the method is
insensitive to linger Cmnpulalion of rcJ"crcncc-nor-
malitL-d proximity ratios in step 572 rather than absolu!c a high spcL-d threshold. nor of l.ogan cl a!. 's n1cthod in U.S.
us 7,812,828 82
47
changes in proximity pn .. '\cnls the large palm heel conlacls
from having undue inHucncc on the weighted average posi-
tion.
Since only the current contact positions arc used in the
mcragc position cmnpulations. the roll and tilt vector is indc-
pcmk:n! of!atcral motions such as hand translation or rotation
48
Step 512 takes a second pass with a related !!her known as
n dead-tone !i!tcr .. \dead-zone !iller produces zero output
velocity Jix input velocities less thnn a speed threshold but
produces output speeds in proportion to the dilkrcncc
'i between the input spL-cd and the threshold ll1r input \docitics
that exceed the threshold. Prckrahly the speed threshold or
width of the dcnd zone is set ton frnction of the mnximum of
current component speeds .. \ll velocity components arc !i!-
as long as the lateral motions do not disturb Hng.cr pressure.
thus once again achieving integrality. I lowcvcr. hand scaling
and differential hand pressure nrc dillicu!t to usc at the same
time because Hexing the !ingcrs generally causes signilicanl lCJ
decreases in Jingcrlip conlacl area and thus intcrkrcs with
inference ofHngcrtip pressure changes. When this becomes a
serious problem. a tow! hand pressure component can be used
tcrcd using this same dead zone width. The !ina! extracted
component velocities arc llHv.ardcd to the chord motion rcc-
ognitcr module 18 which will determine what if any input
events should be gcncrmcd li"om the motions.
]-'](j. 39.-\ shows the details of the !ingcr synchronization
dclL-ctor module 14. The synchnmi111tion detection process
described below is repeated !l1r each hand independently.
Step 600 !Ctchcs proximity mnrkcrs and idcntiEcntions for the
hand's current paths. The idcntiEcations will be ncccssm)' to
ignore palm pmhs and idcntil)' combinations of synchronized
lingers. while the proximity markers rL-cord the time at which
as a sixth degree of frc<..'dom in plncc of the hnnd scnling
component. This total pressure component causes cursor l'i
\docity along a z-axis in proportion to deviations of the
avcrngc of the contnct proximity ratios li"om one . .-\ltcrnmivc
embodiments mny include li.mhcr enhancements such ns
adapting the rc!Crcncc proximitics to slow \ariations in rest-
ing hand pressure and applying a dead zone !!her to ignore
pressure di!krcncc H.-ctors with small magnitudes
"" each contact path !irs! cxcc\.:ds a press proximity threshold
nnd the time nt which each contnct pnth drops below a rclcnsc
proximity threshold prior to totnllilloll' Setting these prox-
imity thresholds somLwhat higher than the minimum prox-
imity considered signi!icant by the segmentation search pro-
"-' cess 264. produces more precise !Inger press and release
times.
Despite the care tnkcn to measure the polar velocity. trans-
lation velocity. nnd hnnd pressure components in such n wny
thm the rcsuhnnt vectors nrc independent of one another.
uneven linger motion during hand scaling. rotation. or trans-
lation can still cause minor perturbations in measurements of
one degree of li"ccdom while primarily attempting to move in
another. Non-linear Ehcring npplicd in steps 510 and 512 of
F!(i. 34 J"C!llllvcs the remaining motion leakage hetwL-cn
dominant components and nearly stationary components. In
30
steps 510 each component velocity is downscaled by the ratio
Step 603 searches Jix subsets ofEngcrs which touch down
at about the same time and !l1r subsets oflingcrs which li!l off
at about the same time. This can he done by recording each
Enger pnth along with its press time inn tcmpornlly ordered
list us it crosses the press proximity threshold. Since the
primal)' li.mction oft he palms is to support the Jixcnrms while
the hands arc resting. palm activity is ignored by the typing 12
and chord motion rccognizcrs 18 except during di!krcntial
hand pressure cxtrnction nnd pnlm heel presses can be
excluded li"om this list and most other tests.
'](J check for synchronization between the two most recent
linger presses. the press times of the two most recent entries
in the list arc compared. If the dif!Crcncc between their press
times is less thnn a temporal threshold. the two !ingcr presses
nrc considered synchronitcd. If not. the most recent !ingcr
press is considered asynchronous. Synchronitation among
thrL-c or more !ingcrs up to li\c is !(mnd by comparing press
times of the three. llmr. or li\c most recent list entries. If the
press time of the most recent entry is within n temporal
threshold oft he nth most r<.x:cnt cnt!)'. synchronitnt ion nmong
of its nverngc spcL'<i to the mnximum of all the component
speeds. the dominnnt component speed:
( 11,--,..-,dll! r'
1741
II, ]11]: = II"]II]X

( 11,--,. . ,]11] r'
1751
II" ]11]: =II., ]II]X

( )'"''
1761
11"]11]: = 11"]11]"

( )'"''
1771
11"]11]: =11.,]11]>< .

where !l,,.,,ccdfnl. !l,,,.cdfnl. and !l,,,,.cdfnl nrc mnorcgrcs-
sivc nveragcs over time of the trnnslmion speed. scnling
speed. and rotational speed. where:
<lonuuanl fll.!X(II,,,w,dln].ll,,, ,.,,!n].
I 1,
1
,",1 n ]I
where pds controls the strength of the !iller . .-\s pdy is adjusted
towards inlinity the dominant component is picked out and all
components less than the dominant tend toward zero produc-
ing the orthogonal cursorclkct wdl-kmw.n in drawingappli-
cntions . .-\s pds is ndjustcd townrds zero the !iltcrs have no
elk-ct. l'rc!Crably. pds is set in between so thm components
signilicantly slower than the dominant arc slowed !lu1hcr. but
components close to the dominant in spL-cd arc barely
all<..x:t<.-d. preserving the possibility of dingonnl motion in
multiple degrees of li"ccdom at once. The mnorcgrcssivc aver-
aging helps to pick out the component or components which
arc dominant mer the long tcm1 and suppress the others even
while the dominant components arc slowing to a stop.
35
"
45
then most rL-ccnt linger presses is indicalL'd. '!l1 accommodate
imprL-cision in touchdown across the hand. the magnitude of
the tcmpornl threshold should incrcnsc slightly in proportion
'0 to the numbcrof!ingcrs being tested for The
lnrgcst set of recent !ingcr presses found to be synchroniz,-d is
recorded as the synchronitcd subset. and the combination of
linger identities comprising this subset is stored comcnicntly
ns a Enger identity bitEeld. The term subset is used bccnusc
'5 the synchronized press subset may not include nil Engcrs
currently touching the surfi1cc. as happens when a !Inger
touches down much earlier than the other !ingcrs yet remains
touching as thLy simultaneously touch down. An ordered list
of Enger relcnsc times is similarly maintained and scnrchcd
60 scpnrately . .-\ltcrnmivc embodiments mny require that n !ingcr
still be touching the surli1cc to he included in the synchro-
nized press subset.
Decision dinmond 602 checks whether n synchronization
marker is pending li"om a previous image scnn cycle. If not.
65 decision diamond 604 checks whether the search 603 ](lund n
nLwly synchronitcd press subset in the current proximity
image. If so. step 606 sets the IL'mporal
us 7,812,828 82
49
marker to the oldest press within the nL'\\" synchroniA:d sub-
set. Additionnl Hng.cr presses may be nddcd to the subset
during future scnn cycles without nffccting. the value of this
temporal synchronization marker. If there is currently no
!Inger press synchroni/alion, decision diamond 605 deter- 'i
mines whether three or more lingers have just been released
simultaneously. Simultnm.ous rdcnsc of three or more Hngcrs
should not occur while typing with n set of !ing.crs but docs
occur when !i!ling !ingcrs olrthc sur bee li'llm rest.
simultaneous release of three or more lingers reliably indi- lCJ
cates that the released !ingcrs arc not intended as kcyprcsscs
and should be deleted from the keypress queue 605. regard-
less of whether these snmc lingers touched down synchro-
nously. Release synchroni121tion oflwo !ingers is nol by itself
a reliable indicator of typing inlcnl and has no eflCct on the 15
keypress queue. The keypress queue is described later with
!-'IGS. 42-4JB.
Once a press synchronization marker for the hand is pend-
ing. !i.u1hcr processing checks the number of !Inger presses
which arc synchroni/L-d and wails ll1r release of the synchro- ""
nizcd ling.crs. If decision diamond 608 linds three or more
lingers in the synchronized press subset the user camlllt pos-
sibly he typing with these lingers. "!.hercl(Jrc step 612 imme-
diately deletes the three or more synchroni1cd presses limn
the keypress queue. This way llu:y cannot cause key symbol "5
trnnsmission to the host. nnd transmission of key symbols
from subsequent nsynchronous presses is not blocked wniting
ll1r the synchronizL-d lingers lo be released.
llowcvcr. when the only invohes lwo !In-
ger presses 608. it is dillicult to know whether the user 30
intended to tap n linger pnir chord or imcndcd to type two
adjaccm keys and accidcntnlly let the key presses occur
simultaneously. Since such accidental simuhanLous presses
arc usually !(JIIowL-d by asynchronous releases of the lwo
lingers. but linger pnir chords arc usunlly released synchro- 35
nously. the decision whether the presses nrc asynchronous
key taps or chord taps must be delayed umil linger release cnn
he checked ll1r In the meantime. slcp 610
places a hold on the keypressqucuelo prL'\enllransmission of
key symbols from the possible linger chord or any subSL'<.JUC!ll 40
linger presses. "](J prevent long. backups in key trnnsmission.
decision diamond 614 will e\enlually release the queue hold
by having step 615 delete the synchronized presses !i-om the
keypress queue if both !ingcrs remain touching a long lime.
Thoug.h this abons the hypothesis thm the presses were 45
intended ns key taps. the presses nrc nlso less likely to be key
laps if the lingers arc nolli!k-d soon a!ier touchdown.
50
laps. the synchronized !ingers cannot he held on the surbcc
more than abom halfn second if they nrc to qualil)'. usn chord
tnp. Decision dimnond 626 tests tills by thresholding the time
between the release of the last rcmni1llng synchronized ling.cr
and the temporal press synchronil<llion marker. :\ chord lap
slllmld also exhibit a limited amount of lateral !Inger motion.
measured either as an average of peak linger spcL-ds or dis-
tnncc trnvelcd since touchdown in dL>cision diamond 628. If
the quick release and limited lateral motion conditions nrc not
met. step 624 clears the synchronitmion mnrkcr with the
conclusion lhallhe synchroni1L-d !ingers were eilherjusl rest-
ing !ingers or pan of a chord slide.
I ftheclmrd lap conditions arc met. step 6JO looks up. using
the synchroni1L-d subset billield. any input events such as
mouse clicks or kcybonrd commnnds assig.ncd to the combi-
nation of lingers in the chord wp. Some chords such us those
includingniii(Jur ling.cnips may be reserved as resting chords
634. in which case dL>cision diamond 6J2 will !ind they ha\e
no associalL-d input e\enls. If the chord docs ha\e lap input
events. step 6J6 appends these lo the main outgoing event
queue of the host communication imcrlUcc 20. Finnlly step
624 clcnrs the synchronization marker in readiness llx li.Jturc
linger synchronizations on the given hand.
As a funhcr prL>caulion against accidental generation of
chord laps while typing. it is also uscli.!l ll1r decision diamond
6.U lo ignore through step 634 the !irsl chord lap which comes
soon alicr a vnlid keypress without a chord slide in between.
Usunlly nlkr typing. the user will need to reposition the mouse
cursor hcll1rc clicking. rL'<.juiringan inlcneningchord slide. If
the mouse cursor happens lo already be in place alkr typing.
the user may ha\e lo lap the !Inger chord a SL>cond lime !l1r the
click to be sent. bm tills is less risky thnn havingnn nccidcmal
chord tap cause an unimcndL'<i mouse bunon click in the
middle of a typing session.
I'J(j, 40:\ shows the detailed steps of the chord motion
rccogni1er module 18. n1e chord motion rL>cognilion process
described below is repented lix each hnnd independently.
Step 650 retrieves the parmnetcrs of the hand's idcmilicd
paths 250 and the hand's extracted motion components from
the motion extraction module 16. If a slide of a linger chord
has nol already sl;u1L"d. decision diamond 652 orders slide
initimion tests 654 nnd 656. "](J distinguish slides from glnnc-
ing ling.cr tnps during. typing.. decision diamond 654 requires
alleasllwo lingers from a hand lo he touching the surliJCc ll1r
slide mode lo stan. rhcre may be some exceptions lo this mlc.
such as allowing a single !Inger lo resume a previous slide
within n second or so after the previous slide chord !ills oll"thc
,
0
surlUcc.
If the synchnmiZL-d !ingers arc nol lilling. decision dia-
mond 616leavcs the synchronization mnrkcr pending so syn-
chronization checks cnn be cominucd with updatL'<i pnth
pnramcters 600 nlkr the next scan cycle. If the synchronized
!ingers arc lilling. hut decision diamond 618 !inds with the
help of the synchroni1alion release search 60J lhallhLy are
doing. so usynchronously 618. step 622 releases any holds on
the keypress queue nssuming any synchronizL'<i ling.er pnir '-'
was intended lo he lwo kLyprcsses. Though the synchronized
!Inger presses arc nol deleted from the keypress queue allhis
point. they may ha\ealrcady been deleted in step 612 if the
pressed subset contnincd more than two .. \!so. step 624 clcnrs
the temporal synchronitntion marker. indicming. thm no li.1r- 60
!her synchroni1alion tests need he done !lH this subset.
("onlinuing lo 1:1c;. J91S. if the !ingers synchronized during
touchdown nlso Jill simulwncously. step 618 removes them
and nny holds li"om the keypress queue in cnsc they were a
pnir nwaiting. n positive relcusc synchronization test. l'unhcr 65
tests ensue lo determine whether the synchronized !ingcrs
meet additional chord lap conditions. :\s with single linger
Inn prdCrrL'<i embodiment. the user cnn start n slide nnd
specify its chord in either of lwo ways. In the !irsl way. the
user slans with the hand !loaling ahmc the surliJCc. places
some ling.crs on the surlUcc possibly asynchronously. nnd
begins moving. nil of these lingers latcrally.J)ccision dinmond
656 initiates the slide mode only when signilicanl motion is
delL>cted in all the touching !ingers. Step 658 selects the chord
li-mn the cmnhinalion of !ingers touching when signi!icanl
motion is detected. rcg.ardless oftouchdown synchronitmion.
In this case cohercm initimion of motion in all the touching
lingers is su!licienl lo distinguish the slide !ium resting lin-
gers. so synchroni1alion of touchdown is nol nL>cessary. Also.
novice users may erroneously tl)' to start n slide by placing
nnd sliding. only one ling.cr on the sur!Ucc. forg.cning that
multiple lingers arc ncccssal)'. "](J!ernncc of nsynchronous
touchdown allows them lo scamlcssly coJTL'Cl this by subse-
quently placing and sliding the rcsl of the !ingers desired !l1r
us 7,812,828 82
51
the chord. The slide chord will then initiate without l(Jrcing
the user to pick up nll Hngcrs nnd swrt over with synchronized
!ingcr touchdowns.
In the second wny. the user stnrts with multiple lingers
n::sling on the surli1cc. !i!is a subset of these !ingcrs. !ouches a 'i
subset back down on the surbcc synchronously lo sdccl the
chord. and begins moving the subset laterally to initiate the
slide. D<..><::ision diamond 656 actually initimcs the slide mode
when it detects signi!icanl motion in all the !ingcrs of the
synchronized subscl. Whe!hcr the !ingcrs which remained lCJ
n::sling on the surliJCc during this sequence begin to move
docs not matter since in this cnsc the selected chord is deter-
mined in step 658 by the combination of !ing.crs in the syn-
chnmizcd press suhseL no! li'llm the set of all touching lin-
gers. This SL"Ctmd way has the advantage that the user docs no! l'i
hnve tolili the whole hnnd from the surlUcc before sinning the
slide. bll! cnn ins! end lcnvc most of the weight of the hnnds
resting on the surlUcc nnd only lili nnd press the two or three
lingers necessary lo identify the most common linger chords.
'!(J provide greater tolerance !(Jr accidental shills in resting ""
linger positions. decision dinmond 656 requires both thnt nil
rclcvnm lingers nrc moving nl signi!icnnl speed nnd 1hn11hcy
arc moving about the same spel.:d. This is checked either by
thresholding the geomclric mean of the linger speeds or by
thresholding the fastest linger's spL-cd and \erifying that the ""
slowest linger's speed is nl lens! n minimum li"nction of the
fnstcst linger's spL>cd. Once n chord slide is inilintcd. s1cp 660
disables recognition of key or chord laps by the hand a! leas!
until either the touching lingers or the synced suhsc! !ills o!f
52
cursor accdcralion paramclers ll1r each degree of li"L"Lximn.
These will be used 10 discrctizc motion imo 1hc unils such ns
nrrow key clicks or mouse clicks expected by existing host
computer systems.
Step 675 ofdmrd motion comersion simply picks the lirsl
slice in the given chord aclivity slmclurc !(Jr processing. Step
676 scnlcs 1hc currcm vnlucs of1hc cxtrnctcd vcloci1y com-
poncllls by 1hc slice's mo1ion scnsitivi1y nnd nccelcrntion
paramclcrs. Step 677 geomclrically projecls or clips the
scaled velocity components into the slice's ddinLxi speed and
dirL-clion range. For the example mouse cursor slice. this
migh1only involve clipping the rotmion nnd scnling compo-
nents 10 tcro. But for nn nrrow key slice. the trnnslntion
velocity vcclor is projL-c!L-donlo the unit vcclorpointing in the
same dircclion as the <llTOW. Step 678 integrates each scak-d
nnd projected component velocity over time in the slice's
nccumulmors umil dL>cision dimnond 680 dc1cnnincs miens!
one unil of motion hns been nccumulntcd. S1cp 682 looks up
the slice's prdCrrcd mouse. kLy. or three-dimensional input
even! !lmnal. allaches the number of accumulated motion
units 10 the cvem: nnd s1cp 684 dispntchcs 1hc cvcm 10 the
omgoing queue oft he hos1 commtmicntion intcrlUcc 20. Step
686 suhlracls the sen! motion events li-mn the accumulators.
and step 688 optionally dears the accumulators of other
slices. If the slice is intendL-d lo generate a single kLy com-
mnnd per hnnd motion. decision dinmond 689 will determine
thnt it is none-shot slice so1hm step 690 cnn disnblc Ji.Jr!hcr
even! generation from it until a slice with a dilkrenl direclion
intenencs. If the gi\en slice is the las! slice. decision diamond
692 returns to s1cp 650 to nwnit 1hc next scnn of the sensor
nrrny Otherwise s1cp 694 cominucs to imcgrntc nnd convert
the currcm motion lix o1hcr slices.
Returning lo F!(j_ 40A. ll1r some applications it may he
desirable lo change the sdec!ed chord w hencvcr an additional
!ingcr touches down or one of the !ingcrs in !he chord !ills oil".
llowcvcr. in 1hc prcJCrrcd embodiment. 1hc selected chord
cnm1o1 be chnngL'd nlicr slide ini1imion by nsynchronous En-
ger !ouch acli\ily. This gi\es the user li-L"L-dom lo res! or Jill
addition lingers as may he nL-cessary lo get the best precision
inn desired degree of freedom. For cxnmplc. even !hough the
!ingcr pnir chord docs no1 include 1hc 1humb. 1hc 1humb cnn
he sci down shm1ly alkr slide initiation lo access the !i.J!I
dynamic rangeofthc rotation and scalingdcgrL-cs ofli"L"Lximn.
In liJCL all remaining lined lingers can always he sci down
Once 1hc slide in.itimcs. the chord mo1ion recognizer could 30
simply begin sending rnw componcm vclocilics pnircd with
1hc selected combinmion of linger idcntilics 1o1hc hos1. I low-
e\er. in the interest ofhackward compatibility with the mouse
and kLy e\enl l(mJJals of comcnlional input de\ ices. the
mo1ion event gcncrmion steps in FIG. 40B conven motion in 35
nny of1hc cx1rnc1cd degrees of li"ccdom into swndnrd mouse
nnd key conunnnd events which depend on the idcmity of the
sdec!cd chord. 'Ill support such motion conversion. step 658
linds a chord acli\ily slmclurc in a lookup table using a
bit!icld of the idcnti1ics of ci1hcr 1hc touching lingers or the 40
lingers in 1hc synchronized. subsc1. DillCrcm linger idcmity
combinations can re!Cr!o the same chord aclivity slmclurc. In
the pre!Crrcd emhodimcnl. all linger cmnhinalions with the
same numhcrofnon-lhumh lingcnips rcJCr lo the samedmrd
nctivi1y stmcturc. so slide chord nc1ivitics nrc dis1inguishcd
by whether the 1humb is 1ouching nnd how mnny non-thumb
lingers are touching. !Sasing chord aclion on the number of
linger! ips rather than their combination still prmides up lo
seven chords per hnnd yc1 mnkcs chords cnsicr for !he user to
memorize nnd perform. The user hns 1hc freedom to choose '0
nnd vmy which linger! ips nrc used in chords requiring only
one: two or three linger! ips. (iivcn this li-eL-dom. users natu-
rally tend lo pick combinations in which all touching linger-
lips nrc ndjnccnt rmhcr 1hnn combinmions in which n linger
such ns the ring linger is lifted bm the surrounding lingers '5
such as the middle and pinky must !ouch. One chord typing
study l(mnd that users can lap these linger chords in which all
pressed lingerlips arc adjacent twice as li1sl as other chords.
45 nlicr inilintion of nny chord 10 nl!ow mnnipulmion by the
whole hnnd. LikLwisc. nil lingers but one cnn be lifted. yet
translation will ctmlinue.
"ll1c cvems in cnch chord nctivily structure nrc orgnnizcd
into slices. Ench slice contnins events to be gcncrntcd in 60
response lo motion in a panicular range of spcL-ds and dirL-c-
tions within the exlrac!cd degrees of li"L"L-dmn. !'or example. a
mouse cursor slice could be nllocmcd nny 1rnnslmionnl speed
nnd dircc1ion. However. tcx1 cursor mnnipulmion requires
four slices. one lix cnch nrrow key. nnd cnch nrrow's slice 65
integrates motion in a narrow dirL-clion range of translation.
I Oach slice c;m also include motion sensili\ ity and so-called
t-hough asynchronous linger !ouch aclivity is ignored. syn-
chronized liliing nnd pressing of muhiplc !ingcrs subsequent
to slide inilintion cnn crcntc n new synchroniZL'<i subsc1 nnd
chnngc the sclcc!cd chord. Pre!Crnbly !his is only nllowcd
while the hand has paused hut its lingers are still resting on the
smbce. I kcision diamond 670 will de!Lx:llhe nL'W subset and
conuncncc mo1ion testing in decision dinmond 67J which is
nnnlogous to decision dinmond 656. If signi!icnnt motion is
!(mnd in all lingersofthc newly synchroni/L-d suhscL step 674
will sdec!lhe new subset as the slide chord and lookup a new
chord aclivity slruclure in analogy lo step 658. Thus linger
ngnin nllows !he user 10 switch ton dif!Crcnt
nc1ivity wi1hoU1 lixcing the user to lili 1hc whole hnnd from
the surface. Integral ion of velocity components resumes hut
the e\enls generated limn the nL'W chord aclivity slmclurc
will prcsumnbly be dif!Crcnt.
It is ndvnntngcous 10 provide visunlor nudi!O!)' lCcdbnck 10
the user nboll! which chord nctivily Slrtlclurc hns been
sek-c!L-d. t-his can he accomplished\ isually by placing a row
oflivc light cmilling diodes across the lop of the multi-touch
us 7,812,828 82
53 54
surli1cc. with one row per hand to he used on the surbcc.
When entering slide mode. step 658 would turn on a combi-
nntion of these lights corresponding to the combinmion of
lingers in the sd<..><::tcd chord. Step 674 would change the
combination of a c l i H ~ lights to match the new chord a eli\ ity 'i
slmclurc should the user sdccl a new. chord. and step 668
would tum them olr Similar lights could be cmulm<..'d on the
host computer display 24. The lights could also be Hushed to
indicate the !Inger combination dc!cc!cd during chord laps in
step 636. rhc imp!cmcnlalion l(Jrauditory !Ccdback would be lCJ
similar. except light cmnhinalions would he replaced with
tone or tone burst combinations.
sd<..>ct<.."t.! chord. The user then n<..""L"t.! add the thumb to the
surlUcc when attempting rotation or scaling.
Finger chords which initially include the thumb can be
reserved lix one-shot conunand gestures. which only gener-
ate input e\ents once ll1r each slide of a chord rather than
repeating transmission each time an additional unit of motion
is detected. For example. the co nun on editing commands cut.
copy and paste can be imuitively a!!ocat<..'d to a pinch hand
scaling. chord tap. and anti-pinch hand scaling of the thumb
and an opposing lingertip.
FJ(j. 41 shows the steps within the k<..y layout delinition
and morphing process. which is pan of the typing recognition
module 12. Step 700 retrieves at system startup a key layout
which has been pre-spcci li<.."t.! by the user or manu bcturcr. J"hc
"111c nccumulmion nnd event gcncrmion process repeats Jix
all array scan cycles until decision diamond 664 detects !i!loff
by a!! the lingers Jfom the initiating cmnhination. Decision
diamond 666 then checks the prc-lilioll" deceleration Hag of
the dominant motion. component. The state of this Hag is
determined by step 556 or 558 of translation extraction (FIG.
37) if translation is dominant. or by corresponding !lags in
step 534 of polar extraction. If there has been signilicant
d<..>cclcration. step 668 simply exits the chord slide mode.
setting the selected chord to null. If the Hag indicates no
signilicant linger dccdcration prior to lilloll". decision dia-
mond 666 enables motion continuation mode ll1r the selected
chord. While in this mode. step 667 applies the prc-!i!loff
weighted average (560) of dominam component velocity to
the motion accumulators (678) in place ofthc current veloci-
ties. which arc presumably tcro since no lingers touch the
surli1cc. Motion continuation mode docs not stop until any of
the remaining Hngcrs not in the synchronized subset arc lifted
or more Hngcrs newly touch down. This causes decision dia-
mond 664 to become false and nonnal slide activity with the
currently selected chord to resume. Though the cursor or
scrolling velocity docs not decay during motion continuation
mode. the host compmcr can send a signa! instructing motion
cominuation mode to be canceled if the cursor reaches the
edge of the screen or end of a document. Similarly. if any
lingers remain on the smbcc during motion continuation.
their translations can adjust the cursor or scrolling velocity.
In the preferred embodiment. the chord motion rccognizcrs
for each hand li.mction indcpcndcm!y and the inplll events lix
each chord can he conligurcd independently. This allows the
system to allocate tasks between hands in many dilkrcnt
ways and to support a \aricty ofhimanual manipulations. For
example. mouse cursor motion can be allocated to the linger-
tip p.1ir chord on both hands and mouse bun on drag to a triple
lingcrlip chord on both hands. This way the mouse pointer can
he moved and dmg with either hand on either half of the
surface. Primary mouse clicks would be generated by a tap of
a Hngcrtip p.1ir on either half of the surface. and double-clicks
could be C!'6onomica!!y generated by a single tap of three
lingcrlipson thcsurli1cc. Window scrolling could he allocated
to slides of llmr lingers on either hand.
15 k<..y layout consists of a scl ofk<..y region data stmclurcs. J Oach
region has associated with it the symbol or commands which
should be scm to the host computer when the region is pressed
and coordinates representing the location of the ccmcr ofthc
region on the surli1ce. In the prcJCrr<.."t.! embodiment. arrange-
"" men! oftlmse k<..y regions containing alphanumeric and punc-
tuation symbols roughly corresponds to either the QWERTY
or the Dvorak key layouts conunonon mechanical keyboards.
In some embodiments of the nmlti-touch surbcc apparatus
it is advantageous to he able to snap or morph the key layout
"" to the resting positions oft he hands. This is csp<..>cially hdpli.ll
lix multi-touch surlUccs which arc several times !a!'6Cr than
the standard keyboard or key layout. such as one covering an
entire desk. Fixing the k<..y layout in one small lix<.."t.! area of
such a surliJCc would he inconvenient and discourage use of
30 the whole available sur!Ucc area. To provide feedback to the
user abom changes in the position of the key layout. the
position of the key symbols in these embodiments of the
multi-touch surbcc would not he print<.."t.! permanently on the
smbce. Instead. the position of the k<..y symbols would he
35 rcprogranunably displayed on the surlUcc by light emitting
polymers. liquid crystaL or other dynamic visual display
means embedded in the multi-touch sur!Ucc app.1ratus along
with the proximity sensor arrays.
(jivcn such an apparatus. step 702 rctriL"\cs the current
40 paths from both hands and awaits what will be known as a
lay om homing gesture. If d<..>cision diamond 704 decides with
the help oL a hand's synchronitation dct<..>clor that all li\e of
the hand's lingers ha\e just been placed on the surli1ce syn-
chronously. step 706 will ancmptto snap the key layout to the
45 hand such that the hand's home row keys lie under the syn-
chronized lingcnips. wherever the hand is on the sur!Ucc. Step
706 retrieves the measured hand ollScts limn the hand posi-
tion estimator and translates a!! k<..y regions which arc nor-
mally typ<..'d by the given hand in proportion to the measured
'0 hand offsets. Note the currently mcasur<..'d rather than Hhcrcd
Alternatively. mouse cursor manipulations could be allo-
cated as discussed above to the right hand and right halfofthc '5
surli1cc. while corresponding text cursor manipulations arc
allocat<.."t.! to chords on the !ell hand. !'or instance. !c!l lingcrtip
pair mmcmcnt would generate arrow key commands corre-
sponding to the dir<..>ction of motion. and thr<..-c Hngcnips
would generate shift arrow combinations for selection oftcxt. 60
estimates of offsets can be used because when all !ivc Hngcrs
are down there is no danger of linger misidentilication cor-
mpting the measured oflScts. This proc<.."t.!urc assumes that the
umranslat<..-d locations of the home row keys arc the same as
the dcJUult Enger locations for the hand.
J kcision diamond 708 checks whether the lingers appear
to he in a neutraL p;u1ially dosed posture. rather clos<.."t.! than
outstretch<.."t.! or pinch<.."t.! together. If the posture is close to
neutraL step 710 may Ji.Jrlhcr oll"sct the keys nonnally typed
by each !ingcr. which fort he most pan arc the keys in the same
column oft he linger by the measur<.."t.! linger onSets. Temporal
liltcring of these linger onSets over se\era! layout homing
gestures will tend to scale the spacing between colunms of
keys to the user's hand site. Spacing between rows is scaled
down in proportion to the scaling between colunms.
For host computer systems suppm1ing manipulations in
three or more degrees of frc<.."t.!om. a !e!l hand chord could he
selected to pan. zoom. and rotate the display background
while a corresponding chord in the right hand could translate.
resize and rotate a foreground object. These chords would not 65
have to include the thumb since the thumb can touch down
anytime a!ler initiating chord motion without changing the
With the key layout ll1r the hand's k9s morph<.."t.! to lit the
size and current position of the resting hand. step 712 updates
us 7,812,828 82
55 56
down or the surliJCc. Decision diamond 752 checks whether
the p.1th proximity rcnchcd a keypress proximity thresh for
the Hrst time during. the current sensor array scan. If the
proximity has not reached the threshold yet or hns n!rcndy
cxcecdL-d it prL'\ iously. control rctums to step 750 to try
kLyprcss detection on the next recent path. If the path just
crossed the keypress proximity threshold decision dinmond
754 checks whether the contact pmh hns bL-cn idcntiHcd as n
linger rather than a palm. "Ill give the users the frcL-dom rest
the displayed position of the symbols on the surli1ce. so that
the user will see thm the key layom hns snapp<..'d to the position
ofhis hand. From this stage the user can begin to type and the
typing recognizer 718 will usc the morphcd key region loca-
tions to decide what h:y regions arc being pressed. '!-he layout 'i
will remain morphcd this way until either the user pcr!lmns
another homing gesture to move it som<..whcrc else on the
surface. or until the user takes both hands olrthc surface Jix a
while. lkcisiondiamond 714will cn::ntually time out so that
step 716 can rcsc!lhc layout to its delimit position in readiness
ll1r another user or usage session.
lCJ the palms anywhere on the surface. palm presses should not
normally cause keypresses. and are therdl1re ignorLxi.
.\ssuming. the path is a !ing.cr. decision diamond 756 clK>cks
whether the hand the idcnti!iL'd !ing.cr comes from is currently
perl(Hming a chord slide gesture or writing \ia the pen grip
!-'or smaller multi-touch sur!Uccs in which the key layolll is
permanently printed on the sur!Ucc. it is ndvanwg.cous to g.ivc
the user tactile !L'"l..xiback about the positions of key regions.
I !owe\er. any tactile indicators placed on the surliJCc must he
cnrcfu!!y designed so as not to impede smooth sliding. ncross
the surface. !'or cxnmp!c. shallow depressions made in the
surface ncar the center of each key mimicking. the shallow
depressions common on mechanical keyboard key caps would
cause a vibratory washboard d!Cet as the hand slides across
the sur!Ucc. To minimize such wnshboard effects. in the pre-
ferred embodiment the multi-touch surface provides for the
ling.ertips of each hand a single. continuous depression nm-
ning. Jiom the delimit index lingertip location to the delimit
pinky ling.crtip location. "!.his corresponds on the QWI ;R"!Y
key lnyolll to shn!!ow. slig.htly nrchcd chmmc!s nlong. home
row from the 'T' key to the":" key fort he right hnnd.and from
the "A" key to the 'T" kl:y ll1r the !ell hand. Similarly. the
thumbs c;m each he provided with a single oval-shaped
depression m their default locations. s!nntcd slightly from
vcrticnl to mntch the dc!Uult thumb oricntntion. These would
prcfcrnbly correspond to "Sp.1cc" and "BnckSpncc" key
regions llHthe right and !ell thumbs. rcspccti\dy. Such mini-
ma! depressions can taetildy guide users' hands hack to home
row of the key layout without requiring. users to look down m
the surface nnd witholll seriously disrupting. Hng.cr chord
slides nnd mnnipu!ntions on the sur!Ucc.
"l11c positions of key regions olfhomc row can he marked
by other types of tactile indicators. Simply roughening the
surface m key regions docs not work we!!. Thoug.h humans
cnsi!y dill"crcntintc textures when sliding. Hng.crs over them.
most textures cannot be noticed during quick taps on a tex-
tured region. Only rdati\dy abrupt edges or protmsions can
15 hand conliguration. Asynchronous linger presses arc ignorL-d
once these nctivitics hnvc stnrtcd. ns nlso indicated in step 660
of FIG. 40.\. Assuming. such hnnd nctivitics nrc not ongoing..
decision dinmond 757 proceeds with dcbouncc tests which
check that the linger has touchL-d the surli1cc ll1r at least two
"" sensor array scan cycles and that it had been offthesurli1cc ll1r
several scan cycles before touching. down. The pmh trucking.
module (FIG. 22) faci!itmcs such lilioll" debouncing. by rcnc-
tivating in step 334 a linger's old path if the linger !ills olLmd
quickly touches back down mer the same spot. Upon !"Cacti-
"" vat ion the time stamp of the last !illoffby the old path must he
prcscrvL-d for compnrison with the time stamp of the new
touchdown.
I fall of these tests arc passLxi. step 758 looks up the current
path position (P,[ n J.P,.[ nJ). and step 760 linds the kLy region
30 whose rcl"crcncc position is closest to the !ing.crtip centroid.
J)ccision dinmond 762 checks thnt the ncnrcst region is within
n rcnsonablc distnncc of the !in g. cr. nnd if not causes the !ing.cr
p1-css to be ignorL-d. Assuming a key region is dose to the
linger. step 764 cJ-cates a kLyprcss dement data stmclurc
35 contnining. the pmh. index identifier nnd Hng.cr identity. the
closest key rcg.ion. and n time stump indicnting. when the
Hng.cr crossed the keypress proximity threshold. Step 766
then appends this dement data stmclure to the tail of a I'!FO
kLyprcss queue. This accomp!ishL-d. processing returns to
40 step 750 to process or wnit for touchdowns by other Hng.crs.
he scnsLxi by the users' lingertips under typing conditions.
Therefore. n small raised dot like a Braille dot is Jixmcd on 45
The keypress queue cll"cctivcly orders Hng.cr touchdowns
by when they pass the keypress transmittL-d to the host. I low-
ever. an dement's kLy symbol is not assurL-d transmission of
the host once in the keypress queue. Any of a number of
conditions such as being. part of n synchronitcd subset of
pressing. Hng.crs cnn cnusc it to be deleted from the queue top of the surface at the center of cnch key region. The user
rccei\es !L'"l..-dback on the accuracyofthcir typing strokes limn
where on the lingertip a dot is !Cit. "!.his kL-dback can be used
bell1re being transmitted to the host. In this sense the keypress
queue should be considerLxi a keyp1-css candidate queue.
Unlike the ordered lists of !ing.cr touchdowns nnd rdcnscs to correct !ing.cr nim during. li.Jturc kcyprcsscs. Since sing.lc
!ing.cr slides arc ig.norcd by the chord motion recognizer. the
user can also slide n !ing.cr nround the surface in tnctilc search
'0 maintained Jix cnch hnnd scpnrmdy in the synchronization
dck>ctor. the keypress queue includes nnd orders the !ing.cr
touchdowns from both hands. of a particular key region's dot and then tap the kLy region
when the dot is llmnd. all without looking at the surliJCc.l ;ach
dot should be just: !arg.c enough to be ldt during. tnpping. bm
not so !arg.c as to impede chord slides ncross the surface. Fvcn '5
if the dots are not large enough to impL-dc sliding. they can
still cormpl proximity and lingertip centroid measurements
by raising the lingertip Jlcsh near the dot off the surbce thus
locally scparming. the Hcsh from the underlying. proximity
sensing. electrode. Therefore. in the preferred embodiment. 60
the portion of each dot above the surbcc dielectric is made of
a comlucti\e material. "l11is imprmes capaciti\e coupling
betwL-cn the raised !ing.crtip Jlcsh and the underlying. ckc-
trodcs.
l'!(j. 43:\ shows the steps within the keypress acceptance
nnd trnnsmission loop. Step 770 picks the clement m the head
of the keypress queue. which represents the oldest !ing.cr
touchdown which has neither been ddetL-d li-om the queue as
an invalid kLyprcss candidate nor transmittL-d its associatL-d
kLy symbol. I kcision diamond 772 checks whether the path
is still idcnti!iL'd ns a !ing.cr. While waiting. in the queue p.1th
proximity could have increased so much thnt the idcntiHca-
tion system decides the path is actually limn a palm heel. in
which case step 778 deletes the keyp1-css dement without
transmitting. to the host and step 770 ndvanccs processing. to
the next dement. Decision diamond 774 nlso invalidmcs the
FIG. 42 shows the steps within the keypress dc!L'ction loop.
Step 750 retrieves lium the current identilicd path data 250
any paths which were recently crcatL-d due to hand p;u1touch-
65 dement if its press hnppcncd synchronously with other Hn-
gers of the same hand. "!lms decision diamond 774 ll1llows
through on deletion command steps 601. 612. 615. 620 oft he
us 7,812,828 82
57 58
hands while lapping long sequences of symbols. "!.his causes
users to mise their hands oll"thc surface and !lom them above
the surJUcc during JUst typing sequences. This is acccpwb!c
typing posmrc except that the users arms wi!! eventually tire
detection process (!'Ki. 39). Decision dia-
mond 776 invalidates 1hc keypress if too much lateral Enger
motion has occurred since touchdown. even if thnt Intern]
!ingcr motion has not yet caused a chord slide to stnrt.
lkcausc users may he !ouch typing on the surli1cc. sc\cra!
millimeters or la!cral motion arc allowed lo accmnmodatc
glancing fingertip motions which ollcn occur when quickly
reaching for keys. This is much more glancing wp motion
than is tolerated by touchpads which employ a single linger
slide I(Jr mouse cursor manipulation and a single !Inger lap I(Jr
key or mouse \muon click emulation.
5 if the user bi!s lo res! the hands hack on the surli1cc hc!wccn
sequences.
To provide an ahcmativc typing posture which docs not
encourage suspension of the hands above the surlUcc. deci-
sion diamond 790 enables a second key acceptance mode
lCJ which docs no! rL'<.juirc quick linger lillolf allcr each press.
Instead. the user must star! with all li\c lingers of a hand
resting on the surlUcc. Then each time a linger is asynchro-
nously raised oll"!hc surJUcc and pressed on a key region. that
k<.y region wil! he lransmit!cd regardless ofsuhsL'<.jUCntlilloff
D<..>cision diamond 780 checks whether the !ingcr whose
touchdown crcntcd the keypress clement has since lillcd off
the surli1cc. If so. decision diamond 782 checks whether it
was !ilkd off soon enough to qualify as a nmma! key lap. If so.
step 784 lrnnsmils the associmcd key symbol to the host and
step 778 deletes i1 from the head of the queue. Note !hal a
keypress is always deleted from the queue upon liftoff. blll
c\cn though it may hmc stayed on the stubcc ll1r a time
exceeding the lap timcoul. it may hmc still caused transmis-
sion as a modifier key. as an impulsive press with hand rest-
ing. or as a typcmmic press. as dcscrib<.-d below.
15 timing. If the surbcc is hard such that Jingc11ip proximity
quickly smurmcs as force is app!i<.-d. decision diamond 792
checks the impulsivity of the proximity prolilc Jix how
quickly the Enger proximity peaks. If the proximity proli!c
increases lo its peak very slowly mer time. no key will he
When a keypress is lransmillcd lo the host it is ad\;m!a-
gcous ll1r a sound generation de\ icc on the nm!ti-louch sur-
liJcc apparatus or host computer lo emil an audible dick or
b<.-cp as k-ed back to the user. Generation of audible dick and
b<.-cp f<..-cdback in response 10 kcyprcsscs is well known in
commercia! touchscreens. kiosks. appliance control panels
and mechanical k<.yhoan.ls in which the kcyswitch aclion is
nearly si!cm and docs not have a make Jixcc threshold which
f<..-c!s distinctive 10 the user. Feedback can also be provided as
a light on the muhi-louch surface apparmus which !lashes
each time a keypress is sen!. K<.yprcsscs accompanied by
modilicr k<.yprcsscs should cause longer !lashes or Iones lo
acknowledge thm the key symbol includes modiEcrs.
"" generated. This al!ows the user lo gently sci down a raised
Enger without generating a key in case the user !ills the linger
with the imcmion of generating a key but then changes his
mind. If the !ouch surli1cc is compressible. decision diamond
792 can more dircc!ly inJCr linger ll1rcc Jium the ratio of
"5 measured Jingcrtip proximity lo ellipse axis lengths. Then it
can threshold the interred force to distinguish dclibcrmc key
presses Ji"om gentle linger rests. Since when imcnding to
generate a key the user will nmmal!y press down on the new
k<.y region quickly allcr lilling off the old k<.y region. the
30 impulsivity and force thresholds should increase with the
time since the linger lilicd off the surlUcc.
Fmulming typcmatic on a multi-touch surlUcc prcscms
special problems if linger resting !(Jrcc cannot he distin-
guished reliably Jium sustained holding !(Jrccona key region.
35 In this case. the special touch timing sequence detected by the
steps of I'IG. 4JB supports reliable typcmatic cmulmion.
.\ssuming decision diamond 798 Ends thattypcmatic has not
started yet. decision diamond 794 checks whc!hcr the key-
lflhc Enger has not yet lifted. decision diamond 786 checks
whether its associa1<.-d key region is a modifier such as
<shill>. <c!rl>. or <all>. If so. slcp 788 ad\;mccs lo the next
clement in the queue without deleting the head. Processing
wi!! cominuc a! step 772 to sec if the next elcmcm is a valid 40
key wp. If the next element succcssli.llly reaches the trans-
mission stage. step 784 will scan hack toward the head of the
queue ll1r any modilicr regions which arc still pressed. Then
step 784 can send the next dement's key symbol along with
the modifying symbols of any preceding modifier regions.
press queue dement being processed represents the most
rcccm Enger touchdown on the surJUcc. If any linger touch-
downs have followed the touchdown rcprcscmcd by this de-
men!. lypcmalic can nc\cr star! Ji-om this queue dement.
Instead. decision diamond 796 checks whc!hcr the dement's
linger has been touching longer than the nmma! lap timeout.
D<.>cision diamond 782 r<.'<.juircs that users 10uch the Enger
45 If the Enger has been touching too long. step 778 should
delete its keypress dement because decision diamond 786 has
dc!crmincd it is no! a modilicr and decision diamond 794 has
dc!crmincd it can nc\cr sl<ll1 lypcmalic. If decision diamond
794 dctcnnincs thm the keypress dement docs not represent
on the surbcc and !ill hack olfwithin a JL-v. hundred milli-
seconds ll1r a key lo he sen!. "!.his Jill off timing rL'<.juircmcnl
subslilulcs Jix the Jixcc activmion threshold of mechanical
kcyswitchcs. Like the force threshold of mechanical kcy-
swi1Chcs. the timing constrain! provides a way for the user to
res! the linger on the key smbccwithoul imoking a k<.yprcss.
The dc!cclor 14 provides another way l(Jrc-
!ingcrs to reston the surface without gcncrming key symbols:
they must touch down at the same time as m !cast one other '5
linger. I low<.'\ cr. sometimes users will s!;u1 resting by simul-
lancous!y placing the central Jingcrlips on the surli1cc. hut
then they l(JI!ow asynchronously with the pinky a second later
and the thumb a second alicr that. These latter presses arc
essentially asynchronous and wi!! not be invalidat<.-d by the 60
dc!cclor. hut as long as they arc no! lilkd
within a couple hundred milliseconds. decision diamond 782
wi!! delete them witholll transmission. But. while decision
diamond 782 provides tolerance of asynchronous Enger rest-
ing. its rcquircmcmthat Engcrs quickly !if! oJI i.e .. crisply
lap. the surli1cc lo cause k<.y generation makes it \cry dillicu!t
lo keep most of the lingers resting on the surface to support the
'0 the most rcccmtouchdown. yet decision diamond 796 indi-
cates the clcmcm has not exceeded the tap timeout. process-
ing rclums lo step 770 lo await either lillolf or timeout in a
Ji.J!urc sensor array scan. "!.his al!ows linger laps lo overlap in
the sense that a new key region can be pressed by a linger
before another linger litis oll" the previous key region. ]]ow-
ever. either the press times or release times of such a pair of
overlapping linger laps must he asynchronous lo prL'\cnllhc
pair from being considered a chord lap.
Assuming the Enger touchdown is the most r<.>ccnt. deci-
sion diamond 800 checks whether the linger has been touch-
ing ll1r a lypcmalic hold sclup interval of he! ween about half
a second and a second. If no!. processing rclums lo 770 lo
await either linger !illoffor the hold sclllp condition to be met
during future scans of the sensor array. When the hold setup
65 condition is met. decision diamond 802 checks whether all
other lingers on the hand of the given linger keypress lilkd off
the stubcc more than a half second ago. If they did. step 804
us 7,812,828 82
59
will lypemalic !l1r the given kl:ypn::ss dement. The
combination of d<..>c is ion diamonds 800 and 802 allow the user
to have other lingers of the hand to be resting (l!l the surface
when a !ing.cr intend<..'<! Jix typcmatic touches down. Blll
lypemalic will not start unless the other lingers !i!l off the 'i
surli1cc within half a second of the desired typcmatic !Inger's
touchdown. and typcmatic will also not swrt until the typc-
mmic !ing.cr has a continu<..-d to touch the surface for at !cnst
half a second alkr the others lillL-d off the surli1cc. If these
stringent conditions arc not mel. the kl::ypn::ss ckmcnl will no! lCJ
st;u11ypcmalic and will eventually be ddetcd through either
wp timcolll 782 when the !ing.cr lilis off or throug.h tnp tim-
com 796) if ano1her 1ouches down nlicr it.
S!cp 804 simply scls a Jlag which will indicalc lo decision
diamond 798 during !l11urc scan cycles !hal lypcmalic has l'i
n!rcndystnn<..'d for1heelcmcnt. Upon1ypcmmic initin!izntion.
step 810 sends om the key symbol for 1he !irs11imc to the hos1
intcr!Ucc conununicntion queue. n!ong wi1h nny modifier
symbols being held down by the opposite h.:md. Step 812
n.:cords the time the kl:y symbol is sen! !l1r !i.J!urc rckrcncc by ""
d<..>cision dimnond 808.1'roccssing then returns 10 step 770 to
nwni11he nex1 proximi1y imngc scnn.
Until the !Inger !i!is o!f or another laps asynchronously.
processing will pass through decision diamond 798 lo check
whether the k<..y symbol should be sen! again. Step 806 com- "-'
pmcs 1he symbol repem intervn! dynmnicnlly 10 be inversely
proponionnl 10 !ingcr proximity. Thus the key will repem
liJslcr as the linger is pressed on the surbcc harder or a larger
pan of the Jingc11ip !ouches the surli1cc. "!11is also reduces the
chnnce 1hm the user will cnuse more repents 1hnn intended 30
since ns !inger proximity begins to drop during liftoff the
repent intervnl becomes much longer. Decision dimnond 808
checks whclhcr the dynamic 1-cpcal interval since the !as!
lypcmalic symbol send has elapsed. and if n<..>ccssary sends
1he symbol ngnin in 810 nnd updntes 1he 1ypcmmic send lime 35
stump 812.
11 is dcsirnb!c to !ctthe users res11he other !in!lers back onto
the surbcc a!icr lypcmalic has initialed 804 and while type-
malic nmlinucs. hut the user must do so without lapping.
Decision dimnond 805 causes 1ypcmmic 10 be cancckd and 40
1he typematic elemcm deleted 778 if the user nsynchronously
laps another linger on the smbcc as if trying lo hit another
key. If this docs no! occur. decision diamond 182 will L"\cn-
lually cause delclion of the lypcmalic element when its linger
!ills off 45
"ll1e typing recognition process dcscrib<..'d above 1hus
allows the multi-touch surli1cc lo crgonmnical!y emulate both
the typing and hand J-esting capabilities of a standard
mechnn.icn! keyboard. Crisp laps or impulsive presses on the
surface gcnernte key symbols ns soon as 1he !inger is released '0
or d<..>cision diamond 792 veri!ics the impulse has pcnkcd.
ensuring prompt JL-cdhack lo the user. 1:ingcrs intended lo res!
on the surbcc generate no k9s as long as lhLy arc members
of a synchroniz...-d !inger press or relcnsc subsc1 or nrc placed
on 1he surface gcm!y nnd remain 1herc n!ong wi1h o1her !in- '5
gcrs !l1r a second or two. Once resting. !ingcrs c;m be lillcd
and lapped or impulsively pressed on the smbcc lo generate
key symbols w ithoul ha\ ing lo !i!i other resting lingers. "!)'pc-
mmic is ini1imcd ether by impulsively pressing and mainwin-
ing distinguishable force on n key. or by holding n !ingcr on a 60
key while other lingers on the hand arc lilkd. (i!ancing
motions of single lingers as lhLy lap kLy regions ;u-c easily
1olerntcd since most cursor mnnipu!ation must be ini1imcd by
synchronitcd slides of 1wo or more lingers.
Other embodiments of the invention will be nppnrcnt to 65
those ski!kd in the an from considcrdlion oft he spccillcalion
and praclicc of the invention disclosed hc1-cin. II is intended
60
thai the spccilicalion and examples he considered as cxcm-
p!m)' only. with n 1me scope nnd spiri1 oflhe invention being
indicm<..'d by 1he following claims.
What is claimed is:
1 .-\ method of processing inplll from n 1ouch-scnsi1ivc
smbcc. the method comprising:
rL>cci\ ing a! leas! one proximity image representing a scan
of n p!urn !ity of electrodes oft he 10uch-scnsitive sur!Ucc:
segmeming each proximity imngc imo one or more pixel
groups !hal indicate signi!icanl proximity. each pixel
group representing proximity of a distinguishable hand
pan or other !ouch objccl on or ncar the louch-scnsili\c
surface: nnd
mmhemmicnlly !ining an ellipse tom kns1 one of the pixel
groups.
2. "!l1c mc!lmd of claim 1 fm1hcr comprising twnsmining
one or more ellipse parnme1crs as a control si!lllill 10 an
electronic or clectromechnnica! device.
J. The method of claim 2 wherein 1he one or more ellipse
paramclcrs is selcc!cd !i-om the group consisting of position.
shape. size. oricnlalion. ccccnlricity. major radius. minor
rndius. nnd any combinmion 1hercof.
4. The method of claim J wherein 1he one or more ellipse
parmncters nrc used to distinguish a pixel group nssocintcd
with a Jingc11ip !i-mn a pixel group associated with a thumb.
5. n1c mc!lmd of claim 1 wherein tilling an ellipse loa
group of pixels comprises compming one or more eigenval-
ues and one or more eigenvectors of n covarinnce mntrix
associated with the pixel group.
6. The method of claim 1 !lu1hcr comprising: !racking a
path ofal !cas! one of the one or more pixel groups through a
time-sequenced series of proximity imngcs:
!it1ing an ellipse 10 the at least one of1he one or more pixel
groups in cachofthctimc-scqucnccd series of proximity
images: and
!racking a change in one or mm-c ellipse paramclcrs
1hrough 1he lime-sequenced series of proximity imnges.
7. "ll1e method of claim 6 fun her comprising 1rnnsmitting
the change in the one or mm-c ellipse paramclcrs as a control
signa! to an ck>clnmic or elcc!romcchanica! dL"\ icc.
8. rhc mc!hod of claim 7 wherein the change in the one or
more ellipse parnme1crs is selcc1cd from 1he group consisting
of position. shape. size. oricntmion. eccentricity. mnjor
radius. minor radius. and any combination thereof
9. The mc!hod of claim 6 wherein lining an ellipse lo the
one pixel group comprises computing one or more cigcma!-
ucs and one or more eigenvectors of n covarinnce mntrix
nssociated wi1h 1he pixel group.
10. A !ouch-sensing dL"\ icc comprising:
a suhslralc:
a plurality of !ouch-sensing elcc!rodcs anangcd on the
subs1rmc:
electronic scmming hardwnrc ndap1cd 10 read 1he plurn!i1y
of touch-sensing ck>ctrodcs:
a calibration module opcrali\ely coupled to the ck>clronic
scanning hardwa1-c and adap!cd to conslrucl a proximity
imnge having a plurn!i1y of pixels corresponding to the
1ouch-scnsing electrodes: and
a conlacl tracking and idcnti!icalion module adap!cd lo:
segment the proximity image into one or more pixel
groups. each pixel group representing proximity of a
distinguishable hnnd pan or o1hcr touch objcc1 on or
ncnr 1he touch-sensitive sur!Ucc:
and
mathematically Ill an ellipse loa!
mm-c pixel groups.
!cas! one of the one or
us 7,812,828 82
61
I I '!-he !ouch-sensing device of claim I 0 !i.Jrthcr cmnpris-
ing. a host communicmion intcr!Ucc adapted to trnnsmit one or
more ellipse pam meters us a control signal to an electronic or
clcctromcchanical device.
12. The touch-sensing tk'\icc of claim 11 wherein the 'i
!ouch-sensing de\ icc is integral with the ck:clronic or dcc-
trom<..>chanical device.
D. The touch-sensing device of claim 11 wherein the one
or more ellipse parameters comprise one or more parameters
sclcc!cd ![om the group consisting of position. shape. size. lCJ
micnlalion. ccccntricity. major radius. minor radius. and any
combination thcn.of
14. The method ofclaim 13 wherein the one or more ellipse
parameters arc used lo distinguish a pixel group associated
with a !ingcrlip from a pixel group associated with a thumb. 15
15. The touch-sensing device of claim 10 wherein the
con wet tracking and idcmiHcation module is adapt<..-d to com-
pUle one or more eigenvalues and one or more eigenvectors to
!it the ellipse.
I6. The touch-sensing dL'\ice of claim IO wherein the ""
comact tracking and identification module is li.1rthcr ad.1ptcd
to:
62
23. :\ cmnputer-rcadab!e medium having embodied
thereon instructions cxccmable by a machine to perlixm a
method according to any of claims 1-9.
24 . .-\ touch-sensing device comprising:
means ll1r producing a proximity image representing a scan
of a plurality of ek>ctrodes of a touch-sensiti\e surli1cc.
the proximity image having a plurality of pixels corre-
sponding to the touch-sensing electrodes: and
means ll1r segmenting the proximity image into one or
more pixel groups. each pixel group representing a touch
objL>ct on or ncar the touch-sensitive smbce: and
means for lilting an ellipse to at least one of the pixel
groups.
25. "J he touch-sensing de\ ice o fclaim 24 wherein the touch
objL>ct comprises at !east a portion of a hand.
26. The touch-sensing device ofclaim 24 wherein the touch
objL>ct comprises at least a portion of one or more Hngcrs.
27. The touch-sensing device ofclaim 24 wherein the touch
objL>ct comprises at !east a portion of a body part.
28. The touch-sensing dcviceofdaim 27 wherein the body
part comprises one or more of a hand. a linger. an car. or a
check.
track a path of one or more pixel groups through a plurality
of time-sequenced proximity images:
!it an ellipse to at !east one of the one or more pixel groups
in a !irst proximity image of the plurality of time-se-
quenced proximity images: and
track a change in one or more ellipse parameters associated
with the lined ellipse through two or more of the time-
sequenced proximity images.
29. n1e touch-sensing device of claim 24 further compris-
ing means ll1r transmitting one or more ellipse parameters as
"5 a control signal to an electronic or ck>ctromL>chanical de\ icc.
30. The touch-sensing device of claim 27 wherein the
touch-sensing device is integra! with the electronic or elcc-
tromL>chanical device.
3 I. n1e touch-sensing device of claim 24 further compris-
30 ing.
17. The touch-sensing device of claim 16 Ji.Jrthcr compris-
ing a host conununication imcrJUcc adapted to transmit the
change in at !east one of the one or more ellipse parameters as
a control signa! to an dectnmic or electromechanical device.
18. The touch-sensing device of claim 17 wherein the 35
touch-sensing device is integral with the electronic or ckc-
tromL>chanica! device.
means Jix tracking a path of one or more pixel groups
through a plurality of time-sequenced proximity
1magcs:
means ll1r lining an ellipse to at !east oncofthc pixel groups
in a plurality successive proximity images: and
means for tracking a change in one or more ellipse param-
eters through a plurality of time-sequenced proximity
1magcs.
32. n1e touch-sensing device of claim 29 further compris-
ing means Jix transmitting the change in the one or more
ellipse parameters as a control signal to an ck>ctron.ic or
dectromL>chanical device.
I9. The touch-sensing dL'\ice of claim I7 wherein the
change in one or more ellipse parameters used as a control
input to an electronic or electromechanical device comprises 40
one or more parameters selected from the group consisting of
position. shape. si/e. orientation. eccentricity. major radius.
minor radius. and any combination thereof. 33. The touch-sensing device of claim 32 wherein the
touch-sensing device is integra! with the dectnmic or clcc-
45 tromcchanical device.
20. The touch-sensing dL'\ice of claim I6 wherein the
contact tracking and idcmiHcation module is adapt<..-d to com-
pUle one or more eigenvalues and one or more eigenvectors to
lit the ellipse.
2I "!.he touch-sensing dL'\ ice of any one of claims I 0- I2
and 16-18 wherein the touch-sensing device is lUbricated on
or integrated with a display device.
22. The touch-sensing device of claim 21. wherein the
display de\ icc comprises a liquid crystal display (I ("]))or a
light-emitting polymer display (I PD).
34. The touch-sensing device of any one of claims 24 and
29-33 wherein the touch-sensing device is bbricatcd on or
integra!L"d with a display device.
35. The touch-sensing device of claim 34. wherein the
'0 display device comprises a liquid crystal display (!.CD) or a
light-emitting polymer display (I .I'D).
' ' ' '












































Exhibit 2
(12) United States Patent
Christie et al.
(54) \1ETIIOD.\:'\I>.\PP.\Il..\TUS FOR
I>ISPL\YI:'\G I:'\FOR\1.\TIO\" DUIU\"G .\:"\
1:\ST\:\T VIESSAGI:\G SESSIO:\
(75) !mentors: Gregory:\. Christie. San Jose. CA
(US): Westen. Menlo Park. C.\
(US): Stephen 0. Lemay. San
Francisco. CA (US): .ft>ns Alfke. San
Jose. CA (US)
(73) Assignee: .\pplc Inc .. Cupcnino. C.\ (US)
( * ) Notice: Suh_icc!lo any disclaimer. the lcrm of this
pntcnt is extended or adjusted under 35
U.S.C. 154(b) by 686 days.
(21)
(22)
(51)
App!. No.: 10/428,523
Filed: \1ay 2, 2003
Int. Cl.
<ifJ6F 3/f)(J
G06F 9100
G06F 17100
(2006.01)
(2006.01)
(2006.01)
(52) u.s. Cl. 715/758:715/762:715/751:
7151753
(5R) Fidd of (:Jassitication Search 7151715.
715/501.1.707.706.751.762.763.764.
7151760: 70<)1204
S<..-c npplicntion H!c for complete senrch history.
(56) Cited
U.S. PA:tI;N"r J)OCUMI;NI-S
4.107.7M A s 197i-: lkmmdcn .................. 900
4.4D.377 A 2 Lustis d a1 200
4.41>.".439
'
"
1<)1>4 ................... 364 200
4.5!3 .. 17<)
'
4 1<)1>5 "I 364 474
A 4 Brandle ...................... 900
4.(>42.790 A 2 V!inshull d al 900
''-21>7.441>
'
2 1<)<)4 d a1 3<)5 1."<)
.".421>.731
'
"
1<)<)5 Powers. lll ................. 3<)5 1.'4
45.'i.'i.77.'i A l2 1995 Pike ...........................
.'iM75W A 4 1997 l.udwigda1 39.'i 20002
.".7."1>.07<)
'
; 1<)<)1; ludwigd"1 _1<)_'; 200 34
111111 1111111111111111111111111111111111111111111111111111111111111
US007669134Bl
(I OJ Patent No.: US 7,669,134 Bl
Feb.23,2010 (45) Date of Patent:
5.1>54.N1
'
12 1<)<)1; 1.udw ig al .1<)5 200 34
5.1>1>4.0_1<)
'
3 1<)<)<) 1.udw ig al .1<)5 200 57
5.1>% . ."00
'
4 1<)<)<) 1.udw ig al .1<)5 200 34
(i.J77.931 Bl $ I 2001 .\lcxandcr ct al 725 52
(>.237.025
'"
.'i 2001 Ludwig cl al .............. 709 204

'"
2 2002 Ludwig cl al .............. 709 204
ti.434.ti04 B1 g 2002 1lara.d.a ............... 70<) 207
ti.4.17.1>11> Ill g 2002 1.udw ig al 341> 14 0<)
(Continu<..'d)
en! II OR
\kynmil>'. '\orman. Bnmin An Adapwb!e
Sl!'a/<'gtfix Jfi'mlo" :WWWJ;<'I'"lirum/ lermiua/ S_nlem.<,
D<1JW'III!<'III o(('omplll<'l' Scie!IC<'. !Jrmm ( "lli>-'<'l'.<il_l'(l9gl ). rr I
ll><J
(Continued)
l'riman Hxaminer Titdcssc llailu
.1ssislanl Hxaminer Anita]) Chaudhuri
(74) Allome_r. Agent. or Finn Williams. Morgan &
.\mcrson. I'. C.
(57) AHSTR\(T
.\ method and nn apparatus arc provided for controlling a
graphical user interbcc lo display inl(mmtlion related loa
commtmicalion session. !nl(mJJalion relating lo data pro-
duced hy a lirsl participant lo the communication session is
displayed on n Hrst displny unit. wherein the infonnation
produc<..'d by the Hrst pnnicipmll is displny<..'d nt a !irst position
on the Jirsl display unit. Data is rccei\L-d from a second
participant lo the communication session. and inl(mnalion
relating. to the dnta received from the second participant is
displayed on the Hrst display unit. wherein the infonnation
rcceiwd from the second participant is displayed m a second
position on the Jirsl display unit. The Jirsl and second posi-
tions arc horizontally spaced apart.
28 Claims, 8 ])rawing Sheets

US 7,669,134 Bl
Page 2
U.S. PA:t-I;N"r J)OCUMI;NI-S
B2 (i 2003 l.udwigdal
6594.(in Bl 7 2003 Smith <:tal 007 104 I
6.594.til>l> B2 7 200.1 ludwigd"l 70<) 204
6.910.11>6 B2 (i 2005 Kim 715706
7.124.164 Bl to 200(i Chcmtob 709 204
7.133.<)00 Bl'' !1 200(> 70<) 206
7.1."2.093 B2 !2 200(> ludwigd"l
7.206.gQ<) B2 4 2007 l.udwigdal
7.4!2.470 B2 :; 2001> \hsuno d ,]
7.433.<)2! B2 !0 2001> ludwigd"l
2003 0097401> AI 5 2003 d al 709 205
2003 OD4(i71> AI 7 2003 -L1Mka 4(i3 42
200_1 011>7<)2." _\]'' !0200.1 !mil" al 70<) 204
200_1 0201>7."6 _\]'' !1200.1 d ,] ............... 72_<. 34
20o3 ong<Jo<J AI 12 2003 -L1Mka Cl a] ................. 4(i3 42
2004 019%63
"
!0 2004 l!onil/ d ,]
2004 02234K'
"
!1 2004 .\rdh!no cl al
200(i 002(i2D AI 2 200(i Tcncmbmun d al 709 20.'i
OT!!I;R PUISIICAT!ONS
R \\". lite X IVindo" S).<l<'m, :11/l labora/01)"/i.>r Com-
pill<'!" Science and Gelt_n . .lim !ltj;iwl!:.qmj;m<'lll Cmporarion am/
Mil" Proje<J Allu.tw. A(:lf liYtn.WtCiiom ott Graphic.\. n>1 5. '\o 2.
-\pr !91>6. pp 79-!09
\Vadlow. VI G . llw Roi<' of l!uman Imel"(ace Ciuiddim'.' in rlw
!le.<tj;ll ofMulrim<dia .1pp/icariom. ("am<"J;ie Jfellon I "nher.,irr(lo
bt. Publi.,hed in Curren! p,.) ,/wlog) Rt."\"t.w,h and Rt.'lieon. Summt.r
/VYO)pp l-22(C:vHIIC-<Jl-101)
Y!oJTis. Jmncs II ct a! . A !li.\lribu!ed l'er.,mta!
Fmimnmem; (fmmllmicatiom <?/ 1/u. A(:lf.(\1ar 1<)1>6). \o1 29 '\o
.1. pp !1>4-20!
!Inward. John II . (.1b.\lraCI)AII 0\ervie" hie S_nlem,
Infi:!nlwlion kdtnolom ("emet: ("am<"J;ie Jfellon I "niwnirr. (("Y!l.-
ITC-1>1>-062). pp l-6 ({"\1!:-ITC-1>1>-062) (To .\ppc"r in" Ji.olure
issue of the .\CVI l ransadions on Computer Systems
Borenstein. '\athaniel S . ln!Ormation Jcdmology ("enter and Com-
puler Carnegie \1d1on t:ni,ersily.
Chris.\ Computing. Cmnegie \1d1on (fmp-
erative Work in the .1mlre" S) .\/em. (l9l>g) pp 306-323
l rowbridgc. David Center !Or Design of Computing.
( arnegie-\1d1on l: ni,ersily. [ :,ing Amken"jor /)t.'!dopmem <?/Fdu-
caliona/.1pp/icatiom. pp J-(i (CY!l .JJ c-g5-06.'i)
Palay .. J. ct al. ln!Onnation Center. Camcgic-
\1d1on l"he Andren l"oo!kit: An 0\"<'ITien. pp 1-15
l.arry K. l"ht. (-.lfl: PC St.tTt."i" l'n?ieCI l <Jl>6) pp l-30
(CVIL-ITC 1>6051)
Gmhn .. \ Y! and llohl. A C .. 1 Re.wanh l'<r.'fNCiiveon Compuler-
//{,./;.( l<J79)1B\1 Sysl J \ol 11>. '\o 3. pp 4.12-4."6
Bkher. J 11 . cl al . A Graphi<lmt.raCiivt. Application Moniwr; l!Hf
::,:r.<l . . !. vol J<J. '\o 3 ( tnO) pp 31>2-402
Gardner. PC . A \ntem fi:!r 1/te Auwmaled Oflwe JOmimnmem, IBM
S).\"1 . .!.. H>1 20. '\o 3. ( l <)l; l) pp 321-.14."
B . cl al . Goal\ and 0/?i<'clil"t.".\" jor [ :,.,.,lntt."i:fiic<' Sc!finmt;
("ompltler Ciraphic.\. vol 21. '\o 2. (Apr 191>7) pp 73-71>
conk1in. A Sunt.J" <<OZ'P'-"1"/<'XI. \1cc
Progra.m 191>7). pp l-31>
llalbcn. DC .l'mJitWIW!iiiJi b) Fwmpi<'; D<1JI. Uei"lrica!I:.IIJii!l<'er-
and ("omp. Science.\, ( "niwnil)" of ("alifomia. ('\ov
191>4)pp 1-76
W"1ker. B. "1. l"ht. !.OCl 'S f)imi/mied Opt."IYiling Sy_\'lf.m' .
( "niver.\il) ofCalifi:!mia /.o.< Anjide.\ (191>3) pp 4'!-70
!Jill. RD. Some lmponam 1-ea/W'<'> and 1.<.<11<'.' in I ".wr Imertace
Managemem Sy,Jem. f)ynamic Gmpliin Pn!it.ct. l."ni,ersily of
loronto.("SRJ.vol 21.'\o 2(Apr 191>7).pp ]J(i-120
Croll.\\' and Lcfkowitf. S. la.\k Suppol'l In an
O!fice S)_,um; and !t!ii)l'maJion S<ience f)qmrtmew. l 'ni-
,.,.,.,ity '!(.lfa_,.,-m/m_\'<"11\'. (1 91>4) pp 22-24
lishncr. S S . ct al . J inua!JOmimmlwl!l Di.,p/a_l S_nlem, lmeracriw
JJ) Cirap/iic.< (<kt 23-24. tn6) pp 77-1>7
/.ieglcr. K . Jr . . 1 Di.\lribu!ed ln/i:!rmarion 2>) .<l<'m Srudr; flJJf S) .\/ .. ! .
n>1 !1>. '\o 3 ( !97<J)
Vlikc. A S/ie/1 for Modem Pel'.<onal ("ompurer.,, l 'uiwr.<il) of
Califi:!mia. pp 13-I<J
Trigg. R 11 mul1rish. P\1 .ll)fX"i"I<"Xilla/;iJal\':l:\perit.mt.\'<?(Wril-
,_.,.,. in .VcmCard,;IZlfX'i'l'-'\'1 '87 Papt."i"\'. Labo-
rato!y. Xerox Palo Alto Rescardl Center. pp l><J-101>
Donahue. Jrunes. Widom. Jennifer. IV/iiieboard.<: A Cirap/iical Data-
/;'"''-" l"oof. -\C\1 on lnJ(mnalion n>1 4.
'\o !.Jan Jn(i.pp 24-41
Rosenberg. J ct a! . An Overvie" of 1/w M<'>.\ajie 2>) .<1<'11!,
li!li>rllwtion l"edmofor..1 Ct.filt."i" Camegie-.lfd!on [."nil"t."i"_\'il)' (.hoi
l<Jl>7). pp <J9-101>
Vlorland. D Verne. lluman Factor.' Guideline.< fi:!r fi>rminal Imer-
jiue f),.,ign; ('ommunicatioti\' <?/the A(:lf (.ho1 19K1) n>l 26. '\o 7 .
pp 41>4-4<)4
Good. Y!khael D. ct al. a I ".'<'1'-D<'I'iwd lmel'(ace; Com-
!lwnicalion.< of rile ACJ!. (<kt tn4) vol 27. '\o 10. pp 1032-1043
Smith. Sidney I_ and .honeD. Guiddine.\'.for f),._,igning [ .-_.,_.,.
fiilei:faceS<?filmre; l-\t.rl.ab. Inc_ (_\ug l<Jl>6). pp 1-.11>4
""A Comparison of Applkation Sharing Y!edmnisms in Real-! imc
Desktop by S R -\huj". al. _\T&T Bd1
I "bmalories. !9<)0
""A Distributed !Or Vlultimedia by
Lve Sdtoolcr. IS! Rescardl Rep011 ('\ov 19'!1)
"_\ \1ullimedia by
Schm>1er. d ;ol. -\C\1 Sigois Bulletin. n>l 1. '\o L pp 12-22 (.hon
]<)g9)
""An .\rdliKdure !Or Vlanagemcnt." by Fvc
Schm>1er. cl al. IFF 4'
1
' C0\1SOC Work-
shop on Communkations. VIVI "92. pp 271-274 (Apr
I<J<J2)
Study Control in"
Journal or lnternctworki ng ;ond
<J<J-120(Jun 1993)
""CFRC Repon Series '\ote." by Kankanahalli
Srini,-,s. al. 1992
collaboration . \warcncss in Supp011 of< "ollaborat ion
Requirements for the '\ext Generation \\'indow Systems."
by J c-hris d al. 1<)90 c111 '90
"CSCW '92 Demonstrations." by \1u1kr. cl al. '\m- !9<)2
""Design of a Y!ulti-VIedia \'chide for Soda! Browsing." by Robc11
Root. Bell Communkations Rescardl. 191;g
"Distributed \1uhiparly Desktop \krmaid."
by K;o;;uo W"t"bc. ,] Csc-w 90 ( kt 1<)90
""FmopaK"s Integrated (]Ill) Larly
by Bull Buxton. ct ai.IJscvier Publishers B \'
('\oo1h-llolland). 1!-34
""Explorations into Real-Time Conkrendng." by !!any
Forsdkk. Computer Message System g). J<Jg(i
"Kermo1 Suppm1 J(lf I i,-e Digita1_\udio ;ond by Ke,in .lc!Tay.
cl al. In oflhe Workshop on ;ond
( lpcmting System Suppolt !Or Digital .\udio and '\ov I '!91
""Y!VI Conf Anlnfmstmdure for Building Shared .\ppli-
by Crowky. al. CSCW 90 (( kl
l<J<JO)
Prindplcs. Systems and Applkations." by Lars
Kjd1dahl. a1. -\pr 1991. or 1<J91
Workshop on \1u11imcdia Systems. ;ond lnlemction.
Stockholm.
Confcrcndng lias it Comcof.\ge''."" by Fvc
cl al. 24'
1
' ll;m,ii International on
707-7!6(.hon !9<Jl)
""'\-Way \Vith Padct Stephen casncr. ct a!.
The J hint International Workshop on Video. VIm 22-23.
]<)<)0
one is '\ot Enough Y!ultiplc Vk-v.s in a Vledia by William
Gaver. ct al.!nterdli. Apr 24-29. 1993
"Piloting of \1ullimedia lnlegmk"<l !(>r European
by PT d al. 1'\ET '<)3
""Po11holcs Supp<H1ing Awareness in a \\'ork Group." by
Paul Dourish. ct a!. Clll I <J<J2. Vlay 3-g
US 7,669,134 Bl
Page 3
Lnvironmcnt Rave System." by Wil-
limTI GmcL d ;!l. R;mx Xerox C;unbridgc Furopan;. 1<)<)2
"lkndcvous An for Syndlmnous Ylulti-Lscr .\pp!ka-
tions." by John J'atlCI'SOJl. Cl aJ. CS{"\V ( kt ]<)90
Replicated -\rchitcclurcs for Slmrcd Window Systems _\Critique,"
by J Chris l.aliwcrs. d ;!l. { >li\clli Rcscan;h catiJ(Jrnia 1<)<)0
"System Suppon !Or lntcgmk'<l Desktop Confcrcncing." by
.\my l'cm-1. Sunmkrosystcms Laboratories. 1992
']Cam Work St,lion Towmds a Sc;unlcss Shared Workspace," by
llimshi. :\T l !Iuman Interface Laboratories. CSCW 90
ings.<kt 1990
The Connection conlorl Protocol .\n:hitcclurc { by hc
Schook!, d al. CSC lnonmtion Institute (Jan 21>. 1<)92)
"The !mpad of on a ('onnedion
by h-e _\(-\1 Jourm!l of\1u1timedia Systems, \ol 1. '\o 1.
pp 2-<J (1<)<)_1)
"I he !min System (1\'S)." by I hicny llll'ktti.
Octob<cr 1<)<)4
"!he V!ulti \\'orkstation." by Dkk Phillips. Siggmph 'g<)
!'and
"Touring _\ Soll\\are l'h!tJ(mn ](,-Distributed \1u1timedia
.\pp1ic,tions," by \1 -\mngo. eta!. 19921F!I' !nterm!tiona! Confer-
ence on Lppcr Layer Protocols. Architcdurcs and .\pplicmions. V!ay
!<)<)2
"'](m;mls" Cni\ersa! Direct01y by Keith L;ml>'. cl a!.
Depm1mem of Computer and Lnginwring.
Stan!Ord l'niversity. I <)K'i
"Videoconferencing '" a T<-chnology to Support Group Work _\
Re,-i<w of Its Failure." by Carmen Fgido. Bdl Communications
Rescardl. J<Jgg
* cited by examiner
U.S. Patent Feb.23,2010 Sheet I of 8 US 7,669,134 Bt
~ Bobby0211-lnstant Message
1-ID!X
File Edit Insert BobbyD211's Warning Level: 0%
BobbyD211: Hey, how was the fishing trip?
Fred1432: It was great! I couldn't believe the
size of the big mounth bass that Jimmy caught
BobbyD211: Wow! Sounds like it was an
awesome trip.
Fred1432: II sure was!
Ail!J ~ ' - ' A ] B /y
!ink 5l 9@
"\
" & 8? ~
7fv
Wam Block Add Buddy Talk Get infQ Send
1111111111111111
404
FIGURE 1
(prior art) /406
()
0 0
Instant Messages with Sue
402
/410 /
I-
~
c
-, . ,_,,.,-
(Hey. what"s up? [.]
Ill iNot too much ) (412
(Did you hear what Bob did
[.]
I-
yesterday? I
Ill iWhat did y o ~ do?!)
(Ohman, it was so great. You
~
shoulda been there. man!
400
408
I
~
FIGURE4
U.S. Patent Feb. 23, 2010 Sheet 2 of 8 US 7,669,134 Bt
sl
00
0
-
"
0 0
: : : ~ , -
( ~ ~ ~ D .
0 .--------.--.--/ D.

D.
~ )
0
"
0
-
~
0
-

D.
-D.w
)'
0
U.S. Patent Feb.23,2010 Sheet 3 of8 US 7,669,134 Bt
300 initiate
--- session, identify parties, and determine icon
for each party
.
304
(

Generating message............_.........._
to other party? ____...//
! to other
'I indicating message is being 1-----------< N
1
__ _ _ '-........ f 306
/ ---.,__

)308 "----. other party is
/-
,-. - ------- -- ' /
' Produce graphical image of
[
message being generated by 1 0
other party / '-.....
//
Received
312 I other party?//
Di:play mes:age and icon 1 -,"-:J///
adjacent of display I
--------- Y to
-----............... party complete? ____..__...---
..... ........_,____
Send message to other party
N
and display message and icon 1----------.j
locally adjacent right side of
window
316
FIGURE3
U.S. Patent Feb.23,2010 Sheet 4 of 8 US 7,669,134 Bt
404
406
\ /
~ 0
0 Instant Messages with Sue
402 '-
(410 /'
-
'\
\.> ,_; ~ i ; :.'1
( Hey. what"s up? [J
Ill i Not too much.) (412
400
r Did you hear what Bob did
[J
-
yesterday? /
[ l ] a o ~
'
500
408
QI
d
FIGURE SA
U.S. Patent Feb.23.2010 Sheet 5 of8 US 7,669,134 Bt
404
406
@) 0 0 Instant Messages with Sue
402
410
400
i Not too much.)
~ Did you hear what Bob did
ISJ yesterday?
408
oO
Ill 500
_iBobwho?)
FIGURESB
U.S. Patent Feb.23.2010 Sheet 6 of 8 US 7,669,134 Bt
300 request to initiate instant messaging
session, identify parties, and determine icon
for each party
.. -- ----
304 302 -----
(
....- Generating
to other party?

j
-/----
indicating message is being ---------1 N
deliver signal to other party
_____ generated . _ ( 306

Receive message that .....
poo party is
I

image ...--
message being generated by 1--------_...J N
other party based on time
created
---. -----
602
- -- Display message and icon
adjacent left side of display _
window based on time
completed
----
Send message to other party
and display message and icon
locally adjacent right side of
display window based on time
_ __ completed
604
FIGURE 6
310
U.S. Patent Feb.23,2010 Sheet 7 of8 US 7,669,134 Bt
404
406
"
r,
0
0 0 Instant Messages with Sue
402
410 ,
-
:-
1\
o::; ::: (
(Hey, what's up? ( [,]
IIJJNottoomuch) (
412

[Did you hear what Bob did, [ [] I--
yesterday? /
Ill :9r2 \__ 700 /704 . :
oO
.
--------
Ohman. it was so great. You "
702 l shoulda been there, manJ []
400
408
Qj
~
404
FIGURE?
406
~ - ~
------ -
~ -
8
0 0 Instant Messages with Sue
.
4G2 :-
. ' (410 v
1--
.. ,'
1\ (Hey, what's up?( 1:!1]
1111 Not too_rn@l) (
412
[Did you hear what Bob did, l[J
I--
yesterday? ....,
I
BOOl\ e - ~ - . Oh man, it was so great. You
Ill shoulda been there. man! [I]
{What dld you do?l ~
704
400
408
c:
-- ----------- ..
- - ~
Q

-
~
FIGURES
U.S. Patent Feb.23.2010 Sheet 8 of 8
13 6
FtF===='jff13
FIGURE9A
The balloon will fit tightly
around this message.
1000
FIGURE9B
20 12
r = + + = = = = ~ g
FIGURE lOA
width-(20+12)
FIGURE lOB
US 7,669,134 Bt
height-(9+13)
US 7,669,134 Bl
1
VIETIIOD X\!) APPARATUS FOR
I>ISPL\YI:'\G DUIU:'\G .\:"\
1:'\ST\:'\T \1ESS.\GI:'\G SESSIO:'\
2
unit. Data is received Jium a second participant lo the com-
munication session. and infonnation rdming to the dmn
receivl.-d from the second p.1rticipm11 is displayed on the Hrst
display unit. wherein the infonnntion receivl.-d from the sec-
ISACK(jROUND OF Till: IN\"I;NT!ON
1. Field of the Invention
.< ond participant is displayed a! a second position on the Jirsl
display unit. The Jirsl and second positions arc spatially dis-
tinct.
"111is invention rdntcs gcncrnl!y to a user intcr!Ucc for dis-
playing an exchange of messages during an instant messaging
session. aml more p;u1icular!y. to a mc!hod and apparatus I(Jr lCJ
displaying instant message exchanges in a manner that
grnphically dillCrcntiatcs the participnnts inn convcrsmion.
2. Description of the Related Art
Networks. such as the Interne!. intrancls. or other pri\aleor
public networks. arc ubiquitous. In beL many compulers arc l'i
conn<..><::tcd to one or more networks nl the same time. For
cxmnplc. n business may have hundreds or even thousnnds of
complllcrs coupled to its own privmc network. which was. m
!cas! initially. used primarily ll1r storage and exchange of
com puler Jiles. AI !cas! some of these same business cmnpul- ""
crs may also be coupled to the internet. Further. with the
dcvdopmcntofwirdcss devices. nd hoc nctworks mny also
he J(mned with properly conligured pm1ah!c de\ ices. !Oven
telephonic dL'\ices. such as cellular phones. pagers and the
!ike. may be coupled lo one or moreofthesc nL""Iworks. Small ""
businesses nnd homes nrc also olien com1ccted in similnr
arrangements.
In another nspL"CI of the prcselll invention. n complllcr
read.1ble program storage device is provided and encoded
with inslmclions that. when cxL-cu!cd by a compuler. per-
limns a mc!hod. rhc mc!hod includes displaying inllmnalion
relating to d.1w produced by a Hrst particip.1ntto the conunu-
nicntion session on n Hrst display unit. wherein the infonna-
tion produced by the Jirsl p;u1icipanl is displayed a! a Jirsl
position on the Jirsl display unit. Data is rccei\L-d Ji-om a
SL"Ctmd participant lo the communication session. and inlllr-
mation relating to the d.1W receivL'<i li-otn the second partici-
pant is displnycd on the !irs! displny unit. wherein the infor-
mation rL-ccivcd Ji-om the second participant is displayed a! a
SL"Ctmd position on the Jirsl display unit. The Jirsl and SL"Ctmd
positions nrc spminlly distinct.
In still nnothcr aspect of the present invemion. n grnphical
user intcrJUcc for displaying inli.mnmion rdmcd ton conunu-
nication session is providL-d. The inlcrliJce is comprised of a
Jirsl and a SL"Ctmd spatially distincl region. The Jirsl region is
ndaptcd to displny nt lenstonc mcssnge from a Hrst participant
to the instant messnging session. The second region is
ndaptcd to display m lenstone message li-otn a SL><::ond par-
ticipant lo the instant messaging session. and the Jirsl and
A!! of this connecli\ ity has naturally led lo cmnmunica-
tions hc!wcen various users mer these nL""Iworks. For
cxmnplc. electronic mail (e-mail). bccnuse of its usdi.J!ness.
is now conunonp!ncc. E-mail is now widely used by busi-
nesses and individuals. nnd in m least some instnnces hns
replaced more lradititma! J(mns of communications. such as
maikd kllers. liJCsimi!cs. telexes. and the !ike. I !owevcr.
e-mail has proven to be somL'\\"]l<ll nwkward when USL'<i to
curry on nn ongoing conversation.
30 sL-ctmd spatially distincl regions p;u1ially merlap and each
include at least n portion thm is li-ce from overlnpping.
In yet another nspectofthe present invention. a method for
displaying infonnation related to a conununication session is
prm idL-d. lnl(mJJalion relating lo data produced by a parlici-
35 panllo the communication session is rccei\L-d. The inllmna-
tion received Ji"om the particip.1nt is then nt lens! p.1rtinlly
displayed witllin n spCL"Ch bnlloon.
Instant messaging. on the other hnnd. allows two or more
users connL-c!ed through these nL""Iworks lo carry on an inlcr-
aclivc conversation. l;xempbry instant messaging sysiL""ms
include Apple iChat. AOI lnswnt Messenger. Microsoli 40
MSN Messenger. and the like. Typically. two or more users
type in messages or selec! icons. which they send lo one
another. n1e rL-cei\ing p;u1y(ies) may immediately respond
with an appropriate message or icon. "J.hcse instant messages
are commonly nll displnycd in serial JUshion. such as shown in 45
I'IG. l. usunlly scrolling the user's screen from top to bo!lom.
Commonly. each message is preceded by a label. such as
JSohhyJ)211 and in l'!(i. I. indicating the identity
In still nnothcr aspect of the present invemion. n grnphical
user iniL""rlilce ll1r displaying inl(mnalion rela!L-d loa commu-
nication session is prmidL-d. The graphical user iniL""rlilcc
comprises n !irs! nnd second region. wherein the !irs! region is
ndaptcd to displny a speech balloon. The second region is
ndaptcd to displny m least one messngc li"om n particip.1ntto
the instant messaging session. wherein the SL"Ctmd region is a!
!cas! partially located within the Jirsl region.
BRII:F J)J:scRJPTION OF Till: DRAWINGS
The imcnlion may he undcrsllmd by rcJCrcncc lo the !(J!-
Iowing description taken in nmjunclion with the accompany-
ing drawings. in which like re!Crence numerals identify like
dements. nnd in wll..ich:
of the nuthor of the messngc. llcretolixc. users hnve relied on
these labels. or other limited indicia. to locme and idemify '0
messngcs from a particulnr party. Accordingly. it will be
apprL-ciatcd that the presenlalion of each message in substan-
tially similar J(mnal makes it dillind!!o readily de!crmine the
authorship of one or more previous messages. I jkL'\Vise. it is
dillicult to go back and quickly locate n previous message
without reading through many pre\ ious messages.
I'IG. 1 illustrmes a vk'\v of n screen representative of n
,
5
graphical user inter bee of a prior art ins! an! messaging sys-
"l11c present imenlion is dircc!cd lo overcoming or a! !cas!
reducing one or more of the problems sci ll1rth above.
SUMM.\RYOI' Till: IN\"I:N"nON
In one aspccl oft he present imenlion. a mc!hod is provided
for displaying inlixmmion rda!L-d to n commutl..icmion ses-
sion. lnformmion relating to d.1w produced by n Hrst pnrtici-
pmll to the communicmion session is displayL'<i on a Hrst 65
display unit. wherein the inl(mJJalion produced by the lirsl
participant is displayed a! a Jirsl position on the lirsl display
!em:
l'l<i. 2 i!!uslralcs a lop-level diagram ofonccmbodimenl of
n hnrdware system on which the present invention mny be
implemented:
l'l<i. J il!uslralcs a Jlowchart of an embodiment of a graphi-
cal user inlerbce !hal may hecxL-cu!L-d by components within
the sysiL""m of I'! G. I lo produce the exemplary screens of
FIGS. 4 nnd 5:
I'J(j. 4 illustrates n Hrst view of nn cxemplmy screen rep-
rcscnlalivc of a graphical user inlcrliJce:
I'! <iS. 5A-51S i!!uslralc a SL"Ctmd and third \iLw ofcxcm-
p!aty screens rcprcsenlali\e of a graphical user inlcrliJcc:
US 7,669,134 Bl
3
I'Ki. 6 illustrates a !low char! of an a!tcrnatin:: cmbmlimcnl
of a graphical user interface that may be executed by compo-
nents wi1hin the system of !-'J(j. 1:
!-'I G. 7 i!lustrntcs a view of an cxcmplm)' screen rcprcscn-
latin:: of a graphical user interface:
!'Ki. 8 illustralcs an ahcmatin:: \iL'\\" of the exemplary
screen of !-'IG. 7:
!-'IG. 9 i!lustrmcs a typical formm for a spc<..>ch bubble: and
I'Ki. I 0 illuslratcs a speech balloon that has been slrctchcd
or mmli!icd to accommoda!c a message.
While the inn::nlion is susceptible lo \arious modi!icalions
and altcmativc fonns. specific embodiments thcn.of have
b<..'Cll shown by way ofcxmnplc in the drawings and arc herein
described in dc!ail. It should be understood. however. that the
description herein ofspccilic embodiments is no! in !ended lo
limit the invention to the particular Ji.mns disclosed_ but on the
comrmy. the intention is to cover nil modi!icmions. cquivn-
!cms.and al!crnmives falling within the spirit and scopeoflhc
imenlion as dclined by the appended claims.
1)1:"!:\IJJ]) DI:SCRIPTION OF SPITIFIC
!:MBODIMI:NTS
4
lion in any environment where two or more users arc inler-
CllllllCC!ed and cap.1ble of commutlicming with one nnolhcr.
Those skilled in !he art wi!lapprccime 1hm nclwork con-
nections may include n vnricly of other equipmelll. such as
5 routers. switches. telephone modems. wireless devices. cabk
modems. digital subscriber lines. and the like. This type of
equipment is no! i!lus1rmcd or discussed in detail herein so as
to nvoid unnccessnrily obli.Jscaling !he inslnlll invention. For
purposes of understanding the instant invention. it is sulli-
lCJ cienllo r<..>cognize that additional conventional equipment of
this type may he uscli.1l in establishing and maintaining com-
munications bclwcen the various users.
At lens! two of! he devices in the system 100 have sollware.
such as an application program. ins!alk-d thereon lo allow an
15 instant messaging session lo he initiated and ctmduc!<..-d. An
instnm messnging session may include renl-time or ncar real-
time communicmions. FIG. 3 illus1rn1cs a !lowchnn of n por-
tion of !he soliwnrc nssoeintcd with itlitiming the instnnt
messaging session and controlling a graphical user inlerbcc
"" (<iUI) us<..-d by the participants lo the instant messaging ses-
sion. In pnnicular. the process begins at block 300 in a con-
vemionnl mmmer with one of the two pnnics sending the
other party an invitation lo initiate an instant messaging ses-
sion. Assuming !hall he other party accepts the im italion. the H!uslralive embodiments of the imcnlion arc described
below. In the imercslof clarity. no! nil features of nn ncnml
imp!cmcnwtion nrc describ<..'d in tills sp<..>ci!icmion. II will of
course be npprccimcd thm in 1he developmcm of nny such
aclual embodiment. numcnms impkmcnlalion-specilic deci-
sions must be made lo achie\e the de\ elopers' specilic goals.
such as complinnce with sys1cm-rela!ed and business-related
constraints. which will vnry from one implemenlmion to
another. Morcmer. it will be appr<..>cial<..'d that such a develop-
men! emll1 might be complex and time-consuming. but
would ne\erthekss he a routine undertaking ll1r those of
ordinmy skill in !he art hnving !he benclil of1his disclosure. 35
"5 sollwarc on each party's computer initiates the (iUI. which
opens n window where both parties' messngcs and other
pcrtincm informmion nnd controls nrc displayed. An exem-
platy rcprcscnlalion ofthe(iUI is shown in 1:1G. 4 and may he
rc!Crenccd simultaneous with the discussion ofl'l(i. 3 herein
30 Ji.1r a more complete understnnding of 1he opcrmion of the
instnm invention.
The mcssnges exchanged by 1hc pnnicipmus may contnin
inllmJJalion regarding an icon lo be us<..-d lo represent each
party. For example. party A may sd<..>cl an icon. such as
"Mnry" 400 as n graphicnl rcprcsentmion of party.\. Party B
may receive and s1orc 1he icon and 1hen display it ndjaccnta Turning now 10 FIG. 2. n block diagram depic1ing n sys1em
I 00 in accordance with embodiments oft he present invention
is illustrated. "l11c system I 00 includes a plurality of cmnpul-
ing d<..'\ices coupkd together through one or more nclwork
conn<..>clions. For cxmnplc. n plurnli1y of devices may be 40
coupled logc!hcr via n private or public ne1work. such as a
local area network (!.AN) 102 or the Interne!. The aclual
conn<..>clion bdwcen the de\ ices and the I.AN I 02 may lake
message delivered by party A "ll1e icon makes it ensier for
party B lo more quickly identiFy those messages associat<..-d
with party A. An exemplary exchange of messages in which
party A hus selcc1cd 1he icon "Mmy" 400 nnd puny B has
scl<..>c!cd 1hc icon "Sue" 402 is shown in FIG. 4. Displaying
unique graphical icons allows a user lo readily identify the
speaker with a quick glance. Additionally. displaying the
icons adjacent each pat1y's message allows the users lo idcn- on one or more ofnny of a vnriely of fonns. such as n network
inlcrJUcc curd (NIC). a modem_ a digilnl subscriber line
(DS!.). a cable modem. a wireless comJcclion_ and the like.
The de\ ices coupled lo the !.AN 102 may include. ll1r
example. desktop computers. such as an Apple Macintosh f\
104. n classic Apple Mnc R 106. nn IBM compmible pcrsonnl
compmcr (PC) 108. and 1he like. Further. these dcsk1op com-
puters. such as the Apple Macintosh R I 04. may he coupled
together\ ia a smaller sub-I.AN I I 0. with the sub- I .AN I I 0
being coupled lo the I.AN I 02. Portable de\ ices. such as the
Apple PowcrBook !\ or iBook !\ 112. mny nlso be coupled to
1he !.AN 102. ci1her directly or ns panof1hc sub-! __ \N 110.
Ftu1hcr. other consumer de\ ices. such as cell phones. per-
sonal data assistants (PDAs). nclwork appliances. and other
cmb<..'dd<..'d devices mny be conn<..>c!cd to !he l .. \N 102 so us to
employ nspcc1s of the inswm invention.
45 til)' !he speaker wi1hout looking nway from !he messnge
region oft he GUI lnnn al!crnmive embodimcnt.thc user may
dec!lo display no! only the icon. hut also the name associat<..-d
with the author of the message. On the other hand. the user
may elect1o displny only the nmne associm<..-d with the author
'0 of the message. preveming the icon Ji"om being displayed
nl!ogc1hcr. if desir<..'d.
Those skilled in the ;u1 will appr<..>ciale thatlhc icons nc<..-d
no! beddi\ercd with each message. That is. party A may send
nn icon during 1he initial portion of1he session. nnd party B
'5 willassocimc !he icon with p.1r1y A. s1orc it locally. and then
rclricvc and display it each time a message is r<..>ccivcd from
party A. Additionally. pat1y A's icon may be merridden
locally by party IS. Thai is. party B may eke! lo display a
different icon ndjaccnt party.\ 's messages. m lens! on the GUI
60 vicw<..'d by party B. Party B may select any of n plurnli1y of
icons slor<..-d locally. and indicate through the local (iUI. such
as by pointing and clicking on \arious pull-down mem1s
provided by the local GUI. that the selec1<..'d icon should be
While 1he invemion hus been i!luslra!ed herein us being
usdi.J! in a network cnvinmmcnl. it also has application in
other conncc!cd em ironmenls. For example. two or more of
!he devices described nbove mny be coupled logc!hcr via
device-to-device com1ec1ions. such ns by hard cabling. radio
frequency signals (e.g .. 802.11 (n). 802.11 (b). 802.11 (g). 65
ISiueloolh. or the like). inlioJr<..-d coupling. telephone lines and
modems. or the like. The instant invention may have applica-
used when displaying p.1r1y .\ 's messngcs.
The GUI mny nlso usc nddi1ional strategies to graphicnlly
difJCrcnliale the parties of the instant messaging session. For
cxampk. a sending p;u1y may send an indication of a color
US 7,669,134 Bl
5
scheme in which his/her messages should be displayed. The
n.>cdving party may then. at his/her discretion. display the
messages li"om the sender in the rcqucst<..-d color scheme.
Altcrnmivdy. the receiving party may ck>ct to override the
sending parties requested prcfen::ncc. and instead display
each p;u1y's message in its own distincl color That is. party A.
during an phase. may indicate through the local
GUithat any message rccciv<..'d from party B should be dis-
played with red k:ncrs and a white background. and that any
messages generated by himsdL should he displayed with a
ydlow background and black k:ncrs. In either case. the color
distinction allows the party to visua !ly determine the author of
a message without the need to read and undcrswnd an iden-
tiFying name, such as is illustrated in the prior art a! FIG. I
(e.g .. BohhyD2ll).
6
nmvcrsalion is preserved, with older messages appearing
nearer the top of the GUI nnd newer mcssag.cs appearing
nearer the bottom of the GUI .. \s the disp!ny region oft he GUI
Hlls. old mcssngcs arc scrolled up nnd om of view .. \ user mny.
5 howc\cr, aclivale a scmllhar mechanism 408 using conven-
tional point and click techniques lo alter the portion of the
conversmion prcscmcd in the GUI. Forcxnmple. the user may
move the scrollbar mechanism 408 upwnrd to view nn older
portion of the conversation. or downward lo \iL-v.- a more
1" recent portion of the conversation.
'Ill !i.u1hcr enhance the readability and lo provide !i.1rthcr
g.mphicn! idcnti!icmion of the author of each mcssngc appear-
ing in the GUL cnch mcssag.c may be displayed in a spL>cch
balloon 410. The balloon 410 includes a tail scclion 412,
15 which generally extends toward the icon associa!L-d with the
mnhor of the mcssag.c. For cxnmple. each mcssngc lfom the
user idcnti!iL'd by the icon Mary 400 nppcars inn balloon 410
that has its wil SL><::tion 412 extending. g.cncmlly townrd the
Allowing. the sender to select the color and style. however.
mny lead to some conli.1sion in the cvemthatanothcr pnrtici-
pmu to the instnnt mcssnging sessions elects a similnr style
and/or !(ml. I Ompowcring the receiver of the message lo over-
ride locally the style and color choices indicated by the sender ""
icon Mary 400. In the cvcnllhal an icon is no! associa!L-d with
the author of the message. the tail sec! ion 412 is still useli.J! lo
g.mphicnlly illustrmc the mnhor. Thm is. since the GUI is
divided into !eli nnd right horizomal regions. 404. 406 a
spL"L><::h balloon 410 !oca!L-d in the !c!l horizontal region 404
with its tail scclion 412 extending toward the !c!l will still
mny help to nllcvimc any conli.1sion. Thm is. the receiving
puny mny elect to displny the mcssngc with n dillCrcm color
and style than indicated by the sending party, a! leas! on the
(jlJJ \ iL-v.Ttl by the receiving party. rhc rccci\ ing party may
selcc! any of a plurality of colors and styles stored locally. and
indicmc through the locnl GUL such us by pointing. and click-
ing. on vnrious pull-down menus provided by the !ocn! GUL
that the selcc!cd color and style sl](m!d he used when display-
ing the rccci\L-d messages. Ahcmali\cly. the (jlJJ may be
progrnnuncd to autommicnlly nssig.n a dillCrcm color to each
pnnicipmu.
"5 prmidc a graphical indication of the author (e.g., Sue 402 in
the embodiment of FIG. 4 ).
The site of the balloon 410 is controlled nccording to the
length of the message. That is, the GUI rccci\cs a message,
dc!crmincs the length of the message. dc!cnnincs the size
An additional grnphical distinction mny be nccomplishcd
by partitioning thc(jlJJ into spaliallydistincl regions and then
dirccling the messages loa region basL-d upon its authorship.
I-' or cxmnplc. the cxcmp!nry GUI of FIG. 4 hns bL-cn g.cncmlly
divided imo two horitonwl rcg.ions. n !eli rcg.ion 404 and a
rig.ht reg. ion 406. I-' or example. nil mcssngcs gcnerntcd by the
local user (party A). rcprcscn!L-d by Mary 400, arc displayed
30 (e.g. .. based on the number of lines of text to be disp!nycd) of
the bnlloon 410 required to displny the mcssag.c. nnd then
draws the bnlloon 410 with text in the npproprimc horitonwl
portion of the GUiusing the colors. style, and icon associa!L-d
with the author of the message. A more dc!aik-d discussion of
35 the sizing nspcct of the speech bnlloons mny be li.ltl!ld below
in conjunction with FIGS. 9 nnd 10.
in the right region 406. and all messages gcncra!L-d by the
remote user (party B). represented by Sue 402. nrc displayed 40
in the !eli region 404. It should be apprccimL'd that the assign-
ment ofle!l and right regions lo parties A and rcspccli\cly.
may he rL'\crsL-d without departing !i-om the spirit and scope
of the instant imcnlion. Morcmcr, it is anticipated that the
various spatinlly distinct reg. ions may over!np. Thm is. a mcs- 45
sng.c g.cncmtcd by Mal)' 400 may extend lfom the rig.ht region
406 and a! leas! partially into the !eli region 404. Similarly, a
message generated by Sue 402 may extend !!om the !eli
rcg.ion 404 nnd nt least partially into the right region 406.
Thus. the mcssngcs mny ntlenst pnrtinlly overlnp. depending '0
on the length of the mcssngcs.
Further, depending upon the numhcrofp;u1icipanls. it may
During nn inswm mcssnging session it is olicn useful to
indicate when a remote party is preparing a message lo he
sen!. l'or example. a!kr party A sends a message requesting a
response. it is uscli.ll to know if party B is prcpnring. the
requested response. Knowing.thm the other party is abomto
respond allows a more natural !low lo the conversation. For
example. if party B docs no! answer a question !i-om p;u1y A
in a timely manner, p;u1y A may send a SL><::ond. related
request. Party B. however. may promptly respond to the Hrst
request. leaving party .\ to g.ucss nt whether the response
applies lo the !irs! request. the SL><::ond request. or both.
Accordingly. in thccmbodimcnl ofthc(jlJI shown in J'](j_
3. nt block 302 the sollwarc determines whether a mcssng.c is
being. g.cncrmL'd. and in the event thm a mcssag.c is being
prcpnrL'd. the software m block 304 delivers a signn! to the
other party indicating that a message is being generated. One
mc!hod !\1r dc!cnnining whc!hcr a message is being gcncr-
mcd is for the remote terminal to monitor an inpmlicld for any
he useful to delinc more than two spatially distincl regions.
1-'or example. where three pnnicipmus arc present. it may be
uscli.llto provide three horizontn! regions. '5 charnctcrs having been cmcrcd. such ns via the kcybonrd. nnd
report hack lo the local!cJminal. For example. if the soH ware
dc!L><::Is that a kLJ-strokc has been entered or that cmolicons
ha\c been sck><::ted, then it assumes that a message is being
In the exemplary (jlJI of l'l(j_ 4, the lex! of the messages
associa!L-d with Sue 402 is displayL-d in the le!l region and is
!eli _iusti!iL-d. Similarly the lex! of the messages associated
with Mm)' 400 is displnycd in the right rcg.ion and is right
justiHcd. Those skilled in the nrt will npprccimc thm other 60
_justi!icalion schemes may he usL-d without departing !ium the
spirit and scope of the instant invention.
In one cmbodimcm of the instant invention. the order in
which the mcssng.csnppcaron the GUI gcncrnlly corresponds
to the order in which they were rcceiwd. I-' or example. in the
illuslra!L-d embodiment of l'l(j_ 4. each message is displayed
below pre\ iously rccci\L-d messages so that the order of the
prcpnrL'<i to be scm to the other party. and the software sends
n signal indicming thm n possible responsive mcssngc is being.
prcparL-d.
AI block 306. the so!lwarc checks !\1r a signal !imn the
other remote party indicating. that n message is being. pre-
paR-d. If such n signal is received. comro! trnns!Crs to block
65 308 where the GUI is activntcd to produce a g.mphicn! indi-
cation that a message is being prcparL'<i by the other p;u1y. An
exemplary rcprcscnlalion of the graphical indicator is shown
-;,sqns am ptm p;ol1,;oJ:J :JJ;o_\'\. k")t1l Wl]l ;,m!l ;,t1l uodn p.")st'q
p.")J:J(l.lO am pm"1 J;ot1JO ;,t1l uo 'S:JJqqnq lt11lnmu_ p.")P1dmtn
;:Jt]l Flt]l ;:Jt]l uodn
r09 pue "('."09 "009
JO ;:ll]l l]l!-\\ "9 JO )!Jljl 0) .(!!P-1)
-tmJsqns f "!)1:1 u! l1JJOJ ;,t1J .\!!m;ou:J!)
.\11m1p1' .\;,t1l l(Jll(.1. ll! J:l(l.lO ;otp ll! !fl!) :ll(J m Jt,:Jdd1'
,\;:Jt]l Wl]l OS SUOO!!P-q ;:ll]l'i!UU;:JpJO;:JJ U! pOtj);:lUl 09
e JO )U;:Jm!poquw ;:JUO 9 JO petp_\\O[l ;:lliJ.
1' 11' ;ot1l
JO _\'\.0[1 :ll1J p1,a1 ;ot1l ;oJ:Jl(.1. .\pt'1lD!J.l1'tf
.10:1 J:ll1JO ;ot1l.JO l1"(J lUOJj
um'i!;-oJ pmJs!p 1' u! :llj r1no:J
symdp!).Wd J)OUWJ "SlUJtmpoqm;-, JJl]IO U] '_\\OpU!.\\
S,)UL>dp!)Jed ;:JtjljOf:'Ot Jj;)] ;:Jtjl UO
Jptjl JO snmdp1ued ;:J)OtuJJ Jo o.\\J
;o.11' aJ;ot1l aJ;ot1"" -w1pm-">\ ;oqqo tOt ;op1s \PI :ll(J
UO p:lJU:JSaJd ;o.11' p.")Jl,J:JU:J1J .([;oJOlU;oJ 1Jl' ptm "lfl!)
;:Jtj) U] _\\OpU].\\ ,\r,]dS]p Jl]l JO 90f:' Jp]S Hj'ilp Jl]l UO
;:JJe ,\j]eJO] !!!J "HWUHpoqtuJ JUO UJ
:JJOUl JO :J.'Ul1J'i!U!\10.\Ul SU()[SS:JS 1Ju!'il1"S:llU
Ul p:J.\01dm:J .\ppt!.")J ;oq .\1'm UO!JU;o\Ul :ll(J
1lu!111"s:Jm m' OJ O.LJ an' aJ:Jt1J 'P!'1"" OJ ;os!J ;o.\!TI .\1'm 1luu;op.1o 'f:'()L :J[qqnq
;:Jt]l lOU "J;:J)JP- ZOL lt]ilnotjl U! ,\Jejdm;:JX;:l U! U;:JJq S!Jtj UO!JU;:lAU]
lUeJSU! ;:Jtj) J]!tj.\\ Jet] I ;:J)epJJddr, !!!.\\ UeJt]l U] ;:JSOllJ. ;:Jt]l t]ilnotp "POL -zoL
-ooL .1;opm :ll(J u! p:JJJn:J::>o ;,wt1 01 .m;,ddt' II!-"" uo!J1".n\mn
:ll]l U:ll(J Tiu!J:l(l.lO-;oJ l]ll">\ ;,1qqnq Jt1Tinm1J :ll]l s:J:Jl'1daJ .\p.1;om
(U_\\Ol]S lOU) Ullll!!P-q ;:ll]l J] ;:ll]l
(U_\\Ol]S )OU) UOO!!eq r, .(q m.\\ ZOL ;:l]qqnq 0>
)l]illllltj) ;:ll]l ;:JtjJ pue ZOr ;:IllS ;:JJUO
"POL ;o1l1"s:Jm 1' JU:Js 111q JO_J JW">\ Jtm P!P
'J;o\;o_Lm1"00P .ZmVi[ '()()L ;ot1l .Lopq ZOP uo::>! :Jl1S ;ot1l
S! ZOL lt]ilnotJl e oot
.\.iejiiJ .\q OOL e OJ e
JO SS:JJtud :ll(J Ul Sl Zl)f:' ;ons 'L '!)1:1 ll! JU;ouupoqm:J
:ll]l Ul 'U()[WSJ:l.\UOJ l' Ul JO U()[Jl'JU;osaJd J:l(l.lO
J0-]110 tm OJ L "!)1:1 Ul
)Ue)sm ;Jt]l JO:J SJ! pue
,\J.md JSJ!J ;Jt]l e ot
pum:Js;ot11Jl1lj :Jsuods:JJ t''ilu!.mda1d S! .\JJl'd JS.l!J 1' tpll(-""
u! :Jsm11 Tiu!mp ml!J1"J;o\mn 1'JO .1;op.1o ;,t1l :JU:JsaJd
OJ S! ;:Jtj) JO )U;:Jm!poquw UJ
S! f '!)]:J U! JtjJJO UO!UOd e
.\J.l1'd
Jl':lo1 ;,t1Jl(J!-"" ;o:Jtmt1u;, OJ JqTiu
S! pur, OJr UOO!!eq r, U! S! )X;:JJ Jl]l
.(!!eUO!)!PP\' "J[E) ;:JtjJ U] .\WpU!.\\ e JO 90t Jt]ilU
;:Jtjl uo p."),\e]ds!p .(ped Jt]l ,\q
;ot1l s ptm p "S!)!:! JO SJU;ouupoqm:J :lt1J u1 lfl!)
Jl':l01 ;ot1l Ul J()[(Jlm :ll(J l(Jl">\ fA")Jm:JOSSl' UO:l! :ll(J l(Jl">\ 1JUO(l'
(J'j!i ;:Jtj) JO "UOJ!)OUJJ JO) JXJ) Jl]l ;:JJe.\\
\IllS ;:Jtj) .(i!P-UO!)!pp\" "UOO!!eq njilnOtjJ ;:Jtj) JO
Ul ()lf:' U001Jl'lJ tp;-o:Jds :ll1J Ul p:J.\t'[dS!P U:ll1J Sl ;o'il1"S:llU
;otJ_L .\J.l1'tf J;ol110 ;otp OJ [[(l!J:l.")UUO:l )'(JO.\'\.PU :ll1J J;o\l) ;o'il1"S:llU
;,t1l S.l:l.\!{JP ;oJ1,">\JjOS ;,t1l ptm 9J :-j:lO(q OJ SJ:J_]Stm.lJ10JJUO:l
os J! S] .\ued ]eJO] ;:Jtjl .\q 'i!uwq
;:Jtjl J! JU]UU;:JJJp OJ ;:JJe_\\ljm Jl]l "f:'J 'jJOJq l\'
:lJOUl;oJ "J:ll(JO :ll1Jl(Jl">\ ;o:Jtml1U;o
OJ l,P1 Sl put' ()Jf:' U001(l'lJ tp;-,;,ds l' Ul p.")JU:JS
-JJd S] JXJ) Jl]l ,\!!eUO!)!pp\' 'J[E) Jl]l m _\\OpU!.\\
e JO rOt Jj;)j ;:ll]l UO ;:JJe .().md J)OtuJJ
"JJtjlll Jl]l mOJJ p;:!AWJJJ ;:Jtjl"S pur, r
-1poqm;, p.")Jl,JJSll[[! ;,tp u1 'JX;oJ 1lm.\tmdmo:J:J1' ptm uoo1
lP- JO )U;:Jtu!poquw J,\!)etUJJje ue 9 OJ .\\OU ilmuJnJ.
;,t1l :JJ;ol(-"" zot :-j:lO[q OJ:-j:lt'q SJ:J_]Sut!lJ[lUJUlD
u;,t1l 1Ju!'il1"S:llU JUl,JSU! ;ot1l.J! ;os!-\\.l:ll(J()
umJiltud Jt]l JO dn 1lmpm.\\ .\p;-,pJo pue JJdOJd e JOJ
-uodsJJ umJiltud JJt]IOtm Ollllll SJJJStWJl pur,
Jl]lJO ilu!ptw ;:Jtjl 8Jf ;:ll]l tmop OC
:lS!_\\.l;ol110 JO :-j.lO_\'\.pU ;ot1l.JOJJO 'i!u!1!1Jo1 ;ol1lJO
l[JOq .10 :Juo .\q st' umss:Js'i!m1!1"s;-,m ;,t1l.J 1
-(l'lJ ;,t1l .\q p."):Jl'1daJ S! ;,1qqnq Jt1Tinm[J
,\tm UJ "JOt]llle tJl!_\\
-mse uoJ! Jt]l tjlL\\ iluo1e Jo Jo)
JXJl ;:Jtjl JJR\\ljm ;:Jtjl ;:JJJt].\\ ZJ 01 SJ:J_JSUeJJ
10JJUO:J os .11 .(JJl'd J:Jtpo :ll(J p:l.\l:J:J.")J u;,;,q ;,1\1"
1' .11 :Ju!mJ;opp OJ am.Luos :ll(J o If )-[:JO]q IV
Jt]l ,\q
;:JtjlJO p;-,Jepdn .\ue::>!pop;-,d UJt]l JJe ]e]ued

Jl]l ]fl!) ;:Jtjl JO JP!S \Pi JtjJ JJO JJpjO
l(J!-"" J;op[O JO Jl(TI!.l :ll(J OJ p.").\qds!p ;oq .\1'm
-s;-,m JU:J:J:JJ a1om ;,1dt1mx;, J0.-1 Jtm]sm
.\mm1 m p.").\o1dm;, Tiu!uons Jl':l!J.l:l.\ uommtn ;,tp OJ p.")soddo
,\!!elUOZ!JOl] 0) UO!)P-SJJ,\UOJ JtjJ _\\O!!!J 0) J11J;)Sil
,:c ;otJ_L ,; , .\q tpns ;,p[dtuo:J p.\ ]tm :ll(J J1"[J
ml!J1':l!PU! :l!t1dt!JTI 1' .\q p:l!tmdmo:J:J1' :J'i!1"s:Jm 1l'!J.ll'd :ll(J
"JU:lm!poqm;, :J.\!Wtu;,J]l' Slt1J u1 'P:JJt,daJd Tim:Jq 1'
JetjJ Ue sr, )U;:J!dpJJ ;:Jtj) OJ ;:JJe
Je!).md Jo nwunpoqm;-, J.\peuJJl]e ue U]

JO j:l-\;o( J:llf!llt11' J1' U()[SS:JS Jlm]SU!tm Ul :JWdl:l!J.l1'tf
,\]:-jJlllfl ;:JJOm OJ ;:Jjqe Jq !!!-\\ pur,
;:Jtjl JO UO!WJJdo pur, UO!PtH1J ;:Jtjl pue)SJJpun ,\em
SJJsn uJ,\;:1 .(]'i!mpJoJJ\" ]enpe ue OJ
Jq OSje ,\em)! SUO!ilJJ tjl!.\\ J[E) r, U] "UO!ilJJ
:l1PP!Ul (lD!).l;o\ 1' tpns um'i!;-,.1 :lJl'!fA")UU;oJU! Ul' p.")UTI!SS1' o<:
':J.\Oljl' 'JO J:ll(PTIOJ p.")dmuTI ;oq .\1'm
JeUO!)!PPV uo!'i!JJ monoq Jt]l ,\em )tmd
-p!)JP-d J)OUWJ ;:Jtj) pue "]fl!) ;:Jtj)JO UO!f.JJ dO) Jl]l
.(etu nmdp!Ued jeJO] ;:ll]l"JjdUWXJJO:J "J[E) ;:lljlJO UO!)JOd
!J.l:l.\ 1' fA")U1\!SS1' 1\U!:Jq ]tmdl:l!J.ll'd l(J1';o l(Jl">\
PU!JS!P .\]]mJt,ds OJU! lfl!) :ll1J'i!U!p!.\lp .\q ;oq
,\em snmdp!ued sno!Je\ Jt]l JO:J uo!HWAm me)sm
,:1 JOSJil:J;tJd 1' .\p\!lll1JU! .\1'm :ll(J
"(){f:' UOOIJl'q l1:l."):Jtfs ;ot1l 'UllUJ :l(lJ1"1S!Il'ilU!JS!P .([!S1,;o Jlllj "OJ
Jenuns OOS njilnot]l pJ]tiJ.
.(uetu OJ Je!J!mP-J UOU!UlOJ
;o.11' s;-,[qqnq Jt]illllll1l pmn")s p:J.(t,1ds!p :llj p[llll_\'\. :J1l1"s;-,m 1'
Jl]l JO JdOJS pur, Jp!dS ;:JtjJ mOJJ )lllltj)!-\\
J;ot1JO u! SJtmdD!J.l1'd sno!.m.\ ;,t1J qsm'ilmJS!P .\tml
J;ol110 J1"(J fA")UO!SL\U:l Sl l! "SUO!'i!:JJ 1l'JUO/!J()[1
PU!JS!P .\[[l'!J1,ds OJU! P:JP!\!P lfl!) :ll(Jl1J!-"" lLLm[S u;o:Jq ;owt1
U!JJ;:Jt] ;:Jtjl ;:lJ!lj.\\ .(!!eUO!l!PP\'
mwpJJt]l
'ilmpu:J]x;o :J[qqnq tp;-o:Jds :ll(J ptm um1!;-,.J(l!lJU;o:J;ot1l
ll! 1\u!.m;,ddt' UO:l! J;ol1fS!l1l(J!-"" um'i!;-,.1 (l'JJU;o:J1' OJ
)UL>dp!)Jed pmjJ e "HS m U_\\Ol]S )U;:Jtu!poquw ;:JtjJ Uj 'PU!)
-S!P PJ!tJl pue "ls.!!J JtJJ upJJt].\\
.(];:!A!PJdSJJ pm]l pur, pUll:)..")$ ")s.!!J U! p;-,,\e]dS!P
Jq r1no:J ;o)om:JJ PJ!l]l ptm puo:J.")S tulUJ
8
fll J1"1J UO!TI:lJ 1m;ou:J'i! :ll!ms :ll(J ll! lfl!);ol1l Ul s.m:Jddt' ()()S :l1qqnq
Jl1'ill1m(J ;otp "JS.ll:1 ":JSUOdS:JJ 1' 1\u!.md;,Jd .()Jl,d J:ll(JO :ll(J
WtjJ ;:Jtj) JOJ 00S Jjqqnq
HJ'ilnot]l ;:Jtjl suoseJJ JO e JO:J e iluped
-:JJd .\[JUawu s1 zot ;ons l1"[J ml!JtDIPU! (tD!l(dm'il1' s:JP!\oJd
,_- ()()S .. ;,[qqnq Jt1TII1m[J, l' "S "!)1:1.]0 JU:lm!poqm;, .\.Jqdm;,x;,
;:Jtj) UJ ;:Jtj) S! Otj.\\ .\ued Jl]l l]ll.\\
-mse ;:ll]l l]l].\\ 'i!UOje f,upjU]ljl S] JJPeJetp r,
WtjJ 01 U] UJIJO s1 se ,;J]qqnq
Jt11lnm[J, 1' ':J[dumx;, J0.-1 s "!)1:1JO Hl!) .\.Jqdm;,x;, ;,t1l u!
L
18 tSJ'699'L sn
US 7,669,134 Bl
9
quently replaced by a speech balloon. Because a thought
bubble may be created wdl before 1hc corresponding speech
balloon is completed. it is possible for other panics to com-
plete messages in the intervening time. Thus. when the cor-
responding spc-.x:h bubble is cmnpk:ted and replaces the cor-
responding thought bubble. the order of the speech balloons
may val)'
At block J06. the sollwarc checks for a signal li"om the
other remote p;u1y imlicaling that a message is being pre-
pared. If such a signal is rcccinxL contmltranskrs lo block
600 where the (jlJJ is acli\alcd to produce a graphical indi-
cation that a message is being prepared by the other party The
order in which the graphical indicntion is displayed is based
upon the time !hal the thought huhh!e was created. The time
thatlhe thought bubble was created may he determined Jium
a time stamp provid<..-d by !he remote user who is in !he process
of preparing. !he message.
"lllcrcalicr. at block J 10. the soliwarc checks lo determine
if a message has been recei\ed from the other party. If so.
control lransJCrs lo block 602 where the so!iwaredisp!ays the
text mcssag.c (or cmoticon. or !he like) along. with the icon
associat<..-d with the author. In this instance. any correspond-
ing thought huhh!e is removed and replaced by the corre-
sponding speech balloon and its accompanying lex!. I low-
e\er. the speech ha!!mm is ordered has<..-d upon the time
complct<..-d. The lime !hal !he speech bubble was completed
may be determined from a time stamp provided by the rem ole
user who genera!L'<i the message.
10
loon and within these margins. Using these margins causes
!he sp<..-cch baBoon lo !it1ig.luly around !he mcssag.c. as illus-
lrat<..-d in !-'IG. 9B. so as lobe acs!hclically pleasing. while no!
unnecessarily consuming. !ar-6c portions of !he GUI. Rcduc-
5 ing the si1e of the spe<..x:h baBoons a Bows more messages lo
appear on the GUI a! one time.
The GUluscs !he informmion regarding. the size of the lex!
message and 1hc desired marg.ins lo produce a speech balloon
of the appropriate size. The process invohes di\ iding a lem-
ICJ plate speech baBoon into nine regions. such as is shown in
1:](;. I OA. rhe nine regions are comprised of J(mrcorners. !ell
and right cdg.cs. 1op and bo11om cdg.cs. and a central rcg.ion.
Initially. 1hc marg.ins arc added lo 1hc rectangular area taken
up by the lex! lo produce a destination reclang!e having
15 dimensions in which the ha!!oon is lo he drawn. The J(mr
corner regions arc drawn dir<..><::!ly into !he corners of !he
destination rcclang.lc without sig.ni!icam chang.c to their
shape or size. The lop and bonom cdg.cs arc tiled horizontally
into nxlang!es of the appropriate width (and original height).
"" \ arying the width oft he lop and hollmn edges has the desir-
able cflCc1 of al!cring. 1hc horizomal dimension of! he sp<..-cch
balloon. The !eli and rig.lu edges arc tiled vcrlicaBy imo
reclang!es of the appropriate height (and original width) lo
produce a slrc!ch<..-d or modili<..-d speech bubble I 000. as
"5 shown in FKi. JOB. II should he noted thatlhe destination
rectangle can be smaBcr 1han !he original 1cmplmc imag.c in
ci1hcr or both its vertical and horizontal dimension. !-'or
example. in the embodiment iBuslra!ed in l'l(i. I 0 1 ~ . the \er-
An exemplary instant messaging session !hal i!!uslrales the
rcsul!s of1his ordering policy is presented in FIGS. 7 and S.ln 30
1hc cmbodimcm i!!uslra!cd in FIG. 7. Sue 402 is in !he process
tica! dimension of the sp<..""\..-ch baBoon is smaller !hal the
vertical dimension of !he 1cmplatc speech balloon of FIG.
10.-\. and !he horizoma! dimension of !he speech balloon is
lai'6Cr !hal !he horizontal dimension of !he lcmplalc sp<..-cch
balloon oJTKi. 10:\.
Once the sp<..""\..-ch ha!!mm is appropriately sized. it is color
H!k-d according lo !he rcquircmcms of 1hc sender or !he
recipient. as discussed above. Coloration and shading of !he
spc<.."Ch baBoon is accomplished by al!crnalivc mc!hodo!o-
gies. In one embodiment. cuslmn artwork is prm id<..-d l\1reach
color lo produce a desired \ariation across the surliJCe of the
of preparing a response 10 a message 700 generated by Mary
400. Accordingly. a thought huhh!e 702 is positioned adjacent
the Sue icon 402 below the message 700. Mary 400. however.
did no! wail for Sue's response. but scm a message 704. 35
Because !he sp<..-cch balloons 700. 704 arc ordered based on
1hc lime completed whereas 1hc !houg.hl balloon 702 is
ordered based on the time created. the order of the messages
wi!! remain as shown in l'!(i. 7. until the message Jium Sue
402 is !inally rccciv<..-d. 40 spc<.."Ch balloon. For example. !he color may be varied so !hal
!he coloration is lighter adjaccm !he bonom cdg.c of !he
sp<..""\..-ch ha!!oon. This coloration scheme has proven lo he
pleasing lo users. providing the appearance of three dimen-
sional qualities lo the sp<..""\..-ch hal!oon. :\hemalively. rather
"llllls. as is shown in FIG. 8. a speech balloon 800 has
rep!ac<..-d the thought huhh!e 702. hut is located alkr (or
below) the speech balloon 704. as the spe\..-ch balloon 800 was
completed a!ier the sp<..""\..-ch balloon 704. In this manner. the
actual order of !he convcrsmion is preserved. 45 !han developing. cus1om artwork for each possible color. !he
spc<.."Ch balloon may be tilled uniformly wi1h !he desired color.
Therealler. standard lemp!a!e shadows may he overlayed lo
produce a similar three dimensional clkc!.
.-\ subswmially similar process occurs with rcspcc1 10 dis-
playing speech ha!!mms associated with the local user. For
example. a! block 314. the sollware checks lo deteJmine if the
message being pre pard by !he local pany is complete. If so.
comro! 1ransfcrs 10 block 604 and !he soliwarc delivers !he '0
!-'inaBy. while FIGS. 9 and 10 illummc !he spc<.."Ch balloon
410 drawn with i1s !ail extending lcliward. so as 10 be used in
!he !ell horizontal region of !he GUI. !he same processes
described herein may he applied in drawing the sp<..""\..-ch hal-
message over 1hc nc1work conn<..><::lion lo !he other pany. The
message is then displayed in a sp<..""\..-ch balloon in an order
based on the time !hal the message was completed.
Turning now 10 FIGS. 9 and 10. one exemplar)' method Jix
formalling. and sizing !he speech balloon 410 is shown. In one
embodiment. a lex! system. such as a standard lex! system
used in Mac OS X is used lo produce !he lex! of the message.
'J.hetexl system prm ides inl(mllalion regarding the si1e of the
1cx1 message 10 1hc GUI The GUluscs 1his site inlixmation
10 conslrl.Jcl a speech balloon of an approprimc size 10 comain
the message. Cenain m!es regarding the J(mllalling of the
speech hallmm alkc! the si1e of the speech baBoon. For
example. in one cmbodimcm. upper. lower. !eli and rig!u
margins arc established. In 1hc cxcmplal)' embodiment i!!us-
1rmcd in FIG. 9.\. !he mal'olins arc selected as Ji.1llows:
upper 3: !ower 5: !ell !3: and right 0. 'l11e lex! of the
message is n.:quired lo he positioned within the speech hal-
loon 410 with its tail extending rightward. so as lo he used in
the rig.h1 horizomal region of1hc GUI. In p.1rticu!ar. when the
'5 spc<.."Ch baBoon 410 wi1h a rightward extending. tail is
required. the sp<..""\..-ch balloon 410 with the !ell ward extending
tail is designed lo he of the proper size l\1r the message. and
then the speech baBoon 410 is Jlipp<..-d horimn!ally or rola!L'<i
a boll! a ccmra! vertical axis 10 produce the appropriately sized
60 spc<.."Ch balloon 410 with a rig!uward cx1cnding. wi!. In this
manner. an <..x:om1my of soli ware coding is realized. as only a
single routine l\1r generating speech baBoon with either !ell-
ward or rig!uward cx!Cnding tails is r<..'<.j_uircd.
The panicu!ar embodiments disclosed above arc i!!uslra-
65 live only. as 1hc invention may be modified and praclic<..-d in
difJCrenl hut equivalent manners apparenllo those skilled in
the art ha\ing the henclil of the teachings herein. Further-
US 7,669,134 Bl
II
more. no I imitations arc intended to the detai Is o fconslruclion
or design herein shown. o1hcr than us described in the claims
below. It is therefore evident thnt the particular embodiments
disclosed above may be altered or modi!i<..'d and all such
\ariations arc considered within the scope and spirit of the 'i
imention. Accordingly. the proh.x:lion sought herein is as sci
fonh in the claims below
What is claimed is:
I. A method ll1r displaying inl(mJJation related to a com- 1"
municmion session. comprising.
displaying information relating to data produc<..'d by a plu-
m lily of participants to the communicmion session on a
Jirsl display unit. wherein the inllmnation produced by a
Jirsl participant of the plurality of participants is dis- l'i
played at a !irst position on the first display unit:
receiving. dmn li"om nt lcnst one of the rcmnining pnnici-
pnnts of the plurn!ity of pnrticipnnts by the communicn-
tion session: and
displaying in!(mnation relating to data n.:cei\ed !!om the at ""
lcnst one of the rcmnining. pnrticip.1nts oft he plurnlity of
pnrticip.1nts. on the first displny unit. wherein the inlix-
mation received !!om the at !east one of the remaining
participants of the plurality of participants is displayed
at a separate position !l1r each said remaining p;u1icipant "-'
of the plurnlity of pnrticipnnts on the first displny unit.
wherein the rcsp<..>ctive scpnrmc positions rdnting. to
each oft he plurality of panicipants arc all non-random.
horimnta!-spatially distinct and chronologically
nrrnngcd. 30
2 .. \method. ns set Jixth in clnim 1. wherein the respective
positions rclming. to the plurnlity ofpnrtieipnnts nrc permitted
to at least partially overlap.
J. A method. as set !l1rth in claim 1. wherein:
receiving data !!om the remaining participants of the plu- 35
rnlity of pnrticip.1nts includes r<..>ceiving dmn li"om n s<..'C-
ond pnnicipnnt of the plurnlity of pnrticipnnts:
displaying in!(mJJation relating to data produced by the
!irs! participant !i.Jrther comprises displaying in!(mJJa-
tion relating to data produced by the !irs! participant in a 40
right justified li.mnm: and
displaying. infonnntion relating to dma r<..>ceived li"om the
second participant !i.Jrther comprises displaying in!(Jr-
mation relating to data received from the s<..>cond p;u1ici-
pant inn left justified formm.
4 . . \method. as set forth in clnim 1. wherein:
receiving. datu li"om the remaining. participants of the plu-
rality of participants includes r<..>cei\ ing data !i"om a s<..>c-
ond p;u1icipant oft he plurality of participants:
displaying in!lmm11ion relating to data produced by the
first pnrticipant li1rther comprises displnying inli.mm-
tion relating to dma produced by the lirst pnrticipant in a
le!l_justi!icd !lmJJal: and
45
'0
displaying in!lmnation relating to data r<..>cei\L-d !i"om the ,
5
second participant !i.Jrther comprises displaying in!(Jr-
mation relming to dnta received from the second pnnici-
pant inn right justilicd formm.
5 . . \method. as set forth in clnim 1. wherein:
displaying in!(mJJation relating to data produced by the
60
!irs! participant !i.Jrther comprises displaying in!(mJJa-
tion relating to data produc<..-d by the !irs! participant in a
spc<..>ch bnl!oon.
I2
displaying in!(mJJation relating to data received !ium the
second participnnt lilflhcr comprises displaying. infor-
mmion relating. to dmn received li"om the s<..>cond partici-
pant in a speech balloon.
7. A method. as set !(H1h in claim I. !luther comprising:
r<..>cei\ ing data li-mn a second p;u1icipant of the plurality of
participants:
detecting. dmn being. entered by the s<..>cond particip.1nt
bdl1rc the data is ddivcr<..-d to the !irs! participant: and
displaying in !lmnation relating to data being entered by the
second participant on the !irs! display unit.
8 . .-\method. ns set li.mh in claim 7. wherein:
displnying informmion relating to datu being entered by the
second participant on the Jirst display unit !i.uther com-
prises displaying a thought bubble on the Jirst display
unit.
9 . .-\ method. ns set li.mh in claim 7. wherein:
displnying informmion relating to datu being entered by the
second participant on the Jirst display unit !i.uther com-
prises displaying the thought bubble on the Jirst display
unit m a second position on the first displny unit.
10 . .-\ method. as set forth in clnim 7. wherein:
displaying in !lmnation relating to data being entered by the
second participant on the Jirst display unit !i.uther com-
prises displaying presck>ctcd graphical in!(mJJation on
the lirst displny unit m n second position on the first
displny unit.
I I. A method. as set !l1rth in claim 7. wherein:
displaying in !lmnation relating to data being entered by the
second participnnt on the first display unit li1rthcr com-
prises displaying preselected graphicnl inli.mmtion on
the lirst displny unit m n second position on the first
display unit: and
removing the prcsek>ctcd graphical inllmJJation !ium the
lirst displny unit in response to receiving. dma li"om the
second particip.1nt.
12 . .-\ method. as set forth in clnim 11. wherein:
displaying prcse!<..>ct<..-d graphical in!(mJJation on the Jirst
display unit at the second position on the Jirst display
unit funhcrcompriscs displnyingn thought bubble at the
second position on the first displny unit.
I 3. A computer readable program storage device encod<..-d
with instmctions that. when cx<..>cutcd by a computer. per-
limns a method. comprising:
displnying informntion relming to dmn produced by a plu-
rality of particip.1nts to the conununication session on n
!irs! display unit. wherein the inllmJJation produc<..-d by a
Jirst p;u1icipant of the plurality of participants is dis-
plnycd m n first position on the first displny unit:
r<..>ceiving dnta from m lcnst one of the remaining. partici-
pants of the plurality of pnrticipants by the communica-
tion session: and
displaying inllmnation relating to data r<..>ccivcd !i"om the at
lcnst one of the rcmnining pnrticipants of the plurnlity of
participants. on the lirst display unit. wherein the infor-
mation rccei\L-d !i"om the at least one of the remaining
participants of the plurality of participants is disp!ay<..-d
at a separate position !(Jrcach said remaining participant
of the plurality of participants on the lirst displny unit.
wherein the respective scpnratc positions relming to
each of the plurality of participants arc all non-random.
horizontal-spatially distinct and chnmo!ogica!ly
arrnng.cd.
6 .. \method. as set forth in clnim 1. wherein:
receiving. datu li"om the remaining. participants of the plu-
rality of participants includes r<..>cei\ ing data !i"om a s<..>c-
ond p;u1icipant oft he plurality of participants:
14 . .-\ computer rcndablc progrmn storag.c device. ns set
65 Ji.1rth in claim 13. wherein the respective positions rdming. to
the plurality of participants arc pclmitt<..-d to at least p;u1ia!ly
overlap.
US 7,669,134 Bl
13
15. An apparatus ll1r displaying inl(mJJalion rdatcd lo a
communication session. comprising:
means for displaying inli.mmnion relating to daw produced
by a plurality of panicipants to the conununication ses-
sion on a !irs! display unit. wherein the inl(mJJalion 'i
produced by a !irs! participant is displayed a! a !irs!
position on the Hrst display unit:
14
21. :\ graphical user intcrliJCc. as sct l(H1h in claim 20.
wherein the !irsl region is adapted to display the text in a left
justified li.mnm.
22 .. \graphical user imcrlUcc. as set forth in claim 17.
wherein a second region of the plurality of regions is adaplL'<i
lo display lcxl !i-om a second participant of the plurality of
particip.1nts in a sp<.-cch balloon in a right justi!icd li.mml.
means for receiving data from m least one oft he remaining
participants of the plurality of participants by the com-
munication session: and
23 .. \graphical user imcrlUcc. as set forth in claim 17.
wherein the !irst region is adapted to display a thought bubble
lCJ in response lo receiving a signal Jium the lirsl participanllhal
a message is being prcparL'<i lo he scnllo a second participant
of the plurality of participants.
means ll1r displaying inl(mJJalion relating to data received
li"om the at least one of the remaining participants of the
plurality of participants on the !irst display unit. wherein
the inl(mJJalion n.:cci\L-d from the a! least one of the
remaining participants of the plurality of participants is
displayed m a separate position for each said remaining
particip.1nt of the plurality of participams on the !irst
display unit. wherein the respective scparmc positions
relating lo each of the plurality of p;u1icipanls arc all
non-random, horizontal-spatially distinct and chrono- ""
logically arrang<.'d.
24 .. \ graphical user imcrlUcc. as set forth in claim 23.
li.1rthcrcomprising a second region of the plurality of regions.
15 wherein the Jirsl and second regions arc adaplL'd lo display the
messages rccci\L-d !ium the lirsl and second participants \cr-
tically in the order in which they were r<.>ceivcd.
16 .. \n app.1ratus. as set li.mh in claim 15. wherein the
rcspccti\ c positions relating lo the plurality ofp;u1icipanls arc
pcrmincd lo allcasl partially mcrlap.
17. A graphical user intcrliJCc !(Jr displaying in!(mJJalion "5
rclal<.'d to a communication session. comprising.
a plurality of regions adapted to display messages li"om a
plurality o fp;u1icipanls lo the inslanl messaging session:
a Jirsl region of the plurality of regions adapted lo display
m least one message from a !irst participmu to the inswm 30
messaging session: and
the remaining regions of the plurality of regions adapl<-'<1 to
display allcasl one message from the allcasl oncoflhc
remaining participants of the plurality of participants
respectively to the instant messaging session. wherein 35
the respective separate plurality of regions rclming to
each of the plurality of participams arc all non-random.
horimnlal-spalially distinct and chronologically
anangcd.
18 .. \graphical user intcr!Ucc. as set forth in claim 17. 40
wherein the respective plurality of regions rclming to the
plurality of participants arc pcrmincd lo al least partially
mcrlap.
19. :\graphical user inlcrbcc. as set l\1rth in claim 17.
wherein the !irst region is adapted to display an icon associ- 45
atcd with the !irst p.1rticipam. and a second region of the
plurality of regions is adapted lo display an icon associated
with a second participant of the plurality of participants.
20 .. \graphical user intcr!Ucc. as set forth in claim 17.
wherein the !irst region is adapted to display text from the !irsl '0
participam in a speech balloon.
25 .. \ graphical user imcrlUcc. as set forth in claim 24.
wherein the Jirsl and s<.><::tmd regions arc adapl<--d lo replace the
thought bubble with a corresponding speech balloon when
the speech balloon is rL><::civcd.
26 .. \ graphical user imcrlUcc. as set forth in claim 25.
wherein the !irst and second regions arcadapl<.'d to display the
corresponding speech balloon in a vertical position corre-
sponding lo the order in which it was rcccivL-d regardless of
the \crtical position in which the thought bubble was dis-
played.
27.:\ system. comprising: a !irst terminal: and
a second terminal. wherein the !irsl and second tcnninals
each include a graphical user inlcrliJCc l(Jr displaying
in!(mnalion rclal<--d lo a communication session
bctw<.-cn the !irst and second terminals. the graphical
user intcr!Ucc comprising:
a plurality of regions adapt<.'d to display messages from a
plurality of p;u1icipanls:
a lirsl region of the plurality of regions adaplL-d lo display
allcasl one message Ji-om a lirsl parlicipanllo the inslanl
messaging session: and
the remaining regions of the plurality of regions adapted to
display allcasl one message !i-mnlhc allcasl one of the
remaining participants of the plurality of participants lo
the inslanl messaging session, wherein the rcspccli\c
separate plurality of regions relating to each of the plu-
rality of particip.1nts arc all non-random. horizontal-
spatially distinct and chronologically <I!Tang<.-d.
28. A system, as set l(Jrth in claim 27. wherein the respec-
tive plurality of regions relating lo the pluralityofp;u1icipanls
arc permitted tom least partially overlap.
'
, , ,
UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
P A r i ~ N T NO. 7,669,134 Bl
APPLIC/\TION NO. 101428523
D/\JU)
INVJ:HIOR(Sl
Febmmy 23, 2010
Christie et a!.
Page I of 1
It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:
On the Title Page:
The first or sole Notice should read--
Subjectto any disclanner, the term of this patent rs extended or adjusted under 35 U.S.C. 154(b)
by 1143 days.
Signed and Sealed this
Seventh Day of December, 20 I 0
David J. Kappos
Jlirecwr o(tll< I ;nired Sial<'> Pat<'lll and li'ml<'mark Offic<'












































Exhibit 3
021 United States Patent
Christensen
(54) METHOD AND APPARATUS FOR
I>ISPI.AYING ANI> ACCESSING CONTROL
ANI> STATUS INFORMATION IN A
COMPUTER SYSTEM
(75) Inventor: Stc,cn W. Christensen, Milpitas, CA
(US)
(73) A-.signcc: Apple Computer, Inc., Cupertino, CA
(US)
( ') Notice: This palco! issued on a continued pros-
ecution application !lied under 37 CFR
1 .53( d), and is subjc<.:t to the twenty year
patent term provisions of 35 U.S.C
J54(a)(2).
(21)
(22)
(63)
(51)
(52)
(58)
(56)
Sub_icc!lo any disdaimcr, the lcrm of this
patent is extended or adjusted under 35
U.S.C. 154(b) byO days.
App!. No.: 08/821,004
Filed: Mar. 20, 1997
Related U.S. Application nata
Continuation of application No. 08/316,2.>7. lilcd on Sep.
30. 1994. now abandoned.
Int. Ct. G06F 3/()(l
u.s. ct. "'"'""'"'"'""'"'"'"'"'"'" 345/779; 345.072
Field of Search ................................. 395/345, 350,
.:1,81>2,389

.:1,8%,291
.:1,9.>1.957


395/352, 354, 326, 339, 340, 341, 973,
974; 345/348, 350, 352, 354, 326, 339,
340,341,973,974,700,764,771-772,
77(r-779, 781,784,788,791,798-800,
X33, X56
References Cih.'d
U.S. PATENT DOCUMENTS
A 81989 Takagi
""'"'""'"'""'"'
.>45/79.:1
A 12/l'JR'J Takagi cl al.

A I 1 'J90 Gc\1 ct al. .>45.'8.:11
A
,,
1 'J90 Takagi et al. .>45.-'45.>
A 2/J'J'I2 Takagi ........................

A 'J/J'J'I2 l!ullol el al. 7'1(1
:(iF_!!_ EDIT VIEW LABEL SPECIAL
EXTENSICNS
t""''""''"""""""' "'"""
'
u

(10) l'atcnt No.:
(45) nate of l)atcnt:
us 6,493,002 Bl
*Dec. 10, 2002
Ll'
FP
FP
Ll'
A 4.'J'I'J3 el al.
A Ander:<.on cl al.
5 . .:128.730 A {)/1995 Baker et al.
5.5&'1.105 A 12/19% et al.
A 4.'J'I'J7 Oran cl al.
5.640 . .:198 A {)/1997 Chew .........................
5.644..>34 A 7/1997 ct al.
A il,'J'I'J7 el al.
A HI. 'J'I'J7 Chew .........................
5.757..>71 A 5.'1998 Orand al.
5.825..>57 A HI/ 1998 Malamud ct al.
kU3.8'JR A HI. 2000 Ludolph cl al.
FOREIGN PATENT DOCUMENTS
0 48.> 777 i\2
-433777 i\
0 _,'II AI
584.W2 i\
HI/ 1 W 1

R.'J'I'I2
3/1'N4
OTIIER PUBI.ICATIONS


3.:15 740
3.:15 779

3.:15 790
3.:15 .:119


3.:15 779
3.:15 779

EPO 0 584 391 AI, Cohausz, English Translation of the
German patent document, Mar. !992.*
"PCT Wrillcn Opinion", Ocl. 4, 1996.
Sleven Harris, cl al. "Inside WordPcrfecl 6 for Windows",
1994, USA, pp. 1104-1108.
Mark A Benge, Matt Smith: --Designing Custom Controls",
Spring !993, USA, pp. 72-85.
lnlcrnational Search Report, PCTiUS 95/11025, Jan. 03,
1996.
' cited by examiner
Priman Emminer---{'rescellc N. dcla Torre
(74) Allorney, Agent, or Finn--Blakely, SokololT, Taylor &
Zafman IJ.P
(57) ABSTRACT
An interactive computer-controlled display system having a
processor, a data display screen, a cursor control device for
interactively positioning a cursor on !he data display screen,
and a window generator that generates and displays a
window on a data display screen. The window region
provides status and control information in one or more data
display areas. The imlividiual data display areas may be
controlled through the usc of controls and indicators on the
control strip itself using cursor control keys.

'""'c=-r
50 Claims, 13 I>rawing Sheets
'
' '
:.t.,.,
..... "fM<>W!<
""'""'""""
O..<<Tit<i' ..
<<R'""-""'-

"-"' -i I _
'" ----.--,----
.,,.' '''"
"
Q '" '""- ""
"P::o);;ll!J]L '>', '' . II.'' :;-J:-"""'""Jc<i>:c]' ]G - H.'
Iilli
I'""'"!
,00
FIG. I
MASS
STORAGE107
!Y-
MODEM
108 j'r--
LAN V'-----
109 l'r-
HARD COPY
~
DEVICE 124
DISPLAY
V'-----
121
POWER
TO OTHER
SUPPLY152
COMPONENTS
I
SWITCH VDD
DECODER
153 154
TIMER
C L K ~
150 HOT INDICATION
SIGNAL 183 ---- PROCESSOR
~
--"
1'181
,--
103
POWER
SIGNAL
vDD
r-
182
-
I/0 BUS 101_\
--"-
BUS
-"'
MAIN
TRANSLATOR/
LOCAL BUS 1 00
MEMORY
, INTERFACE
104
SOUND
~ I/O
CHIP 125 ' v CONTROLLER
FLOPPY DISK -"----"
130
DRIVE
126
' -
UNIT
LL184
140 TO
CLOCK
GENERATOR
160
NON-
--" VOLATILE
MICRO-
MEMORY
CONTROLLER
106
-
TEMP. ~ J
127
1
SENSOR
1
2
7
A
r .
l FROM :
z..
} INTERF1
KEYBOARD
CURSOR
: ..J_ CLOCK 1 60
122
CONTROL 123
CLK GENERATOR
/0
CE
140
e

Cfl

;r
;:;-
=
-
0
" r-
-
.0

CJJ
"'
" a
-
"'
-
"'
c
"'
i
0
0
N
1:0
....
QFILE EDIT VIEW LABEL SPECIAL
Cfl iCl
EXTENSIONS

NAME SIZE
APPLICATIONS
10 ITEMS 12.6 MB IN DISK 313.6 MB AVAILABLE
[
[J [J
"
APPLEUNK POWERBOOK EDmON EXPRESS MODEM


[J

'

-
MACTERMINAL
REMOTE ACCESS CUENT FOLDER SIMPLE TEXT

0
[J [J
\
MOVIEPLAYER
RES EDIT TOOLS WORD

PRINTER SHARE
83K
I I
PRINTt.tONITOR
66K
"liT
0

QUICKTIME
111
831K 21TEMS
12.6 MB IN DISK 313.6 MB A

SERIAL TOOL
33K 0
mJ TEXT TOOL 28K I- &
>& TTY TOOL 61K 1111
c 0111 o ooooo1ooooo w :0049 TRASH 1
'
FIG. 2A
200
e

Cfl

;r
;;-
=
-
0
" r-
-
.0

CJJ
"'
" a
N
"'
-
"'
c
"'
i
0
0
N
1:0
....
U.S. Patent nee. I 0, 2002 Sheet 3 of 13
MODULE DISPLAY AREA
202
US 6,493,002 Bl
(
CLOSE BOX
201
DRAG/SIZE TAB
203
LEFT SCROLL ARROW
204 RIGHT SCROLL ARROW
205
FIG. 28
HIDE BATIERY LEVEL
HIDE BATIERY CONSU"'TION
HIDE TIME REMAINING
J ~ 0 OOODDIDDOOO W' ~ I H < - DODO(
FIG. 2C
BATIERY MONITOR
THE POWER ADAPTER IS
CONNECTED AND RECHARGING
ONE OF THE BATIERIES.
FIG. 2D
U.S. Patent nee. 10,2002 Sheet 4 of 13 US 6,493,002 Bl
g: .. W .. 00000000 ., ., [k
.a.
b:..ID W .. 11-:9.- 00000000 .. .. IJIL
.a.
[ .. liD ooo[Jl1ooooo oo[J]oooo lllo L
.a.
bAD ODDDDIDDDDO W .. hJ- .. JJ!o .. JJ!Ih
.J,
bA -:9--00000000 .. .. .. ij]l,
FIG. 2E
U.S. Patent nee. I 0, 2002
210
(
+
(211
o
-
-
oooootooooo
Sheet 5 of 13
FIG. 2F
::J C CONTROL STRIP
,81 @ SHOW CONTROL STRIP
'qL Q HIDE CONTROL STRIP
FIG. 3
US 6,493,002 Bl
U.S. Patent nee. I 0, 2002 Sheet 6 of 13 US 6,493,002 Bl
MAIN LOOP
(CALLED BY OPERATING SYSTEM)
SAVE PREVIOUS CONTEXT,
SETUP STRIP'S CONTEXT
SECONDARY
INITIALIZATION
COMPLETE?
YES
NO
402
401
404
DETERMINE THE
USER EVENT TYPE
NULL EVENT
(IDLE)?
405
NO
YES
FIG. 4
RUN SECONDARY
INITIALIZATION
RUN IDLE TASKS
406
403
MOUSE CLICK?
>----1 PROCESS MOUSE CLICK I--!
407
NO
RESTORE PREVIOUS
CONTEXT
EXIT
YES
408
409
U.S. Patent nee. I 0, 2002 Sheet 7 of 13
CONTROL STRIP MAIN LOOP
/RUN SECONDARY INITALIZATION
'
SECONDARY
INITIALIZATION
HAS THE FINDER
STARTED UP?
403
NO
5D1
CREATE THE STRIP'S
WINDOW 502
SUCCESSFUL?
YES 503
INITIALIZE STRIP'S FONT,
COLOR
OPEN,INITIALIZE EXTERNAL
MODULES
SET STRIP'S DEFAULT
SCREEN LOCATION,SIZE
LOAD USER
CONFIGURATION
DRAW THE STRIP'S
CONTENTS
FIG. 5
504
505
506
507
508
US 6,493,002 Bl
U.S. Patent nee. I 0, 2002 Sheet 8 of 13
CONTROL STRIP MAIN LOOP
/OPEN. INITIALIZE EXTERNAL

MODULES
( OPEN, INITIALIZE
EXTERNAL MODULES
US 6,493,002 Bl
505
i
MORE MODULE
NO-
FILES?

y
,s
OPEN MODULE FILE _j\.
602
LOAD MODULE CODE -f\
INTO MEMORY
603

CALL MODULE TO
-j\604
INITIALIZE ITSELF
CALL MODULE TO GET
MODULE s FEATURES-\
605
CALL MODULE TO GET
ITS DISPLAY WIDTH ,_ \
606
CLOSE MODULE
607
(EXIT
FIG. 6
U.S. Patent nee. I 0, 2002 Sheet 9 of 13 US 6,493,002 Bl
N
YES
CONTROL STRIP MAIN LOOP
/..DRAW THE STRIP'S CONTENTS
DRAW THE STRIP S
CONTENTS
IS THE STRIP
VISIBLE?
y s
MORE MODULE
TO DRAW?
y s
THIS MODULE
NEED TO BE
REDRAWN?
y s
IS
THIS MODULE
A BUTTON?
>-------NOI--
701
> - - - - N O , - ~
702
703
704
NO
DRAW BUTTON
BACKGROUND
GRAGHICS
DRAW STATUS-ONLY
BACKGROUND
GRAGHICS
705
706
L ---"' CALL MODULE TO '-t----'
DRAW ITSELF r
707
EXIT
FIG. 7
U.S. Patent nee. I 0, 2002 Sheet 10 of 13 US 6,493,002 Bl
RUN IDLE TASKS
8D1
CONTROL STRIP MAIN LOOP
LRUN IDLE TASKS
WINDOW UPDATE YES
PENDING?
DRAW THE STRIP'S
CONTENTS
NO
HAS STRIP'S
CONFIGURATION YES
CHANGED?
803
NO
CALL CURRENT
IDLE MODULE TO ~
RUN ITS IDLE TASK
806
UPDATE ID OF NEXT
MODULE TO IDLE
DOES MODUL
NEED TO SAVE
SffiiNGS?
809
NO
UPDATE STRIP'S
HELP MESSAGES
EXIT
807
808
YE
812
IS IT
SAFE TO
SAVE NOW?
802
YE
NO 804
IS IT SAFE TO
YES
SAVE NOW?
NO
810
FIG.
SAVE STRIP'S
CONFIGURATION
TO DISK
805
SAVE MODULE'S
SffiiNGS
TO DISK
811
8
U.S. Patent nee. I 0, 2002
PROCESS MOUSE
CLICK
MOUSE CLICK
INSIDE STRIP?
Sheet 11 of 13 US 6,493,002 Bl
CONTROL STRIP MAIN LOOP
/PROCESS MOUSE CLICK
901
DETERMINE WHICH
MODULE WAS CLICKED ON
MOVE
OPERATION?
NO
CLICKABLE
MODULE?
y s
902
YES MOVE MODULE'S DISPLAY,
OR MOVE ENTIRE STRIP
904
> - - - N O - ~
905
TRACK THE MOUSE CLICK
MOUSE
STILL IN
MODULE'S
BOUNDS?
CALL THE MODULE TO
PROCESS THE CLICK
POST PROCESS
06
> - - N O - ~
907
908
THE CLICK 909
EXIT
FIG. 9
U.S. Patent nee. I 0, 2002 Sheet 12 of 13 US 6,493,002 Bl
MODULE WANTS TO
UPDATE SETTINGS?
1001
NO
MODULE
NEEDS TO
RESIZE
DISPLAY?
1003
NO
MODULE WANTS
TO BE CLOSED?
1005
1007
NO
RESIZE OR
CLOSE?
NO
MODULE HELP
STATE CHANGE?
1009
NO
( E ~ I T )
CONTROL STRIP MAIN LOOP
/POST PROCESS THE CLICK
YES
YES
""-YES
"--YES
YES
FLAG THAT THIS MODULE
HAS A SAVE PENDING
CALL MODULE TO
UPDATE ITS WIDTH
CLOSE THE
MODULE
1002
1004
1006
REPOSITION AND REDRAW
MODULE DISPLAYS
1008
INVALIDATE OLD
HELP STATE
1 10
FIG. 10
U.S. Patent nee. I 0, 2002 Sheet 13 of 13 US 6,493,002 Bl
....,
..c:
"'
c=J
i2
Q)
DilJ
a.
0
ITIIIl
Ul
IIIIill
..c:
a.
IIIIIIII
0
'-
-
~

0
CD
~
OJ
..c:
......,
..c:
....,
..;
....,
..c:
....
'0 a>
C\J
Q)
I I
~
i2 ....J
a.
111111 111111
....,
..c:
0
0
1111111
a.
- 1111111
u..

f-
0
<.!)
.:
1111111111
'-
.
1111111111
.:
a.
~ <.!)
a.
-
0
llllllllllll
0
l.L. L
"
-
'-
~ 1111111111111111
CD
l.L. ~
....,
"
MIIIUII
Q)
0
.0-
<.:> CD
I
CD
Ul
+'
>
....
..;
.,
....
....1
., .,
....1
-
a.
a.
0
0
IIIlllill
Ul
f-
rrrrrn
..c
..c
a.
a.
0 rnm
0
'-
L
c=J
<.:>
~
'-
0
0
.!) CD
us 6,493,002 131
I
METHOJ) ANn APPARATUS FOR
!)IS PLAYING AND ACCESSING CONTROL
ANI> STATUS INFORMATION IN A
COMPUTER SYSTEM
2
The present invention overcomes these problems by pro-
viding a stal\ls and control information display. The display
of the present invention is in an easily ac<:cssible format.
Also, the display may be <:on!igurcd to permanently display
.< m a visible manner control and status indicia.
This is a continuation of application Scr. No. OX. 3! 6,237,
tiled Scp. 30, !994, now abandoned.
I'IELD 01- TilL INVENTION
SUMMARY OF "ll!E INVENTION
An interactive computer-controlled display system is
described. In the present invention, the display system
The present invention relates to the !icld of computer
syslcms; particularly, !he present invention rcla!cs to dis-
playing a status and control furu:lion bar or window to
enable access of user sdc<.:tcd indicia to a computer system
user.
Ill includes a processor, a data display screen, and a cursor
control device for interactively positioning a cursor on the
data display screen. The present invention also includes a
window generator that generates and displays a window
(e.g., a <:ontrol strip) on a data display screen. In one
BACKGROUND OF TI!E INVENTION
Typically, a com puler system contains a processor, a bus,
and other peripheral devices. The processor is responsible
for executing insln1<.:tions using the data in the computer
system. The bus is used by the processor and the peripheral
devices for transferring information between one another.
The information on the bus usually includes data, address
and control signals. The peripheral dcvi<.:cs <.:omprisc storage
dcvi<.:cs, input/output (! '0) devices, etc.
1.< cmlmdimcnl, the window comprises a control and. or slat us
window for display on the desktop of the computer system
The window displays graphics depicting at least one display
area of indicia. The individual data areas may be controlled
through the usc of controls and indicators in the window
:w itself using cursor control keys.
BRIEF DESCRIP"IlON OF TilE DRAWINGS
The present invention is illustrated by way of example,
"-' and not by way of limitation, in the ligures of the accom-
panying drawings and in which like reference numerals refer
to similar clements and in which:
Computer systems also include information management
systems that coordinate the display of information to the
user. Currcnl!y, the arl in computer display management
provides the capability of displaying data in rectangular
portions (commonly referred to as windows) of a display .>o
screen. Such information management systems include the
1:indcr '" interface of the computer systems manufactured by
Apple Computer, Inc. of Cupertino, Calif. Controls arc
typically provided lO resize and move windows within the
con!incs of the physical display boundaries.
.>5
Windows may be used to display information regarding
application programs, as well as information produced by
system programs, that arc run on the computer system.
Many of these system and <.:ontrol programs provide status
and control information and functionality. Some of the 40
system control programs also provide options with respect to
the information they provide and the functions they perform.
These options can be accessed and. or selected by moving a
cursor at a predetermined point in the window and click-
ing" a mouse or performing requisite key strokes. Access to 45
these programs may require locating the program (e.g.,
locating ami entering a folder) before execution. The time
necessary to access such programs may he unduly long. ll is
desirable lO provide a less obtrusive manner of accessing
such system and control programs.
The computer system is often capable of displaying
multiple windows or data areas on the display screen at the
same time. Windows may overlap each other. The informa-
tion contained in the portion of the window that is over-
lapped is not visible. The window that is entirely visible to 55
the computer user is typically the active window. Therefore,
a program, such as a system or control program may he
running, while another program displaying information in
another window is selected as active and thereafter covers,
partially or <:omplctely, the windows or data areas displayed 60
by the system control program. Sometimes the user may
wish to have an unobstructed view of the system/control
data area, regardless of the window selected as active (even
when the windows overlap each other). Thus, it is desirable
at times to have windows that arc always visible to the user. 65
However, it is also desirable to be able to eliminate that
window at times based on the uscr"s requirements.
FIG. I is a block diagram of one embodiment of
computer system of the present invention.
the
FIG. 2A displays a computer desktop illustrating the
control strip of the present invention as well as opened
windows.
1:1G. 21l illustrates one cmlmdimcnl of the control strip of
the present invention.
FIG. 2C illustrates a pop-up menu displayed
control strip of the present invention.
FIG. 2D illustrates a help balloon displayed
control strip of the present invention.
from the
from the
FIG. 2E illustrates the process of moving a display area
from one position lo another in the control strip.
FIG. 21' shows !he control strip window graphics gener-
ated by processing logic being combined with graphics
generated by a module to illustrate the creation of the
resulting control strip.
FIG. 3 illustrates one embodiment of the control panel of
the present invention.
FIG. 4 is a Jlow chart of one embodiment of !he process
for the control strip of the present invention.
FIG. 5 is a !low chart of one embodiment of the sc<:ondary
initialization process of the present invention.
FIG. 6 is a !low chart of one embodiment of the open and
initiali...:alion process for !he cxtcmal modules of !he present
invention.
FIG. 7 is a llow chart of one embodiment of the pro<:css
for drawing the contents of the <:ontrol strip of the present
invention.
FIG. S is a Jlow chart of one embodiment of !he process
for running idle !asks in the sent invention.
FIG. 9 is a Jlow chart of one embodiment of !he process
for responding to a mouse dick occurring in the control strip
of the present invention.
FIG. 10 is allow chart of one embodiment of the pro<:css
for post processing a mouse dick in !he present invention.
FIG. II illustrates a bar graph for display in a data display
area in the control strip of !he present invention.
us 6,493,002 131
3
F!Ci. 12 il!uslratcs a bar graph that rcsuhs after using
arrow direction icons.
4
programs in accordance with the lcachings herein, or it may
prove convenient to construct more specialized apparatus to
perform the required method steps. The required structure
for a variety of these machines will appear from the dcscrip-
DEIAII.ED DESCRIPTION OF TilE
INVLNilON
-' lion below. In addition, the present invention is no!
described with rc!Crcncc lo any particular programming
language. It will be appreciated that a variety of program-
ming languages may be used to implement the teachings of
the invention as described herein.
A mc!hml and apparatus for providing slai!Js and control
indicia. In !he following detailed description of the present
invention numerous spcci!k details arc set forth, such as
types of status indicia, instruction names, etc., in order to
provide a thorough understanding of the present invention.
111
llowcvcr, it wi!! be apparent to one ski !led in !he arllhallhc
present invention may be pradiccd without these
details. ln other instances, well-known structures and
dcvi<.."CS arc shown in black diagram form, rather than in
dc!ail, in order lo avoid obscuring !he prcscl invention. 1.<
The present description includes malcrial prolcdcd by
copyrights, such as illuslralions of graphical user interface
images which the assignee of the present invention owns.
The assignee hereby reserves its rights, including copyright, :w
in these materials, and each such material should be
regarded as bearing !he following notice: Copyright Apple
Computer, Inc., 1003. The copyright owner has no objcdion
to the facsimile reproduction by anyone of the patent docu-
ment or the patent disclosure, as it appears in the Patent and "-'
Trademark O!Ticc Jl!c or records, but otherwise reserves a!!
copyrights whatsoever.
Some portions of !he dc!ailcd descriptions which fo!!ow
arc presented in terms of algorithms and symbolic repre-
sentations of operations on data bits within a computer .>o
memory. These algorithmic descriptions and representations
arc !he means used by !hose skilled in !he data processing
arts lo most clTcc!ivcly convey !he substance of !heir work
to others skilled in the art. An algorithm is here, and
generally, conceived to be a self-consistent sequence of steps _,
5
leading lo a desired result. The steps arc !hose requiring
physical manipulations of physical quantities. Usually,
!hough no! necessarily, these quantities lake !he form of
electrical or magnetic signals capable of being stored,
transferred, combined, compared, and otherwise manipu-
40
la!cd. II has proven nmvcnicnl a! limes, principally for
reasons of common usage, lo refer lo these signals as hils,
values, clements, symbols, characters, terms, numbers, or
the like.
OVERVIEW OF TilE COMPU"rER SYSTEM OF
"lliE PRh'iENT INVENTION
Referring to FIG. 1, an overview of a computer system of
the present invention is shown in block diagram form. The
present invention may he implemented on a general purpose
microcompulcr, such as one of !he members of !he Apple
family of personal computers, one of !he of !he
IBM personal computer family, or one of several other
computing and assistant dcvi<.."CS which arc presently com-
mercially available. Of course, !he present invention may
also implemented on a multi-user system while encoun-
tering all of the costs, speed, and function advantages and
disadvantages available with these machines. The preferred
embodiment of the present invention is implemented on an
Apple Power Book'-'' computer system developed by the
assignee of !he present invention.
i!!uslra!cd in FIG. 1, !he computer system of !he
present invention generally comprises a local bus or other
communication means roo for communicating information,
a processor ro3 coupled with local bus roo for processing
information, a random access memory (RAM) or other
dynamic storage device ro4 (commonly referred to as a
main memory) coupled with local bus roo for storing
information and instructions for processor 103, and a read-
only memory (ROM) or other non-volatile storage device
106 coupled with local bus IIHI for storing non-volatile
information and instructions for processor 103.
The computer system of the present invention also
includes an input/output (I 0) bus or other communication
means 101 for communication information in !he com puler
system. A data storage device 107, such as a magnclic !ape
and disk drive, including its associated controller circuitry,
is coupled to l/0 bus 101 for storing information and
It should be borne in mind, however, that all of these and
similar terms arc lo be assm:ialcd with !he appropriate
physical quantities and arc merely C\mvcnicnllahcls applied
lo these quantities. Unless spccilka!!y slated otherwise as
apparent from the following discussions, it is appreciated
that throughout the present invention, discussions utilizing
lcrms such as "processing or "computing or "caleulaling
or "dclcrmining or "displaying or !he like, rcl"cr lo !he
action and processes of a computer system, or similar
electronic computing device, that manipulates and trans-
forms data represented as physical (electronic) quantities
within !he computer system's registers and memories into
other data similarly rcprcscn!cd as physical quantities within
45
instructions. A display devi<.."C 121, such as a cathode ray
lube, liquid crystal display, clc., including its associated
controller circuitry, is also coupled lo 1/0 bus 101 for
displaying information to the <..Xlmputer user, as well as a
hard copy device 124, such as a plotter or printer, including
the computer system memories or registers or other such
information storage, transmission or display devices.
50
its associated <..xmtroller circuitry for providing a visual
rcprcsenlalion of !he compulcr images. !lard copy device
124 is coupled with processor 103, main memory 104,
non-volatile memory ro6 and mass storage device 107
through I 0 bus 101 and bus translator "interface unit 140. A
modem 108 and an c!hcmc!local area nclwork 109 arc also
55
coupled lo 1/0 bus !Ill.
Bus interface unit 140 is coupled lo local bus 100 amll/0
bus 101 and acts as a gateway between processor ro3 and
the I"() subsystem. Bus interface unit 140 may also provide
lranslalion bel ween signals being sen! from units on one of
!he buses lo units on !he other bus lo allow local bus 100 and
I 0 bus 101 loco-operate as a single bus.
The present invention also relates to apparatus for per- 60
forming !he herein. This apparatus may be
cia!!y conslruc!cd for !he required purposes, or it may
comprise a general purpose computer selectively activated An I 0 controller 130 is coupled to I 0 bus 101 and
controls access to <.."Crtain I 0 peripherals in the computer
65 system. For instance, l/0 <..xmtroller 130 is coupled to
controller device 127 !hal controls access lo an alpha-
numeric input device 122 including alpha-numeric ami other
or rc<..onJigurcd by a computer program stored in the com-
puter. The algorithms and displays presented herein arc not
inherently related lo any particular com puler or other appa-
ratus. Various general purpose machines may he used with
us 6,493,002 131
5
keys, clc., for communicating information and command
selections to processor 103, a cursor <.:ontro! 123, such as a
trackball, stylus, mouse, or trackpad, etc., for controlling
cursor movement, and a temperature sensor 127/\ for mea-
suring !he inlcrnal syslcm lcmpcraturc. The system also .<
indudcs a sound chip 125 coupled lo 1/0 conlro!!cr 130 for
providing audio recording and play back. Sound chip 125
may indudc a sound circuit and its driver which arc used to
gcncralc various audio signals from the computer system.
6
the display areas is individually and variably si:tcd. The size
of the control strip itself may also be variably sized. In one
embodiment, the size may be adjusted such that none, all, or
only a portion of the display areas within its boundaries arc
visible. The si.-:c of the control strip may also he varied such
that only a portion of one display area is visible in !he control
strip. FIG. 2A displays a computer desktop illustrating the
control strip of the present invention as well as opened
windows. FIG. 2B illustrates one embodiment of the control
I. 0 controller 130 may also provide access loa Jloppy disk
and driver 126. The processor 103 controls I. 0 controller
130 with its peripherals by sending commands to l/0
controller 130 via local bus 100, interface unit 140 and l/0
bus 101
111 strip of !he present invention.
Each of !he variably si:tcd data areas may be sensitive lo
user input for control. That is, a user may interact with the
individually display data areas. Dill"crcnt parts of the control
strip either display information or act as buttons, or both.
Ballcrics or other power supply 152 may also be included I.< Note thai hullons may display information on !heir surface.
When !he user clicks a hullon, it is highlighted. In one
embodiment, buttons may also display additional clements
such as pop-up mcnus(shown in FIG. 2C) or help messages
(e.g., balloons shown in FIG. 2D). Thus, in one embodiment,
"II control of the individual data areas is accomplished, in part,
through the usc of small butlon controls and indicators in the
form of various i<.ons.
lo provide power necessary lo run !he various peripherals
and integrated circuits in the computer system. Power sup-
ply 152 is typically a DC power source that provides a
constant DC power to various units, particularly processor
103. Various units such as processor 103, display 121, etc.,
also receive clocking signals to synchroni.-:c operations
within the computer systems. These clocking signals may be
provided by a global dock generator or multiple clock
generators, each dedicated to a portion of the computer
syslcm. Such a clock genera lor is shown as dock generator "-'
160. In one embodiment, clock generator 160 comprise a
phase-locked loop (PLI.) that provides clocking signals to
processor 103.
Each of the display areas is associated with a program-
ming module. Each of the modules provides a spccilic status
or control function. In one cmlmdimcnl, !he module is
rcprcsenlcd by a disk Jllc containing the code necessary for
In one embodiment, processor 103 is a member of the
68000 family of processors, such as !he 68040 processor
manufactured by Motorola Corporation of Schaumberg, Ill.
The memory in the <.:omputcr system is initialized to store
the operating system as well as other programs, such as Jilc
directory routines, control programs, system programs and
application programs, and data inputlcd from I. 0 controller '
5
130. The operating syslcm running on processor 103 takes
care of basic tasks such as starting the system, handling
interrupts, moving data to and from memory 104 and
peripheral devices via inpul/oulpul interface unit 140, and
managing the memory space in memory 104. In one
40
embodiment, the operating system is stored in ROM 106,
while RAM 104 is utilized as the internal memory for the
computer system for accessing data and application pro-
grams.
the module to interact with the <-Xlfltrol strip as well as other
clements such as text, icons, pictures, etc. Modules may be
designed to be responsive to selection from cursors via a
'
0
mouse, trackpad, or cursor control keys, such as on a
keyboard. Many of !he modules arc able to provide control
to various system functionality, and may provide menus to
do the same.
Processor 103 accesses memory in !he computer system
The control strip is a control panel that provides the
operating environment for control strip modules. In one
cmlmdimcnl, !he control strip runs on any Macintosh'"
computer using a System 7.0 or later operating system. The
control strip of the present invention may be designed to run
on <-Xlmputcr systems using other operating systems.
In one embodiment, !he control strip is implemented in a
privalc window layer that appears in front of the windows of
all !he application layers. That is, !he control strip window
appears on top of all application programming windows that
45
may be generated as part of the execution of an application
program. This prevents other windows from obscuring it. In
one embodiment, processing logic in !he computer system
may maintain a list of windows ordered from the frontmost
via an address bus within bus 100. Commands in conncclion
with the operation of memory in the computer system arc
also sent from the processor to the memory using bus 100.
Bus 100 also includes a hi-directional data bus lo commu-
50
nicalc data in response to the commands provided by
processor 103 under the control of the operating system
window on the screen being at the top of the list and the
bottommost window being at the bottom of the list. Pro-
cessing logic can maintain !he control strip window a! the
top of the list.
running on it.
Of course, certain implementations and uses of the
present invention may neither require nor include all of the 55
almvc components. For example, in certain implementations
The control strip of the present invention may include
windowing conligurations that arc shown as being horizon-
tal or vertical on the screen. Furthermore, the present
invention is not limited to a single row or column of slat us
a keyboard or cursor control device for inpulling informa-
tion to the system may not be required. Furthermore, the
computer system may include additional processing units.
OVERVIEW 01 TilL PRESENT INVENTION
The present invention provides a control and/or status
window for display on the desktop of the computer system.
The control and status window will be referred to herein as
the <:ontrol strip. The control strip of the present invention is
a window of graphics depicting one or more display areas
for control and/or status indicia. In one cmhodimcnl, each of
and control data areas. In other words, multiple rows and
columns of module data areas may be included in the
window of the control strip.
CONTROL STRIP MANIPULATION
The control strip, such as shown in FJ(i. 2B, may also be
moved to di!Tcrcnt portions of the display screen. I lowcvcr,
in one embodiment, the window for the control strip may be
65 moved to any location on the display as long as the right
and. or left edge of !he strip is allachcd lo the right or left
edge, respectively, of !he display. The user may also hold
us 6,493,002 131
7
down !he option key and drag !he lab 203 of the control strip
200 with the usc of a cursor control dcvi<.:c (e.g., trackpad,
trackball, mouse) to move the <.:ontrol strip to a new position
on !he display.
8
The control strip may also include a ballcry monitor !hal
displays the status of the battery or batteries. In one
embodiment, the battery monitor displays the current power
drain in a manner similar to a cars miles per gallon (MPG)
In one embodiment, !he user may adjust the si.-:c of the
control strip window. Ad_iuslmcnls to the si.-:c of the window
may <.:omprisc either an increase in the height of the window,
the width of the window, or both. In one embodiment, only
!he width of the control strip window may altered. The
dell nit ion and usc of windows is well-known in the arl. In
one embodiment, the control strip 200 has a tab 203 on its
unattached end. The user can drag tab 203 to adjust the
length of the strip. By "clicking on tab 203, i.e. selection
through !he usc of !he lrackpad, mouse, cursor control keys,
clc., !he user is able lo shirt from a minima! control strip si.-:c
to a maximum control strip size, and vice versa. In its
minima! size, the graphics of the modules in the control strip
.< indicator. The needle for !he power drain indicator imlicalcs
!he drain relative lo !he maximum possible. The control strip
of the present invention allows this display to be updated
frequently so if the user increased the IFD display screen's
brightness level, !he needle would animate lo denote !he
111 consequence of !he action.
Another control strip module displays !he slate of File
Sharing (e.g., on, oil", or users connected) that may be
currently employed on the computer system. The Ji!c sharing
module also lets the user tum !He sharing on or oil" and lets
1.< the user open a control panel lo control processing lo sci up
!He sharing on !he com puler system.
arc not visible and only the tab is showing. In its maximum
si.-:c, a!l of !he modules in !he control strip arc showing.
Rccogni.-:ing cursor con!rol!cd sclec!ions through !he usc of
trackpad, trackba!!, mouse, cursor control keys, etc., as wei!
as the tracking of movements of the cursor made by the same
The control strip of the present invention may also
provide a module to allow the internal hard disk power to be
turned off (to save power), and to indicate whether is
"II currently on or o!T.
arc well-known in !he arl.
Scro!! arrows, such as !crt scroll arrow 204 and right scm!!
arrow 205, arc provided on !he control strip !hal enable !he "-'
window of the control strip to be scrolled to the left or right,
respectively. Usc of scroll arrows with windows is well-
known in !he arl.
The user may also hide !he control strip. In one
The control strip may also provide power scllings that
a !low the user to select hclwccn maximum ballcry conser-
vation or maximum computer performance without opening
a control pane!. In one embodiment, the power settings
portion of the control strip also allow the user lo open up the
power savings control panel. The control strip of !he
invention may also include a function that p!a<."CS the com-
puter in sleep mode or allows the user to select the sound
volume.
Other modules, for example, may provide lime and/or
dale information, may list currently running programming
applications, may indicate !he amount of available memory,
may control a CD drive, may provide access to audio
embodiment, to make the control strip disappear comp!c!ely, .>o
the user can dick the !!ide button in the control strip control
panel, as described later in conjunction with FIG. 3. A dose
box 201 is also indudcd in control strip 200 In one
cmbmlimcnl, by holding !he option key and clicking a
display area, !he user can drag !he display area lo another
position in the <-Xlntro! strip. An example of the process of
moving one display area to another position on the control
strip is shown in FJ(i. 2L. Referring lo FIG. 2L, !he user
sclecls one of the display areas by, for instance, positioning
the cursor over the display area. When the user ""clicks" the
display area, its border becomes highlighted. While clicking,
_,
5
controls and status information. Therefore, the control strip
acls as a slat us and control function bar, or windowing area,
that provides running modules lo he displayed in an arrange-
ment that is to be displayed, such an arrangement being
modi!iab!c such that the size of the window or bar may be
40
changed.
the display area is dragged to another location in the control
strip module display area. When !he user has moved !he
display area lo !he location of his choice, !he user slops
""dicking The <-Xlntro! strip display areas arc then rear- 45
ranged.
In one embodiment, the control strip is con!ro!lab!c
through a control pane!. An exemplary display of such
control panel is shown in FJ(i. 3. Usc of control panels is
well-known in the art. Using the control panel in FIG. 3, the
user is able to hide or show the control strip by dicking the
corresponding bullon in the control panel. Note !hal in one
cmlmdimcnl, the control panel may also he lo change
the font and size of the text in the control strip window.
PROCESSING LOCilC FOR TilL PRESENT
INVEN"I"ION
The present invention includes computer prm:cssing logic
for generating !he control strip of the present invention. rhis
pro<."Cssing logic is described, in part, in the !low charts
After the user rearranges the parts of the control strip, the
new arrangement is saved. The saving operation may he
deferred until resources, such as !he hard disk is ready (e.g.,
spinning) or until just before the <-Xlmputer system is shut 50
down or restarted. In other words, in computer systems in
which the hard disk is not turned on a!l the time in order to
save power, !he saving operation may be deferred until !he
hard disk has been lumed on by another.
55 shown in FIGS. 4-10. In addition to the computer resources
described earlier, !he present invention relics upon the
availability of an operating system and system llmc!ions
capable of displaying windows, information in windows,
characters, and cursor symbols on the display devices.
EXEMPI.ARY C:ONTROI.iS.Ii\TUS
INI:ORMAI"ION
The control strip of !he present invention provides a
standard screen location for a <.Xl!!ection of individual mod-
ules that provide status and control functions. In one
embmlimcnl, !he control strip functions include a network
switch !hal shows whc!her a nclwork conncclion for !he
computer system, such as an AppleTa!kn' network
connection, is on or oil" and lets the user tum the network
60 System functions for interfacing with the cursor control
devices and cursor funclion keys, including !he !racking of
cursor location within a window, arc also required. These
resources arc standard processing components known in the
art.
connection on or otT without having to !ocate and execute 65
other nclwork connection software on !he com puler system
(e.g., without having lo open !he Chooser.,").
When the pro<."Cssor of the present invention is Jirst
powered up, the operating system logic obtains control and
initializes the system components such as read/write
us 6,493,002 131
9
memory, the display device, !he cursor control device, the
cursor function keys, and keyboard. During this initializa-
tion pro<.:css or in response to a user command, the operating
system displays the control strip of the present invention.
In one embodiment, !he control strip initiali.-:alion is
performed in two stages. The !irs! stage begins by initially
loading at least one routine a! star! up. Upon loading
necessary routines, the operating system allocates storage
for global variables usc. Next, resources arc loaded for usc
10
1\t processing block 407, a lest dc!crmincs if !here has
been a ""dick" of the mouse within the area ddincd by the
control strip. If there has been a dick of the mouse within
the control strip, the mouse dick is processed (processing
.< block 408), and processing continues a! processing block
409. The mouse dick processing dc!crmincs the location of
the mouse dick, which module in the <.:ontrols strip was
selected, or ""clicked-on", if any, and any action to be taken
based on !hal location. One embodiment of the mouse dick
by the control strip processing logic. These resources Ill
indudc !he visual components or indicia that is lo appear in
processing is described in FIG. 9. If a mouse dick has not
occurred within !he control strip, processing continues
directly to processing block 409.
the control strip, such as pictures, i<.:ons, text, etc. The
processing logic for the control strip is patched into the
operating system.
I.atcr, as a second stage of the initiali:talion during the sci
15
up process, !he control strip processing logic causes each of
!he module !lies lo opened one a! a time. The code for !he
module is loaded. An initialization routine is run in response
1\t processing block 409, the previous context is restored
and the processing logic exits to rcturn control to the
operating system.
One cmbmlimcnl of !he secondary initiali:tation process
called by !he control strip main processing logic is described
in a !lowchart in FIG. 5. Referring to FIG. 5, the sc<.:ondary
to a call, during which time, the module itsdf determines if
it can run. This information is conveyed to the control strip. :w
The processing logic !hen causes !he window to be displayed
and calls the modules to run lhcmsdvcs and appear in the
control strip. FIG. 2F shows the control strip window
graphics generated by processing logic being combined with
graphics genera led by a module to illuslrale !he creation of "-'
!he resulting control strip.
initialization pro<..-css begins by testing whcthcr the Finder'"'
has slar!cd up (processing block 501 ). If the Finder'-" has not
slar!cd up, the secondary initiali:tation process ends.
llowcvcr, if the I:indcr""' has slar!cd up, the window of the
control strip is created (pro<..-cssing block 502).
Then a test determines whcthcr the creation of the win-
dow of the control strip was successful (processing block
503). If !he creation of !he window of the nmlrol strip was
no! successful, the process ends. The creation of the window F!Ci. 4 is a Jlowcharl of the processing logic responsible
for generating the control strip of the present invention and
processing events that occur involving the control strip. In
one cmbmlimcnl, the control strip main processing is called
by the system. Referring lo FIG. 4, the processing
begins by saving !he previous conlexl and sets up the conlexl
of the control strip (processing block 401). The previous
context refers to the state of the <.:omputcr system prior to
performing control strip processing. The previous conlcxl
may correspond lo an application program running imme-
diately prior to the control strip processing being called. The
context may include settings up its memory space, providing
access to its global variables, etc.
Next, a lest dc!crmincs if the secondary initiali:tation has
been done (processing block 402). If !he secondary initial-
i:talion has no! been done, processing continues a! process-
ing block 403 where a sc<.:ondary initialization process is
run, and pro<.:cssing thereafter continues at processing block
409. The secondary initiali:talion process causes !he pro-
cessing logic lo initialize the control strip. One embodiment
of the secondary initiali.-:alion process is described in FIG. 5.
On the other hand, if the window of the control strip is
allocated, processing continues at processing block 404
where !he processing logic awaits a user even! and dctcr-
mincs !he type of such an even!.
Then a test dc!crmincs if the uscrcvcntlypc is a null even!
(processing block 405). That is, a test dctcrmincs whether
the user event type is idle or not. If the user event type is a
null even!, processing nmlinucs at processing block 406
where idle tasks arc run, and processing thcrca!"tcrnmtinucs
at processing block 409. Thus, during idle periods, tasks
involved with the control strip window may be ntn as wdl
as tasks of the modules. Examples of module tasks may
include updating hdp messages (e.g., due loa help !Ca!urc
being enabled on !he compulcr system) and saving updated
state information (e.g., display area on screen moved to new
location, display area resized, module made invisible; mod-
ule indicates state is changed and that it must be saved). On
!he other hand, if the even! type is not a null event,
processing continues at processing block 407.
may not be successful because, for instance, there is not
.>o enough memory, missing system rcsour<..-cs, etc. On the other
hand, if !he creation of the window of the control strip was
successful, !he font and color of !he control strip arc initial-
i.-:cd (processing block 504). Then external modules arc
opened and initialized (processing block 505), the default
_,
5
screen location and size of the control strip arc set
(processing block 506), the user conllguralion is loaded
(processing block 507), and the conlcnls of the control strip
arc drawn (pro<..-cssing block 508). The user con!iguration
may include screen location for the control strip, the saved
40
display order of the modules, the window size of the control
strip, de. Then the secondary initiali.-:ation process ends.
The default screen location and si.-:c of the control strip arc
stored in memory and accessed. In one cmhodimcnl, these
values may be changed by the computer user, such as by
4
5 interacting with the control strip itsdf In another
cmlmdimcnl, the dc!"aull values arc dctcrmincd and perma-
nently sci by a system designer. Nole !hal spccilkation of the
font, color, default screen location and si:tc may not be
required in lieu of the user con!iguration. l.ikcwisc, by using
50
solely the default settings, the user con!iguration is not
required.
One embodiment of !he process for opening and initial-
i.-:ing cxlcrnal modules such as may be invoked by the
secondary initialization process is shown in FIG. 6. Refer-
55 ring to FIG. 6, the processing logic begins by testing whether
there arc more module Jllcs to he opened (processing block
601). If there arc no more module Jlles(c.g., all !he modules
have been and loaded), !hen the process ends. The
modules arc opened and initialized one at a time. If there arc
60 more module tiles, the processing logic opens the module
Jllc (processing block 602) and loads the module code into
memory (processing block 603). The processing logic calls
the module to initialize itsdf (processing 604). The module
is then also called by the pro<..-cssing logic to obtain the
65 features of the module (pro<..-cssing block 605) and to obtain
the width of !he module"s area, as well as features of the
module (processing block 606). The features of !he module
us 6,493,002 131
II 12
perform a save operation. This de!crminalion is based on
whether the rcsour<.'CS arc available (i.e., II.D. is turned on)
to perform the save operation. If it is not safe to perform a
save operation, processing continues at processing block
include help messages to be displayed when the module is
"dickcd on with the cursor. Then the module Ji!c is dosed
(processing block 607) and the processing loops back to
processing block 601. By looping back to processing block
601, the processing logic is able to provide !he initia!i.-:alion
procedures lo a!! !he modules, such !hal when a!! the
modules have been processed the process ends. When the
process ends, it returns in a manner well-known in the art to
!he processing logic !hal ca!!cd (e.g., initialed) it.
.< 806. However, if it is safe lo save control strip, processing
continues a! processing block 805 where !he conllguralion of
the control strip is saved to disk. Thereafter processing
continues to processing block 806.
At pro<.-cssing block 806, the current idle module is called
One cmhodimcnl of a process for drawing !he contents of
!he control strip (processing block 508), such as used a!
processing block 508 of FIG. 5, is described in FIG. 7.
Referring to FIG. 7, the processing logic initially determines
if the control strip is visible (processing block 701). If the
control strip is no! visible, processing ends. Thai is, if !he
user has hidden !he control strip, !he present invention will
not draw its contents.
Ill to run its idle !ask. In one cmhodimcnl, the processing logic
of !he present invention allows only one module lo run its
idle tasks during each a call to the processing of FIG. 8 (e.g.,
the currently designated module) to reduce overhead time.
ldcnti!kation of the current module is based on an ID
1.< associated with each of !he modules.
Then !he idle !ask undergoes post processing (processing
block 807), and the ID of the next module is updated to idle
(processing block 808). That is, the module designated as the
current module for the next call to the processing of FIG. 8
On the other hand, if the control strip is visible, processing
continues at processing block 702 enters a looping structure
where !he processing logic lcsls whc!hcr !here arc more
modules lo draw. If !here arc no more modules lo draw,
processing ends and control returns lo !he process !hal called
"
11
will be the next module in the list of modules. An example
of !he post processing is shown in FIG. 10.
it. If there arc more modules to draw, processing continues
at processing block 703 where the processing logic tests
whether !he particular module needs lo he redrawn. A "-'
module may need lobe drawn when !he information being
displayed needs to be updated. For example, as the amount
Then, !he processing logic determines whc!hcr !he mod-
ule needs to save settings for usc later (processing block
809). If the module docs not need to save its settings,
processing continues lo processing block 812. On !he other
hand, if !he scllings of !he module arc lo be saved, !he
of energy in the battery is changing due to energy <.:onsump-
tion from the computer system, an update to the battery
indicia in !he control strip must be made. If the module docs
no! need lo he redrawn, processing loops back lo processing
block 702 where the more modules test is repeated. On the
other hand, if the module needs to be redrawn processing
continues a! processing block 704 where !he processing
logic determines whc!hcr !he module is a bullon. If !he
module is a hullon, processing continues a! processing block
705 where the background graphics of the button arc drawn,
and pro<.:cssing continues to pro<.:cssing block 707. If the
module is no! a hullon, !he slalus-only background graphics
arc drawn (processing block 706) and processing continues
40
at processing block 707. Note that in one embodiment, the
type of background graphi<-'S may be obtained using a
message sent to the module requesting its features.
pro<.-cssing logic tests whether the settings may be saved at
this time (pro<.-cssing block 810). One reason the settings
may not be saved is that the hard disk may be powered down
.>0
or turned olT. lf !he sellings can no! be saved a! this lime,
processing continues a! processing block 812. lf !he scllings
of the module may be saved at this time, the processing logic
causes the module settings to be saved to disk (processing
block 811). Thcrca!lcr, processing continues a! processing
'
5
block 812.
AI processing block 812, !he help messages for !he control
strip arc updated, and processing ends and returns to the
control of the main processing logic.
One embodiment of the mouse dick processing of the
present invention, such as used in FIG. 4, is shown in a
Jlowcharl in FIG. 9. Referring lo FIG. 9, !he processing logic
determines whc!her a mouse click has occurred inside !he
control strip (processing block 901). In one embodiment,
AI processing block 707, !he processing logic makes a call
lo !he module lo draw itself. Thai is, it is !he responsibility
of !he module itself lo draw its sial us for control indicia.
Thereafter processing loops back to processing block 702.
One embodiment of the processing for running idle tasks
in FIG. 4 is described in a llowchart in FIG. 8. Referring to
FJ(i. 8, !he processing logic l1egins by dclermining whc!hcr
a window update is pending (processing block 801). If a
window update is pending, the processing <.Xlntinucs at
processing block 802 when the <-Xlfltcnts of the control strip
45
this determination may be made by <-Xlmparing the current
]\JCalion of !he cursor with !he ]\JCalion of !he control strip
(e.g., slalus bar). If a mouse click has no! occurred inside !he
control strip, then processing loops back upon itself, retest-
ing repeatedly until a mouse click docs occur. When a mouse
50
dick occurs within the control strip, processing continues at
processing block 902 where a de!crminalion is made as lo
upon which module !he cursor was during !he click.
arc drawn. Window updates may be required due to a change 55
in slalus in one of !he modules. Changes may also be due lo
a reordering of !he control strip entries. Note !hal one
embodiment of the process to draw the control strip is shown
in FIG. 7. Thereafter processing continues at processing
block 803. If a window update is no! pending processing
60
continues direc!ly lo processing block 803.
AI processing 803, processing logic lcsls whc!hcr !he
con!iguration of the control strip has changed. If the con-
!iguration of the control strip has not changed, processing
continues at processing block 806. If the con!iguration of the 65
control strip has changed, processing continues a! process-
ing block 804 where a lest determines whc!her it is safe lo
The processing logic !hen determines whc!her a move
operation is being selected by the mouse (processing block
903). If a move operation has been chosen, the display of the
module is moved or !he entire control strip is moved
(processing block 904) and !he processing logic ends !he
mouse dick process and exits to <-Xlfltrol of the processing
logic that called this procedure. The determination of
whether to move a module or the entire strip is based on the
uscrs keystrokes or mouse movements. On !he other hand,
if a move operation is no! lo occur, processing continues a!
pro<.-cssing block 905.
At processing block 905 the processing logic determines
whether the module on which the dick occurred is a
"clickahle module, as opposed lo a slalus only module,
(processing block 905). Thai is, !he processing logic lcsls
us 6,493,002 131
13 14
clc., arc contained in a Jllc on a disk. The control strip
pro<..-cssing logic draws the strip which acts as the back-
ground for the individual modules. Each module is respon-
sible for drawing the i<..Xlns and other objects that make up its
whc!hcr the module provides any additional furu:lionality
when a mouse moves the cursor to select an clement in the
control strip. If the module is not "clickablc," processing
ends. If the module is dickablc, processing <.:ontinucs to
process processing hlm:k 906 where the mouse dick is
!racked, i.e., !he lm:alion of !he cursor.
.< user interface.
Conlcnls of Module Files
In one embodiment, !he module !lie includes only a single
resource <..Xlfltaining the <..XKlc necessary for the module to
interact with the control strip. A module Jilc may contain
Next, a test determines if the mouse is still within the
bounds of the module (processing block 907). If the mouse
is not within the bounds of the module, pro<.:cssing ends.
llowcvcr, if !he mouse is within !he boundaries of the
module, !he module is e<J!!cd lo process the dick (processing
block 908) and the dick undergoes post processing
(processing block 909). Thereafter, the pro<.:css ends.
Ill more than one code resource if it is to provide mul!ifunc-
lional support. In !hal case, each module in !he !lie is loaded
and initialized scparalcly and lrca!cd as an
entity.
One embodiment of the process for post pro<..-cssing the
mouse dick is shown in FIG. 10. Referring to F!Ci. 10, the 1.<
Module Interface
The interface of the module to the control strip comprises
processing logic by determining whether !he module
desires to update its settings (processing block 1001). If the
settings for the module arc to be updated, pro<..-cssing con-
tinues at processing block 1002 where a llag is set to indicate
!hall he module has a save and processing contin-
ues lo processing block 11Hl3. The sellings for a module may
have to be updated due to user interaction, such as in the case
a code resource. In one embodiment, using the Macintosh'"
computer, !he type of !he code resource is "sdcv This code
is responsible for performing all of the functions required by
the control strip as well as any functions that arc custom to
"
11
the module ilscl!". The mmlulc"s entry point is allhc begin-
ning of !he resource and is defined as
of an option for a module being turned o!T or a module
acquiring data as part of its functionality. If the settings of
a module do no! have lobe updated, processing continues "-'
dircdly lo processing block 1003. Selling may need lo he
updated when the module is displaying information that is
changing frequently.
At proccssinl!. block 1003, a test determines whether the
.>0
module needs lo rcsi...:c !he display. If !he display of !he
pascal long Conlro!StripModulc (long message, long
params, Reel 'slalusRcd, Ciraf!Pir slalusporl);
Interactions between a module and the <..Xlfltrol strip arc
managed by passing messages to the module to tell it what
lo do or lo obtain information about !he module and its
capabilities. In one cmlmdimcnl, each module is required lo
observe Macintosh""' Pascal register saving conventions;
that is, it may trash 680x0 processor registers DO, D1, D2,
AO, and A1, but must preserve all other registers across its
call. Note !hal other operating systems and implcmcnlalions
of !he present invention may have di!l"crcnl rcslridions.
module must be rcsi...:cd, processing continues a! processing
block 1004 where the module is called to update its width.
Then processing continues at processing block 1005. On the
other hand, if !he display of !he module docs no! need lo he
resized, processing continues dircdly lo processing block
1005.
The message !kid comprises a message number from the
_,
5
list in the section "Control Strip Module Messages" that
indicates to the module the action to perform.
The params !icld signi!lcs !he result returned by !he
iniliali...:c call lo !he module. This would typically be a
pointer to a pointer (e.g., the handle) to the private variables
At processing block 1005, the processing logic deter-
mines whether the module desires to be dosed. If the
module desires to be dosed, processing continues at pro-
cessing block 1006 where !he module is dosed immediately.
Then processing continues a! processing block 1007. If !he
module docs not desire to be dosed, processing continues to
processing block 1007.
40
to be used by the module since modules cannot have global
variables. This result is passed lo !he module on all subse-
quent calls. Note !hal in embodiments where modules can
have global variables, such a Jicld may he climina!cd.
The statusRcct !kid comprises a pointer to a rectangle
45
within the control strip ddining the area that a module may
draw within.
At processing block 1007, a test determines whether the
control strip is lo be rcsi...:cd or dosed. If !he control strip
needs lo be rcsi:tcd or dosed, processing continues a!
processing block 1008 where the module displays arc repo-
sitioned and redrawn, and then processing continues at
processing block 1009. If the control strip is not to be resized
50
or dosed, processing continues dircdly lo processing block
1009'
AI processing block 1009, a lest determines whether !he
help state of the module is to be changed. The help state
refers to help messages that the modules provide to users 55
generally. If !he help stale of !he module is lo changed,
processing continues a! processing block 1010 where !he old
help state of the module is invalidated and the process ends.
If the module help state docs not need to be changed,
processing ends. Changes to the help state may occur due to 60
a global change in !he computer system, such as when a
particular help feature (e.g., help balloons) is enabled.
The slalusPort Jicld spccillcs a pointer lo !he graphics pori
of the control strip. The graphics port may be either a color
or black-and-white graphics port, and depends on the com-
puter system on which the <..Xlfltrol strip is running.
The result value rclurncd by !he module
on !he message sen! lo it. Results for each message arc
described below in the sections on the individual messages.
CONTROL STRIP MODULE REFERENCE
In one embodiment, control strip modules interact with
the <..Xlfltrol strip processing logic in three ways: by accepting
messages, by calling utility routines, and by calling the
operating system manager (e.g., a call lo Cics!al! sclcdors).
The next three scdions each of !hose inlcradions.
Control Strip Module Messages
ADDINCi CONTROL STRIP MODULES
In one embodiment, the <..Xlntrol strip of the present
invention operates as a shell with individual control and
slalus modules added. Each module and its icons, pidurcs,
In one embodiment, all <..Xlfltrol strip modules respond to
messages from the control strip processing logic, which is
65 responsive to user interaction with the control strip dis-
played on !he screen. The following messages have been
dcllncd:
us 6,493,002 131


,Uc,-1-caltllcs
;dc,-(<ctDL-'pla\Wldth
MJc,PcrimJid idle
llclp
sdcvlni!Modu!c
15
lnltlali/.c the llH><iulo
Clean up bot{!lc boi11g do;od
Return lhc i'caltllc hils
Rotum tho currolll width oltho
1no<Julc-, Ji,pla,-
l'crio<Jic tidlc \\hen nothing else is

l:txJalc I he inlcli"acc in lhc conllol
;trit>
t.,cr ha; dickod on the modulo';
Ji,pla\ a1ca
Sa>o ,,,. cha11god ;ctting; in the
moJulc., tllclo,o,Kc., file
Di,plav a help hal loon. if the
moJulc ha; '"'"
'"
1.<
The sdcvlnitModuk message is the Jirst message sent to :w
a module after the module has been loaded from its Ji!c.
Initialintion allows the module to initialize its variables and
16
embodiment, this hit issei when, for example, a module
has a pop-up menu associated with it. If this bit is
cleared, the control strip tracks the cursor until the
mouse button is released, then sends an sdcvMousc-
Ciick message, dcscrilJcd below, to !he module lo
notify it that there was a mouse-down event.
c) sdcvl IasC:ustoml lelp (2)-If this hit is set, the module
is responsible for displaying its own help messages.
These help messages may be customized depending on
its current slate. If the bit is cleared, !he control strip
displays a generic help message when !he cursor passes
over the its display area and Balloon Help, or other
help-based information provider, is on.
d) sdcvKccpModu!d.ockcd (3)-If this bit is set, the <.:ode
of !he module is kepi locked and prolcclcd. In one
embodiment, this hit is set only if the module is passing
!he address of one of its routines to a routine cxlcrnal
to the module (e.g., installing itself in a queue).
sdcvGctDisp!ayWidth
lo dclcrminc whether it can run on a particular machine. For
example, if !he fund ion of !he module is lo display ballcry
information, it may be only able to n1n on a portable
computer, such as the Powcrbook manufactured by Apple
Computer.
In response lo receiving !he sdcvlnitModule message, the
module loads and detaches any resources (e.g., text, code,
icons, etc.) in its resource !ile that will be used. Also, space .>o
is allocated in the global variables for handles to those
detached resources.
The sdcvGctDisplayWidlh message is sent to a module lo
determine how much horizontal space (in pixels) its display
currently requires on the control strip. In response to the
message, the module return the number of pixels as its result.
In one embodiment, the returned width docs not comprise
"-' the maximum width required for any con!lguralion, hut
instead, rc!lcc!s how much space it currcn!ly requires. Note
that this useful because, in one embodiment, its possible for
a module to request that its display be resized.
The sdcvlnitMmlule message returns a result depending
on its success at installing itself. In one embodiment, a
positive result indicates successful installation. The _,
5
processing logic passes this result value to the module on all
subsequent calls. A negative result indicates an error
condition, and installation of the module is aborted by the
control strip processing logic. Also if a negative result
occurs and installation has been aborted, the module docs
40
no! receive a close message.
sdcvC'IoscModule
The sDcvCioseModule message is scnllo a module when
it should be dosed. In one embodiment, the module itself
decides when to be dosed. A module may be dosed when
4
5
it no longer is required to be running, such as when a ballcry
level indicator no longer needs to be running when the
com puler system is receiving its power from an outlet. When
the module receives this message, it disposes of all the
detached resources it loaded as well as its global storage. No 50
result is cxpcclcd.
sdcvl.-calurcs
The sdcvFcaturcs message queries the module for the
features it supports. This message returns as its result a
bitmap consisting of 1 bits for supported features and 0 bits
55
for unsupported !"ca!urcs. In one cmhodimcnl, there arc 32
bits returned. All undefined bits arc reserved for future
features, and, in one embodiment, arc set to 0. The bits arc
ddincd as:
a) sdcvWantMouscC!icks (0}--lf this bit is set, the control 60
strip noti!lcs !he module of mouse down If this
hit is not set, the control strip assumes !hat !he module
only displays status information with no user interac-
tion.
b) sdcvDonv\utoTrack (1}--lf this bit is set, the control 65
strip highlights !he display of the module and then calls
the module to perform mouse tracking. In one
sdcv Per iodicTickle
The sdcvPcriodicTick!c message is passed lo !he module
periodically to allow !he module to update its display due lo
changes in its state. In one embodiment, this message occurs
at regular intervals, while in other embodiments, there is no
minimum or maximum interval between "tickles."" In
response lo !he sdcvPcriodidkkle message, the module
returns, as its result, some bits that signal requests for ad ions
from the control strip processing logic. In one embodiment,
there arc 32 bits returned. All unddincd bits in the result arc
reserved for future usc and, in one cmhodimcnl, arc set to 0.
The bits arc defined as:
a) sdcvRcsi.-:cDisplay (0)-lf this hit is set, the module
resizes its display. The control strip processing logic
sends a sdcvGctDisplayWidth message to the module
and !hen updates !he control strip on the display.
b) sdcvNccdToSavc (I)-If this hit is set, !he module
needs to save changed settings lo disk. The control strip
processing logic marks the request but may defer the
actual save operation to a better time (e.g., when the
hard disk is spinning).
c) sdcvllelpS!atcChangc (2)-lf this bit is sci, the help
message of !he module needs to be updated due to a
change in state. If a help balloon is being displayed for
the module, the control strip processing logic removes
the previous help balloon with a new help balloon for
!he current stale.
d) sdcvCioscNow (3)-If this bit is sci, !he module is
requesting to be closed. The control strip processing
logic calls the module to save its settings, then calls the
module again to dose itself by, for example, disposing
of any loaded resources, disposing of private storage,
clc.
sdcvDrawStatus
The sdcvDrawStatus message indicates that the module
has to redraw its display to rcllcct the most recent state. In
one embodiment, this message is sen! when !he user clicks
on !he display area of the module, when any of !he display
us 6,493,002 131
17
18
pascal Boolean SB!sContro!StripVisib!c ( ); of !he module is n:si:tcd, or when !he control strip itself
needs to be updated, perhaps in response to a screen saver
deactivation.
The SB!sControlStripYisible routine returns a Boolean
value indicating whether or not the <..Xlfltro! strip is currently
visible. ll returns a value of ""true if the control strip is
5
visible, or a value of false if it's hidden.
The statusRcct parameter points to a rectangle bounding
!he display area of !he module, in lm:al coordinates. A!!
drawing done by a module within !he bounds of !he control
strip is limited to the module's display rectangle. In other
embodiment, drawing may extend outside the display rcd-
angc of the module. rhc dipping region of the control strip s
window is scllo the visible portion of !he display rectangle 111
of the module so thai a!! !he clcmcnls in !he display may he
drawn. If the dipping region is to be changed, the initial
dipping region should be observed to avoid drawing over
other ilcms in the control strip.
sc!cvMouscC'Iick
When the user clicks in a display area of the module, the
control strip pro<.:cssing logic calls the module with the
sdcvMouscCiick message if the sdcvWantMouscC'licks bit
is set in the features of the module.
If the sdcvDontAutoTrack hit is also set, the control strip
processing logic draws the display of the module in its
highlighted state and then sends the sdcvMouscC'lick mes-
sage to the module. If the sdcvDontAutoTrack bit is not set,
the control strip processing logic tracks the cursor until the
mouse bullon is released. If the cursor is still within the
display area of the module, the control strip pro<..-cssing logic
sends the sdcvMouscC'lick message to notify the module
that a click occurred. In either case, the module can then
perform the appropriate function in to a
down event.
This message returns the same result as the sdcvPcriod-
icTiddc message.
sc!cvSavcScttings
The sdcvSavcScllings message is passed to the module
when the control strip processing logic has determined that
the <..XlflJiguration information may be saved to the disk (e.g.,
liD turned on, etc.). In one embodiment, the sdcvSavcSct-
tings message is sent only if the module had previously
the sdcvNccd.J(JSavc bit in the rcsull of a sdcvPcriollicTickle
1.<
In one embodiment, the SB!sControlStripYisiblc call
returns a value of true" even when the control strip is not
visible. That happens whenever the control strip is not
accessible in the current environment. soon as that
condition changes, the control strip becomes visible again
and the returned value correctly rcllccts the actual state.
S BShow II idcC'omro!Strip
The SBShowl lideContro!Strip routine shows or hides the
control strip. An exemplary call follows:
pascal void SBShowllidcC'ontro!Strip (Boolean show h);
The SBShowllidcContro!Strip routine determines the vis-
ibility stMc for the control strip 1-utscd on the v;tluc of the
"showlt"" parameter. Passing a value of ""!rue makes the
control strip visible, and passing a value hil!cs it.
:w Modules may not need to call this routine. llowcvcr, the
SBShowllidcC'ontroiStrip routine provides a means for
other so!"twarc to hide the control strip when it is in the way.
Calling the SBShowl!ideConlro!Strip routine with a
"show it"" value of "true may or may no! show the control
"-' strip, depending on the current environment. If the control
strip is not accessible, it docs not become visible. If a
"show it"" value of ""true is passed lo this routine, !hen the
control strip becomes visible when the environment
changes.
.>0
S llSa fc"lbAcccssStartu p ]) isk
The SBSafc"lbAcccssStartupDisk routine determines
whether !he intcmal hard disk is turned on so that processing
logic of the present invention can determine whether lo
make a disk ac<.."CSS or postpone it until a time when the disk
.>s is already spinning. An exemplary call follows:
pascal Boolean SBSafe"lbAcccsStartupDisk ( );
The SBSa!CToAcccss..'ltartDisk routine returns a Boolean
value of"truc if the disk is turned on ami if it is nol.
SBOpcnModuleRcsourccFilc
The SBOpcnModuleRcsourccFile routine opens a module
resource !lie. An ex<1mp!ary call follows:
pascal shorl SllOpcnModulcResourcl'ilc (OSType
!lleCrcalor);
or sdcvMouscCiick message. The call returns an error code 40
(File Manager, Resource Manager, or the like) indicating the
success of the save operation. The control strip processing
logic continues to semi this message to the module until the
module returns a rcsuh of 0, indicating a successful save. The SBOpcnModulcRcsourccFile routine opens the
45 resource fork of the module !ilc whose creator is sdcvShowBalloonllclp
!iJcCrcator", and return the Jilc's reference number as its
result. If the Jl!c cannot he found or opened, the SBOpen
Mdu!eResourccFile routine returns a rcsuh of-!
The control strip processing logic calls the module with
the sdcvShowB<JIIoonl!clp message if Balloon I!clp is
turned on, the module has previously set the sdcvl !asCus
tomllelp bit in its features, and the cursor is over the
module's display area. In such a case, the module calls the
Help Manager to display a help balloon describing the
current state of the module. The module returns a value of
The SBOpcnModuleRcsourccFile routine also provides a
50 means for a module to load in large or infrequently used
resources !hat it doesn"tusually need, hutlhal it requires for
a particular operation.
0 if successful or an appropriate error rcsuh if not. SBI AJadPrefcrences
The SBLoadPrcfercnccs routine loads a resource li"om a
UTILITY ROUTINES
55 preferences !ile. An cxamplary call follows:
In one embodiment, the control strip processing logic
provides a set of utility routines that arc available to control
strip modules. They arc provided to promote a <.."<lflsistcnt
user interface within the control strip and to reduce the
amount of duplicated code that each module would have to
60
include to support common functions. Therefore, in an
cmbmlimcnt that docs not include these utility routines, a
portion or all of the modules may include duplicated code
supporting <..Xlmmon functions.
SlllsContro!StripVisib!c 65
"]he SBl-;Contro!StripVisib!c routine determines whether
the control strip is visible. An exemplary call follows:
pascal OSErr SB!-<:JadPrel"crcnccs (Cons!Sir255Param
prc!SResourccNamc, Iland!e 'preferences);
The Sill .oadPrcl"crences routine loads a resource contain
ing a module's conJiguration information from the prefer-
ences Jilc of the control strip. The PrefsRcsour<..-cNamc
parameter points lo a Pascal string containing !he name of
the resource. The "Preferences parameter points to a vari
able that holds a handle to the rcsour<.."C read from the !ile.
The handle docs not need to be preallocated.
If either prefsRcsourccNamc or preferences contains a nil
pointer, the SBI.oadPrc!Ocrcnu:s routine docs nothing and
returns a result of paramErr. If the resource is successfully
us 6,493,002 131
19
loaded, the SBLoadPrcfcn.:nccs routine rei urns a rcsuh of 0.
The SBLoadPrcfcrcn<..'CS routine also returns other Memory
Manager and Resource
some art of the process.
SllSavc Pre fcrc nccs
Manager errors if it fails during
The SBSavcl'rcfcrcnccs routine saves a resource to a
prcfcrcn<.:cs Ji!c. An exemplary call follows:
pascal OSLrr SBSavcPrcfcrcnccs (ConstStr255Param
prcfsRcsourccNamc, llandk prdCrcnu:s);
20
module is open. This is typically !he case during a module's
initialization ca!!.
SBTrackpopupMcnu
The SBTrackpopupMcnu routine manages a pop-up
5
menu. 1\n exemplary ca!! fo!!ows:
pascal short SBTrackpopupMcnu (const Reel
moduleRcct, Mcnullandlc thcMcnu);
The SBSavcPrc!Crcnccs routine saves a resource contain-
111
The SBTrackpopupMenu routine handles selling up and
displaying a pop-up menu associated with a module. The
module passes a pointer to its display rectangle and a handle
to the menu to usc.ln one embodiment the menu is displayed
immediately above and adjacent lo the display rectangle of
the module, yc!lhis is no! required. By doing so, !he user is
a !lowed to view the current <.'onJiguration or to change the
settings. The SBTrackpopupMenu routine returns an in(li-
ing a module s con!iguration information to the prcfcrcn<..-cs
!ilc of the <.:ontrol strip. The PrcfsRcsourccNamc parameter
points loa Pascal string containing !he name of the resource.
The "prc!Crcru:cs parameter contains a handle lo a hlm:k of
data which will he wrillen lo !he Jl!c.
t.< calion as to which menu item was selected, or 0 if no item
was selected (e.g., because !he user moved the cursor outside
the menu's bounds).
If either prcfsRcsourccNamc or preferences has a nil
value, the SBSavcl'rcfcrcnccs routine docs nothing and
returns a result of paramErr. if the resource is successfully
saved, the SBSavePrefcrences routine returns a result of 0.
The SBSavcl'rcfcrcnccs routine can also return other :w
Memory Manager and Resource Manager errors if it fails
during some part of the process.
Sll( iet I )etachcdString
The SBGetDetachedlndS!ring routine obtains a string
from a detached resource. 1\n exemplary ca!! follows:
pascal void SBGetDctachcdlndString (StringPtr the
String, Handle stringList, short whichString);
SBTrackSlider
The SBTrackS!ider routine displays and scls an arbitrary
paramctcr. 1\n exemplary call follows:
pascal short SBTrackSlider (<-onst Reel *modu!cRecl,
short ticksOnS!idcr, short initialValue);
The SBTrackS!idcr routine displays an unlabeled slider
above !he module's display reclangle. The slider may be
"-' used for displaying and setting the state of an arbitrary
parameter. The parameter "ModuleRcct' contains a pointer
to !he module's display rcclangle; "licksOnSlidcr is the
upper hounds of the value rclurned by the slider; and
"initial Value" is the starting position (0 to ticksOnS!idcr -I).
The SBGetDetachedlndS!ring routine is the dctachcd
resource version of GetlndString. The parameter theslring
points to a Pascal string, the stringList is a handle to a
detached STR#' rcsour<.:c; and whichString is the index
(I-n) into the array of Pascal strings contained in the
detached resource. The SllCict!)ctachcdlndString routine
copies the string whose index is whichS!ring into the space
pointed to by thcString. If whichString is out of range, the .>5
SBGctDetachcdlndString routine returns a zero-length
.>o When the user releases the mouse button, the SBTrackSlider
routine returns the !ina! position.
SBShowllelpString
The SBShowllelpS!ring routine displays a help balloon.
1\n exemplary call follows:
pascal OSErr SBShowllclpString (const Reel
*moduleRecl, Stringl'tr hc!pstring);
string.
Sll( ict I )ctachkonSuilc
The SBGctDetachkonSuitc routine sets up a detached
icon suite. 1\n exemplary call fo!!ows:
pascal OSLrr SBGctDctachlconSuitc (Handle
'lhckonSuilc, shorttheResiD, unsigned long sdcclor);
The SBGetDetachkonSuitc routine creates a new icon
The SBShowllelpString routine displays a module's help
balloon. rhe module passes a pointer to its display reclangle
and a pointer to a Pascal string, and the routine displays the
40
balloon if possible. If the help dstring has a length ofO or the
Ilclp Manager is unable lo display a balloon, an error result
is rcturncd. If the SBShowllelpString routine successfully
displays !he help balloon, it returns a result of 0.
suilc, loads a!! of !he requested icons, and then dctachcs the
icons. The parameter thclconSuite points to the location 45
where the handle to the i<.:on suite is stored; the parameter
lhcRcslD is the resource ID of !he icons that make up the
icon suite; and the parameter "seleclor indicates which
icons arc to be loaded into the suite. In one embodiment, the
SBGctBarGraphWidth
The SBGetBarGraphWidth routine determines how wide
a bar graph drawn by !he SBDrawBar(iraph routine
(descril1cd below) will he so thai a module can cakulalc its
display width. 1\n exemplary call follows:
pascal short SBGetBarGraphWidth (short barCount);
The SBGetBarGraphWidth routine returns the width of a
"selector' parameter contains one (or a combination ol) the :;o
following values:
bar graph containing harCounl segments. IfharCounl has a
value less than 0, the SBCictBar(iraphWidth routine rcturns
a width of 0.

"/\IISmalll )ala

load largo 32-lw-.>2-pixol ico11;
("IC.'\#". "icl II{' I
''"'J '""'II l<>h\lf>pixcl icon'
("ic.,#". 'ic.,I!'J
''"'J mini 12-h\12-pi;,.cl icon'
("icon#". "iconi!"J
These values may be ORed logcthcr to load combinations
of icon sizes. The SBGetDetachkonSuitc routine rcturns an
appropriate error code if it's unsuccessful, or 0 if it was able
to load the icon suite. Note that if none of the icons
comprising the icon suite could be found, the call returns the
error "resNoll'ound. ln one embodiment, !he SllCict!)ctachl-
conSuilc routine is ca!!cd only when !he resource Jl!e of the
SBDrawBarGraph
The SBDrawBarGraph routine draw as bar graph. 1\n
exemplary call follows l1c!ow:
pascal void SBDrawBarGraph (short level, short
barCount, short direction, Point barGraphTopl.eft);
The SBDrawBarGraph routine draws a bar graph <.Xln-
60 taining the number of segments speci!icd by the barCount
paramctcr in a module's display area. If the value of
harCounl is less !han or equal to 0, the SBDrawBarGraph
routine docs nothing.
The bar graph is drawn relative to the location speciJied
65 by barGraphTopLeft. FIG. ll illustrates the manner in which
the point barGraphTopl.eft dclermines !he position of the
bar graph.
us 6,493,002 131
21 22
description, it is lobe understood lhallhc particular embodi-
ment shown and described by way of illustration is in no
way intended to be considered limiting. Therefore, refer-
ences lo ddails of the preferred embodiment arc not
The "!ever paramclcr dc!crmim:s how many segments
arc highlighted. The value of "!cvo.r should be in the range
ofO to barCount -1 If the value of "level" is less than 0, no
segments in the bar graph arc highlighted; if "level" is
greater !han or equal to barCmml, all segments in the bar
graph arc highlighted.
The direction parameter spcciJics which way the bar
graph will be drawn to show a larger level. In one
cmbmlimcnl, the din:clion paramclcr spccillcs om: of the
following values:
5
in!cndcd lo limit the scope of the claims which in themselves
recite only those features regarded as essential to the inven-
tion.
#define BarGraphSlopcLcfl -1 //max end of sloping
graph is on the left #ddinc BarGraphF!atRight 0 //max
end of !lat graph is on the right #ddinc BarGraphSlo-
pcRight 1 //max end of sloping graph is on the right
'"
F!Ci. 12 illustrates the rcsuhing bar graph for each dircc- 1.<
lion value. The arrows indicalc which way an increasing
level value is displayed. In one embodiment, for sloped
versions of the bar graph, the number of segments spcci!kd
by the barCounl value may no! be larger thanK If a larger
barCounl value is passed, the SBDrawBar(iraph routine :w
draws nothing.
SBModalDialogln("ontcxt
The SBModa!DialoglnContcxt routine may be used in
place of !he Moda!Dialog routine lo prevent background
applications from being run while the modal dialog window "-'
is visible. An exemplary call is as follows:
pascal void SBModa!Dia!oglnContcxt
(ModalFi!tcrProcPtr Ji!tcrProc, short item! lit);
The SllModa!!)ialoglnConlcxl routine is a special version
of MmlalDialog that docsn "t allow background applications '
0
lo be run while a modal dialog window is visible. The
SBModa!Dia!oglnContcxt routine is used when the
occurcn<.:c of context switching is not desired.
GL'iTAIT SU.ECTOR
The control strip processing logic installs two Gestalt"
selectors to return information to locations external to the
com puler system. One sdcdor rdurns software allributcs,
ami the other rdums the current version of the processing
logic (e.g., software).
gcsta!tControlStripi\ttr
.>5
"'
Thus, a mdhod and apparatus for generating a window
displaying control and status indicia has l1ccn described.
I daim:
l An interactive computer-controlled display system
compnsmg:
a processor;
a data display screen coupled lo the processor;
a cursor <.:ontro! device coupled to said processor for
positioning a cursor on said data display screen;
a window generation and control logic coupled to the
processor and data display screen lo create an operating
environment for a plurality of individual programming
modules associated with dill"crcnt application programs
!hal provide status ami/or control !lmc!ions, wherein
!he window generation and control logic generales and
displays a !irst window region having a plurality of
display areas on said data display screen, wherein the
!irs! window region is indcpcndcn!ly displayed and
independently aclive of any application program, and
wherein each of the plurality of display areas is asso-
ciated with one of the plurality of individual program-
ming modules, the !irs! window region and the plurality
of independent display areas implemcnlcd in a window
layer that appears on top of application programming
windows that may be generated; and
an indicia generation logic coupled to the data display
screen lo cxeculc a! least one of the plurality of
individual programming modules to gcneralc informa-
tion for display in one of the plurality of display areas
in the !irst window region, wherein at least one of the
plurality of display areas and its associated program-
ming module is sensitive to user input, and further
wherein the window generation and control logic and
the indicia generation logic usc message-based com-
munication to exchange information to coordinate
aclivities of the indicia generation logic to enable
intcradivc display adivily.
The selector "gcstaltContro!StripAHr ('sdcv") rdurn 32
bits describing the attributes of the current version of the
control strip processing logic. In one embodiment, only the
following bit is dell ned:
2. The display system defined in daim I wherein the Jlrsl
45 window region comprises a control strip.
gcstai!ConlrolS!ripExists 0 !=control strip is installed
gcstai!ControlStrip Version
3. The display system defined in claim l wherein said at
least one display area is variably si...:ed.
4. The display system defined in daim I wherein si:te of
the Jirst window region is variable.
5. The display system ddincd in claim 4 wherein the Jirst
window region is sized such that none of the plurality of
display areas is visible.
The selector gcsta!tControlStripVcrsion ("csvr') returns
!he version of control strip processing logic !hal is installed.
50
The formal of !he rdumcd version is the same as !hal of the
numeric part of a Macintoshn' computer system resource,
that is:
6. The display system defined in daim 4 wherein the Jlrsl
window region is sized such that all of the plurality of
55 display areas arc visible.
Bi"
Bib C.>-CO
1\i" l<i-H>
Bib
\1ajor oi" the ,.c"ion. in BCD
pa1t of the ,o,;ion. in BCD
Bug 1dca'c ,.c"i'"' in BCD
Rolca;c ;tago:



s Cl c !Ojl lllC " l
Rc\ision lc\cl oi" nonrclcaMOJ in hina"
7. The display system defined in daim 4 wherein the Jlrsl
window region is si...:ed such that a portion of the plurality of
display areas is visible.
8. The display system ddincd in daim 1 wherein at least
60 one of the plurality of the display areas only displays
information.
9. The display system ddincd in daim I wherein a! least
one of the display areas acts to provide access to control
information when selected.
Whereas many alterations and modiJications of the 65 10. The display system ddincd in daim 9 wherein said at
least one of the plurality of display areas displays an
additional display dement.
present invention will no doubt become apparent to a person
of ordinary skill in !he art after having read !he foregoing
us 6,493,002 131
23
11 The display system dell ned in claim 1 wherein each of
the plurality of display areas is individually and variably
sized.
12. The display system ddincd in claim 1 wherein the !irst
window region always appears in front of application win- .<
dows.
13. The display system ddincd in claim 1 wherein the !irst
window region is implemented in a private window layer
!hal a p p e a r . ~ in front of windows for a!! applications layers.
14. An intcraclivc computcr-conlro!!cd display system 111
compnsmg:
a processor;
a data display screen coupled to the processor;
a cursor control device coupled to said processor
positioning a cursor on said data display screen;
for
15
window generation and control logic coupled to the
processor and data display screen to crcalc an operating
environment for a plurality of individual programming
modules associated with di!Icrcnt application programs :w
that provide status and/or <.:ontrol functions, wherein
the window generation and control logic generates and
displays a Jlrsl window-region having a plurality of
display areas on said data display screen, wherein the
Jirst window region is independently displayed and "-'
independently active of any application program, and
wherein each of the plurality of display areas is asso-
ciated with one of !he plurality of individual program-
ming modules, the !irs! window region and the plurality
of independent display areas implemented in a window .>o
layer that appears on top of application programming
windows !hal may be generated; ami
a! least one indicia graphics generation logic coupled to
the processor and the window generation and control
logic, wherein said at least one indicia graphics gcn- .>5
cration logic generates user sensitive graphics for dis-
play in at least one data display area by executing a!
least one of !he plurality of individual programming
modules;
wherein the window generation and <:ontrollogic deter-
4
o
mines when said at least one data display area has
been sclcc!cd by the user and signals said at leas! one
indicia graphics generation logic in response lo user
selection, and further wherein said at least one indi-
cia graphics generation logic initiates a response 45
from said at leas! one of !he plurality of program-
ming modules.
15. The display system defined in claim 14 wherein !he
lirst window region is always visible to the user.
16. The display system ddincd in claim 14 wherein the 50
Jlrsl window region comprises a control strip.
17. The display system dell ned in claim 14 wherein said
at least one display area is variably sized.
18. The display system ddincd in claim 14 wherein each
of the plurality of display areas is individually and variably 55
si...:cd.
19. The display system defined in claim 14 wherein !he
lirst window region always appears in front of application
windows.
20. The display system defined in claim 14 wherein !he
6
o
Jlrsl window region is implemented in a private window
layer thai appears in front of windows for all applications
layers.
21 A method for generating control information compris-
mg:
creating an operating environment for a plurality of
individual programming modules associated with dif-
65
24
l"crcnl application programs that provide status and/or
control functions;
generating a lirst window sized to accommodate a plu-
rality of display areas for indicia resulting from execut-
ing at least one of !he plurality of individual program-
ming modules, wherein each of !he plurality of display
areas is associated with one of the plurality of in(li-
vidual programming modules, and wherein the Jirst
window is independently displayed and independently
aclivc of any application program, !he Jlrsl window
region ami the plurality of imlcpendcnl display areas
implemented in a window layer that appears on top of
application programming windows that may be gener-
ated;
displaying !he indicia in each of said plurality of display
areas by executing one of a plurality of individual
programming modules corresponding to each indicia;
selecting one of the indicia, wherein the selecting com-
prises a lirst programming module determining which
of said plurality of display areas is sclec!cd and sending
a message loa programming module of said plurality of
individual programming modules responsible for gen-
erating a display of a selected indicia;
said programming module performing a function in
response to a selection.
22. The mclhod defined in claim 21 wherein one of said
plurality of indicia comprises status information.
23. The method ddincd in claim 21 wherein one of said
plurality of indicia comprises control information.
24. The mclhod defined in claim 21 further comprising:
!he Jlrsl programming module requesting a sci of features
supported by said programming module, wherein said
requesting comprises sending a lirst message to said
programming module; and
said programming module returning a second message
indicative of l"ca!ures supported by said programming
module, such !hal said Jlrsl programming module inter-
acts with said programming module in response to user
interaction with the Jirst programming module based on
indicated l"ca!urcs as sci forth by said programming
module.
25. A system comprising:
a window generation and control logic to create an
operating environment for a plurality of individual
programming modules associated with dill"crcnt appli-
cation programs that provide slat us and. or control
llmc!ions, wherein !he window generation and control
logic generates and displays a Jirst window region
having a plurality of display areas, wherein the Jirst
window region is indcpemlently displayed and imle-
pcndcntly aclive of any application program, and
wherein each of !he plurality of display areas is asso-
ciated with one of the plurality of individual program-
ming modules, the lirst window region and the plurality
of independent display areas implemented in a window
layer !hat appears on lop of application programming
windows that may be generated;
an indicia generation logic coupled to the data display
screen to execute at least one of the plurality of
individual programming modules to generate informa-
tion for display in one of !he plurality of display areas
in the lirst window region, wherein at least one of the
plurality of display areas and its associated program-
ming module is sensitive to user input, and further
wherein the window generation and control logic and
!he indicia generation logic usc message-based com-
us 6,493,002 131
25
municalion lo exchange information lo coordinate
activities of the indicia generation logic to enable
interactive disp!av activitv.
26. An intcraclivc display system
compnsmg:
a means for positioning a cursor on a data display screen;
a means for creating an operating environment for a
plurality of individual programming modules associ-
alcd with dilTcrcnl application programs that provide
status and/or control functions, wherein a Jlrsl window
111
region is displayed having a plurality of display areas
26
a!cd with di!l"crcnl application programs !hal provide
status and/or control functions, wherein a !irst window
region is displayed having a plurality of display areas
on said data display screen, wherein !he Jlrsl window
region is indcpcmlcnlly displayed and indcpcndcn!ly
active of any application program, and wherein each of
the plurality of display areas is associated with one of
!he plurality of individual programming modules, !he
!irs! window region and !he plurality of imlcpcndcnl
display areas implemented in a window layer that
appears on top of application programming windows
!hal may he generated;
a means for generating user sensitive graphics for display
in at least one data display area;
a means for determining when said at least one data
display area has been selected by the user; and
a means for initialing a response from said a! leas! one of
!he plurality of programming modules.
on said data display screen, wherein the !irst window
region is independently displayed and independently
aclivc of any application program, and wherein each of
the plurality of display areas is associated with one of
15
the plurality of individual programming modules, the
Jirst window region and the plurality of independent
display areas implemented in a window layer !hal
appears on lop of application programming windows
that may be generated; and
40. The display system dcllncd in claim 39 wherein !he
"
11
Jirst window region is always visible to the user.
a means for executing at least one of the plurality of
individual programming modules to generate informa-
tion for display in one of !he plurality of display areas
in !he Jlrsl window region, wherein a! leas! one of !he
plurality of display areas and its associated program- "-'
ming module is-sensitive to user input, wherein an
inlcraclivc display aclivily is enabled.
41. The display system ddincd in claim 39 wherein the
Jlrsl window region comprises a control strip.
42. The display system defined in claim 39 wherein said
at least one data display area is variably sized.
43. The display system dc!incd in claim 39 wherein each
of the plurality of display areas is individually and variably
sized.
27. The display system defined in claim 26 wherein !he
!irs! window region comprises a control strip.
28. The display system ddincd in claim 26 wherein said
at least one of the plurality of display areas is variably sized.
29. The display system defined in claim 26 wherein si.-:c
44. The display system dcllncd in claim 39 wherein !he
Jirst window region always appears in front of application
.>o windows.
45. The display system dcllncd in claim 39 wherein !he
of !he !irs! window region is variable.
30. The display system ddincd in claim 29 wherein the
!irst window region is sized such that none of the plurality .>5
of display areas is visible.
31 The display system defined in claim 29 wherein !he
!irs! window regions is si.-:cd such !hal all of !he plurality of
display areas arc visible.
32. The display system ddincd in claim 29 wherein the
4
o
!irs! window regions is si.-:cd such !hal a portion of !he
plurality of display areas is visible.
33. The display system dell ned in claim 26 wherein said
at least one of the plurality of display areas only displays
information.
34. The display system dell ned in claim 26 wherein said
a! leas! one of !he plurality of display areas acls lo provide
access lo control information when selcc!cd.
35. The display system ddincd in claim 34 wherein said
45
at least one of the data areas display an additional display
50
clement.
36. The display system defined in claim 26 wherein each
of the plurality of display areas is individually and variably
sized.
37. The display system ddincd in claim 26 wherein the 55
!irs! window region always appears in front of application
windows.
38. The display system ddincd in claim 26 wherein the
!irst window region is implemented in a private window
layer !hal appears in front of windows for all application
60
layers.
39. An inlcraclivc compulcr-conlrollcd display system
comprising:
a means for positioning a cursor on said data display
screen;
a means for creating an operating environment for a
plurality of individual programming modules associ-
65
Jlrsl window region is implemented in a private window
layer !hal appears in front of windows for all applications
layers.
46. A computer readable medium containing executable
computer program instructions, which when executed by a
data processing system, cause !he data processing system lo
perform a method for generating <:ontro! information com-
prising.
creating an operating environment for a plurality of
individual programming modules associated with dif-
!"crcnl application programs !hal provide slalus and/or
control functions;
generating a !irst window sized to accommodate a plu-
rality of display areas for indicia resulting from execut-
ing a! leas! one of !he plurality of individual program-
ming modules, wherein each of !he plurality of display
areas is associated with one of !he plurality of indi-
vidual programming modules, and wherein the Jirst
window is independently displayed and independently
aclivc of any application program, !he Jlrsl window
region ami !he plurality of imlcpcndcnl display areas
implemented in a window layer that appears on top of
application programming windows that may be gener-
ated;
displaying !he indicia in each of !he plurality of display
areas by executing one of a plurality of individual
programming modules corresponding to each indicia;
and
selecting one of the indicia, wherein the selecting com-
prises a !irs! programming module dclcrmining which
of !he plurality of display areas is selected and sending
a message to a programming module of the plurality of
individual programming modules responsible for gen-
erating a display of a selected indicia, and the program-
ming module performing a funclion in response lo a
selec!ion.
us 6,493,002 131
27
47. The compulcr readable medium as set forth in daim
46 wherein ones of the indicia comprises status information.
48. The computer readable medium as set forth in daim
46 wherein one of the indicia <.:ompriscs control information.
49. The compulcr readable medium as set forth in daim .<
46 further comprising:
!he Jlrsl programming module requesting a set of !Calmes
supported by said programming module, wherein
requesting comprises sending a Jirst message to said
programming module; and Ill
said programming module rcluming a second message
indicative of features supported by said programming
module, such that said Jirst programming module inter-
acts with said programming module in response to user
inlcraction with the !irs! programming module based on
1
-'
indicalcd features as sci forth by said programming
module.
50. A system comprising.
a means for window generation and control to create an :w
operating environment for a plurality of individual
programming modules associated with di!lCrcnt appli-
cation programs !hal provide status and/or control
functions, wherein the means for window generation
and control generates and displays a !irst window
28
region having a plurality of display areas, wherein the
!irs! window region is independen!ly displayed and
independently active of any application program, and
wherein each of the plurality of display areas is asso-
ciated with one of the plurality of individual program-
ming modules, the !irs! window region and the plurality
of independent display areas implemented in a window
layer !hat appears on top of application programming
windows that may he generated;
a means for indicia generation coupled to the data display
screen to execute at least one of the plurality of
individual programming modules to generate informa-
tion for display in one of the plurality of display areas
in the !irs! window region, wherein a! !cas! one of the
plurality of display areas and its associated program-
ming module is sensitive to user input, and further
wherein the means for window generation and control
and the means for indicia generation usc message-
based communication to exchange information to coor-
dinate aclivities of !he means for indicia generation lo
enable interaclive display aclivily.












































Exhibit 4
111111 1111111111111111111111111111111111111111111111111111111111111
(12) United States Patent
Ording
(54) LIST SCROLLI\"G A\"D DOCU\1E\"T
TH.. \:'\SL \TIO\", SC\LI\"G, A :'I'D H.OT. \TIO\"
0:\ A TOUCH-SCREE:\ DISPLAY
(75) !mentor Ording. San Francisco, CA (US)
(73) Assignee: .\pplc Inc .. Cupcnino. C.\ (US)
( * ) Notice: Suh_icc!lo any disclaimer. the lcrm of this
pntcnt is extended or adjusted under 35
U .S.C. 154(b) by 0 d.1ys.
(21) App!. No.: 111956,969
(22) Filed:
(65)
I>cc. 14, 2007
Prior Publication Oata
(60)
(51)
(52)
(58)
(56)
US 2(XlR/OlOX404 AI Jul. 10. 200X
H.clatcd U.S. Application I>ata
Provisional applicmion No. 601937.993. tiled on Jun.
29. 2007. provisional applicmion No. 601<)46.971.
tiled on Jun. 28. 2007. provisional application No.
(i()/()45.R5R. !!led on Jun. 22. 2007. provisional appli-
cation No. 601870.460. tiled on Jan. 8. 2007. provi-
sional application No. 601883.801. Hied on Jan. 7.
2(X)7. provisional application No. 60/879.253. Hied on
Jan. 7. 2(Xl7.
Int.("].
G06F 3101
u.s. Cl.
(2006.01)
7151702:715/764:715/863:
715/864: 715/76<)
Fil.'!d of (:Jassitication Scarch 7!51764.
715/7(,<). 702. 863. 864
S<..-c application Hie for complete search history.
Rcfcrcnccs Citcd
.".4<).".566
'
2 1<)<}(, Kw,tind;- ................... 3<)5 1."7
.'i.i-:44547 A l2 J<J<)g d
.'i.i-:67.l.'ii-: A 2
]<)<)<)
d 341
6.034.(igg
'
.12000 345 yq
FP
US007469381B2
(I OJ Patent No.:
(45) Date of Patent:
US 7,469,381 B2
Dec. 23, 2008
ti.41N.951 B1 122002 Wongcl"l .................. 14." t7.1
ti . ."ti7.to2 B2 ." 200.1 Kung .......................... 14." 6(>0
(Continued)
I:ORIOICiN I'Al"ION"I"I)OC"UMI;N"l"S
063577<} .\t 1 1<)<)5
(Continued)
OI"IIIOR
\1inoso!l Word 2003 Shols
(Continued)
Primar_r Ewminer Boris Pcsin
(74) A/lome\: Agent. or Firm Morgan. I <.."\vis & ISockius
I 1.1'
(57) AHSTJ.t.\CT
In accordance with some embodiments. a compuler-implc-
mcntcd method l(Jr usc in conjunction with a device with a
touch screen display is disclosed. In the method. a movement
of an object on or ncar the touch scr<..-cn display is dct<..>ctcd. In
response to detecting the movement. an ck>clronic document
displayed on the touch screen display is transla!<.."d in a Jirst
dir<..>clion. If an edge of the d<..>clronic document is reached
while 1ranslating the electronic documcm in 1hc !irs1 direction
while the objcc1 is s1ill deJected on or ncar 1hc 1ouch screen
display. an area beyond 1hc edge oft he documcm is displayed.
Aller the ohjccl is no longer dc!<..>cted on or ncar the !ouch
screen display. the document is lransla!<.."d in a second dir<..>c-
tion until the area hL")""tmd the edge of the document is no
longer displayed .
20 Claims, JS Drawing Sheets
1- I ,;:,11 :2::'<1
... .
'
"''"'""" '"'

- -

-
-

'
>"'
-
""'"'
.
"""" ..
-
""H
Woo""'
-
=
""""
=
-lli
US 7,469,381 B2
Page 2
.......................... 34.'i J5g
6.(>61.409 B2 122003 Dcmm1incsda1 345 !73
20060Jg].<;JO .\1
20060!1>7215 -\1
200601'!7753 .\1
2006 ong4<J5 .\1
2007 OOOl>Otiti .\1
2007 002464(, .\1
2007 004ti64ti .\1
2007 00ti7745 .\1
2007 0!0<}275 -\1
2007 01201>35 .\1
2007 Ol501>2ti .\1
2007 0!501>42 -\1
2007 0!52971> -\1
2007 015297<) .\1
2007 0!55434 -\1
2007 0!5ti364 -\1
2001> Ol04.'i44 .\1 $
:; 200(,
g 200(,
9 200(,
10 200(,
l 2007
2 2007
3 2007
3 2007
-' 2007
.'i 2007
........... 14-' !7_1
llotd1ing .................... 34.'i 173
B2
6.707.449 B2
6.go<J.724 Bt
6.907575 B2
6.<Jl2.462 B2
6.<J72.776 B2
6.975.306 B2
7JJO<J.59<J B2
7JJ46.230 B2
7.075512 Bl
7.102.(>26 B2
7.1."4.534 B2
7 .1SS.041> B2
7.t:;un B2
7.1 g,J.7<J6 B2
7.240.291 B2
2003 009.'iD.'i At
2003 0095(>97 A 1
200_1 011>452." _\1
2004 002J(i76 At
2004 0027391> A 1
2004 001>0541 _\ 1
2004 01."51>1>1> _\1
2004 026341>6 A 1
2005 00127D At
2005 01451>07 _\1
2005 OJMWil> At
2oo:; otn:;:;:; At
2oo:; on no:; _\ 1
2005 027026<) _\ 1
2005 0275(>11> At
200ti 00252U< _\1
200(> 002(>521 _\1
200(i 004imn At
200(i OO.'i.'i(i69 At
200(> 00615."1 _\1
200(> 0077544 _\ 1
200(i 001>2549 A 1
200(> ()()<)4502 _\1
200ti 01Sti24<J _\1
2 2004
.12004
to 2004
(i 2005
6 2005
12 2005
12 2005
.1200(>
5 200(>
7 200(i
9 200(i
12 200(>
12 200(>
2 2007
2 2007
7 2007
5 2003
5 2003
!0200_1
2 2004
2 2004
4 2004
:; 2004
12 2004
t 2005
7 2005
g 2005
<) 2005
!0 2005
!2 2005
12 2005
2 200(,
2 200(,
3 200(,
3 200(,
.1200(,
4 200(,
4 200(,
5 200(,
7 200(,
et 345 tiM
llinckkyct;!l 345173
Shiraishi ct 345 16'!
Dum1<: ...................... 7157g4
Ogaki ........................ 701 20g
])"'is 345 r;g4
llinckky d 345 173
l'ihl"j" ...................... 345 173
3451-'ti
et 345 l.'iti
Denny. tH ................... 345 17'!
ct ,1 _141> 207 t
........................ 31>2 132
I.e Cocq ct 703 t
K"ridis d ............... 455 56ti
C.ml
ct
Wood
715 776
345 (, 13
3g2 13 t
........................... 345 173
Chen ct 345 tiM
........................ 345 U>3
3451>0-'
345til9
Seni ........................... 345 173
................... 345 173
I "PSllin d ,] 250 56(,
................... 345 ti.'i<J
7157M
345173
................... 345 1."6
Juh ct 345 l.'i6
4ti3 37
llotelling d ,] 715 702
el 715 71;4
............................ 345 L'iti
h1teh ......................... 345 1."1>
St,rk ......................... 35<) 441>
Hoshi no et 345 l.'i7
K"t"ym11" ............. 4ti3 31
71571>!
......................... 34.'i l5ti
Fuku<hi . 340 -' 52
S;mrinend"l ............. _14."6ti0
................ 34.'i 173
Choi .................. 7l.'i l>ti3
Ch,mg ....................... 14-' !7_1
Smo ........................... 34.'i 173
62007 ............. 7J.'i 772
62007 Ch;mdhri ............ 7!." l>ti_1
7 2007 Kociend.a d ,] ............. 14-' !7_1
7 2007 Jobsct ................... 34.'i 173
7 2007 Jobsd ................... 4-'-' Sti5
7 2007 Rothkopf .................... 702 !17
.'i20og ............... 7l.'il>4ti
FOR!]GN P.-\T!:N"r DOCUM!:NTS
Fl' 0 701 220 _\ .119%
Fl' 0 1>1>0 091 _\2 11 1991>
WO \\'002 0133g At I 2002
\\'0 \\'0 03 Oti0622 _\2 7 200_1
\\'() \\"()200."05277_1 _\2 ti2005
WO \\'0 2006 003.'i<J I A I 200ti
W<l \\'() 2006 020305 A2 2 200(,
en! II OR
Photo 3 1 2 200ti Shots''
Repo11 and Written Opinion for
_\pplic"tion '\o PCT CS2001> 0."0292. 19. 2001>
lB\1. "SnoH conlrol Box." IB\1 Discloslire n>l
3g_ '\o 04 .. \pr 19'!3. pp 3'!9-403
Tidwell et Designing lntel'lilces. '\ov 2005. Sec-
lion 1>5
lo P"y _\ddiliom!l for _\pplic"lion '\o
I'C"I t:S2oog 0502'!2. Jut IK 2oog
\1ilkr. ]) . _\pplication Em ironm@l." Sun
hllp sun com prodlicls pcrsom!ljmalouch-
ablc. Jun 1>. 19<J<J. 13
* cited by examiner
U.S. Patent Dec.23,2008
Memory
1.Q;t

Operating System
y-126
Communication Module
'V-128
Contact/Motion Module
]f130
Graphics Module
v-132
Text Input Module
J._r134
v-135
GPS Module
j..r 136
Applications
Contacts Module
v-137
Telephone Module
v-138
v-139
Video Conference Module
E-mail Client Module
v-140
Instant Messaging Module
v-141
Slogging Module
v-142
v-143
Camera Module
Image Management Module
v-144
Video Player Module
v-145
Music Player Module
v-146
Browsing Module
v-147
r1o3
Sheet I of 38 US 7,469,381 B2
P t bi M In r o e or a e U IUOC 100 eVIC
1Jll!
Applications (continued)
Calendar Module
Widget Modules
Weather Widget
Stocks Widget
Calculator Widget
Alarm Clock Widget
Dictionary Widget

.
.
User-Created Widget(s)
Widget Creator Module
Search Module



Power
f.r162
System
r"
External V124
Port r
( 118
"'103
v1
j_.r1




j_.r1
36
48
49
49-1
49-2
493
494
49-5
49-6
50
51
104_/
/
v-10
8
RF Circuitry
122 _./
Controller

110\, s
"103
Peripherals
1103
...,.,v
Interface

It

120 _./
Processor(s)
"" Microph
+
....
Proximity
113
f"103
Sensor
l'---166
peaker
111
one
110 Subsystem /158
106 .../
Display
Optical
Other Input
Controller
sensor(s)
Controller(s)
v
Controller
160
156-----
f-'103 f-'103 f-'103
Touch-Sensitive
Optical Other Input
!J1
112 .../
Display System
Sensor(s) Control
164 Devices
16
Figure 1
U.S. Patent
<0
0
N
I
I
206
Dec.23,2008 Sheet 2 of 38 US 7,469,381 B2
Prtbl Miff f D. o a e U IUnCIOn ev1ce
~
100
(Speaker 111)
l Optical J l Proximity J
Sensor 164 Sensor 166
.1
.....
r
L
[
202
Touch Screen 112
c.

l Microphone J
Home
113
204
Figure 2
U.S. Patent
300

I

Dec.23,2008 Sheet 3 of 38
Portable Multifunction Device
100
(Speaker ill)
Optical
Sensor 1M
Proximity
Sensor.l.Q
Current 3oa
310 Day and 312
Wallpaper image
314
US 7,469,381 B2
[[ ]
Slide to unlock
]
$ $
302
Microphone
113
306
Home
204
Figure 3
Touch Screen 112
U.S. Patent
400
I
Dec.23,2008 Sheet 4 of 38 US 7,469,381 B2
Portable Multifunction Device
100
(Speaker 111)
Optical
Sensor 164
Proximity
Sensor .1.2.
IM
Text
141
M
75"
Weather
lllU
+-X7
Calculator
~
t
Phone
138
Current Time 404
Photos
144
Stocks
lllU
Alarm
1494
6
~
Mail
140
410
408
Camera
143
Videos
145
Jan
15
Blog Calendar
ill ill
User
ABC Created
Widget
Dictionary Widget
14M 1496
0
fl
Browser Music
147 146
Touch Screen 112
Microphone
113
Home
204
Figure 4
U.S. Patent Dec.23,2008 Sheet 5 of 38 US 7,469,381 B2
'02
~ )
Detect a movement of an object (e.g., a finger) on or near a touch screen
display of a device.
+
~ 5
Scroll a list of items displayed on the touch screen display in a first
direction (e.g., vertical or horizontal).
Scroll the list at a speed corresponding to a speed of movement of '-'
the object.
Scroll the list in accordance with a simulation of an equation of motion'-'
having friction.
t
Is a
terminus of the
r
r
04
506
5 0 ~
list reached while scrolling the
list in the first direction while the object
is still detected on or near
No
--
Process
Complete
the touch screen
display?
+ Yes
~ 5
Display an area beyond the terminus of the list.
Display the area in white.
'--
r
Display the area as visually indistinct from the background of the list. '-' r
+
~ 5
After the object is no longer detected on or near the touch screen display,
scroll the list in a second direction opposite the first direction until the area
beyond the terminus of the Jist is no longer displayed.
Scroll the list using a damped motion.
'--
r
Make the terminus of the list appear to be elastically attached to an
r
edge of the touch screen display or to an edge displayed on the touch'-'
screen display.
Figure 5
14
5 I I>
SIX
20
522
524
U.S. Patent Dec.23,2008 Sheet 6 of 38 US 7,469,381 B2
~ I
~ I
Portable Multifunction Device
2!& 100
3500A
I
(Speaker 111)
Optical
Sensor 1...4.
Proximity
Sensor.1.QQ
Current Time 404
3502
Mailboxes s inbox
6
5
3504
~ 1 ~ J ~ M ~ ~
Bruce Walker January 30, 20o6s
Lunch meeting todayf
3508
.
2
3530
3506-3 3510-3
Kim Brooi(f January 30, 2006
5
_,3508-3
Draft agreement
3512-2
Bob Adams
Project Orion
e Darin Adler
fantasy football
I
: January 29, 2006
I
:r3514
I
I
V January 29, 2006
Aaron Jones
Fwd: status report
January 29, 2006
Kim Brook
Re: proposal

I
-.v
3520 3522
Microphone
ill
January 28, 2006
T
'-3518
0
~
/
3524 3526
3310
Home
204
Figure 6A
U.S. Patent Dec.23,2008 Sheet 7 of 38 US 7,469,381 B2
Portable Multifunction Device
~ 100
3500A
s
(Speaker 111)
Optical
Sensor1.i
Proximity
Sensor12Q
Current Time 404 a:!> 406
s3502
Mailboxes lnbox
6
s3504
3506-1 3510-1
Aaron J a n e ~ January 30, 2006s
s3508-1
Re: Project Orion 3534
3512-1 3506-2 3510-2
Bruce Walkers January 30, 2005s
. 3508-2
Lunch meetmg today.!' ~
3506-3 I j910-3
K1m Brook' : January 30, 2006
.)3508-3 I
Draft agreement :r3514
Bob Adams
Project Orion
I
I
1 January 29, 2006
v
3512-2
e Dann Adler January 29, 2006
fantasy football
Aaron Jones
Fwd: status report
January 29, 2006
...
'-3518

0
~
/
I
~
3520 3522 3524 3526
3310
Microphone Home
113
204
Figure 6B
U.S. Patent Dec.23,2008 Sheet 8 of 38 US 7,469,381 B2
Portable Multifunction Device
2QQ 100
3500A
s
(Speaker 111)
Optical
Sensor 1..4
Proximity
Sensor 1...QQ
Current Time 404
Mailboxes .r
3502
lnbox
6 .r3504
_r3506-1 3510-1
Aaron Jone:s January 30, 2006.r
.r3508-1
Re: Project Orion 3534
3512-1 3506-2 3510-2
Bruce Walkers January 30, 2006.r
Lunch meeting today>
3508
-
2
=
, 3506-3 I 3510-3
K1m Broo!<f : January 30, 2006.r
.[3508-3 1
Draft agreement :r3514
I
I
V January 29, 2006
Bob Adams
Project Orion
tr
2
Darin Adler January 29, 2006
fantasy football
'Y
'-3518

0
~
/
I
""
3520 3522 3524 3526
3310
Microphone Home
113
204
Figure 6C
U.S. Patent Dec.23,2008 Sheet 9 of 38 US 7,469,381 B2
~ I
~ I
Portable Multifunction Device
2QQ 100
3500A
\
(Speaker 111)
Optical
Sensor 1.Q1
Proximity
Sensor .1..QQ
Current Time 404
. .f3502
Mailboxes Inbox
6 .f3504
3506-1 3510-1
Aaron Jones.J January 30, 2006.J
. . .f35081
Re: ProJect Onon 3534
~ ~ ? ~ ~ ~
Bruce Walker January 30, 2006.J
Lunch meeting today.f
3508
.
2
3530
K
. B k.f35o6-3 351 o-3
Im roo January 30, 2006.f
3508-3
Draft agreement.!
Bob Adams
Project Orion
3

2
-
2
Darin Adler
fantasy football
Aaron Jones
Fwd: status report
T
January 29, 2006
January 29, 2006
January 29, 2006
'-3518

0
~
/
I
w
3520 3522 3524 3526
3310
Microphone Home
113
204
Figure 60
U.S. Patent Dec.23,2008 Sheet I 0 of 38 US 7,469,381 B2
~
702
Detect a movement of an object (e.g., a finger) on or near a touch screen
display.
+
704
~
Translate an electronic document displayed on the touch screen display in
a first direction (e.g., vertical, horizontal, or diagonal).
Translate the electronic document at a speed of translation
'-'
corresponding to a speed of movement of the object.
Translate the electronic document in accordance with a simulation of '----
an equation of motion having friction.
+
Is an
edge of the
electronic document reached
r 706
r 7 0 ~
while translating the electronic document in
the first direction while the object is still
detected on or near
No
Process
-
Complete
the touch screen
display?
+ Yes
Display an area beyond the edge of the document.
Display the area in black, gray, a solid color, or white.
Display the area as visually distinct from the document.
+
714
~
'-'
r
716
'-'
r 7 1 ~
720 ~
After the object is no longer detected on or near the touch screen display,
translate the document in a second direction (e.g., opposite the first
direction) until the area beyond the edge of the document is no longer
displayed.
r
'-'
Translate the document using a damped motion.
722
Make the edge of the electronic document appear to be elastically "--
r 724
attached to an edge of the touch screen display or to an edge
displayed on the touch screen display.
Figure 7
U.S. Patent Dec.23,2008 Sheet II of 38 US 7,469,381 B2
~ I
~ I
Portable Multifunction Device
206 100
3900A
s
( ) [ Optical
Speakeri11 Sensor164 [
Proximity
Sensor166
402 Current Time 404
Welcome 3904
http://www.company.com/start 3908
Block 3
3914-3
Block 1
3914-1
39251\:t,
Block4
3914-4







Block 5
3914-5
Block 2
3914-2
Block 6
3914-6

p::J
+
I
3916 3918 3920
Microphone Home
113
204
Figure SA
0910
Web page
3912
Block
7
3914-7
Block
8
3914-8
3914-9
U.S. Patent Dec.23,2008 Sheet 12 of 38 US 7,469,381 B2
Portable Multifunction Device
~ 100
3900A
(Speaker ill)
Optical
Sensor.1..1
Proximity
Sensor1.QQ
Current Time 404
Welcome 3904
http://www.company.com/start 3908
Block 1
3914-1
Block 2
3914-2
Web page
3912

I
Block 4
3914-4
Block 5
'\)
3925
3914-5








Block 6
3914-6
Microphone Home
ill 204
Figure SB
+
Block 7
3914-7
Block 8
3914-8
Block 9
3914-9
U.S. Patent Dec.23,2008 Sheet 13 of 38
Portable Multifunction Device
2QQ 100
(Speaker ill)
Optical
Sensor .1i
Proximity
Sensor 1Qfi
Current Time 404
Welcome 3904
http://www.company.com/start 39os
3914-1
Block
2
3914-2
Web page
3912

I
Block4
3914-4
Block 5
3914-5







3914-6
Microphone Home
ill 204
Figure 8C
+
Block 7
3914-7
Block 8
3914-8
Block 9
3914-9
US 7,469,381 B2
U.S. Patent Dec.23,2008 Sheet 14 of 38 US 7,469,381 B2
~ [
~ [
206
Portable Multifunction Device -....._
100 -..
( S eaker
111
) [ Optical J [ Proximity J
P - Sensor 164 Sensor 166
402 Current Time 404 .::lo 406
Welcome 3904
~ ,3910
http://www.company.com/start 3908 V
Block 1
3914-1
Block 2
3914-2
Web page
3912

I
Block4
3914-4
Block 5
3914-5
Block 6
3914-6
( Microphone J
\ 113
Home
204
Figure SD
+
Block 7
3914-7
Block 8
3914-8
Block 9
3914-9
U.S. Patent Dec.23,2008 Sheet 15 of 38 US 7,469,381 B2
~
902
Display an electronic document at a first magnification on a touch screen
display. The electronic document has a document length and a document
width.
~
904
Detect a gesture (e.g., a pinching gesture) on or near the touch screen
display corresponding to a command to zoom out by a user-specified
amount.
~
906
In response to detecting the gesture, display the electronic document at a
magnification less than the first magnification.
Is the
908
(
(910
document length or
document width entirely displayed whlle
the gesture is still detected on or near
the touch screen
No
Process
-
Complete
display?
Yes
~
912
Display the electronic document at a magnification wherein areas beyond
opposite edges of the electronic document are displayed.
~
914
Upon detecting termination of the gesture, display the electronic
document at a magnification wherein the areas beyond opposite edges of
the electronic document are no longer displayed.
Figure 9
U.S. Patent Dec.23,2008 Sheet 16 of 38 US 7,469,381 B2
~ I
~ I
Portable Multifunction Device
206 100
3900A
(
(Speaker ill)
Optical
SensorM
Proximity
Sensor .1...2
Current Time 404
Welcome 3904
http://www.company.com/start 3908
Block 1
3914-1
Block 2
3914-2
Block 3
3914-3
3914-5
Block 6
3914-6

Pl +
I
3916 3918 3920
Microphone Home
113
204
Figure 10A
()3910
Web page
3912
Block
7
3914-7
Block
8
3914-8
3914-9
U.S. Patent Dec.23,2008 Sheet 17 of38
Portable Multifunction Device
100
(Speaker ill)
Optical
Sensor 1.Qi
Proximity
Sensor 1.2.2
Current Time
1
Welcome 3904
http://www.company.com/start 3908
.
I
Block 1
3914-1
Block 3
3914-3
3951' ,Block 4
/
3914-4
Block 5
:3953 3914-5
Block 2 ........ '
3914-2
Microphone
113
Block 6
3914-6
Home
204
Figure 108
+
Block 7
3914-7
Block 8
3914-8
Block 9
3914-9
US 7,469,381 B2
U.S. Patent
3900A
~ I
~ I
Dec.23,2008 Sheet 18 of 38
Portable Multifunction Device
100
(Speaker ill)
Optical
Sensor 164
Proximity
Sensor 166
Current Time 404
Welcome 3904
US 7,469,381 B2
.::!> 406
http://www.company.com/start 3908
()3910
Web page
3912

I
Block 1
3914-1
Block 2
3914-2
Microphone
113
Block 3
3914-3
Block 4
3914-4
Block 5
3914-5
Block 6
3914-6
Home
204
Figure 10C
+
Block 7
3914-7
Block 8
3914-8
Block 9
3914 9
U.S. Patent Dec.23,2008 Sheet 19 of 38 US 7,469,381 B2
~
1102
Display at least a first portion of an electronic document at a first
magnification on a touch screen display.
~
1104
Detect a gesture (e.g., a de-pinching gesture) on or near the touch screen
display corresponding to a command to zoom in by a user-specified
amount.
~
1106
In response to detecting the gesture, display decreasing portions of the
electronic document at increasing magnifications.
(1110
Upon detecting
termination of the gesture, does the
magnification exceed a predefined
magnification?
Yes
No
-
Process
Complete
, - - - - - - - - - - - - - - - - - - - - - ~ - - - - - - - - - - - - - - - - - - ~ ~ ~ 1112
Display a respective portion of the electronic document at the predefined
magnification.
Figure 11
U.S. Patent Dec.23,2008 Sheet 20 of 38 US 7,469,381 B2
~ I
~ I
Portable Multifunction Device
206 100
3900A
I
(Speaker 111)
Optical
Sensor1i
Proximity
Sensor 1.2.
Current Time 404
Welcome 3904
http://www.company.com/start 3908
411' ... ,
Block 139311
'"'
3914-;
, ... ,
139331
'"
Block 2
3914-2
Block 3
3914-3
Block 4
3914-4
Block 5
3914-5
Block 6
3914-6

p:J
+
I
3916 3918 3920
Microphone Home
113
204
Figure 12A
.::!> 406
0910
Web page
3912
Block
7
3914-7
Block
8
3914-8
3914-9
U.S. Patent Dec.23,2008 Sheet 21 of 38 US 7,469,381 B2
~ I
~ I
206
Portable Multifunction Device -...._.
100 '\,
( S eaker
111
) ( Optical J ( Proximity
P - l Sensor 164 l Sensor 166
402 Current Time 404
Welcome 3904
http:/lwww.company.com/start 39os

I
3916
, ... ,
139311
;--
, .... ,
t39331 , __ ,
Block 1
3914-1
r:q
3918
( Microphone
J
Home
113
204
Figure 128
Web page
3912
+
3920
[[j
3922
U.S. Patent
~ I
~ I
3900A
I
Dec.23,2008 Sheet 22 of 38
Portable Multifunction Device
100
(Speaker ill)
Optical
Sensor 1M
Proximity
Sensor1..
Current Time 404
Welcome 3904
http://www.company.com/start 39os

I
Block 1
3914-1
Microphone
113
Home
204
Figure 12C
Web page
3912
+
US 7,469,381 B2
3914-
;i
3914-

U.S. Patent Dec.23,2008 Sheet 23 of 38
... <
/ \
. .
\,b
.. J .....
. .
. .
. .
. .
. .
\..(
9 9 ~ 10SU8S
/ijiW!XOld
-
j ; > g ~ msuas ) (ITT 18>18BdS)
re:mdo
US 7,469,381 B2
J
J
U.S. Patent Dec.23,2008 Sheet 24 of 38 US 7,469,381 B2
r ;mr 1 (;;;'\ r m )
l (s)Jajewom]e:xrv J V \._ euoljdOJO!Jf\J
/. ... ~
\ ... <'
(
~ JOsues em msues) ( )
filiWIXOld Je:;llldO TIT m ~ e e d s
U.S. Patent Dec.23,2008 Sheet 25 of 38 US 7,469,381 B2
g g ~
@c ffi J
( S)J9j9WOJ9j9::l0\f
auoLJdOJO!!N
7
0
"'
_/
\
/
~
9 9 ~ Josuas t 9 ~ Josuas
( ~ ~ ~ Je>jeads)
Al!WJXOJd reondo
U.S. Patent Dec.23,2008 Sheet 26 of 38 US 7,469,381 B2
, - - - - - - - - - - - - - - - - - - - - - - - - " ~ - - , 1402
Detect a multifinger twisting gesture on or near a touch screen display.
The multifinger twisting gesture has a corresponding degree of rotation.
Does the
corresponding degree of
rotation exceed a
predefined degree of
rotation?
No
( 1406
Yes Execute a goo
-------.. screen rotation
command.
~
1408
Execute a screen rotation command with an acute angle of rotation.
~
1410
Upon ceasing to detect the multifinger twisting gesture, executing a
screen rotation command with an angle of rotation opposite to the acute
angle.
Figure 14
U.S. Patent Dec.23,2008
(Speaker 111)
Optical
Sensor 1M
/ "\
\.., ./
Sheet 27 of 38
Proximity
Sensor 1.2
L J
,.. ... 0 ............. ..
-..... (___)_./
A
1506
I.......JI \ ..........
Microphone r2\
~ 1 1 3 C.)
Accelerometer(s)
168
Figure 15A
US 7,469,381 B2
To Figure 158
U.S. Patent
c
Dec.23,2008 Sheet 28 of 38 US 7,469,381 B2
( Speaker 111 )
-
Optical Sensor
164
.(::::\
Mrcrophone ill \J
Figure 158
Proximity
Sensor 1Q6_
0
Accelerometer(s)
168
U.S. Patent Dec.23,2008
(Speaker 111)
Optical
Sensor .124
/'
' ./
Sheet 29 of 38
Proximity
Sensor 12fr
L J
"
... - ...
. .

> ... ./
1508
I \
I.......J/ \.____.
Microphone ~
'---...=ill"'---/ \:V
Accelerometer(s)
1l&
Figure 15C
US 7,469,381 B2
To Figure 150
U.S. Patent Dec.23,2008
(Speaker ill)
Microphone
lli
Optical
Sensor .12:!
~
\!V
Sheet 30 of 38
Proximity
Sensor .1QQ
....... ~
: :
... ---------.....
. .
......... ---
. .
> ... /
1508
Accelerometer(s)
1.!i!l
Figure 150
US 7,469,381 B2
To Figure 15E
U.S. Patent Dec.23,2008 Sheet 31 of 38 US 7,469,381 B2
( ) l Optical
. Speaker 111, Sensor 1M
Proximity J
Sensor .1.22.
1502
/
'
\.. ./
L J
"
\
@
( Microphone
Accelerometer(s)
) 113 168 4
Figure 15E
U.S. Patent
1604-L
5
....
I
Dec.23,2008 Sheet 32 of 38 US 7,469,381 B2

1.; p -') Sensor J..orl.j Sensor .!!&.. I \
.. ----...
,: \
+;
. .
.. .

0
L J
(Microphone) M ( Acceleromeler(s)
113 Q \.. llii
Figure 16A
To Figure 16B
..
U.S. Patent Dec.23,2008 Sheet 33 of 38 US 7,469,381 B2
1604-R
)
ll
()
,c--
'
0
'
" .
- --
L J (
1608-R
0
'-'
\
0
1604-L
"
1502
0
(
1608-L
_)
---'\ , \:
\._
.... __ .... ./
~
\..
Figure 168
U.S. Patent Dec.23,2008 Sheet 34 of 38
'
'
\
.
\ 1606-L
.

:
I /
! ~
\
I '
_;. .. ---
--------------;
'
Figure 16C
'
'
US 7,469,381 B2
U.S. Patent Dec.23,2008 Sheet 35 of 38 US 7,469,381 B2
1604-L
5
( Proimi<yJ ~
Sensor 166
( ) ( Opiicol
Speaker ill Sensor 164
......
..... --..
:... ___________ .,:;
\.,.., ______ ,/
L
0
u
J
(
Microphone ~ Accelerometer(s)'
'----"'' '"-' _.. Q 168 )
~ - - - - = - - - - - - - '
Figure 16D
To Figure 16E
..
U.S. Patent
1604-L
Dec.23,2008
(speaker ill)
1606-L
Sheet 36 of 38
1606-R
Accelerometer(s)
168
Figure 16E
US 7,469,381 B2
To Figure 16F
1604-R
U.S. Patent Dec.23,2008
( Speaker 111 )
Microphone
113
Sheet 37 of 38
Optical Sensor
1M
(::::\
\V
Figure 16F
Proximity
Sensor .122
Accelerometer(s)
168
US 7,469,381 B2
U.S. Patent Dec.23,2008
Device 1lQQ
\.
1710---.._
CPU(s)
1 7 ~
1730\
User interface
Touch-Screen
,_.1740
Display
Keyboard/Mouse
1'fso
Network
1 7 ~
Communications
Interface
Sheet 38 of 38 US 7,469,381 B2
Memory 1770
' ~
Operating System
Communication Module
ContacUMotion Module
Graphics Module
Text Input Module
Applications
Contacts Module
Telephone Module
Video Conference Module
E-mail Client Module
Instant Messaging Module
Slogging Module
Camera Module
Image Management Module
Video Player Module
Music Player Module
Browsing Module
Calendar Module
Widget Modules
Weather Widget
Stocks Widget
Calculator Widget
Alarm Clock Widget
Dictionary Widget

.
.
User-Created Widget{s)
Widget Creator Module
Search Module


.
..r
;_
...r
-;
...r
;_
...r
...r
-;
-;
-;
...r
-;
-;
-;
...r
;_
126
128
130
132
134
136
137
138
139
140
141
142
143
144
145
146
147
148
149
149-1
149-2
149-3
149-4
149-5
149-6
150
151
Figure 17
us 7,469,381 82
1
I.IST S<:ROIJ.I:\G A:\D DO<:UVIEYr
TH.. \:\SL \TIO:\, SC \LI:\G, A:\D ROTATIO:\
0:\.\ TOU('II-._\.iCREE:\ DISPLAY
2
to access. store and manipulate data. t-hese conventional user
interfaces often result in complicm<..'<i key S<..'<.jucnccs nnd
menu hierarchies that must be mcmoritcd by the user.
Many conventional user interfaces. such ns those that
RI:J Ari:J)API'!.ICArJONS
"111is npplication claims priority to U.S. Provisional l'mcnt
Application Nos. 601937.993. "l'ortnb!c Mu!tilimction
5
include physicnl pushbuttons. arc nlso inHcxible. This is
unl(H1unatc because it may prevent a user intcrli1ce from
being conligured and/or adapted by either an application nm-
ning on the portable de\ ice or by users. When coupled with
I kvicc." Jilcd Jun. 29. 2007: 601940.97 I. "!.is! Scrolling and
Document Translation. Scaling. and Rotation on a Touch-
Screen Display." Jilcd Jun. 2R. 2(XJ7: (i()/()45.R5R. "List
Scrolling and Document Trnnslmion on a Touch-Screen Dis-
piny." !i!<..'d Jun. 22. 2007: 60/879.469. "Portable Mu!tili.mc-
tion Device." Jilcd Jan. R. 2007: 60/RiG.ROL "I is! Scrolling
and I )ocumcnl rranslation on a 'llmch-Scn.:cn I )isplay." !ik:d l'i
Jun. 7. 2007: and 601879.253. "Portable Mu!tilimction
Device." H!<..'d Jun. 7. 2007 .. \ll of these npplications nrc
incorporntcd by relCrcncc herein in their entirety
1
, the time consuming requirement to memorize multiple key
sequences and menu hicrnrchics. and the dilliculty in nctivm-
ing n dcsir<.'d pushbutton. such inHcxibility is frustrming to
most users.
As a result of the small size of display screens on pm1able
electronic devices and the potentially large size of electronic
Jiles. Jicqucntly only a portion of a list or of an electronic
document of interest to a user cnn be displnycd on the screen
m n given time. Users thus will frequently nc<.'d to scroll
displayed lists or to translate displayed electronic documents.
Users also will need to rotate and to scale (i.e .. magnify or
de-magnify) displny<.'d electronic documents. However. the
limitmions of conventional user interfaces can cause these
nctions to be nwkward to perform.
Furthermore. scrolling display <.'<I lists and translating dec-
"-' Ironic documents can he awkward on both portable and non-
portahledectnmic devices with touch-screen displays. A user
may become lh1stratcd if the scrolling or translntion docs not
rc!kct the user's intent. Similnrly. n user mny become lhts-
'!l1is application is related to the ll1llowing applications: ( l)
U.S. patent application Ser. No. I 01 I RR, !R2, "'!lmch Pad 1:or ""
I lnndhdd Device." Hied on Jul. I. 2(Xl2: (2) U.S. pmcnt nppli-
cntion Scr. No. 10/722.948. "'](Juch Pad For llandhdd
Device." Jiled on Nov. 25. 2<Xn: patent application
Scr. No. I "Movable '!lmch Pad With Added l:unc-
tionality," Jiled on Aug. JR. 2<Xn: (4) U.S. patent application
Scr. No. 101654.1 OR. "Ambidextrous Mouse." HI <.'<I on Scp. 2.
2003: (5) U.S. pmcnt applicmion Scr. No. 101840.862. "Mul-
tipoint -lhuchscn.:cn." Jilcd on May (i_ 2004: (0) U.S. patent
application Scr. No. l 01903.964, "(jcsturcs For '!lmch Sensi-
tive Input Devices." tiled on Jul. 30. 2004: (7) U.S. pmcnt 30
applicmion Scr. No. 11/038.590. "Mode-Bused (jraphicnl
User Interfaces !-'or Touch Sensitive Input Devices" tiled on
Jan. JR. 2005: (R) U.S. patent application Ser. No. l 11057.
050, "Display Actuator." lilcd on Feb. I l. 2(Xl5: (9) U.S.
Provisionnl Pmcnt .\pplicntion No. (i()/058.777. "Multi- 35
!-'unctionnlllnnd-lldd Device." Hied Mnr. 4. 2(X)5: (10) U.S.
pntcnt npplication Scr. No. 11/307.749. "Multi-1-'unctionnl
I land-! kid lk\icc," liled Mar. 3, 2006: and (I I) U.S. Provi-
sionaiPatent Application No. 60/R24.769. "Pm1ahle Multi-
function Device." tiled Scp. (i_ 2(Xl6 .. \II of these npplicntions 40
arc incorporated by reference herein in their entirety.
T!:C!!NICAI FI!:J D
tratcd if rotation and scaling ofdcctronic documents does not
rdlcct the user's intent.
Accordingly. there is a need ll1r de\ ices with touch-screen
displays with more trnnspnrcnt and intuitive user interJUccs
Jix scrolling lists of items nnd Jix trnnslming. rotming. nnd
scaling electronic documents that arc easy to usc. conHgurc.
and/or adapt.
SUMM.\RY
The nbove ddicicncics and other problems associm<..'<i with
user interbces ll1r pm1able de\ ices and devices with touch-
sensiti\e displays are reduced or eliminated by the disclosed
device. In some embodiments. the de\ ice has a touch-sensi-
tive displny (also known as a "touch screen") with n grnphical
'111c disclosed embodiments rdmc gcnernlly to devices
with touch-screen displnys. nnd more particulnrly to scrolling
lists and to translating. rotating, and scaling dectnmic docu-
ments on de\ ices with touch-screen displays.
45
user intcrlUcc (GUI). one or more processors. memory nnd
one or more modules. programs or sets of instmctions stored
in the memory ll1r perllmning multiple Ji.mctions. In some
embodiments. the user interacts with the (jUJ primarily
through Enger contacts nnd gestures on the touch-sensitive
B.\CKGROUND
,
0
display. In some embodiments. the li.mctions mny include
telephoning, video conll:rcncing. e-mailing, instant messag-
ing. blogging, digital photographing. digital \ idcoing. web
browsing. digiwl music plnying. nnd/ordigital video plnying.
Instructions Jix pcrfonning these li.mctions mny be included
As portable electronic devices become more compact. and
the number of Ji.mctions pcrl(mm:d by a gi\en de\ice
incrcnscs. it has become a signi!icnnt challenge to dcsig.n a
user interface thnt nllows users to easily internet with a mul-
tili.mctionde\ icc. n1ischallenge is particularly signilicantll1r
handheld portable de\ ices, which have much smaller screens
than desktop or laptop computers. This situation is unlllrlu-
nntc bccnusc the user interface is the gnt<..'way through which
users receive not only content bm also responses to user 60
actions or behaviors, including user attempts to access a
d<..'\ ice's Jl:aturcs. tools, and Jimctions. Some portable com-
municmion devices (e.g .. mobile telephones. sometimes
culled mobile phones. cell phones. cellular telephones. and
the like) have resorted to adding more pushbuttons. incrcns- 65
ing the density of push buttons. overloading the Ji.mctions of
pushbuttons, or using complex menu systems to allow a user
,
5
in n compmcr progrmn product conHgur<..'<i Jix execution by
one or more processors.
In accordance with some embodiments, a computer-imple-
mented method ll1r use in conjunction with a device with a
touch screen displny is disclosed. In the method. a movement
ofnn object on or ncar the touch scr<..-cn displny is del<..><: ted. In
response to detecting the movement. an ek>ctronic document
displayed on the touch screen display is translated in a Jirst
dir<..><:tion. If nn <..'<igc of the ck>ctronic document is reached
while trnnslming the electronic document in the Hrst direction
while the object is still detected on or ncar the touch screen
display. an area beyond the <..-dgeofthedocument is displayed.
Aller the object is no longer dct<..>ctcd on or near the touch
us 7,469,381 82
3
screen display. the document is translated in a second din.x:-
tion until the area beyond the edge of the document is no
longer displayed.
4
area hL")'"tmd the edge of the document is no longer displayed.
alier the object is no longer de!ec!L-d on or near the 1ouch
scrL-cn display.
In accordance wi1h some embodiments. a compmer-imple- In accordance with some embodiments. a graphical user
inter bee on a device with a touch screen display is disclosed.
comprising a portion or an dcclnmic document displayed on
the touch sen.-en display and an area beyond an <..-dgc of the
document. In the grnphical user interface. in response to
detecting a mmcmcnl of an object on or ncar the touch screen
display. the dcclnmic document is translated in a !irs! din.x:-
tion. If the edge of the dcclronic document is reached while
translnting the dcctronic document in the Hrst direction while
the object is still detected on or ncar the touch screen display.
the area hcyomlthc L'tlgc oft he document is displayL'tl. Aller
the ohjccl is no longer dc!cc!L'tl on or ncar the !ouch screen
display. the document is 1ranslated in a second dirL>clion until
1he area beyond 1he edge of 1he documem is no long.er dis-
playL'tl.
5 men!L'tl me!hod ll1r use in conjunclion with a device with a
!ouch screen display is disclosed. In the mc!lmd. a mmemcnl
of an object on or near 1he touch scrL-cn display is de!L>c!ed. In
response 10 de1ec1ing. 1he movemen1. a lis! of items displayed
on the !ouch screen display is scrolled in a !irs! dirL>clion. If a
1" lcJminus of the list is reached while scrolling the list in the
!irs! dircclion while the ohjccl is still dc!cc!ed on or ncar the
touch screen display. an area beyond 1he 1erminus oft he list is
displayed . ..\lier the objec1 is no longer deJected on or near the
!ouch screen display. the list is scrolled in a second dirL>clion
15 opposite the !irs! dircclion until the area beyond the terminus
of the lis! is no long.er displayed.
In accordance wi1h some embodimems. a g.raphicaluser
imerface on a device with a touch screen display is disclosL'tl.
comprising a pm1ion or a list or items displayed on the !ouch
"" screen display and an area hL")'tmd a terminus or the list. In
response 10 de!ec1ing a movemem of an object on or near the
touch scrL-cn display. 1he lis! is scrolled in a lirs1 dirL>clion. If
the terminus oft he list is reachL'tl while scrolling the list in the
!irs! dircclion while the ohjccl is still dc!cc!ed on or ncar the
In accordance with some embodiments. a device is dis-
closL'tl. comprising a !ouch screen display. one or more pro-
cessors. memory. and one or more programs. The one or more
programs are slorL'tl in the memory and config.ured 10 be
exccu!L'tl by the one or more processors. The one or more
programs include inslruclions !l1r de!cclinga movement of an
ohjccl on or ncar the !ouch screen display and inslmclions !l1r
1ranslming. an elec1ronic document displayed on the touch
screen display in a !irs! direction. in response to de!ecling the
movement. The one or more programs also include instruc-
tions !l1r displaying an area hLJ'tmd an L'tlgc of the ck>clnmic
document if 1he edge of the electronic document is reached 30
while mmslaling. 1he ek>clronic document in the !irs! direc1ion
while !he object is s1ill de!ec!ed on or near the touch screen
display. The one or more programs further include instruc-
tions !l1r translating the document in a SL>cond dircclion until
1he area beyond 1he edge of 1he documem is no long.er dis- 35
playL'<i. aller 1he objL'CI is no longer deJected on or near the
1ouch screen display
"5 !ouch screen display. the area hL")'tmd the terminus of the list
is displayed .. \lkr 1he object is no long.er de1ec1ed on or near
the touch screen display. 1he list is scrolled in a SL>cond direc-
tion opposite the !irs! dirL>clion until the area hL")'tmd the
lcJminus of the list is no longer displayed.
In accordance with some embodiments. a device is dis-
closed. comprising a touch screen display. one or more pro-
cessors. memO!)'. and one or more programs. The one or more
programs arc slorL'tl in the memory and configured lo he
executed by the one or more processors. n1e one or more
prog.rams include instructions for deJecting. a movement of an
objL'CI on or near the !ouch screen display and ins1n1c1ions for
scrolling. a lis! of ilems displayed on the 10uch screen display
in a !irs! d i n ~ c l i o n in response lo de!L>cling the mmemenl. rhe
one or more programs also include inslruclions !(Jrdisplaying
an area beyond a terminus oflhe list if1he terminus oflhe list
is reached while scrolling the list in !he !irs! direc1ion while
the ohjccl is still de!cc!L'tl on or ncar the !ouch screen display.
The one or more programs !i.1rlher include inslmclions ll1r
scrolling the list in a second dirL>clion opposite the !irs! dircc-
In accordance with some embodiments. a computer-pro-
gram produc! is disclosL'tl. comprising a computer readable
slorag.e medium and a compmer program mechanism embL'<i- 40
dL'<i !herein. The compmer program mechanism comprises
inslruclions. which when exL>cuted by a device with a !ouch
screen display. cause the device lo deice! a movement or an
ohjccl on or ncar the !ouch screen display and lo translate an
elec1ronic documem displayed on 1he !ouch screen display in 45 lion until1he area beyond 1he terminus of1he list is no long.er
displayed. alier the objec1 is no longerde!ec1L'tl on or near the
!ouch screen display.
a !irs! direction. in response to de!ec1ing !he movemenl. The
inslruclions also cause the device lo display an area beyond an
edge of the clcc!nmic document if the L'tlgeofthc ck>clnmic
document is reached while lranslaling. !he electronic docu-
ment in the !irs! direction while the object is s1ill de!ec!ed on
In accordance with some embodiments. a computer-pm-
g.ram product is disclosed. comprising. a computer readable
or near 1he 1ouch scrL-cn display. The instmc1ions li.mher
cause the device lo translate the document in a second dirL>c-
tion until the area beyond the edge or the document is no
longer displayed. alkr the object is no long.er deJected on or
near 1he touch screen display.
'0 s1orage nK'<iium and a computer program mechanism embed-
ded therein. The computer program mechanism comprises
inslmclions. which when exL>cuted by a de\ icc with a !ouch
screen display. cause the dL"\ice lo dc!ccl a mmemenl oLm
objL'CI on or near1he 10uch screen display and 10 scroll a list of
'5 items displayL'tl on the touch scrL-cn display in a !irs! direction
in response lo de!ccling the mmcmenl. The inslmclions also
cause the device lo display an area beyond a terminus or the
list if the terminus oft he list is reachL'tl while scrolling the list
In accordance with some embodiments. a de\ icc with a
!ouch screen display is disclosed. The device comprises
means ll1r de!ccling a movement or an ohjccl on or ncar the
1ouch scrL-cn display and means Jix 1ranslming. an electronic
document displayL'tl on the touch screen display in a lirs1 60
dirL>clion. in response lo de!ccling the movement. rhc de\ ice
also comprises means ll1r displaying an area beyond an L'tlge
of! he electronic document if1he edge oflhe ek>clronic docu-
ment is reaciK'<i while translating 1he electronic documem in
!he !irs! direction while the object is s1ill de!L>c!ed on or near 65
the !ouch screen display. n1e de\ ice !i.1rlher comprises means
ll1r translating the document in a second dirL>clion until the
in !he !irs! direction while the object is s1ill de!ec!L'tl on or near
the touch screen display The ins1mc1ions li.mher cause the
device lo scroll the list in a second dircclion opposite the !irs!
dirL>clion until the area beyond the tenninus of the list is no
long.er displayed. aller 1he objL'CI is no longer de!ec1L'tl on or
near the 1ouch screen display
In accordance wi1h some embodimems. a device wi1h a
!ouch screen display is disclosed. The dL"\ ice comprises
means !l1r dc!ccling a mmemenl of an ohjccl on or ncar the
us 7,469,381 82
5
!ouch screen display and means !(Jr scrolling a list of items
displayed on the touch screen display in a !irst direction in
response to detecting the movement. "111c device also com-
prises means for displaying an area beyond a tcnninus of the
list if the terminus oft he list is reached while scrolling the list
in the !irs! dircclion while the object is still dc!cclcd on or ncar
the touch screen display. n u ~ de\ icc Ji.Jrthcr comprises means
for scrolling the list in a second direction opposite the !irst
direction until the urea beyond the terminus of the list is no
longer displayed, alkr the object is no longer dc!cclcd on or
ncar the touch screen display.
In acconlancc with some embodiments. a cmnpulcr-imple-
mented me! hod for usc at a device with n 10uch screen displny
includes detecting. n multi!ingcr twisting. g.csturc on or ncnr
the !ouch scn.x::n display. rhc nmhilinger lwistinggeslurc has
a corresponding degree or rotation. If the coJTcsponding
degree or rotation exceeds a preddined degn.::e or rotation. a
90 screen rotmion conunnnd is executed. If the correspond-
ing. dcg.rec of rotmion is less thnn the predefined degree of
rotation. a screen rotation command with an aculc angle or
rotation is executed and. upon ceasing lo dc!ec!lhe muhilin-
gcr lwistinggcslure. a screen rotation command with an angle
of rowtion opposite to the ncme nngle is executed.
In nccordnnccwith some embodimcms. n device includes n
touch screen disp!ny. one or more processors. memO!)'. nnd
one or more programs. rhc one or more programs are stored
in the memory and nmligured lo he executed by the one or
more processors. The one or more prog.rmns include: instruc-
tions for detecting. n multi!ing.er twisting gcsmrc on or ncnr
the !ouch scn.::en display. wherein the muhilinger twisting
gesture has a corresponding degree or mlalion: inslmclions
ll1r executing a 90" screen rotation cmnmand. if the corre-
sponding. dcgr<:e of rotmion exceeds n predefined dcg.rec of
rotmion: nnd instructions for execmingn scr<.-en rowtioncom-
mand with an acute angle or mlalion and ll1r executing. upon
ceasing lo detccl the muhilinger twisting gesture. a screen
rotmion commnnd with nn nng.le of rotmion opposite to the
ncme nngle. if the corresponding degree of rowtion is less
thnn the predefined dcg.rec of rowtion.
In accordance with some embodiments. a computer-pro-
gram produc! includes a computer readable storage medium
and a computer program mechanism embedded therein. The
compmer prog.rmn mcchnnism includes instructions. which
when cx<:cutcd by n device with n touch screen displny. cnuse
the dL"\ ice to: deled a muhilinger twisting gesture on or ncar
the !ouch screen display. wherein the muhilinger twisting
gesture has a corresponding degree of rotation: execute a 90"
screen rotmion commnnd. if the corresponding degree of
rotmion cxceL'<is n pr<.-deHned degree of rotmion: nnd execute
a screen rotation command with an acute angle or mlalion
and. upon ceasing lo detcc!lhe muhilingcr twisting gesture.
exec men screen rotmion conunnnd with nn nngle of rotmion
opposite to the acme nng.le. if the corresponding degree of
rotmion is less thnn the predefined degree of rowtion.
In accordance with some embodiments. a de\ icc with a
!ouch screen display includes: means ll1r dc!ecling a multi-
linger twisting gesture on or near the !ouch screen display.
wherein the multiHng.er twisting. gesture has a corresponding
dcg.rec of rotmion: means Jix executing. a 90 screen rotmion
command. if the coJTcsponding degree of rotation exceL-ds a
prcdclinL-d degree of rotation: and means ll1r executing a
screen rowtion conunand with nn acme angle of rowtion nnd.
upon censing. to detect the multi!inger twisting g.esture. Jix
execming n screen rotmion conunand with nn nnglc of rotn-
tion opposilc lo the acute angle. if the corresponding degree
of rotation is less !han !he prL-deJinL-d degree of rotation.
6
In accordance with some embodiments. a compulcr-imple-
mented method of displaying nn electronic document hnving
n document length and n document width. for usc m n device
withn touch scr<.-en displny. includes displnying. the ekctronic
5 document a! a lirsl magnilicalionanddetcclinga gesture on or
near the !ouch screen display corresponding loa command lo
zoom om by a user-speci!k-d amoum. In response to detecting
the g.esturc. the electronic documcm is displnyd nt a mag.ni-
Jication less than the lirsl magnilicalion. If the document
lCJ length or document width is entirely displayL-d while the
gesture is still dc!ec!L-d on or near the !ouch screen display. the
electronic document is displayed m a mng.niHcntion wherein
nrens beyond opposite edges of the ekctronic document nrc
displayed. and upon detccling termination of the gesture. the
15 elec!nmic document is displayed a! a magnilicalion wherein
the nrens beyond opposite edges of the ekctronic document
nrc no long.cr displayed.
In accordance with some cmbodimems. a g.raphicnl user
in!crliJCc on a de\ ice with a !ouch screen display includes an
"" elec!nmic document having a document length and a docu-
ment width. to be displnyed on the touch screen display at
multiple mag.niHcations including. n Hrst magni!icmion. nnd
areas bLytmd opposite edges of the elec!ronic document. In
response lo dc!ecling a gesture on or ncar the !ouch screen
"5 display coJTcsponding lo a command lo zoom out by a user-
speci!id amount. wherein the gcsmrc is dek'cted while dis-
playing the electronic documem nt the Hrst mng.niHcntion. the
elec!nmic document is displayed a! a magnilicalion less than
the lirsl magnilicalion. If the document length or document
30 width is cmirdy displnyd while the gcsmre is still detected
on or nenr the touch screen displny. the electronic documem is
displayed nt a magni!icmion wherein the nrens beyond oppo-
silc L-dges of the elec!ronic document are displayL-d. and upon
de!ccling lcnninalion of the gesture. the elec!ronic document
35 is displnyed m a mag.ni!icmion wherein the nreas beyond
opposite L'<igcs of the electronic document arc no longer dis-
played.
In accordance with some embodiments. a device includes a
!ouch screen display. one or more processors. memory. and
40 one or more prog.rams. The one or more prog.rams nrc stored
in the memory nnd con!ig.ur<.-d to be executed by the one or
more processors. rhc one or more programs include: inslmc-
tions ll1r displaying an elec!ronic document a! a Jirsl magni-
Jicalion: inslmclions ll1r de!ccling a gesture on or ncar the
45 touch screen displny corresponding to a conunand to zoom
om by a user-speciHednmount: instmctions for displaying. the
elec!nmic document a! a magnilicalion less than the Jirsl
magnilicalion. in response lo dc!ecling the gesture: inslmc-
tions for displaying. the ekctronic document m a mag.niHca-
'0 tion wherein areas beyond opposite cdg.cs of the ekctronic
documcm nrc displayed. if a documcmlcng.th or a document
width is entirely displayL-d while the gesture is still dc!ec!L-d
on or near the !ouch screen display: and inslruclions ll1r
displaying the electronic document m n mag.niHcation
'-' wherein the areas beyond opposite L'<igcs of the ekctronic
document arc no longer displayed. upon detccling lcrmina-
tion or the gesture.
In accordance with some embodiments. a compulcr-pm-
g.ram product includes a compmer rcadnble stornge medium
60 nnd a compmer program mechanism embL'<idL'<i therein. The
computer program mechanism includes inslmclions. which
when cxecu!L-d by a de\ ice with a !ouch screen display. cause
the device to: displny nn ekctronic documcm nt a !irst mng-
ni!icmion: detect n g.csture on or nenr the touch screen display
65 corresponding to a commnnd to zoom om by a uscr-speciHed
amount: display the elec!ronic document a! a magnilicalion
less than the lirsl magnilicalion. in response lo dc!ecling the
us 7,469,381 82
7
gesture: display the dcc!nmic document a! a magni!icalion
wherein nrcas beyond opposite <..-dgcs of the electronic docu-
ment nrc displayed. if a document length or a document width
is entirely displayed while the gesture is still detected on or
ncar the !ouch screen display: and display the dcc!nmic -'
document a! a magni!icalion wherein the areas beyond oppo-
site edges of the dcc!ronic document arc no longer displayed.
upon detecting. termination of the gesture.
8
In accordance with some embodiments, a compulcr-pro-
g.rmn product includes a compmcr rcadnblc storng.c medium
nnd a compmcr program mechanism cmb<.-dd<.-d therein. The
computer prog.rnm mechanism includes instmctions. which
when cxL><::U!L-d by a de\ icc with a !ouch screen display. cause
the de\ icc to: display a! kasl a Jirsl portion of an d<.>clronic
documcm ill n !irs! mag.ni!icillion: detect n gesture on or ncar
the touch screen displny corresponding. to a conunand to
mom in by a uscr-sp<.><::ilicd amount: display decreasing por-
In accordance with some embodiments. a device with a
touch screen display includes: means for displaying an ck>c-
tronic document at a !irs! magnilication: means !(Jr detccling
a gesture on or ncar the !ouch screen display cmn::spomling to
a command to zoom om by a uscr-speciHed amount: means
lCJ lions ofthcclcc!nmic document a! increasing magnilicalions,
in response lo dc!ccling the gesture: and display a rcspccli\c
portion of the ck>ctronic documcm ill n prcdc!incd mng.ni!i-
cillion i[ upon detecting. tcrminntion of the g.csturc. the mng.-
nilication cxccL-ds a prL-ddincd magnilicalion.
In accordance with some embodiments. a device with a
!ouch screen display includes means l(Jr displaying a! kasl a
Hrst portion of an electronic documcm ill n Hrst mag.niHcation:
mcnns Ji.1r detecting. n g.csmrc on or ncar the touch screen
display coJTcsponding lo a command lo /oom in by a uscr-
spccili<.-d amount: means !(Jr displaying decreasing pm1ions
of the ck>clronic document a! increasing magnilicalions, in
response to detecting. the g.csturc: and mcnns for displnying. n
respective portion ofthc electronic document ill a prcdc!incd
magnilicalion iL upon dc!ccling termination of the gesture,
for displaying. the electronic document ill a mag.niHcation less
than the !irs! magnilicalion. in response to dc!ccling the gcs-
15
lure: means !(Jr displaying the clcc!ronic dm:umcnl a! a mag-
nilicalion wherein areas h:yond opposite edges of the dec-
Ironic documcm nrc displnycd. if a document length or a
document width is entirely displayed while the g.csturc is still
dc!cc!cd on or ncar the !ouch screen display: and means !(Jr ""
displaying the clcc!ronic document a! a magnilicalion
wherein the areas h:yond opposite L-dgcs of the clcc!nmic
document nrc no long.cr displnycd. upon dcl<.'cting. tcnninn-
tion of the g.csturc.
"-' the magnilicalion exceeds the prL-dclinL-d magnilicalion.
In nccordnnccwith some cmbodimcms. n computcr-implc-
mcntcd mel hod of displaying an ckc!ronic document, l(Jr usc
a! a device with a !ouch screen display. includes displaying a!
lens! n Hrst portion of the electronic document ill n Hrs1 mng.-
niHcntion and detecting. a g.csturc on or ncar the touch screen
30
display corresponding lo a command lo /oom in by a uscr-
spcciliL-d amount. In response lo dc!ccling the gesture.
decreasing portions of the clcc!nmic document arc displayed
at incrcnsing. mng.ni!icillions. Upon detecting. tcnninillion of
the gesture. ifthc mag.ni!ication cxcc<.-ds a prcdelincd mng.ni-
35
Jicalion, a rcspccli\c portion of the clcc!ronic document is
displayL-d a! the prcdclincd magnilicalion.
"
In accordance with some embodiments, a graphical user
intcrJUcc on n device with a touch screen displny includes
d<.>crcnsing. portions of an electronic document. to be dis-
play<.-d on the !ouch screen display a! increasing magnilica-
tions. The decreasing pm1ions of the clcc!ronic document
include a Jirsl portion. In response lo dcl<.><::ling a gesture on or
ncnr the touch scr<.-cn displny corresponding. ton conunand to
zoom in by a user-specified nmount. wherein the g.csmrc is
45
delL><:: led while displaying a! leas! the Jirsl portion of an d<.>c-
lronic document a! a Jirsl magnilicalion, the dL>crcasing por-
tions of the clcc!ronic document arc displayed all he increas-
ing. mag.niHcations. Upon detecting. termination of the
g.csmrc. if the mag.niHcation cxcc<.-ds a prcdelincd mng.ni!icn- ,
0
lion, a rcspcclivc portion of the clcc!ronic document is dis-
play<.-d a! the prL-ddincd magnilicalion.
'1-hc disclosed embodiments provide l(Jr easy and intuili\c
scrolling. of lists and trnnslilling. of electronic documents on n
device with n touch scr<.-cn display. and Ji.1r cnsy and intuitive
rotation and scaling ofck>clnmic documents on a device with
a !ouch screen display.
BRII:t.-1)1.-SCRIPTION OF Till: DR.-\ WINGS
I-' or a better undcrstnnding. of the nforcmcntion<.-d embodi-
ments of the imcnlion as well as additional embodiments
thcrcoL rdCrcncc should he made lo the Description of
J:mbodimcms below. in conjunction with the following. draw-
ings in which like rcJCrcncc numerals rcJCr to corresponding.
paris throughout the ligures.
l'l(j. 1 is a block diagram illuslraling a portahk multifunc-
tion device with a !ouch-sensitive display in accordance with
some cmbodimcms.
]-'](j. 2 illustrillcs a portable multi!i.mction device hnving. n
touch screen in nccordnncc with some embodiments.
I-'IG. J illustrates an cxcmpln!)' user interface for unlocking.
a portahk clcc!ronic de\ icc in accordance with some cmhodi-
mcnls.
l'l(j. 4 illuslralcs an exemplary user inlcrliJCc l(Jr a menu of
npplications onn portable multi!i.mction device in nccordnncc
with some cmbodimcms.
I-'IG. 5 is n !low diagram illustrilling. n method of scrolling.
through a list in accordance with some embodiments.
In accordance with some embodiments. a de\ icc includes a
touch screen display. one or more processors. memO!)'. and
one or more prog.rnms. The one or more prog.rnms arc stored
I'J(jS, 6.-\-6]) illuslralc an exemplary user intcrliJCc l(Jr
'
5
manng.ing. an inbox in accord.1ncc with some cmbodimcms.
in the memory and conligurL-d lo he cx<.><::u!cd by the one or
more processors. '1-hc one or more programs include: instruc-
tions l(Jr displaying a! kasl a Jirsl pm1ion of an ck>clnmic
document ill a !irs! mng.niHcntion: instmctions for dck>cting. a
g.csmrc on or ncnr the touch screen display corresponding. to 60
a command lo /mlm in by a uscr-sp<.><::ilicd amount: instruc-
tions l(Jr displaying decreasing pm1ions of the ck>clnmic
document nt increasing. mng.niHcntions. in response to detect-
ing. the gesture: and instmctions for displaying. a respective
portion of the electronic document at n prcddincd mng.niH- 65
calion iL upon dcl<.><::ling termination of the gesture, the mag-
nilicalion cxcL"L-ds the prL-ddinL-d magnilicalion.
I-'IG. 7 is a How diag.rmn illustrilling.a method of translating.
nn electronic documcm in accord.1ncc with some cmbodi-
mcnls.
I'J(jS, 8.-\-8]) illuslralc an exemplary user interface l(Jr a
browser in accordance with some embodiments.
l'l(j. 9 is a !low diagram illuslraling a process ofdisplaying
nn electronic documcm ill multiple mng.niHcillions in nccor-
dancc with some embodiments.
1-'lGS. 10.-\-IOC illustrillc the displny ofnn electronic docu-
ment a! multiple magnilicalions in accordance with some
embodiments.
zn .!:llJO.!jUO:l ,(JOlU:llU :ll[J.{q (h'"l[(lUJUO:l :llJ
'8I I :Jt(J ptm on fld.J :Jt(J ooi
JO ,\q ZOI .\Jouwm 01
.Uouwm .!0
'jS!p Jo
tp11s .Uom:Jm :l(IW]O.\-uou :Jp11pm .\1'm ptm .\.mm;,m
moptmJ :Jp11pm .{I'm zo I .\.rom:Jj!IJ
Jl':l]K{qd uommo:J :Juo w :J:JI \:JP
:ll11 UO (h'"IJI1:l:JX:l :ll( S110J.m.\ :lllJ.
"UO]leJ
-!]dde ]mtf!]p e Jo;pue uo]WJHdde J]snm
jeJ]f!!p r, "UO]leJ!!dde e "Ull]WJ!!ddr,
O:lpj.\ I' 'UO]JlDI(lflfl' l!J:llUlD 'UllljlDI(lfdl' flmfl
-1J01l( 'UO]JI':l][dlfl,flU]fli"S:llU tm 'UllljlDI(lfdl' [mtu-:J
JOjptm
]P-tril]S .!0 flmpn]JU] pue
-p.!et] t]llll(JO uonemqmo::> e Jo m
-;,1dmt ;,q I "!)1:1 m :ll!J.
:ll11 JO JO UOljl!lllfl!JUO:l jU:J.!:J_JJ]p
e ,\em e Jo Jo O.\\l .(em
tmt]J Jo .(em 00 I
Wl]l pur, '00 I UO!jJU!1J]l]ntU r, Jll
:JUO .{[UO S] 00 I :l:l].\:lp :ll(J ;,q p[110l1S 1!
09 ue uo]WJHdde e uone::>Hdde
r, se Jll r,
"'jJ!Js.\of
:JSI10lU I' 'p.!tmq.\:J)\ (l':l]K{l(d I' '1:l..'"ll(.l )j:ll(:l 1lDIS.{t1d
(lDIS,\t1d J:ll(JO :l.!OlU .!0 :JUO
.(em JP-t]J doJde] pue
se OJ .(em
ptm S:l:JIOJ.!:lJU! J:JSI1 :ll(J .JO 'J:l.\:l.\'\.Ol(
;,q 11 'JU:lm]poqm;, tm s1' p:Jsl1 U:J..'"IJ:Js
"0I S:JU]1
.!0 UllljlD!UI1UllUO:l JO :JUO J:J\0 :lJI':l]UI1UllUO:l O>
"t-91 jeJ]JdO .!0
.\em 00 I rz1 pod ue pue 91 I
(lUJUlD JO Jl1lfU] J:ll(JO '90I (0/1) Jl1lfJl1llfJ11tfut
'II :JUOl1dO.!:l]Ul I' 'III '()II .(Jjlll:JJD mpm' .
'80I ,\.!JjlD.!]:l '8II S(I,J:lt1d].!:ld on ;t
Jo zn e
Jo .(em
tptt1\\) ZOI .Umu;,m 1' :Jp11pm OOI :J:J].\:JP :ll!J. ':J:JU:J]U
-:J\Ull:l .!O.J .,U:l..'U:JS l(:JI10J, I' p:l[Jl':l S:llU]PUlOS ZI I
-S]p t]l!.\\ Ot
l(Jllllll' S:lp11(JU] 11"11 :l:JL\:lp
Wl]l U] .!0:] OJ f>mpuods
puemmo::> e .(em tpnoJ
uo (:l:ll1\\ l(JI\\ PI,JUO:l
'(:l:ll1\\ :ll(J.JO J:JjU:l:l :ll(J .!0 1:l.'"ll(.l :ll(J.JO UO]JJOd I' UO
U.\\Op tWtj.\\ .!OJ
Jo JO 01 puemmo::> e
OJ OSje ,(r,m )-(J!!J \' r,
.\q ]J:ll(.\-\.:ll(Jl(J].1. PI'JUO:l.JO JU!Od I' JO (:l:ll1\\ :ll(J.JO jU:JUl:J:Jqd
tm uo UO]JI'TI].\tm :lp].\OJd .\1'm
Wt]l e S] 'jJ!]J \" tpnoJ
'jJ!]J wnu],\ e Jo tpnoJ eo1 UO!J!ppe
m 'jJ!]J jeJts.\tjd e .(em
U] Zll .(ejdS]p e l]Jl.\\ 001 Ull!PU!1J!j
-jntU r, f.U]W.!JS!l!l! tueJf!etp 'jJlljl( r, 1
:ll(J .JO (h'"OJJ:l.!]p \\llU UO]JU:lJJV
uos :ll(J flms11 U:ll(.1.
m ;,uo Sill(! ptm .10 ;,uo
e JO .\Jmqeqo;d e 01
,\em
.JO Jl'I1PI \!put .{l(lfi,.RllD]X:l[} ,(JOJS]l1 p.m.\'\.
J;o,\qd m;ptm
\'( ld J:ll(JO ll'l(l :llllll1d:l(:lJ
r, l]JHS SUO]WJ]U!ltutuOJ r,
U] l]JHS
.H:I 'S:l:l] \:lp .!O.J
J:JSI1 ':J:JI\:lp :l1l(I')JOd I' .JO
"SjU:ltujpOlJUl:l :ll(JJO
-:J.'"IUU!1 OJJOU OS U] JOU
I' UO .!0:] ".!:JSI1 :l\IJJ:JdS:J.! OJ (h'UO[mJ :llJ
Jo u1
W OJ (h'"l.!ll]!e! ,(r,m jtWUl
-]poquw sntLL nwJqJ
J!P m;ptm uommm :l/1]]111 :J:J].\:JP :l(l]l'J-Illd :ll(J uo
-tDt[ddl' ;,mm .10 :JU() s[oqm.\s Tiu]puods:J.utn :JJom Jo ;,uo
Jo;ptm Jo se suo]JJe t]l!.\\
U] ,\em SUOJ!
.!0:1 ;,q .\1'm :Jt].L
-m{s Timpuods:JJJtD :JJom Jo ;,uo ptm 'p.!tmq.\;-,1 :ll11 m
oc pur, snwuodmo::>
J:ll1JO Ul "S(mpp :JS:ll1Jll1lll1J].l :llJ
,\I'm UO]JU:l\U] jU:JS:JJd :ll(J JJI' :ll11 U! 1(1)-(S ,(JI"l]p.!O .JO :JUO
]]!.\\ 1! f.U]
t]f!HO.!llt]l r, OJ lj).!OJ
,:c f!u].\\O!IOJ u1 sf.u]
-.w'IP :Jl(J m tptl(.L.JO s;,[dumx;,
'SJU:llmpoqm:J OJ 1JI'PP m :JPI'm :ll( \\llU ][1\\
:l.!OUl JO :JUO JJ:l]JS OJ S.!:JSI1 .!O.J J] ,\I'm flC
e .!OJ Jet]l tpns m
JO OJ \IllS Jo) suo::>] JO
;op11pu] .{I'm p.!tmq.\;-,1 :ll!J. .{pJIJU:J
.!]:ll11 U] ut:J.!:ll( :J:JU:J.!:J_p.! .\q .\qar:Jt1 am tp]t1"'" JO
:ll(J '9001: 'tZ: '111f ,;s:J:JI \:l(] 1 :l(lJI'JJOd ,
1
.!ll:J tpno.L, II 'ON pue "9(XlZ -
rz 1nr JO:J
'')()9'6S"J7/]] 'ON J:JS "S"fl U] p:llJ].!:lS:lp
:Jsm11 'p.!tmq.\;-,1 :Jl(JJO :ll(J uo
JO p.!epueJs-uou Jo;ptm (AUJ:!/1:\<))
01
,\em
\IllS Jo .(em
"JU:J.mdsut!J] ptm :l\]J]I1JU! .!:JS11l]J].L
.JO .{put!\ :ll(J ,\I'm :l:JL\:lp :ll(JJO
l(:lllllJ :ll11 tp11s) Jl':l]K{qd uommo:J 1' .\1,.1. S]l11 ,.-
U] 'UO!j!JJ!]dde e U!l]Jl.\\ .!OjpUe OJ UO!Jr,J
-Hdde tuO.!J Jo;pue ,\em uo
uoneuuoJm tpnoJ
:ll(JJO :JJOlU .!O:JU() "U:J..'U:JS lj:ll10j :ll11 lj:li1S ':l:lj.\:lp
Ol
S.I.N!ll"l!C!OHI"l!! :10 NOLLdnUS!lC!
'SjU:llU!pOl(lU:l :llUOS ljl].1. :J:Jtmp.!O:l:ll' U] U:J:J.!:JSlj:lllllj
e t]Jl.\\ e flmWJJsnm um;<le]p 'jJO]q e L1
t]l!.\\ U]
Ull!WJtU ue :191-\'9I
"SjU:ltujpOlJUl:l :JUlOS
l(JI\\ U] j:J.'"Ifl(O J:ll(JO JO jU:JUlll:lOp :l]UOJj
ue Jll <lU!jeJOJ !!S1-\"S1

-tpoqm;, :lUI OS l(JI\\ :J:Jtmp.HD:ll' U! ptmmmtD UO]JI'JlU U:J:J.!:JS
\\Ot) I' S] t"I "!)]:1

l]J].\\ U! JP-
-n::>op ,)fl -\"1

l1J].l U! :l(d]J(I1Ul W jU:llU
-n::>op ,)ZI -\"Z1
t]Jl.\\
m suone::>mtlilem JP- ue <lu]
.JO ss:J:JoJd 1' ttmJi!mp .LOU I I "!)!:1
6
zs !8t'69v'L sn
us 7,469,381 82
II
'!lu:: peripherals interliJce I 18 couples the input and output
peripherals of 1hc device to the CPU 120 and memO!)' 102.
The one or more processors 120 nm or execute various soil-
ware programs and/or sets of instmctions ston.-d in memory
I 02 lo pcrl(mJJ various !lmctions ll1r the device 100 and to 'i
process data.
I2
dials. slider switches. joysticks. click wheels. and so llH1h. In
some ahcmate embodiments. input comrol!cr(s) 160 may be
coupled 10 any (or none) of the following. a keyboard. infra-
red pon. USB port. and a poimcr device such as a mouse. The
one or more bul!ons (e.g .. 208. FJ(j_ 2) may include an
up/down button !l1r volume control of the speaker I I 1 and/or
the microphone 113. The one or more buttons may include a
push bunon (e.g .. 206. 1-'lG. 2) .. \quick press of the push
bun on may disengage a lock of the !ouch screen I 12 or begin
In some embodiments. the peripherals intcr!Ucc 118. the
CPU 120. and the memO!)' controller 122 may be imple-
mented on a single chip. such as a chip 104. In some other
embodiments. they may be impk:mcnted on separate chips.
'!lu:: RF (radio Jiequcncy) circuitry I 08 rccci\cs and sends
lCJ a process that uses gestures on the !ouch screen lo unlock the
device. as described in U.S. paten! application Scr. No.
111322.549. "Unlocking a Device by l'crlixming. Geslllrcs on
an Unlock Image." H!<.'d Dec. 23. 2005. which is hereby
RF signals. also ca!l<..'d clcctromag.nctic signals. The R!-' cir-
cuit!)' 108 converts electrical signals to/from dcctromag.nctic
signals and communicates with communications networks
and other communications de\ ices \ia the dcc!rmnagnetic 15
sig.nals. The Rl' circui1ry 108 may include well-known cir-
cuill)' Jix perfonning. these li.mc1ions. including but no1 lim-
iled 10 an amenna system. an RF transceiver. one or more
amplilicrs. a tuner. one or more oscillators. a digital signa!
processor. a CODH" chipset. a subscriber identity module ""
(SIM) card. memory. and so Jixth. The Rl' circuill)' 108 may
communicate with networks. such as the lmcmct. also
rc!"crrcd lo as the World Wide Web (WWW). an inlrancl
and/or a wireless nclwork. such as a cellular telephone ncl-
work. a wireless local area network (I AN) and/or a metro-
poliwn area network (M.\N). and other devices by wireless
communication. The wireless commutlicmion may usc any of
incorpora!L'd by rc!Crcncc herein in its entirely. A longer press
of the push \muon (e.g .. 206) may !urn power lo the device
100 on or olr The user may be able 10 cus1omize a li.mc1ion-
ali1y of one or more of the but1ons. The 1ouch screen 112 is
used 10 implcmcn1 virtual or soft buttons and one or more soft
k9boards.
The !ouch-sensitive display system 1 I2 prm ides an input
imcrface and an oll1pll1 imerface bc1wecn 1he device and a
user. The display comrollcr 156 receives and/or sends elec-
trical signals !i"omllo the display system 1 I2. The display
system I12 displays \isual oulpullo the user. '1-hc \isual
"5 output may include graphics. !ext. icons.\ ideo. and any com-
bination !hereof (collectively termed "graphics"). In some
embodimcms. some or all of 1he visual omput may corre-
spond lo uscr-inlcrbcc objccls. !i.n1hcr dc!ails of which arc
described below.
a plurality ofcmnmunicalions standards. protocols and lcch-
no!ogics. including but no! limited lo (i!oba! Syslcm !(Jr
Mobile Conununications (GSM).I:nhanc<.'d Dma GSM J:nvi- 30
ronmem (EDGE). wideband code division muhip!c access
(W-CDM.\). code division muhiplc access (CDM.\). lime
di\ ision nmhip!c access Wireless l:idcl-
ity (Wi-Fi) (e.g .. !! :1 ;! : X02.lla. !! ;I :1; X02. l I h.!! :1 ;I: X02. II g.
and/or IEEE 802.lln). voice over Interne! Protocol (\'oil'). 35
Wi-M.-\X. a pr01oco! for email. instant messaging. and/or
Short Messag.c Service (SMS)). or any o1hcr suitable com-
munication protocol. including communication protocols no!
yc! de\ eloped as of the Ji!ing date of this document.
.-\ touch screen in display sys1cm 112 is a 1ouch-scnsi1ivc
sur!Ucc 1ha1 accepts inpmli"om the user based on hap1ic and/or
tactile comac1. The display system 112 and the display con-
troller I 56 (along with any associa!cd modules and/or sets of
inslmclions in memory 102) dc!ccl conlacl (and any mmc-
mcnt or breaking of 1he contact) on the display sys1cm 112
and converts !he deJected contact into intcrac1ion wi1h uscr-
imcrface objL><::Is (e.g. .. one or more soli keys. icons. web
pages or images) that arc displayed on the !ouch screen. In an
exemplary embodiment. a point of conlacl between a !ouch
scrL-cn in the display sys1em 112 and the user corresponds 10
a Enger of 1he user.
'1-hc !ouch screen in the display system I 12 may usc I.CI)
(liquid crystal display) technology. or I PD (light cmiuing
polymer display) technology. although other display IL'ch-
"111e audio circuitry 110. the speaker 111. and the micro- 40
phone 113 provide an audio intcr!Ucc bc1wccn a user and the
dL'\ icc 100. '1-hc audio circuitry I 10 rccci\cs audio data Jium
the peripherals inlcrbcc 1 I8. converts the audio data loan
elcc!rical signal. and transmits the elcc!rical signa! lo the
speaker 111 The speaker 111 converts the clec1rical sig.na! to
human-audible sound waves. The audio circuill)' llO also
rccci\cs electrical signals ctm\crtcd by the microphone I 13
from sound wmcs. rhc audio circuitry I 10 converts the clec-
1rical sig.nalto audio daw and 1ransmi1s 1he audio daw 10 the
peripherals intcr!Ucc 118 Jix processing. .. \udio dma may be
retrieved from and/or transmilk'd 10 memO!)' 102 and/or the
45 no!og.ies may be used in other embodimcms. "111e 1ouch
scrL-cn in the display system ll2 and 1he display comrollcr
156 may delL><::! conlacl and any movement or breaking
thereof using any of a plurality of touch sensing lcchno!ogics
now known or !mer developed. including. but not limited 10
Rl: circuitry I08 by the peripherals inlcrbcc 1I8. In some
embodiments. the audio circuitry I I 0 also includes a head sci
jack (not shown). The headse1 jack provides an interface
betWL'Cn the audio circuit!)' 110 and removable audio input/
output peripherals. such as output-only headphones or a head-
-;o capacitive. resistive. infrared. and sur!Ucc acous1ic wave k><::h-
no!og.ies. as well as o1her proximily sensor arrays or other
dcmcnls !(Jr detctmining one or more points of con lac! with
a !ouch screen in the display syslcm 1 I2. A louch-scnsili\c
display in some embodiments of the display system 112 may
sci with both output (e.g .. a headphone ll1r one or both cars)
and input (e.g .. a microphone).
'-' be ana!og.ous 10 1he muhi-touch sensitive tablc1s described in
the !l1llowing U.S. Pal. Nos. 6.323.X46 (Wcslcnmm cl a!.).
6.570.557 (WcslcmJ<m cl a!.). and/or 6.677.932 (Westerman).
and/or U.S. l'a!cnl l'uhlicalion 2002/00!5024Al. each of
"111e 110 subsystem 106 couples input/output peripherals on
!he device 100. such as the display system ll2 and o1her 60
input/control devices 116. lo the peripherals intcrliJcc 1 I8.
'1-hc 1/0 subsystem I 06 may include a display con!rol!cr I 56
and one or more input controllers 160 for o1her input or
comro! devices. The one or more inplll comrol!ers 160
R><::civc/scnd elcc1rica! sig.nals from/to other input or comro! 65
dL'\ ices I 16. rhc other input!contro! devices 116 may include
physical \muons (e.g .. push bul!ons. rocker bul!ons. clc.).
which is hereby incorporated by reference herein in their
emirc1y llowevcr. a touch scrL-cn in the display sys1em 112
displays\ isual output !i"om the pm1ahle device 100. whereas
!ouch sensitive tablets do no! prm ide\ isual output. '1-hc !ouch
SCR'Cn in !he display system ll2 may have a resolution in
excess of 100 dpi. In an exemp!al)' embodiment. 1he 1ouch
scrL-cn in the display system has a resolmion ofapproximately
l6X dpi. '1-hc user may makcconlacl with the !ouch screen in
the display system 112 using any suitable objL><::I or append-
us 7,469,381 82
13
age. such as a stylus. a linger. and so J(n1h. In some embodi-
ments. the user interface is designed to work primarily with
!ingcr-bascd contacts and gestures. which nrc much less pre-
cise than stylus-based inplll due to the larger area ofcontact of
a linger on the touch screen. In somccmbmlimcnls. the de\ icc
translates the rough linger-based input into a precise poin!erl
cursor position or command for pcrfonning the actions
dcsir<..'d by the user.
A touch-scnsitin:: display in some cmbmlimcnls of the
display system I 12 may be as described in the lllllowing
applications: (l) U.S. patent application Scr. No. t t n X l . ~ 13.
"Multipoint Touch Surface Controller." H!<..'d on May 2. 2006:
(2) U.S. patent application Scr. No. 10/840.862. "Multipoint
'!lmchscrecn." !!led on May 6. 2(XJ4: (3) U.S. palenl applica-
tion Ser. No. !01903.964. "(jeslures For '!lmch Sensitive
Input ])cviccs.'' tiled on Jul. 30. 2(X)4: (4) U.S. patent appli-
cation Scr. No. 11/048.264. "Gcslllrcs 1-'or Touch Sensitive
Input Devices.'' tiled on Jan. 31. 2005: (5) U.S. patent appli-
cation Ser. No. ll/03X.590. "Mode-Based Ciraphica! User
lnlcrbccs For '!lmch Scnsili\c lnpul DL"\iccs." Jiled on Jan.
18. 2005: (6) U.S. patent application Scr. No. 11/228.758.
"\lnua! lnplll Device Placement On..\ 'li.1uch Screen User
lnlcrbcc." Jiled on Scp. I 6. 2005: (7) U.S. patent application
Scr. No. ll/22X.7(XJ. "Operation Of A Computer With A
'!lmch Screen !nterl'iJCc." !!led on Scp. 16. 2005: (R) U.S.
patent application Scr. No. 1 !/228.737. ".\ctivating \lnua!
Keys Of.\ ']i.Juch-Scrccn Virtual Keyboard." tiled on Scp. 16.
2005: and (9) U.S. palcnl application Scr. No. l 1/367.740.
"Multi-Functional !land-! !c!d I )L"\ icc." Jiled on Mar. 3. 2006.
14
sources (e.g .. hallcry. ahcmaling ctuTcnl (AC)). a recharging
system. a power failure detection circuit. a power convener or
inverter. a power status indicator (e.g .. a light-emitting diode
(LFD)) and any other components associated with the gcn-
5 cralion. management and distribution of power in pm1ahle
devices.
The device 100 may also include one or more optical
sensors 164. 1-'lG. 1 shows an optical sensor coupled to an
optical sensor controller 158 in 1/0 subsystem I 06. The opli-
lCJ cal sensor 164 may include charge-coupled device (C'C! )) or
comp!cmcnlary metal-oxide semiconductor (CMOS) pho-
totransistors. The optical sensor 164 receives light from the
environment. projected through one or more !ens. and con-
vcJ1s the light lo data representing an image. In conjunction
15 with an imaging module I43. the optical sensor I64 may
capture still images or video. In some embodiments. an opti-
cal sensor is !ocak'd on the back of the device 100. opposite
the touch screen display 112 on the from of the device. so that
the touch screen display may he used as a \ iL"\v!indcr ll1r
"" either still and/or video image acquisition. In some embodi-
ments. an optical sensor is located on the front of the device so
that the user's image may be obtained Jix vidL'<.lConfcrcncing
while the user views the other\ ideo conJCrcncc p;u1icipanls
on the touch screen display. In some embodiments. the posi-
"5 lion of the optical sensor 164 can he changed by the user (e.g ..
by rotating the !ens and the sensor in the device housing) so
that a single optical sensor 164 may be used along with the
touch screen display ll1r both \ideo conJCrcncing and still
and/or\ ideo image acquisition.
AI! of these applications arc incorporated by reference herein 30
in their entirety.
The device 100 may also include one or more proximity
sensors 166. 1-'lG. 1 shows a proximity sensor 166 coupled to
the peripherals intcrJUcc 118. Alternately. the proximity sen-
sor I 66 may he coupled loan input controller 160 in the 110
subsystem 106. 'l11c proximity sensor I66 may pcrllm11 as
In some embodiments. in addition to the touch screen. the
dL"\ icc 100 may include a louchpad (nul shown) ll1r acti\aling
or dcacti\aling p;u1icubr functions. In some embodiments.
the touchpad is a touch-sensitive area of the device that.
unlike the touch screen. docs not display visual outpUl. The
touchpad may be a touch-sensitive surface that is separate
from the touch screen in the display system I 12 or an exten-
sion of the touch-sensitive surl'i1cc J(mned by the touch
screen.
In some embodiments. the device 100 may include a physi-
cal or \inual dick wheel as an input control dL-,.icc I16. A
user may navigate among and interact with one or more
graphical objects (hcncd(Jnh rcJCrred lo as icons) displayed
in the display system 112 by rotating the dick wheel or by
moving a point of contact with the dick wheel (e.g .. where the
amount ofmmcmcnl oflhcpoinl ofconlact is measured by its
angular displacement with respect lo a center point of the
click wheel). The click wheel may also be used to select one
35 described in U.S. patent application Scr. Nos. 11124UG0.
"Proximity Detector In llandhcld Device." tiled Scp. 30.
2(XJ5. and 11/240.788. "Proximity Detector In llandheld
lk\icc." Ji!cd Scp. 30. 2005. which arc hereby incorporated
by rckrencc herein in their entirely. In some embodiments.
40 the proximity sensor turns olr and disables the touch screen
112 when the multili.mction device is placed ncar the user's
car (e.g .. when the user is making a phone call). In some
embodiments. the proximity sensor keeps the screen offwhcn
the de\ icc is in the user's pocket. purse. or other dark area lo
45 prevent unncccssm)' battery drainage when the device is a
locked state.
In some embodiments. the soli ware cmnponcnls stored in
or more of the displayed icons. 1-'or example. the user may '0
press down on at !cast a portion of the dick wlK-c! or an
associakd \muon. Uscrcmnmands and na\ igalion commands
provided by the user\ ia the click whee! may he processed by
memory 102 may include an operating system 126. a com-
munication module (or set of instructions) 128. a contact/
motion module (or set ofinstructions) DO. a graphics module
(or set of instructions) B2. a text input module (or set of
inslmctions) I 34. a (jlohall'osilioning System ((jJ>S) module
(or set ofinslmctions) I 35. and applications (or set ofinslmc-
tions) D6. an inplll controller 160 as well as one or more oft he modules
and/or sets of instmctions in memory 102. For a virtual dick '5 The operating system 126 (e.g .. Darwin. RTXC. LINUX.
wheel. the dick whee! and dick whee! controller may he pan
of the display system 112 and the display controller 156.
rcspecti\ely. !:ora vi11ua!dick whee!. the click wheel may he
either an op.1quc or semitransparent object that appears and
disappears on the touch scrL-cn display in response to user
interaction with the dL-,.icc. In some embodiments. a vi11ua!
click whee! is displayed on the touch screen of a ponahle
muhili.mction device and operated by user comacl with the
touch screen.
'll1c device 100 also includes a power system 162 for pow-
ering the \arious components. The power system 162 may
include a power management system. one or more power
UNIX. OS X. WIN!)OWS.or ancmhcddcdopcralingsyslcm
such as \'xWorks) includes various sollwarc components
and/or tlri\crs ll1r controlling and managing genera! system
tasks (e.g .. memory management. storage device control.
60 power management. etc.) and facilitates conununication
between various hardware and sollwarc components.
The communication module 128 bci!ilalcs communica-
tion with other devices over one or more cxtcma! ports 124
and also includes various software components for handling
65 data received by the Rl-' circuitry 108 and/or the cxtcmal port
124. rhc cxlcrna! port I24 (e.g .. Uni\crsa! Serial !Sus (US!S).
FI!{I;WI!{I;. ctc.) is adapted ll1r coupling directly lo other
us 7,469,381 82
15
tk'\ ices or imlircctly over a network (e.g .. the !nlemcl. wire-
less LAN. etc.). In some embodiments. the cxtcmal port is a
multi-pin (e.g .. 30-pin) connector that is the same as. or
similnr to and/or compatible with the 30-pin cOJUlCCtor used
on il'od (trademark of Apple Computer. Inc.) de\ ices.
'!lu:: nmlacl/molion module BO may detccl conlacl with
16
alarm dock widget 149-4. dictionary widget 149-5. and
other widgets obtained by the user. ns well as user-
crcmed widgets 149-6:
widget crcmor module 150 for making user-crcnted wid-
gets 149-6: and/or search module I 51.
the touch screen in the display system 112 (in conjunction
with the display comrol!cr 156) and other touch sensitive
devices (e.g .. a touchpad or physical dick wheel). The con-
lact!molion module 130 includes various so!lwarc compo- lCJ
ncnls ll1r pcrllmning various operations related to dch.x:lion
I Oxamples or other applications I36 that may be stored in
memory 102 include memo pad and other word processing
npplications. JA\:\-cnabled applicmions. encryption. digital
rights mnnngement. voice recognition. nnd voice rcplicmion.
In conjunction with display system I 12. display controller
156. contact module 130. graphics module I 32. and text input
module I 34. the contacts module I 37 may be used to manage
nn nddrcss book or contact list. including. adding namc(s) to
the nddrcss book: deleting name(s) from the nddress book:
of contact. such as determining if contact hns occurred. deter-
mining if there is movement of the contnct nnd trucking the
movement across the touch screen in the display system 112.
and determining ir the contact has been broken (i.e .. ir the
contact has ceased). I )etermining mmement or the point or
contnct mny include determining speed (mngnitude). velocity
(magnitude nnd din.><::tion). and/or an acceleration (n change
15
associating telephone numher(s). e-mail address(es). physi-
cal address(es) or other inl(mJJation with a name: associating
nn imngc with n name: cntegorizing and sorting names: pro-
viding telephone numbers or e-mnil addresses to initiate nnd/
or JUcilitatc conununicmions by telephone 138. video confer-
in magnitude and/or dinxtion) or the point or contact. These
operations may be applied to single contacts (e.g .. one linger
contncts) or to multiple simultaneous contncts (e.g .. "multi-
touch''lmultiple !inger contacts). In some embodiments. the
contnctlmotion module 130 and the display controller 156
also dehxts contact on a touchpad. In some embodiments. the
contact/motion module I30 detects mmement or one or more ""
objects on or nenr the touch screen nnd/or the touchpnd. In
some embodiments. the contact/motion module 130 nnd the
controller 160 detects contact on a dick wheel 116.
"" ence 139. e-mail140. or IM 14I. and so I(Jrth.
In conjunction with RF circuitry 108. audio circuitry I 10.
speaker I I 1. microphone 1 I 3. display system 112. display
controller 156. contact module 130. graphics m<Jdulc 132.
nnd text inplll module 134. the telephone module 138 mny be
used to enter a sequence of characters coJTesponding to a
telephone number. access one or more telephone numbers in
the nddrcss book 137. modi!) a telephone number that has
been enten.>d. din! a respective telephone number. conduct n
conversation nnd discomJcct or hang up when the conversa-
'l11e graphics module B2 includes various known so!iware
components I(Jr rendering and displaying gwphics on the
displny system 112. including components for changing the
intensity of graphics thm arc displnyed. As used herein. the
term "graphics" includes nny object that cnn be displny<..'d to
30 tion is completed. :\s notL>d abme. the wireless communica-
tion may use any of a plurality orcmnmunications standards.
protocols and technologies.
a user. including without limitation text. web pages. icons
(such as user-interl'iJCe objects including soil keys). digital 35
images. vid<.os. nnimations nnd the like.
In conjunction with RF circuit!)' 108. nudio circuitry llO.
spenker 111. microphone 113. display system 112. display
controller I 56. optical sensor I 64. optical sensor controller
158. contact module 130. graphics module B2. text input
'111c text input module 134. which mny ben component of
grnphics module 132. provides soli keyboards for entering
text in \arious applications (e.g .. contacts I 37. e-mail140. IM
I41. hlogging 142. browser 147. and any other application
40
thm needs text inplll).
module 134. contact list 137. nnd telephone module 138. the
vid<.oconfcrencing module 139 mny be used to initiate. con-
duct. nnd terminate a video conference between a user nnd
one or more other participants.
In conjunction with Rl' circuitry I08. display system I I2.
display controller 156. contact module 130. graphics module
132. and text input module 134. thee-mail client module 140
'111c G!'S module 135 determines the locmion of the device
and provides this inlixmation for usc in vnrious npplicmions
(e.g .. to telephone 138 I(Jr use in location-based dialing. to
camera I43 and/or blogger I42 as picture/video metadata.
and to applicmions that provide location-based services such
as weather widgets. local yellow p.1gc widgets. and mnpl
navigation widgets).
'l11e applications B6 may include the J(JIIowing modules
(or sets or instmctions). or a subset or superset thereof
a contncts module 137 (sometimes cnlkd nn nddrcss book
or contact list):
a telephone module 138:
a video conferencing module 139:
an e-mail client module 140:
an instant messaging (1M) module I4I:
a blogging module 142:
a camern module 143 for still and/or video imngcs:
an image manngcment module 144:
a video plnyer module 145:
a music plnycr module 146:
a browser module 147:
a calendar module 148:
""ldget nmdules 149. ""hich may include ;eather ""idget
149-1. stocks widget 149-2. calculator widget I49-3.
45
may be used to crente. send. receive. and mnnnge c-mnil. In
conjunction with image management module 144. the e-mail
module 140 makes it very easy to create and send e-mails with
still or vid<.'O imnges tnken with cnmcra module 143.
In conjunction with RF circuit!)' 108. displny system 112.
,
0
display controller 156. contact module 130. graphics module
132. and text input module I 34. the instant messaging module
14I may he used to enter a sequence or characters corre-
sponding to nn instant message. to modi!) previously entered
charnctcrs. to trnnsmit a respective instant message (for
'5 example. using n Short Message Service (SMS) or Multime-
dia Message Sen ice (MMS) protocol). to rL><::ei\e instant
messages and to view rL><::eived instant messages. In some
embodiments. transmitted nnd/or n.><::cived instnnt mcssnges
may include grnphics. photos. audio Jiles. video Jiles nndlor
60 other attachments as are supported in a MMS and/or an
l;nhanced Messaging Service (I;MS).
In conjunction with Rl' circuitry I08. display system I I2.
display controller 156. contact module 130. graphics module
132. text inplll module 134. imnge manngement module 144.
65 nnd browsing module 147. the blogging module 142 mny be
used to send text. still images.\ ideo. and/or other to
a blog (e.g .. the user's hlog).
us 7,469,381 82
17 18
I Oach or the above identilied modules and applications cor-
respond to a set of instructions for pcrlixming. one or more
Ji.mctions described above. These modules (i.e .. sets of
instructions) nc<.>d not be implemented as separate software
In conjunction with display system 1 I 2. display controller
156. optical sensor(s) 164. optical sensor controller 158. con-
wet module 130. graphics module D2. and image manage-
ment module 144. the camera module 14J may be used to
capture still images or\ ideo (including a video stream) and
store them into memory I 02. modify characteristics of a still
image or video. or delete a still image or video from memory
102.
In conjunction with display system 1 I 2. display controller
156. contact module 130. graphics module 132. text input
module 134. and camera module 143. the image management
module 144 may be used to arnmgc. modify or otherwise
manipulate. labeL delete. present (e.g .. in a digital slide show
5 programs, procedures or modules. and thus \arious subsets or
these modules may be combined or otherwise re-arranged in
various embodiments. In some embodiments. memory 102
may store a subset of the modules and data structures identi-
Jicd abmc. Furthermore. memory I02 may store additional
lCJ modules and data stmctures not described ahme.
or album). and store still and/or video images.
In conjunction with display system 1 I 2. display controller 15
156. contact module 130. g.raphics module 132. audio cir-
cuit!)' 110. and speaker 111. the video player module 145 may
be used to display. present or otherwise play back vid<.os (e.g. ..
In some embodiments. the device 100 is a device where
operation of a prcddined set of Ji.mctions on the device is
pcrlixmed exclusively through a touch scr<.>cn in the display
system 1 I 2 and/or a touchpad. ISy using a touch screen and/or
a touchpad as the primary input/control device I(Jr operation
of the device 100. the number of physical input/control
devices (such as push buttons. dials. and the like) on the
device 100 may be reduced.
The predelincd set of Jimctions that may be pcrl(mncd on the touch screen or on an cxtemaL connected display via
cxtema! port 124). "" exclusively through a touch screen and/or a touchpad
includes navig.ation between user intcrJUces.ln some embodi-
ments. the touchpad. when touched by the user. navig.ates the
device I 00 to a main. home. or root menu Jium any user
In conjunction with display system 112. display system
controller 156. contact module 130. graphics module 132.
audio circuitry I 10, speaker 11 I. RF circuitry 108, and
browser module I47. the music player module I46allows the
user to download and play hack rccon.kd music and other "5
sound Hies stor<.>d in one or more !ile fonnats. such as MPJ or
A \C !iles.ln some embodiments. the device 100 may include
the Jimctionality or an MPJ player. such as an il'od (trade-
mark or Apple C"omputer. Inc.).
interliJCe that may be displayed on the device 100. In such
embodiments. the touchpad may he relCJTed to as a "menu
button." In some other embodiments. the menu button may be
a physical push button or other physical input/control device
instead ora touchpad.
I'J(j. 2 illustrates a portable nm!tilimction device 100 ha\-
30 ing a touch screen 112 in accordance with some embodi-
ments. The touch screen may display one or more graphics. In
this embodiment. as well as others dcscrib<.>d below. a user
may select one or more of the graphics by making contact or
In conjunction with RF circuit!)' 108. display system 112.
display system controller 156. contact module 130. graphics
module 132. and text input module 134. the browser module
I47 may be used to browse the lntemct. including searching.
!inking to. recci\ing. and displaying web pages or portions
thereof. as well as attachments and other Hies !inked to web 35
pag.cs.
touching the graphics. I(Jr example. with one or more lingers
202 (not drawn to scale in the Hgurc). In some embodiments.
sckction ofoneor more graphics occurs when the user breaks
contact with the one or more graphics. In some embodiments.
the contact may include a gesture. such as one or more taps.
one or more swipes (li-om !eli to right. right to !eli. upward
In conjunction with RF circuit!)' 108. display system 112.
display system controller I 56. contact module 130. graphics
module 132. text input module 134. e-mail module I40, and
browser module 147. the calendar module 148 may be used to
create. display. modi!). and store calendars and data associ-
ated with calendars (e.g .. calendar entries. to do lists, etc.).
40 and/or downward) and/or a rolling of a !inger (from right to
ldi. left to rig.ht. upward and/or downward) that has made
contact with the de\ ice I 00. In some embodiments, inadveJ1-
ent contact with a graphic may not select the graphic. For In conjunction with RF circuitry 108. display system 1 I2.
display system controller I 56. contact module 130. graphics
module 132. text inplll module 134. and browser module 147. 45
the widg.ct modules 149 arc mini-applications that may be
downloaded and used by a user (e.g .. weather widget I49-I.
stocks widget I49-2. calculator widget 149-3, alarm dock
widget 149-4. and dictional)' widget 149-5) or created by the
user (e.g. .. user-created widget 149-6). In some embodiments. '0
a widg.ct includes an IITMI (llypcrtcxt Markup Language)
Jile. a CSS (Cascading Style Sh<.-ets) Jile. and a JavaScript Ji!c.
example. a swipe gesture that sweeps mer an application icon
may not select the corresponding application when the g.cs-
ture corresponding to selection is a tap.
The device I 00 may also include one or more physical
buttons, such as "home" or menu button 204. As described
previously. the menu button 204 may be used to navigate to
any application 136 in a set of applications that may be
executed on the device 100 .. \!tcrnatively. in some embodi-
ments, the menu button is implemented as a soli kl:y in a (jlJJ
in touch screen 1 I2.
In one embodiment. the device 100 includes a touch screen
In some embodiments. a widget includes an XMI (I Oxtensible
Markup Language) !ile and a JavaScript !ile (e.g. .. Yahoo!
Widg.cts). '-' 112. a menu button 204. a push button 206 for powering. the
device on/off and locking the de\ ice, and \o!ume adjustment
button(s) 208. The push button 206 may he used to tum the
power on/off on the device by depressing the button and
In conjunction with RF circuitry 108. display system 1 I2.
display system controller I 56. contact module 130. graphics
module 132. text input module 134. and browser module 147.
the widget creator module 150 may be us<.>d by a user to create
widgets (e.g .. turning. a uscr-spcci!i<.>d portion of a web pag.c 60
into a widget).
In conjunction with display system 1 I 2. display system
controller 156. contact module 130. graphics module 132.
and text input module 134. the search module 151 may be
used to search for text. music. sound. imag.c. video. and/or 65
other Jiles in memory 102 that match one or more search
criteria (e.g .. one or more uscr-spccilied search terms).
holding. the button in the state for a predcHn<.>d time
interval: to lock the device by depressing. the button and
releasing the button bel(Jre the prcdelined time interval has
elapsed: and/or to unlock the device or initiate an unlock
process. In an alternative embodiment. the device 100 also
may accept verba! input for activation or deactivation of some
Ji.mctions throug.h the microphone 113.
Attention is now directed towards embodiments of user
interliJCes ("UI") and associated processes that may he imp!e-
us 7,469,381 82
19
mcntcd on a pm1ahk: mul!i!lmction device 100 and/or on a
device 1700 wi1h a touch-screen display (FIG. 17).
!-'I G. 3 i!lustrntes an cxcmplm)' user intcr!Uce Jix unlocking
a ponab!cdcctronic device in accordance with some embodi-
ments. In some cmbmlimenls. user in!crliJCc 300 includes the
l(JI!owing ek:mcnls. or a subset or supersc!lhcrcor:
Unlock image 302 that is mov<..'d with a Hng.cr gesture to
unlock the device:
Arrow 304 that provides a visual cue lo the unlock gesture:
Channel 306 that prm ides additional cues lo the unlock
gesture:
Timc308:
Day 310:
Datc312:and
Wallpaper image 314.
In some embodiments. the device detects contact with the
touch-sensitive display (e.g .. a user's Hng.cr making contact
20
menu bunon enables a user to access any desired application
with at most two inpllls. such as activnting. the menu button
204 nnd then activming. the desird applicmion (e.g. .. by n tap
or other ling.er gesture on the icon corresponding. to the appli-
5 cation).
In some embodiments. Ul 400 provides integrat.._-d access
to both widget-basL-d applications and mm-widget-hased
npplications. In some embodiments. all of the widg.ets.
whether user-crented or not. are displayed in Ul400. In other
lCJ embodiments. acti\ating the icon ll1r user-creak-d widget
149-6 may lead to another Ul (not shown) that contains the
user-crented widg.ets or icons corresponding to the user-cre-
med widg.ets.
In some embodimems. a user may renrrange the icons in Ul
15
400. e.g .. using processes descrih.._-d in U.S. patent application
Ser No. II 1459.602. "Portable I ;kx:tronic lkvice With Inter-
on or ncar the unlock image 302) while the device is in a
uscr-interbce lock stale. The de\ ice moves the unlock image
302 in accordance with the contact. rhe de\ ice transitions to ""
a user-imerface unlock stme if the detecl<-'<1 contnct corre-
bee RLx:tmliguration Mode." lik-d Jul. 24. 2006. which is
hereby incorporated by reference herein in its entirety. 1-'or
example. n user may move application icons in and om of tray
408 using linger gestures.
In some embodiments. Ul 400 includes a gauge (not
shown) that displays an updated account usage metric ll1r an
nccount associal<-'<1 with usage of the device (e.g. .. n cellular
sponds to a predefined g.esture. such as moving. the unlock
image across channel 306. Conversely. the device maintains
the user-interbce lock state if the deh.:cted contact does not
correspond to the preddined gesture. As noted above. pro-
cesses that use g.estures on the touch screen to unlock the
device nrc described in U.S. patelll npplication Ser. No.
II 1322.549. "Unlocking a I kvice by Perll1rming (iestures on
an Unlock Image.'" Jiled DLx:. 23. 2005. which is hereby
incorpornted by reference herein in its entirety.
1-'lG. 4 illustrntes an exemplnry user inter!Uce for a menu of
applicmions on n portable muhili.mction device in accordance
with some embodiments. ln some emhmliments. user inter-
liKe 400 includes the ll1llowing dements. or a subset or
superset thereof:
Sig.nal strength indicator 402 for wireless communicmion:
Time404:
ISallery status indicator 406:
1-ray 408 with icons ll1r lfequently used applications. such
ns one or more of the l(lllowing.:
Phone 138:
1;-mailclient140. which may include an indicator410of
the number of unread e-mails:
Browser 147: and
Music plnyer 146: nnd
Icons ll1r other applicmions. such ns one or more of the
ll1llowing:
IM 141:
lmag.e mnnag.ement 144:
Camern 143:
VidL'O plnyer 145:
Weather 149-1.
Stocks 149-2:
Blog. 142:
Calcndnr 148:
Calculator 149-3:
Alann dock 149-4:
Dictionary 149-5: and
User-cremed widg.et 149-6.
In some embodiments. Ul 400 displays all of the nvnilable
applications 136 on one screen so that there is no ne.._-d to
scroll through a list of applications (e.g .. \ ia a scroll bar or via
a swipe g.esture). In some embodimems. as the number of
applicmions increases. the icons corresponding to the nppli-
cntions may decrease in size so thm nil npplications mny be
display.._-d on a single screen without scrolling. In some
embodiments. having all applications on one screen and a
"
5
phone nccount). ns described in U.S. patem npplicntion Ser.
No. 111322.552. ".-\ccoumlnlixmmion Display 1-'or l'ortnblc
Communication De\ice." Jiled Dec. 23. 2(X)5. which is
hereby incorporat.._-d by rekrence herein in its entirety.
As discuss.._-d above. Ul400 may display all of the mailable
30
npplications 136 on one scrL>cn so that there is no JK>cd to
scroll through a list of npplications. However. in some
embodiments a touch-sensitive display may include a (iUI
with one or more windows that display only a portion of a list
of items (e.g .. inlixmmion items) or of nn electronic docu-
35 ment. In response to detecting a movement of an object on or
near the touch-sensitive displny. the list mny be scrolld or the
dectnmic document may be translated. I kt.._x:ting the mme-
ment of the object may include determining spLx:d (magni-
tude). velocity (mag.nitude and direction). and/or an accelera-
40 tion (including mngnitude and/or dirLx:tion) of the object.
Scrolling through the list or translating the document may he
accderat.._-d in response to an accelerated movement of the
ohjLx:l. In some embodiments. the scrolling and acceleration
of the scrolling.. or translntion and nccderntion of the trnns-
45 lmion. may be in accord.1nce with n simulation of n physical
device hming liiction. i.e .. damp'--d motion. For example. the
scrolling or translation may correspond to a simulation of a
ll1rce law or equation of motion having a mass or inertial term.
ns well as a dissipmive term. In some embodimems. the
,
0
simulation mny correspond to a cylinder rotming. nbom its
aXIS.
In some embodiments. accelerated movement of the
dekx:t.._-d obj'-x:l may include an accderat.._-d movement of a
point ofcomnct followed by n brenkingofthe point ofconwct.
'-' For exnmple. the user may make comact with the touch-
sensiti\e display. swipe or sweep one or more of his or her
lingers along the display (i.e .. move and/or accelerate the
point of contact). and optionally. break the point of contact
with the displny. i.e .. move the one or more ling.ers away from
60 the display The swipe or sweep mny be nlong n predefined
axis of the touch-sensitive display or may be within a prede-
letmined angle of a preddined direction on the touch-sensi-
tive displny. In other embodiments. the accelerated move-
ment of the point of contnct may include a lirst user g.esture
65 oriemed alongn predefined nxisofthe touch-sensitive display
or oriented within a predetetmin.._-d angle of a prL-ddined
dirLx:tion on the touch-sensitive display.
us 7,469,381 82
21 22
letter in the alphabet that has one or more entries. In some
embodimcms. the scrolling may be in accordance with a
simulation of an equation of motion having friction.
The scrolling may include scrolling through a respective
Scrolling through the list of items or translating the dec-
Ironic document may be li.1rthcr accdcrm<..-d in response to
detection of a second movement of an object on or ncnr the
touch-sensitive display. such as a second sweeping motion of
the point of contact along the preddincd axis or within the
predetermined angle of a preddincd din.x:lion on the touch-
sensitive display and/or a s<..>cond user gesture oriented along
the prcdc!incd axis or within the prcdctcnnincd angle of a
prcddim:d direction on the touch-scnsitin:: display. For
example. the user may swipe one or more ofhis or her !ingcrs
along the louch-scnsiliH:: display two or more times.
5 inl(mJJalion item subset if the point of conlacl mmes o\er a
corresponding respccli\e index item in the index items. The
scrolling may have an associmcd scroll speed based on a
spc<.-d of movemem oft he point of contact over the respective
index item and the number of items in the inl(mJJalion item
lCJ subset corresponding lo the resp<.><:li\e index item. For
example. the scroll speed may he li1sler ll1r subsets that ha\e
more cmries than subsets with fewer cmries. The scrolling
may include scrolling through all items in a plurality of the
inl(mJJalion item subsets in response lo the point ofconlacl
"111c scrolling throug.h the list of items or the translmion of
the electronic document may be stopped in accordance with
the user hrcaking the point ofconlacl and then cslahlishing a
suhslanlia!!y stationary point of contact with the touch-sen-
sitive display form least a pre-determined period of time. For
example. alicr swiping one or more of his or her Hngers along
the touch-sensitive display and breaking the poimofcontact.
the user may touch the display and hold the one or more
!ingcrs that arc touching the display stationary (or appmxi- ""
mmcly stationary) Jixone or more seconds. or fractions of a
SL'COnd.
15 mm ing over the corresponding index items in the displayL'<i
index.
]fit is determined that the point of contact with the index
corresponds to a respective index item in the index. the list of
inl(mJJalion items may he scrolled loa corresponding subset
of the list of inl(mnalion items. For example. if the user
sck>cts an index item. such as the letter 'R'. in the set of index
symbols. the list of items may be smoothly scrolled to the
corresponding subset ll1r the letter 'R' in the list of items.
:\llernalivdy. the displayed list of inl(mJJalion items jump
"!lJc direction of scrolling or translation may he reversed in
response lo inters<.>cling a \ irlual boundary corresponding lo
a leJminus of the list or an edge of the electronic document.
The scrolling reversal or translation reversal may correspond
to a d.1mpcd motion. For example. during scrolling. a dis-
played pm1ion oft he list of items may appear lo bounce off of
"5 dir<-><:lly Jiom a cuJTenl scroll position loa scroll position in
which information items corresponding to the index item 'R'
arc displayed.
In the present document. the term "il" may he construed lo
mean '\\hen." or "upon." or "in response lo determining." or a boundary of the window in the louch-sensili\edisplay when
a beginning or an end oft he list of items is rcach<.'d. Similarly.
during translation. a display<.'d portion oft he electronic docu-
ment may appear to bounce oll" of a boundm)' oft he window
in the touch-sensitive display when an edge of the document
is reached. The apparent bounce may coJTespond loa simu-
lation of a viscous or clastic ball having momemum in a !irsl
direction striking an immovable and/or inelastic object. such
as a wall. The subs<.'<.juem motion oft he documem (the motion
30 "in response to detecting." depending on the comcxt Simi-
larly. the phrase "if it is determined" or "if[a stated condition
or event 1 is detected" may be construed to mean "upon deter-
mining" or "in response lo determining" or "upon detecting"
the slated condition orevenl. or"in response lo del<->cling" the
35 stated condition or event. depending on the comext.
If the poim of contact with the touch-sensitive display
corresponds to a user selection of a respective infonnation
item in the list ofinl(mJJalion items. inl(mnalion coJTespond-
ing lo the respL><:live inl(mJJalion item may he displayed on
of which coJTesponds lo the hal! in the anal-
ogy) may he damped. !(Jr example. by including a friction or
dissipmivc term in the simulation . .-\ parameter corresponding
to the friction term in the simulation may be adjustable.
allowing the document lo reach <-'<.juilihrium in contact with
the vi11ual boundary. or displaced limn the\ irlual boundary.
40 the touch-sensitive display. For example. if the user selects a
respective name. the corresponding comact information may
he displayL'<i.
While scrolling through resp<.><:li\e inl(mJJalion subsets. an
index symbol may displayed in nmjunclion with each respcc-
45 tive informmion item subset. In some embodiments. respec-
tive index symbols may be display<.'d adjaccmto correspond-
ing subsets (such as displayed lexl) oft he list of inl(mnalion
items. In some embodiments. a resp<.><:li\e index symbol may
be displayed at an upper edge of a window comaining the
In smneemhodimenls movement of the point ofconlacl by
the user over an index on the touch-sensitive display may be
determined. In some embodimems. the index may be dis-
played in a lirsl region or a !irsl window of the louch-sensili\e
display while the list of items or in!(mnalion items during the
scrolling may be displayed in a second region or a second
window of the touch-sensitive display. The displayed index
may have a S<-'<.jUenceofindcx items. In an exemplm)' embodi-
ment. the SL'<.jUence of index items may include letters in the
alphabet. i.e .. the index may include an alphabetical index.
The list of informmion items may include an alphabetically
ordered list of infonnation items. The alphabetically ordered '5
list ofin!(mnalion items may include conlacl in!lmnalion. !(Jr
example. in a user's conlactlisl or address book.
'0 displayed text of the resp<.><:tivc inli.mnation item subset.
In response lo movement oft he user's point ofconlacl over
a displayed index. the list of inllxmation items on the touch-
sensitive display may be scrolled. The list of inlixmation 60
items may include a sequence of inllmnalion item subsets
corresponding lo the sequence of index items. The subsets
may include one or more cmcgorics. For example. a resp<->c-
tivc cmcgory may include contact inllxmmion for one or
more individuals whose Hrsl and/or last names begin with one 65
or more respective letters. such as the letter 's'. In an exem-
plary embodiment. there is a subset corresponding lo each
The index symbol corresponding to a resp<.>ctivc inli.mna-
lion subset may he translucently displayed mer the respccli\e
inl(mJJalion item subset. The translucently displayed index
symbol may have a different li.1nt color than that used to
display text in the inli.mnation item subset. and/or it may be
displayed using a larger Jlmllhan the l(ml usL'<i lo display texl
in the inl(mnalion item subset.
If the list of inl(mnalion items contains no items ll1r a
particular index symbol. i.e .. no emrics li.1r a particular subset.
a lirst index symbol prcc<.'ding a particular index symbol and
a second index symbol l(JIIowing the index symbol may he
displayed in ctmjunclion with scrolling through the list of
information items from the inli.mnation subset corresponding
to the lirst index symbol to the information subset corre-
sponding to the second index symbol. The particular index
symbol may nol he displayed in nmjunclion with the dis-
played lcxl of the list ofinl(mJJalion items during the scroll
:np puo.\:1q IO:.'"OJI' tm'i!m.\qdsl( 1 p:1q:m:u s1 SI1U!Ul.l:ll :lt]l :1mpq
f:"JSf :l.JI1)S:l'i! :ld!'<\.S :l![JJO .\qds1p u:nDs tp110J :l![l uo fhh'"Ods I'
lll puods:JJJO::I ,\r,m ([9-\"9 "S!)[:J U! U.\\llljS ;)ljlJll
;)tjl'i!Uppr,;).J 0) .!O!Jd ilUHJ<.U::IS r, ";)jdUWX;) .!O:J ");)..'"Ofqo
;)ljl JO )lWUWAlllU Jll p;);)tfS r, lll Wljl p;);)tfS 'i!m
-nons fA'"Oli'!::IOSSI' Sl'l( :ll[lJO SI1U!UU:lll' 'i!U!lPI':lJ OJ
JO!Jd UO!P:tHp :ll(l U! 'i!mnons :llUOS Uj
-p;)tpe;).J 09
;)tjl tTSf ;)dt.\\S ;)tjl ,\q ,\ejds]p
U:l:Jns tp11o1 :lt]l uo 1' tmtp SS:lj :1q .\1m1
tp!l]\\ "8fSf I' .H:J S! JS!J :ll[l p.-..tpt,:l.J SI1U!Ul
J:l) :lljl .!:llJl' ".)9 "!)):] ll! ":ljdlUI,X:l JO:] "fA'"Olpi!.'"OJ Sl
;)ljl .1;)\I!J )J;)fqo ;)ljl JO )U;)lU;)MllU Jll ;);)Ur,JS!p r, U!Jljl
Wljl r, .!OJ UO]l;)..'"O.!]p )s.!!J
:ll[l u! JSI( :lljlJO :lt]l ptm.\:Jq
1':1.11' tm 'ilm.\l'[dS!(! p.-..tpl,:tl s1 :l![l :tHlpq tJSf
;).Jil)S;)f. ;)lj) ,(q U;);).J::IS tplllll ;)ljl UO
;)::IUr,Jstp e 01 .(em ([9-\"9 m U.\\Ot[S lS!i ;)tjl 0>
Jll SllU]m.J;)) ;)ljl f.U]tpP-;).1 0) .!Opd r, ";)jdtu!JX;)
.10:] :ll(l.JO :ll(l Timtpl,:l.J OJ .Hl!.!d p:1fqo :lljl JO
I' OJ Wl[l
p;)Jr,pmse ls.!!J e ;)t[l JO snu]m.J;)l ;)tjl 'ilmtpe;).J 01
.!Opd Ull]l::l;).!!p ;)ljl U! ;)lUOS U[

(pzsl
.\l'[dS!P U:J.."U:lS tp110J :ll[l uo p.-...\qds1p :11!p.-.. tm OJ JO .\l'[dS!P
U:l:lJ::IS tp110):lljlJO:l'i!fA'"O tm OJ fA'"Ol(:li'Jll' O).ll,:lddl'
)S]j ;)ljlJO SllU]m.J;)) ;)ljl p.-..,\ejdS!p .J;)f.UO[ OU lS!j ;)ljl
Jll SHU]UU;)J ;)ljl puo.(;)q !J;).J!J ;)ljl j!)U!l UO!lJ;).!]p pUO;)..'"OS ;)ljl U! Ot
JS!( OJ UO!)J:l.I!P :lljl U! JS!J UllU.J
:J'i!tmtp :l![l s]u:Jtmpoqm:J :1mos UJ (zzs) uo!1om p.-..dump 1'
Ull])J;).!!p pUll;)..'"OS ;)ljl U] S! lS!j ;)lj)"SHWtmpoqtu;)
;)mos Uj "(OZS) p.-...\ejdstp .J;)'i!UO[ OU S] lS!i ;)ljl Jll
;)ljl puo.(;)q e;).Je ;)ljl [!lUll Ull!)::l;).!!p ;)ljl ;))!SOddo Ull!)
-:J:JJIP pum:1s 1' u! p:lJFU:ls S! stu:JJI .JO JS!J :lt]l
![:llllll :ll[J.m:lU JO UO fA'"OJ:h'"OPp J:l'i!UOJ OU S! p:1fqo :lljl J:ll.JV
":l:l!.\:!p
[11m :l]qlnJod 1' :l:l! \:!p :lt]l "SJU:ltmpoqm:J UJ 1'
S! J;)..'"Ofqo ;)ljl Uj "(ZOS) ;)::I!A;)p r, JO
tpnoJ e .10 uo p;)JJ;)J;)p S] J::l;)fqo ue.1o JU;)tu;)Mljii,J
p;)tpe;).J U;);)q SP-tj r, Jll SllU]m.J;)) e )etjl r,
OJ JO]IDipU! I' \ll.ld nOS plll(PlU :ll(.!.
-1poqm:1 l[ll\\ :l:Jtmp.Jm:ll' u! JS![ 1' qTimut]l
JO nOS poqpm I' 'ilU!li'JlSI1[[! Ull!l'ili'!P \\OtJ I' S "!)!:]
"().J;)Z
;)q ,\em )! pjOljS;).Itjl .J;).\\0[ r, ,\)1::10[;)1\ ;)ljl tWlJ/1:\
"'}(,<;" .{q .{lml.{J!:lOj:l.\ :ll(l Jl'\J:l)U!
tplo:.'"O u! .ro.-1 p.u:1d1'1 :1q .\1m1 .10
:lljl .JO ,{]DOj:l.\ :ll(l "Pl')Ull:l :lljl.JO 'i!U!)jl':l.llJ :lljl OJ JO!Jd
-.J;))U] ;)lUll ;).JOlU .Ill ;)Ull U] .\)!:Ill];)\ ;)ljl .!OjpUe UO]Jr,.!;)j;)::l::le ;)ljl
UO p;)sr,q ,\)1::10[;)\ ;)ljlJO UO]WU]m.J;)j;)p
m :1mos U[ uo!Jqstm.JJ m Tiu!
[[ODS :ll[l.JO UO!P:l.I!P JO S!XI' tm 'i!UO(I' p:lJ:h'"Ol(ud :llJ .\I'm UO!l
e tpns m .1o;pue ;)tiJ. je.U;)JU]
;)m]J ;)UO se tpns ";)lUll p.-..sdep tm S] 111 pue U;))jO.Jq
;)ljl U;)lj.\\ UO]Jr,.!;)j;)::IJe ;)ljl JO ;)!lje,\ )lW.J.JnJ r, e "lW'jO.Jq
Sl PI'JUO:l :lljl U:llj\\ ,{]DOj:l.\ :li(JJO :li1Jl'\ ]U:tl.JI1:ll' S!
0
.\ :l.J:llj\\
"1\W+".< 't

p;)U]m.J;)j;)p ;)q .(em pe)Ull::l 'ilm'je;).Jq .!;)lJe Sje,\.J;))U] ;)lU]l ;).JOlU
.10 ;)UO .10 'i!U!iill.!::IS JO fi\ ,(J]JO];)\ ;)ljl ";)jdUWX;) .!ll:J
"PI')UllJ:lljlJO'i!U!'jl':l.JlJ :lljl O).lllUd :llUI) :tlOlU JO:lUO
m .{J!:lO(:l\ :lt]l .10 p.-..:Jds :lt]l put' UO!ll'.l:JP:l:Jt' :lt]l u! :J'i!tmtp :l![l
tjll.\\ ;)::ltmp.!O::IJB m ;)q ,\em ;)tjl .\ueuo]Jdo
r, .1;)\I!J .10 ;)ljl ;)mOS U[
"[P-A.I;)JU] ;)m]JJU;)nb..-..sqns e .1o;pue ilu]p;)J;).Jd e U]
p:lU!Ul.!:!pp Ull fA'"OSI'lJ p:lp:Jfll.ld :llJ .\1m1 JOfPut' (1!\.J:l)U!
:lUl!l :l![l .!OJ S:llU!l :lJOUl JO 0\\l fA'"OU!llll:lPp :llJ .\I'm
UO]Jr,.!;)j;)::IJB .!OjpUe ".\)!:Ill];)\ ;)ljl ;)S;)ljl
UJ )llU .10 p.-..dd!'jS ;).Je S;)tue.!J ;).Jllm .10 ;)Ull U;)lj.\\
se p;)Sil ;)Je.J ;)mP-.!J ;)jqeueA r, U;)lj.\\ tWA;) p;)U]m.J;)j;)p
";)J!tj.\\ ;).Je p;)JS][ ;)tjl JO ;)tjl pue 9fSf
;)tjl t[Joq .)9 "Di:i m ";)[dmex;) .!O:J (8JS)
;)ljl lUO.!J )JU!)S]pU] ,\nenst,\ ;)ljlJO SHU]UU;)J ;)ljl puo,\;)q
1':1.11' :ll(l ptm pu110.Rl'j:Jl'q I' SI"J SUl:l)!JO JS!( :ll(l "S)U:ltU!pOlJlU:l
:lUlOS UJ "(9JS) :lll![.1. :lljl JO :lljl ptm.\:Jq
e;).Je ;)tjl s)U;)m]poquw ;)tum UJ s[mm;) JO 'ilmpuods;).J.JO::I
;)ljlJO ;)ljl pur, m;)J] lS.!!J ;)ljl S] S;)Ullf UO.IP-\" mll.!J
tfSf J!l'lU:l :l![l . !)!:! U! .lll_.j "Ul:ll! :lljl JO lU:l)l
)S.l!J :ll(l J:ll]l!:l :ll(l put' Ul:ll! I' ptm lU:l)l )S.l!J I'
:Jt]l "SJU:Jtmpoqm:J UJ (tiS s:JA-o IS) p:1.\1'[ds!P
OC :1q .\1m1 UO!JIO.!:!p:l:ll' .Hl;ptm .\J!:lOj:l.\ :lljl
UJ .\l'[dS!P 1qo s L<J !()"() s1' 1p11s ":JWJ :JUH!l.JI'JO :JSJ:l\U!
ue 01 Je.U;)JU! ;)tU!l ;)A]l;)..'"OdS;).J e ;).J;)tj.\\
;)m]l ;),\!) .J;)Ml p;)U!lU.J;)J;)p ;)q ,\em UO]Je.!;)j;)::IJe .!Ofplm .()!JOj;)A
"P;);)dS ;)ljl ")U;)tU!poqtu;) .\Jejdtu;)X;) Ue U[ ;)m]l
,:c jlOJ:l\:lS 'i!U!.!11p fA'"OU!lUJ:!pp :J'i!l!l:l.\1' :llU!ll' :llJ ,\I'm
:ll(.!. "PI'JUO:l.JO )U!Od I'.JO )U:lUl:l.\OUl.JO p:h'"OdS l(:li1S
p;)fqo p;)JJ;)J;)P e JO JU;)tu;)Mltu JO e tjl!.\\
U! ;)q ,(r,m UO]lejSlW.J) .10 ;)mOS U[
lSI[ ;)t[l JO ;)tjl puo.\;)q e;).Je ue
l[Jillll ;)ljl .J!J;)U .10 UO fJ<'"OlJ;));)p ii!lS )J;)fqo ;)ljl ;)jllj.\\ Ull!)
-:J..')JIP :Jt]l u! :Jt]l :!]It]\\ :Jt]JJO ""
:ltp'i!mtpl,:tl uod11 TI:1) p.-..tpl,:tl S! JSI( :l![l.JO SI1U!UU:lll' .11
"(80S) Ull!)::I]JJ
'i!m,Wtj Ull]lOlU JO UO!Wilb;) UP- J 0 r, lj)].\\ ;)::ltmp.!O::I::Ie
U] S! lS!i ;)ljl "SJU;)tutpoqtu;) ;)mos U[ "(90S) J;)..'"Ofqo
:ll[l.JO JU:lUl:l.\OUl.JO I' O]'i!mpuods:JJ.lll:l fA'"O:lds >'l
fA'"Oll'!:lOSSI' tm sm.] )SI( :ll(l .JO SI1U!UU:l) I' Timtpt,:l.J OJ JO!.!d
Ull!)::l;).!!p )s.!!J ;)ljl U] )S!i ;)ljl "SHWtmpoqtu;)

:1q :lljl U! .l:lll'( :lSlll':l ,\I'm .mq :ll(l UO PI'JUO:l
JO JU!Od :lljlJO )U:llU:l\OlU p.!I,\\U\\Op I' Sl,:l.J:llj\\ :llJ
;)ljl U] S;)UJU;).J;)!j.!e;) ;)SileJ ,(r,m .Jeq iill.!::IS ;)ljl UO J::leJUOJ
JO 1mod e JO JU;)tu;)Alltu ue U;)t[l p;)sn .1eq e
Jl :lUI OS Uj "JI'lJ nons l(:li1S .{tm .JO )U:lpU:Jd:!pU!
:1q tm:J :tl:l![ .10 :Jt(l "lSI( p.-...\qd
:l![l 1(11\\ uo!pu11fmn m p:l.\l'[dS!P :1q .{[[l'!lU:lJOd pj110:J
.Jeq r, ;)jllJ/1:\ )lWUHlJOp ::I!UO.JP;)j;) .10 ;)ljl
tp!lj.\\ U] .\\OpU].\\ ,\ejdS]p r, U]lj)].\\ ;).J;)lj.\\,(Ue lj!ed r, 'i!UOJ e ";)::I!Jj
s,.\l'[dS!P :lljll]l!.1. PI'JUO:l ll! JO JI'!.'"OU "Ulli]Otu Tiu!plj'il JO
ol 1' :1p11pu! .\1m1 ptm :Jm<.UJ:h'"O(:l
.10 )S!i ;)ljlJO dO) UO .(jlJ;).!]p p;)UUOj.J;)d ;)q ,\em UO]l
-e.J;)dO ;)lj) Ull Ull!)Om
;)liJ. iill.!::IS 'i!m,\ejdS!p lllOljll.\\ S;))!J.J;)dO ;).J;)lj
U[ "UO]l;)..'"O.!]p je)UOZUlllj r, UO]l;)..'"O.!]p lS.!!J ;)ljl "SHWtmpoqtu;)
J:llj)O U! :UO!P:tHp Jl':l!P:l.\ Ulll]:l:l.!lp :ll(l "S]U:lUl
!POlJUl:l :lUlOS UJ "S'j.Jiml'jOOlJ .JO I' JO umq]l' JO
JS!(I' "S:lUO)'i!U!.I.JO JO )SI(I' "S:lSS:l.!pp1'
I!P-UWJO lSI[ e JWm;)JO )S!i r, )S]j r, "(lSI[ 'jllllq
.Ill lS!i )J!JlUO::I e Ull!)P-UUOJU] pe)
Ull:l .JO I' "S.l:llJUlllU :lUO!jd :ll!.I0\10J.JO I' "SUO!li"J:l.\Ull:l
:J'i!I"S:llU ]lW]SU! JO I' SUl:ll! JO JS!( :ll(l "S)U:ltU!pOlJlU:l ,:
;)mos Uj "([9-\"9 "SDJ:i U] j!etu;)
Jll e S] lS!i ;)ljl ;)tum Uj "(f:"OS) UO]l;)..'"O.!]p
ls.!!J e U] ,\ejds]p U;);).J::IS tpno) ;)tjl uo p;),\ejds]p
.JO )SI( I' "]U:llU:l.\OUl :ll(l 'i!U!P:!Pp OJ Uj
t<:
S)U:lUlll:lOp :l!UOJP:l(:l .JO UO!JqSutOJ) :ll(l
ll! .{(JI'(IlUIS "JI'lJ [[ODS I' 'i!m.\qtfs1p ]110l(l!l S:l)IOJ:ldO
;).J;)t[ lSI[ ;)t[l snwunpoqtu;) ;)tuos U[
X;)pU! .Jej!l::I!UL>d ;)ljl
OU SUJelUOJ SUWl! Ull])etu.!OJU!Jll ;)ljl U;)lj.\\ p.-..dd!'jS ;)q ,\em
]Oqm{s X:!pU! I' .JO .{l,jdS!P ":ljdlUI,X:l .Hl:[ "l('illllUI(l
<:
zs !8t'69v'L sn
us 7,469,381 82
25
terminus of the list includes scrolling the !is! in the !irs!
direction nt n second nssocintcd scrolling speed. The second
associm<..-d scrolling sp<..-cd is slower than 1hc !irst nssociatcd
scrolling spc<..'d. For cxnmp!c. in FIG. 6C. displaying the area
3536 beyond the terminus of the list may include scrolling the
list at a spc'--d that is slower than the scrolling speed b c l l m ~ the
terminus is reached. In some embodiments. the second nsso-
cimcd speed is a fmction (e.g .. one-half or one-third) of the
!irs! associated spc-.xi. In some cmbmlimcnls. the second
associated speed is the square nml of the Jirsl associated
speed.
]fa terminus of the list is not reached while scrolling the list
in the !irs! direction while the object is still detected on or ncnr
the !ouch screen display. the process 500 is complc!c (510-
No, 512). The process 500 may he rc-initiatcd upon subse-
quent detection of another movement of an object on or ncar
the touch screen display (502).
26
upward and/or vc11ically downward swipe gestures on the
touch screen. In the example o!TIG. 6..\. a portion of a list of
emails is displayed in the screen area. including a top dis-
played email 3530 from Bmcc Walker and a bottom displayed
.< email 3532 !ium Kim Brook. :\ user pcrl(mJJs a \crtically
downward swipe gesture 35141o scroll toward the lop of the
list. The vertically downward gesture 3514. which may be a
linger gesture. corresponds to the movement of an obj<..'CI on
or ncar the !ouch screen !hal is dekc!cd in operation 502 of
lCJ process 500 (FIG. 5). The vc11ically downward gesture 3514
nL"--d no! he exac!ly vc11ical: a substantially \erlical gesture is
suflicient. In some embodiments. a gesture within a predeter-
mined angle of being pcrlCctly vertical results in vertical
scrolling. In one embodiment. a gesture within 27 degrees of
l'i being per!L'Ctly \erlical results in \ertical scrolling.
As a result of dck'Ciing the vertically downward gesture
3514. in FIG. 6B the displayed cmails have shilk-d down.
such that the previous boll om displayed cmai13532 from Kim
Brook is no longer displayed, the previous lop displayL-d
"" email 3530 limn Bmce Walker is now second !i-om the lop.
and the email 3534 li"om .\aron Jones. which was not dis-
played in 1-'lG. 6..\. is now display<..'<im thetopofthe list. This
shilling of emails is an example of the scrolling dcscrihL-d in
1-'lGS. 6..\-6]) illustrate the scrolling of a list of itcms to a
lcrminus of the lis!. a! which point an area h:yond the lcJmi-
nus is displayed and the list is then scrolled in an opposite
direction until the area beyond the terminus is no longer
displayed. in accord.1ncc with some embodiments. While
Fl(iS. 6:\-6]) illuslralc this scrolling in the context of a por-
table mul!i!i.mclion de\ icc I 00, this scrolling is not limited lo
pm1ahlc muhili.mclion devices. In the example of Fl(iS. ""
6.\-6D. the listed items arc email messages: FIGS. 6.\-6])
illustrmc an cxcmplmy user intcrlUcc 3500.\ for managing an
inhox in accordance with some embodiments. An analogous
user inlcrliJCc may he used lo display and manage other mail-
boxes (e.g .. dralis. sent. trash. personal. etc.). In addition.
other types of lists arc possible. including but not limited to
lists of install! message conversmions. favorite phone num-
bers. conlacl in!lmnalion. lahds. email ll1ldcrs. email
addresses. physical addresses, ringloncs. album names or
bookmarks.
operation 504 of process 500 (FIG. 5).
In this example. the email 3534 limn Aaron Jones is the
!irs! email in the list and thus is the terminus ofthc list. Upon
reaching this cmai13534. in response to continued detection
of the vc11ically downward gesture 3514. an area 3536 (FKi.
6C) above the lirsl email 3534 (i.e .. beyond the k'rminus of
30 the list) is displayed. as described in opcrmion 514 of process
500 (1-'lG. 5). In some cmbodimems. the area displayed
beyond the tcnninus of the list is visually indistinctli"om the
background of the list. as dcscrihL-d in operation 518 of pro-
cess 500 (FKi. 5). In FKi. 6C. both the area 3536 and the
35 background of the cmails (e.g .. cmails 3534 and 3530) arc
white and thus arc visually indistinct. In some embodiments. user interface 3500..\ include the
ll1llowing dements. or a subset or supcrsctthcrcof:
402. 404, and 406. as described ahmc:
Once vertically downward gesture 3514 is complete. such
a creak email icon 3310 that when acliva!cd (e.g .. by a
linger tap on the icon) initiates display of a Ulto create 40
a n<..'W email message:
!hal a cmTcsponding object is no longer dc!ec!L-d on or ncar
the !ouch screen display. the list is scrolk-d in an opposik'
dir<..'Ction umilthe area 3536 is no longer displayed. FIG. 6])
illustrates the result of this scrolling in the opposite direction.
mailboxes icon 3502 !hal when acliva!cd (e.g .. by a linger
lap on the icon) initiates the display of a Ul listing email
mailboxes (i.e .. ll1ldcrs):
which corresponds lo operation 520 of process 500 (FKi. 5):
the email 3534 li"om Aaron Jones is now displayed a! the lop
of the screen area alloi!L-d lo the list and the area 3536 is no!
unread messages icon 3504 thm displays the mnnbcr of 45 displayed.
unread messages in the inbox:
names 3506 of the senders of the email messages:
suhjcc!lincs 3508 !l1r the email messages:
dates 3510 of the email messages:
In the example of FIGS. 6..\-6]). a vertically downward
gesture resulted in display of an area beyond the lirsl item in
the list. Similarly. a \erlically upward gesture may result in
unread message icons 3512 that indicate messages thm '0
have not been opened:
display of an area beyond the last item of the list. if the
vertically upward gesture continues once the list has been
scrolled to the last item. The last item may be considered a
pr<..'\ i<..w pane separator 3518 that separates the list of mes-
sages !ium a pr<..'\i<..w of a sekc!cd message in the list:
scllings icon 3520 that when activated (e.g .. by a linger tap
on the icon) initimcs the display of a Ul to modify '5
lcJminus of the list. similar lo the lirsl item. As discussL-d
ahmc. the gesture need no! he exac!ly \crtical!o result in
vertical scrolling. a gesture within a prcdelincd range of
angles li"om per!Cctly vertical is sullicicnt.
scllings:
move message icon 3522 thai when acliva!ed (e.g .. by a
linger lap on the icon) initiates the display of a Ul lo
move messages:
Del etc symbol icon 3524 that when activated (e.g .. by a
linger lap on the icon) initiates display of a Ul!o con linn
!hal the user wants lo dde!c the sekc!cd email:
Reply/Forward icon 3526 thm when activated (e.g .. by a
linger tap on the icon) initiates display of a Ulto sckct
how to reply or lixward the sdcck-d email:
If the list ofcmails lills more than the allollcd screen area.
the user may scroll through the cmails using \erlically
In some embodiments, instead of scrolling a list of items in
one dimension. a user may desire lo translate an electronic
document in two dimensions. If the ekc!ronicdocumcnl lills
more than the screen area allolled to display the document.
60 the screen will only display a portion of the document. The
user may translate the ekc!ronic document lo viLw pm1ions
of the document thai are no! initially displayed.
1-'lG. 7 is a !low diagram illustrating a method 700 of
translating an electronic document in accordance with some
65 embodiments. The method 700 provides a simple visual indi-
cator lo a user !hal one or more edges of an dec!nmic docu-
ment are being display<..-d.
us 7,469,381 82
27
Mon::mcnl of an objccl is dc!cc!cd on or ncar a !ouch screen
display of a device (702). In some embodiments. the obj< .. ><::t is
a !ing.cr. In some embodiments. the device is a ponab!c mul-
tili.mction device.
28
associated translating distance is less than a distance ofmme-
mcnl of 1he object nlicr reaching. 1he edge of 1he ck><::lronic
documem. For exnmplc. in FIG. 8C. allcr the cdg.c is reached
the web pag.c 3912 is translated by a dislnnce indicmcd by
In response to dc!ccling the mo\cmcnl. an dcc!nmic docu-
ment displayed on the touch screen display is lranslatcd in a
!irst direction (704 ). In some embodiments. the electronic
document is a web page. us i!lustrntcd in FIGS. 8..\-8]). In
some embodiments. the dcc!ronic document is a digital
image. In some embodiments. the ck:clronic document is a
word processing. sprcadshccl. email. or pn::scnlalion docu-
ment. In some embodiments. the !irst direction is a vertical
direction. a horizontal direction. or a diagonal direction. In
some embodiments. the Jirsl direclion corresponds lo the
direclion of movement of the objccl dc!ec!ed on or near the
disp!ny bll! is no! necessarily idemica! to the din.><::tion of
movement of the objecl.
5 opposing arrows 302R-I and 302R-2. which may be less than
a distance traversed on the !ouch screen display by the swipe
g.cslurc 3925 nlicr 1hc 1cnninus is renched.
In some cmbodimems. translating in the Hrsl direction
prim lo reaching an L'tlgcofthc ek><::lronicdocumcnl has a !irs!
lCJ associated translating speed that corresponds loa speed of
mmemenl of the objccl. For example. a spcL'tl of translation
prior 10 renching !he edge of !he web pag.e 3912 shown in
FIGS. 8:\-SD may correspond to a speed of movcmemofthc
swipe gesture 3925. Displaying an area beyond the edge of
15 the elcc!ronic document includes translating the ek><::lronic
documem in the !irs! din.><::lion m n second nssocintcd 1rnns-
ln some embodimems.lranslaling 1he electronic documem
in the Jirsl dircclion prim lo reaching an L'tlgeoftheck><::lnmic
document has an associa!L'tl speL'tl of translation correspond- ""
in g. to n speed of movcmem of the object (706 ). In some
embodiments. the electronic document is 1ransln!cd in accor-
dance with a simulation of an equation of motion ha\ing
friclion (708).
lming. sp<.-cd. The s<.><::ond nssocintcd 1rnnslming speed is
slower than !he !irs! nssocintcd 1rnns!nting. speed. For
example. in FKi. 8C. displaying the area 3930 beyond the
edge of the web page 3912 may include translating the web
page 3912 ala speed that is slower than the speed of transla-
tion bclixc the edg.e is rcnchcd. In some embodiments. the
SL><::ond associated spL"L'tl is a Ji-aclion (e.g .. one-half or one-
third) of the !irs! associated spL"L'tl. In some embodiments. the
!fan edge of the elcc!ronic document is reached (e.g .. upon
renching the edge of the documem) while 1rnnslming the
c!ec1ronic document in the Hrsl din.><::lion while !he object is
still de!L><::IL'tl on or ncar the !ouch screen display. an area
bL")'tmd the L'tlge of the elec!nmic document is displayed
(710-Ycs. 714). In some embodiments. the area beyond the
cdg.c of1hc c!cc1ronic document is black. g.ray. a solid color.
"5 SL><::ond associated speed is the square roo! of the Jirsl associ-
mcd sped.
If an cdg.c of the electronic documelll is no! reached while
translating thedec!nmic document in the Jirsl direclion while
the objecl is still detcc!L'tl on or near the !ouch screen display.
30 the process 700 is complete (710-No. 712). The process 700
may be re-ini1intcd upon subs<.'<.juem deJection of nnothcr
movement of nn object on or ncar the touch screen display
(702).
or while (716). In some embodimems. the area beyond the
L'tlge of the elec!ronic document is visually distincl Ji-om the
document (718). I'm example. the area 3930 b9ond the L'tlge
of1he web png.c 3912 in FIG. 8C is black. in comrnsllo the 35
while bnckg.round of the web pngc 3912. In some o1her
embodiments. a wn!!paper imag.e such ns n picmrc or pnnern
may be displayL'tl in the area beyond theL'tlgc oftheck><::lnmic
document.
Aller 1he object is no !ong.cr de1cctcd on or ncar 1he touch 40
screen disp!ny. the c!cc1ronic documem is 1rnnslm<.-d in a
s<.><::tmd dircclion until the area bL")'tmd the edge of the el<.><::-
lronic document is no longer displayL'tl (720). l'mcxample. in
FJ(i. 8]) the web page 3912 has been translated such that the
area 3930 beyond its cdg.c is no longer displnycd. In some 45
embodiments. the s<.><::ond direction is opposite 1he Hrsl din.><::-
tion. In some embodiments. the elec!nmic document is lrans-
la!L'tl in the second dircclion using a damped motion (722). In
some embodiments. the chnnge from translating the ck><::-
1ronic document in the Hrs1 din.><::lion 10 1rans!nting. 1he ck><::- '0
Ironic documem in 1he second direction umil1he nrcn beyond
the L'tlge of the ek><::lronic document is no longer displayed
makes the edge of the elcc!ronic document appear lo be
clns1ica!!y mwchcd to nn cdg.c of 1he touch sen.-en disp!ny or
10 nn edg.e displayed on the 1ouch screen display (724). '5
In some embodiments. translating in the Jirsl dinxlion
prim lo reaching an L'tlgeoftheelec!nmic document has a Jirsl
associa!L'tl!ranslating distance that corresponds loa distance
of movcmem of the object prior to reaching. the edg.e of the
c!ec1ronic documenl. For example. n distance oflranslationof 60
the web page 3912 shown in FJ(iS. 8A-8D prim lo reaching
the edge of the document may correspond loa distance Ira-
versed on !he !ouch screen disp!ny by the swipe gcsmre 3925
before the cdg.c is renched. In some embodiments. displnying
an area beyond the cdg.c of the c!cclronic document includes 65
translating the elcc!ronic document in the Jirsl dircclion ll1r a
SL><::tmd associa!L'tl translating distance. wherein the second
FIGS. 8A-8D illuslralc the translation of an ek><::lronic
documemto an edge ofthc documem. m which point nn nrcn
beyond the <.-dge is displny<.-d and 1hc documcm is then 1rnns-
lmcd inn second din.><::lion until 1hc area beyond 1he edg.e of
!he document is no longerdisplayL'tl. in accordance with some
embodiments. While FJ(iS. 8:\-8]) illuslralc this translation
in 1he contcx1 of n porwblc muhili.mction device 100. this
translation is notlimi1<--d to portnb!c muhili.mc1ion devices. In
the example of FIGS. 8:\-8]). the document is a web page
3912: FKiS. 8A-81) illuslralc an exemplary user intcrli1ce ll1r
a browser in accordance with some embodiments. :\n <malo-
g.ous user intcrJUcc may be us<.-d 10 display o1hcr types of
electronic documems. such ns word processing.. spreadshcel.
emaiL prescnlalion documents. or digital images.
In some embodiments. user inlerbce 3900:\ of FIGS.
8:\-SD includes the following. c!emcms. or n subset or super-
sci !hereof:
402. 404. and 406. as describ<.-d above:
Previous page icon 3902 that when acliva!L'tl (e.g .. by a
linger lap on the icon) initiates display of the pre\ious
web pag.c:
Web pag.e name 3904:
Next page icon 3906 !hal when aclivated (e.g .. by a linger
lap on the icon) initiates display of the next web page:
URI. (Unillmn Resource I ocalm) entry box 3908 ll1r
inpulling URI .s of web pag.es:
Refresh icon 3910 1hnt when nc1ivated (e.g .. by n Enger tap
on the icon) initiates a reli-esh of the web page:
Web page 3912 or other slmclurL'tl document. which is
mndc of blocks 3914 of1ex1 content nnd other grnphics
(e.g .. imag.cs):
Senings icon 39161hm when nc1ivated (e.g .. by a Hng.cr tap
on the icon) initiates display of a sellings menu ll1r the
browser:
us 7,469,381 82
29
ISookmarks icon .WI8 that when acli\ated (e.g .. by a linger
tap on the icon) initintes display of a bookmnrks list or
menu for the browser:
30
until the area J9JO is no longer displayed. I'Ki. 81) i!!uslrales
the result of this trnnslmion. wllich corresponds to operation
720 of process 700 (I'IG. 7): block 9 (3914-9) is now dis-
played in the lower right-hand corner of the portion of the
Add bookmark icon 3920 that when activmcd (e.g .. by a
linger lap on the icon) initiates display of aU! l(Jr adding
bookmarks: and
New window icon 3922 thm when activated (e.g .. by a
Enger tap on the icon) initiates display of a Ul for adding
nL'\\" windows to the browser.
In some embodiments. the tk'\ icc analy /CS the render tree
of the web page 3912 lo de!cJminc the blocks 3914 in the web
pag.c. In some embodiments. a block 3914 corresponds to a
render node thm is: replaced: a block: an inlinc block: or an
5 screen a!!oncd to display the web page J91 2 and the area J9JO
no! displayed. In some embodiments. the dircclion of trans-
lation is no! necessarily opposite lo the original direclion hut
may be in any direction such thm. upon completion of the
translation. the nrcn beyond the <..'dgc(s) of the ck><::tronic
lCJ document is no longer displayed.
FIG. 9 is a !low diagram illustrating a process 900 of
displaying an dec!nmic document hm ing a document length
nnd a documcm width. in accordance with some cmbodi-
in! inc table.
In I'Ki. 8A. the web page Ji!!s more than the a !lolled screen
area: only the left sides of block 7 (J914-7) and block 8
(J914-8) arc displayed and only the top !eli corner of block 9
(J914-9) is display<..'<!. To view the pania!!y display<..'<! blocks.
mcnts. "ll1c process 900 provides n simple visunl indicmor to
15
a user that an elec!ronic document is being disphtyL-d a! a
minimum magnilicalion (e.g .. the ek><::lronic document can-
no! he /oomL-d out and/or dcmagni!iL-d !i.1rlher).
a user may translate the displayed document by gesturing on
the !ouch screen in accordance with some cmhodimenls.
ln some cmbodimcms. in response to a substantially verti-
The process 900 is pcrllHmcd a! a dL'\ ice with a !ouch
"" scrL>en display In some embodiments. the device is n portnbk
mu!tili.mction device. In some embodiments. the ck><::tronic
cn! upwnrd (or downward) swipe gcsmrc by the user. the web
page (or. more generally. other elec!ronic documents) may
lransla!e one-dimensionally upward (or downward) in the
\ertica! dircclion. In some embodiments. a gesture is consid- "5
crcd substantially verticnl if it is within a prcdctcrmincdnngle
of being pcr!Cctly vertical. For example. in response to nn
upward swipe gesture by the user that is within a prcde!cr-
mined angle (e.g .. 27") of being pcr!Cc!ly \erlical. the web
pngc mny scroll onc-dimcnsionnlly upward in the venicn! 30
direction.
document is a web page (e.g .. web page J912. FJ(iS. IOA-
10C). In some embodiments. the dec!nmic document is a
digital image. In some embodiments. the electronic document
is a word processing. sprcndshcct. cmnil or prcscmation
documcm.
Thedec!nmic document is displayL-d (902) a! a !irs! mag-
ni!icalion on the !ouch screen display. A gesture is dc!ec!L-d
(904) on or ncar the !ouch screen display corresponding loa
conunand to zoom out by a uscr-spcciHcd amount. In some
cmbodimcms. the gesture is n pinching gcsmrc (e.g .. gesture
J951/J95J. FJ(i. lOA).
In response lo de!ccling the gesture. the dec!nmic docu-
ment is displayL-d (906) a! a magni!icalion less than the !irs!
magnification. For example. the web page J912 is shown m a
lesscrmngniHcation in FIG. lOB thnn in I'IG. 10.-\.
Conversely. in some embodiments. in response ton gesture
that is no! within a pn.-de!cJmined angle (e.g .. 27") of being
per kelly vertical. the web page may translate two-dimen-
sionally (i.e .. with simultaneous movement in both the verti- 35
en! nnd horizontal directions). For cxmnplc. in response to nn
upward swipe gesture by the user thm is not within a prcdc-
lermined angle (e.g .. 27") of being per kelly \crlica!. the web
page may translate lwo-dimcnsiona!!y along the din.:clion of
the swipe.
If the documcm length or document width is not cmirdy
displayed (908-No) while the gesture is sti!! dc!ec!L-d on or
near the !ouch screen display. the process 900 is comple!e
40 (910).
In the example of I'IG. 8.-\. nn upwnrd swipe gesture 3925
is no! within a predc!ermincd angle of being per!Cc!ly \erli-
cal. nlercll1rc. as a result of dc!ecling the upward swipe
gesture J925. the web page is lransla!L-d in two dimensions. In
this cxnmplc. the trans!ntion is approximmcly diagonal. FIG. 45
8B illustrates the result of tills trnnslmion: blocks 8 (J914-S)
and 9 (J914-9) arc now !i.J!!ydisplayed: blocks 1 (J914-1) and
2 (J914-2) are now only parlia!!y displayed. and block J
(3914-J) is no longer displnycd at all. This trnnslmion is nn
cxmnple of the trnns!mion described in opcrmion 704 of pro- '0
cess 700 (I'IG. 7).
In FJ(i. 81S. block 9 (J914-9) is in the lower right-hand
corner oft he web page J912: both the bol!om and right edges
of the web pngc have been rcnch<..'d while trnns!nting the web
pngc. Upon reaching these edges of the document. in '5
response lo continued deh.:clion of the upward gesture J925.
an area J9JO (FJ(i. 8C) beyond the hollmn and right edges of
the web page is displayed. In some embodiments. the area
disp!nycd beyond the cdgc(s) of an electronic documcm is
visually distinct from the document. ns described in opcrmion 60
718 of process 700 (l'l(i. 7). In l'l<i. 8C. the area J9JO is black
and thus is visua!!y distincl !i-om the while background of the
web page J912.
Once the upwnrd gesture 3925 is complete. such that a
corresponding obj<.><::t is no longer detected on or ncnr the 65
!ouch screen display. the web page J912 is lransla!L-d (e.g .. in
a direclion opposile lo the original dircclion of translation)
][however. the document length (e.g .. J957.1'1G. lOB) or
documcm width (e.g .. 3959. FIG. lOB) is entirely displayed
(908-Yes) while the gcsmrc (e.g .. 3951/J95J) is still detected
on or near the !ouch screen display. thedec!nmic document is
displayed (912) a! a magni!icalion wherein areas beyond
opposite edges of the electronic document (e.g .. areas 3955.
FIG. lOB) arc displayed.
In some embodiments. the nrcns beyond opposite L'<igcs of
the elec!ronic document include an area beyond a lop Lxige of
the document and an area h9ond a hollmn Lxige of the docu-
mcnt.ln some cmbodimcms. the areas beyond opposite edges
of the electronic document include nn nrcn beyond a right
edge of the documcm nnd nn nrcn beyond a !ell edge of the
document. In some embodiments. the areas h9ond opposile
edges oftheck><::lnmic document include an area beyond a lop
edge of the document. an arcn beyond n bot1om edge of the
document. nn nrcn beyond a right edge of the document. nnd
nn nrcn beyond n !eli edge of the documcm (e.g .. I'IG. 1 OB).
In some embodiments. the areas hL')"tmd opposile Lxigcs of
the ck><::lronic document are black. gray. a solid color. or
white. In some embodiments. the areas beyond opposile
edges of the electronic document nrc visunlly distinct from
the document. For example. the nrcas J955 (FIG. lOB) nrc
black and thus arc \isually distincl !ium the web page J912.
Upon de!ccling lenninalion of the gesture. the ekclronic
document is displayed (914) a! a magnilicalion wherein the
us 7,469,381 82
31
areas beyond opposite edges of the dcclnmic document arc
no longer displayed. For example. 1hc areas J955 arc not
displayed in FIG. lOC
!-'IGS. 10.\-IOC illustrate thcdisp!ayofan ck>ctronic docu-
ment at multiple magnilications in accordance with some 'i
embodiments. While FKiS. IOA-IOC i!!uslralc displaying
these multiple magni!icmions in the context of a portable
muhili.mction device 100. displaying these multiple magniH-
cations is no! limited lo portable muhilimclion de\ ices. In the
example of FJ(jS. IOA-IOC the document is a web page lCJ
3912: FKiS. I OA-1 OC (like !'!(iS. SA-81 )) i!!uslratcan cxcm-
plmy user interface for a browser in accordance with some
embodiments. An analogous user interface may be used to
display other types of dcc!ronic documents, such as digital
images or word processing. spreadsheet. emaiL or prcscnta- l'i
tion documents.
In FIG. 10.\. the web page 3912 is disp!ay<.'d at a !irst
magnification. The web page 3912 !ills more than the allotted
screen area: only the !dl sides of block 7 (3914-7) and block
8 (39I4-8) arc displayed and only the top !ell corner of block ""
9 (3914-9) is display<.'d.
32
gesture 3931/3933, a pm1ion of the web page 3912 is dis-
played at the prcddincd magnification. as illustrat<.'d in I'IG.
12C. and a portion oft he digital image 1302 is displayed at the
prcddincd magnification. as illustrated in FIG. 13C.
In some embodiments, immediately prim to detecting ter-
mination of the gesture, a last decreased pm1ion of the elec-
tronic document is displayed at a !irst rcso!mion. Upon
dct<:cting tcnnination of the gesture. the respective portion of
the electronic document is at a s<.>cond resolution
that is greater than the Jirst resolution.
I'J(jS, 12:\- I2C illustrate the display of an ck>ctronic docu-
ment at multiple magnifications in accordance with some
embodiments. While I'IGS. 12:\-12C illustrate displaying
these multiple magnifications in the context of a portable
multili.mction device 100, displaying these multiple magni!i-
cations is not limited to pm1ab!c nmltili.mction devices. In the
example of I'IGS. 12.\-12C. the document is a web page
3912: I'IGS. 12:\-12C (like FIGS. 8.-\-8])) illustrate an exem-
p!aty user intcrlitcc ll1r a browser in accordance with some
embodiments. :\n analogous user interbcc may be used to
display other types of electronic documents. such as digital
images or word processing. spreadsheet. email. or presenta-
tion documents.
In FIG. 12.\. a !irst portion of the web page 3912 is dis-
played at a Jirst magnilication. n1cwcb page 39I2 !ills more
than the allotted screen area: only the !ell sides of block 7
(3914-7) and block 8 (3914-8) arc display<.'d and only the top
Jell corner of block 9 (3914-9) is display<.'d.
In response to detecting a pinching gesture 3951/3953
(I:J(j. 1 0:\), the web-page is displayed at a magnilication less
than the Jirst magnilication. as shown in J'](j, lOB. !fa docu-
ment length 3957 or a document width 3959 is entirely dis- "-'
play<.'d while the gesture 395113953 is still detected. areas
3955 beyond opposite edges of the web page 3912 arc dis-
played. Upon detecting termination of the gesture 3951 13953.
the web page 39I 2 is displayed at a magni!ication wherein the
areas 3955 arc no longer displayed. as shown in I'IG. 1 OC.
I'IG. 11 is a How diagram illustrating a process 1100 of
displaying an electronic document at multiple magni !ications
in accon.bncc with some embodiments. n1c process !IOU
provides a simple\ isual indicator to a user that an electronic
document is being displayed at a maximum magnification 35
(e.g .. the electronic document cannot be zoomed in and/or
magnified Ji.1rthcr).
In response to dct<.><::ting a de-pinching gesture 393113933
30
(J'](j, I2:\). decreasing pm1ions of the web-page 3912 arc
displayed at increasing magni!ications compared to the mag-
nification shown in FIG. 12.\. For example. the portion of the
web p.1ge 3912 shown in FIG. 12B is smaller than and has a
'llJc process IIOO is pcr!(mJJcd at a device with a touch
screen display. In some embodiments. the de\ icc is a portable
multili.mction device. In some embodiments. the electronic 40
higher magnilication than the pm1ion of the web page 39I2
shown in J'](j, 12A.
In the example of FJ(j, 12B. the magni!ication exceeds a
prcddincd magnification. Upon dct<.><::ting termination of the
gesture 3931/3933. a portion of the web page 3912 is dis-
played at the preddincd magnilication, as illustrated in J'](j,
12C
l'!(jS, 13:\- I3C illustrate the display of an ck>ctronic docu-
ment at multiple magnifications in accordance with some
embodiments. While I'IGS. 13:\-13C illustrate displaying
document is a web page (e.g .. web page 3912. FIGS. 12.\-
I2C). In some embodiments, the electronic document is a
digital image (e.g .. digital image 1302. FJ(jS. I3:\-13C). In
some embodiments, the electronic document is a word pro-
cessing. spreadsheet. email or presentation document. 45
these multiple magni!ications in the context of a pm1able
multili.mction device 100, displaying these multiple magni!i-
cations is not limited to pm1ab!e nmltili.mction devices. In the
example oJTIGS. 13.\-13C. the document is a digital image
1302 that includes an image of a person 1304.
At least a !irst portion of the electronic document is dis-
played (1 I02) at a Jirst magni!ication. :\gesture is detected
(11 04) on or ncar the touch screen display corresponding to a
command to zoom in by a user-specified amount. In some
embodiments. the gesture is a de-pinching gesture (e.g .. '0
3931/3933. FIGS. 12.-\ and 13.\).
In I'IG. 13:\. a digital image 1302 is display<.'d at a !irst
magni!ication. In response to detccling a de-pinching gesture
393113933. d<.>creasing portions of the digital image 1302 arc
displayed at increasing magnifications compar<.'d to the mag-
nification shown in FIG. 13.\. For example. the portion of the
In response to detecting the gesture, decreasing portions of
the electronic document arc displayed (1 I 06) at increasing
magnifications. For example. in I'IG. 12B a dccreas<.'d portion
of the web page 3912 is displayed at a higher magnification
than the portion in l'!(j, 12:\, and in FJ(j, 131S a decreased
pm1ion of the digital image I 302 is displayed at a higher
magni!ication than the pm1ion in l'!(j, 13A.
,
5
digital image 1302 shown in FJ(j. 13B is smaller than and has
a higher magni!ication than the portion of the digital image
1302 shown in l'!(j, 13A.
If. upon detecting tcnnination of the gesture. the magnifi-
cation docs not cxcc<.'d a preddin<.'d magnification (1108-No ). 60
the process 1100 is cmnpletc (I 1 I 0).
IL however. upon detecting tennination of the gesture. the
magnification exceeds a preddin<.'d magnification
(11 08-Yes). a respective portion of the ckctronic document is
displayed (1112) at the pr<.'ddincd magnification. In the 65
examples ofl'!(jS, 121S and 13B. the magni!ication exceeds a
prcddim:d magnilication. Upon detecting termination of the
In the example of FIG. 13B. the magnification exceeds a
prcddincd magnification. Upon dct<.><::ting termination of the
gesture 3931/3933, a portion of the digital image 1302 is
displayed at the prcddined magnilication. as illustrated in
FJ(j, 13C.
I'IG. 14 is a !low diagram illustrating a process 1400 of
executing a screen rotation conunand in accordance with
some embodiments. The process 1400 provides a simple
visual indicator to a user that the user has not provided a
sullicicnt gesture to initiate a 00" screen rotation command.
-1poqm;, u1 zi I IIJ.'"Ons tpnoJ ;,tp uo
Sl UO!J1,JlU JO ;osu;os U! ptm lf:'()9I
sqmntjl 0.\\Jjll Ull!)P-10.! "UO]WJUJUO JWJ)JOd r, U]
ZOSI ;,ilem] JeJ!il]pJt]l I JJ],\JP Jt]l"\"91 . DLI U]
zo51 ::>iletm ]m]iltp ::>tJJ JJeJOJ oJ
Sl UO!J1,JlU UJ.'"OJ:lS SlljJ ":19I-V9I "S!)!:! JO ;,(dttmx;,
;,t]J UJ \ap -'"']q1,J.lOd OJ P-'"'J!lU!jJOII
OLL I .\.rom ;om UJ oIL I
mOJJ p;,JtDO( .\pJottnJ \;op JJom Jo ;,uo ;,pnpm
.\]]eUO!)dO .(em OLLI .\.Jouww
.')]!JejOA-UllU J.')l]lll Jll ,\Jom;,m l]Se[l
'jSip jeJ!)dO s;oJ!AJp J!).')U"i!em .')Jllm JO JUO se
.\.mm;,m ""'JIW]O.\-IIOU ;,pnpu! .(r,m \;op .\.rmu;,m
;oJ1,JS p1(0S moptmJ J:li(JO JO J;\IV"H
se tpns ,\Jom;,m mopueJ
OLL1 .\JomJV'i[ "05L1 (JJI,\;op ilunu!od J.')tjJO .10)
p.!tmq.\;o)\1' .\1nu OS]!' ()fl. I :l:l10]J""'JU! :ll[.!.
otL I.(t,]ds!pua.-..ns-tpnoJ ()fl. I :l:ll,J.!:lJU! J:lSl11'
()()l, I \;op ""''1.1. ua.-...\"\pq
-e::>mnumJoJ S]<.UJUOJ pue Jet]! (JJsdpp e p;,!Je:l
.\.!J!IlJJP ::>pnpm ,\em OZLI s;osnq uope::>mmu
-mo:J :ll[.!. TIII!p;,umnJ;oJm JOJ OZL I
utliJtDmnmmo:J ;omm Jo ;ouo ptm "OLL I "09L I
-JJJU! J;Jt]JO Jll 'j.!O.\\J;JU JJOm .!0 JUO "01LI
(S,fld.)) SJ!lHl "i!U!SS;JJOJd JJOm JO ;JUO s;,pnpm .(!JP-:lld.(J
00L1 JJIA;op Jti.L ;,]qe)Jod ;,q lOU p::>;ou 00L1 JJIAJ(] "SJUJm
!poqm;, ![J!!>\ II! ll.'):lJ:JS![:lllllJ 1'
![J!.\"\ OOL I :l:l! \;op )j:lO]q 1' s1 L I . !)1:1
09 UO!)eJ<.U UJJJJS Sltjl "00 I JJ!,\Jp UO!)JU!1J!)Jl1m J]I(P-l.!Od e Jll
lXJJUOJ Jl]l U! JJnJSJf! Ull!)P-10.! UJJJJS S]ljJ JWJJS!l!!! ;]9 I-\'91
"S!)I:l -'"'[l![i't\ "SJU;otmpoqma ;,mos t]J!.\"\ a:Jtmp.ro:J:J1' II!
IIO!JlWU u;,ans .\.rqdm;,x;, tm :19I-V9I "S!)I:l
. !)]:] II! 11.\"\0!(S S1'
st WSI ;,ilem] JeJ]iltp JtjlJO UO!)W!Juo J]eJuod
JtjJ t]ll.\\ p;,JnJ.""\XJ st Jjilue JJnJe JtjJ 01 JJ]Soddo Jjiltm ue
l(JI\\ ptmmtull:l IIO!Jl'JlU "80S I "i!U!JS!.\"\J
-IJ(lltu -'"'![J J:l.'"OPp OJ IIOd[] "(lSI "!)1:1 m
JtjJl]ll.\\ "fi<""\JilJ.""\XJ UO!)P-!O.!JO Jjiltm JJilJlJ Ue l]ll.\\ ptmUHUOJ
0> UO]WJ<.U UJJJJS \' "UO!)elllJ JO JJJf!Jp pJU!]JpJ.!d e fi<""\JJXJ lOU
s;,op WtJJ UO]JelllJ JO ;o..""\JilJp ilu]puodsJJJOJ e set] 80S I
"ilu!JSI\\J -'"'![J "J;o\;o.\"\m] .)S I "!)1:1 JO -'"'![J
II] "!)1:1 m 11.\"\0!(S II!
Sl Jilem] ]eJlfl]p Jl]l Jtjll]ll.\\ "PJlllJJXJ Sl ptmUHUOJ .\Jet\ ,\r,m ;JJ!\;,p ;Jt]l .(q p::>UUOJ.!Jd SUO!)JUI11
.10 UOIPtH1J JtjJ pue ;oJqepod ;,q wu .(em .10 ,\em .\\O]Jq OOL I
.'):JL\:lp S1' ":l:l!\;op :ll(.J. :llUOS l(J!!>\ .'):lut'p
m t]J!.\"\ ;o:JL\JP .\tm 110 p;,mn:J
-J;od :llJ .\1'm "()()I :l:l!.\;op
;Jjqeuod e Jll JXJJUOJ Jtjl U! SUO!)eJJdO Ull!)eJOJ pur,
uo!w]sueJJ ;]91-\"9 I pue :151-\"SI .)1
Vfi ".)ZJVZI ".JOI-VOI "CI8-V8 "CI9-V9 "S!)!:! :l(!ll/'t\
":19 I !)!:1 m l!.Wll]S "UlliWJU:l!JO Jll!!J.md OJ ZOSI
J"i!em! jeJ!illp JtjJ ,\qJJJtjl pJJilJJXJ Jjiltm JJHJe
Jl]l OJ JJ!SOddo Ull!)eJOJ JO JjilUe tm ljj!.\\ pueUJU!OJ Ull!)eJOJ
UJJJJS e }l-t09I pur, "J-t091 JtjJ Jll Ull!)
:l:lll() "'d9I "!)]:] m :ll]J l[ll\\ :lJll:l1'
.\q fA'"OJl,JOJ ( 19 I !)l:l m ZOS I Jl'J!"illp ;ot]J ":l]dmt,X:l
JO;] -p::>lllJJXJ S! UO!)P-JOJ JO Jj"i!ue JJnJe ue tjl!.\\ puem
-mo::> UO!WJOJ e uo!)eJ<.UJO JJJilJp fi<""\U!]JPJJd e pJJJXJ
JOU s;,op uopm0.1 JO JJJ"i!::>p Jtjl "JJAJ.\\lltj ]!
"!)J:I U! UO!J1,JII:l!JO
-puq OJ \'9I "!)]:] JO UO!J1,JII:l!JO JWJjJOd .')ljJ lUOJJ fh'"OJ1,JOJ
Z051 ;,"ilem! ]eJlil!P JtjJ JO "J]dumx;, JO:J p;,JnJJXJ
puemmoJ Ull!)P-10.! UJJJJS
0
()(i r, "UO!)eJ<.U Jll JJ.ril:Jp p::>U]jJp
-ard 1' sp.h'"O:JX:l UO!J1,JlU JO "ilmpuodsamn :lt]J .11
tpoq .H:J :lS!.\"\)j:lOJ:l.!:lJIIlllD ;,q 11!\"\ ZI I
l(JllllJ .')ljJ .(q UO!J1,JlU JO :lSII:lS .')l[l U.')l[l ".!:li(JO
moJJ pedep::>Aom JJe sqmmJJ Jt]J pue (\"9I "DLI U] Jtjilu
JJ.\\O] JtjJ uetjl JJt]WJ) ZTI UJJJJS tpnoJ JO Jt]ilu JJddn
;,tp UO .\[[l'!J!U! Sl qmm[J "(V9I "!)]:]II! :lP!S Jp] J:lddn
n I J.PJ jJ.\\o
1
""''lJ m1


-]Ul "J-t09I qmntjlJ] "JjdttmXJ .!O:J "Ull]WJOJ
-UilOJ e UO]WJ<.U JO Jl]l "SJUJm]poqm;, Jmos U]
UO!WJ<.U
0
()(i e "Snt].L "Ull!)P-10.! Jll JJJf!Jp fi<""\U!]JpJ.!d r,
SfAh'"O:lX.') UO!J1,JlU JO.').'"OJ"i!;op .')JllJ
-'"'![J vs I !J!:I .10 ""'ldumx;, ""''lJ 111
( s.ra"ilii!J O\\J ;,t]J .1 o
suo]ilJJ pewo::> JtjJJO .10 snnod JJJUJJ Jt]J u;o..""\.\\JJq
Ot SlXlJ Ue f,;,) JJilJSJil JJf!U!J!)]ntU Jl]l U] SJJf!U!J ll.\\J Jl]l
JO UJ.'"OJ:lS lj:lllOJ .'"OI(J Ull P1,JIIO:l.')ljJ UJ.'"O.\"\pq
tm JO UO!J1,JlU JO J.'"O.J1Jap 1' OJ Ullljr,JOJ JO J.'"O.J1Jap
JtjJ sw::>tmpoqm;, UJ uonewJ JO ;o..""\.ri!Jp
r, Setj 8051 .!0 9051 JJillSJil JJf!U!J!)]ntU Jli.L
tpnoJ Jtjl uo PJPJIJP st (.)5I . Di:O 8051 .!0(\"51 . Di:O
90S I "i!U!JSI\\J J.'"O"ilii!JIJjlllU \' "IIO!Jl,JU.')!JO 1' II!
ZOSI a"il1'm! Jl'JI"illp-'"'![J .)SI ptmVSI "S!)I:l Ill
wsi ;,ilemJ ]mtil]p e JU::>um::>op
::>moJPJ!J ::>tJJ :!si-V51 s!JU .10 ::>]dumx::> JtJl u1
0[ UOllJU!1J!)jntU Jji(P-l.!Od OJ pJ)llU!] JOU UO!WJOJ
"()()I .'"O:l! \;op -'"'[lJl'!-llld JX.')JUO:l ;otp m IIO!Jl,JOJ
I -vsi s!JI:l -'"'l!'V't\ ;,mos
l]ll.\\ JJtmp.!OJJlJ U] )JJfqo jej!il]p JJl]JO JO JUJUHlJOp J]UOJJ
'"
-J.""\jJ ue Jll JtjJ ilU!)eJOJ JWJJS!l!!! !151-\"51 "S!)Li
(:]9 I
ptm "lSI "S!)I:l II! ;ot]Jl]J!.\"\ 1!;,) -'"'["iltm
.')ljJ OJ -'"'J!SOddo UO!J1,JlU JO ;,(1\tm tm ljJI\\ ({)If:' I) fA'"OJll:l;)X.')
Sl puemmoJ UO]WJ<.U UJJJJS r, "JJillSJf! f!U!JSl.\\l JJf!UH!lJ nm Jl]l
lJ.""\lJp OJ ilU]SeJJ UOdfl "!!91 pue (!SI "S!)LJ U]
"" "a["iltm tm .\q P-'"'WJOJ S! CI9I ptm .JSI "S!)I:l
JO ZOSI ;,TI1,m1 (l'JITI!P -'"'l]J ;,Jdmt,xa Jll:] "(&Uti) paJn:Jaxa S!
(
0
()(i Uetjl ;,1) UO]WJ<.U Jll Jjiltm JJilJlJ tm l]l].\\ ptmUHUOJ
UO!)e)<.U UJJJJS r, "(ON-f:"Of:'T) UO!WJOJ JO JJ.ril:Jp fi<""\U!]JpJ.!d
e p::>JJXJ JOU uotWJOJ JO JJJil::>p ilmpuodsJJ.JOJ JtjJ J! "JJ!AJP JtjlJO ::>ptS'jJeq JtjJ ilmpBJUOJ .\q 00 I JJ],\JP
lt09I sqmnt]J mn]J .!:l![JO >'l
"".')"!) J.')ljJO JO :llUOS W
Jmos U] "UO!)P-10.! Jll JJJf!Jp JtjJ
JSd!]p JtjJJO JtjJJO Ull]JeJUJ].!O Jtjl U] Jf!Uetp JtjJ "SJUJm
!poqm;, :llUOS U] "(.)9I "!)]:] U! ZII lj:lllllJJO UlliJ.!lld
p.-..1\Jqll:l tm Ull S1' "!)!:!II! 1809I ;osdqp P1'JUlD fll
OJ V9I "!)]:]II! "]9()9I :lSd![[""' Pl'JUlD lUlUJ fA'"OJ:l.'"OPp
HSI "S!JI:l
U! "IIO!Jl,JU.')!JO OJ UO!WJ
-UJ]-10 JW.!JJOd e mO.!J PJJP-10.! V9I pur, \'51 "SDLJ.Ill W51
::>ileun JeJ!il]p Jt]l"J]dumx::> JO;] (90tT) pJJHJJXJ S] ptmunuoJ
IIO!JlWU u;,ans
0
{)() tOt I) IIO!Jl,JOJ JO :l.'"O.R!;op P-'"'II!J-'"'P
-aJd 1' utliWJOJ JO J.'"OJTiap ;,tp .11
(CI9I V9I "S!JI:l)
pur, "J-t09 I sqmntjl 0.\\l ,\q ilmJS].\\l
JJilUH!lJilm JtjJ Jmos UJ "UO]WJ<.U JO JJJf.Jp
;oJnJsa"il "ilmJS!.\"\J -'"'lll.
IIJ.'"O.JJS JOUO (ZOti) p.-..p;,pp S! (.)SI "!)]:]"80S I
.!0 "\'51 "!)LJ "9051 f!J) JJillSJil JJilUH!l]llm \'
Jq ,(r,m JSdl]JJ Jl]l JO S]XP- Ue Jll Ull]WJUJUO Jl]l U! Jf!Uetp
Jl]l "jeJ!)d!]p qmntjl JtjJ Jll eJJe )JBJUOJ Jl]lJl "JjdmeXJ
.!0:] ll.'):lJ:JS lj:lllOJ ;otpt]J!.\"\ qmmp :l![JJO l'!.'"OJ1' P1'JUlD
:ll(JJO IIO!Jl,JU.')!JO U! J"i!tmtp :li(J 1\U!JOJ!UOlU .(q p:lJ:l.'"Opp ,:
UO]WJ<.UJO JtjJ"SJUJtU!poqm;, Jmos U]
"Ull]WJOJ JO JJJf.Jp flmpuodSJJ.!OJ e
sqmmJl o.\\J JtjJJO UO]WJOJ Jti.L "(.)9 I D!:i
1!;,) Ullljr,JOJ UO!J1,JlU JO :lSII:lS :li(J "SJU.')UJ
t
"JJ]AJp Ull!)JUHJlljllm
J]I(P-l.!Od r, JJlAJp Jmos U] UJJJJS
lj:lllOJ ![JI\\ .'"O:l!.\;op 1' pamn:p;od Sl ()()f:'I SS.'"O:Jtud .'"Ol].].

zs !8t'69v'L sn
us 7,469,381 82
35
programs. modules. and data slmclures analogous lo the pro-
grnms. modules. and dma stmcturcs stored in the memory 102
36
of portable multifunction device 100 (!-'IG. 1). or a subset
thereof. !-'unhcrmorc. memO!)' 1770 may store additional
programs. modules. and data slnJclures (not shown) no! 'i
present in the memory 102 ofpm1ahk: mu!ti!lmction de\ ice
100.
5. rhccompulcr-implemen!cd method of claim I. wherein
the !irs! dir<.><::lion is a vertical direction. a horizontal dircc1ion.
or a diagonal dircc1ion.
6. The computer-implemented mc1hod of claim 1. wherein
the elec!ronic document is a web page.
7. rhccompulcr-implemen!cd method of claim I. wherein
the clcc1ronic documcm is a digiwl image.
Fach of the above idcntiHcd clements in !-'!G. 17 may be
stored in one or more of the previously mentioned memory
tk'\ ices. I Oach of the abmc idcntilicd mmlulcs corresponds to
8. The computer-implemented mc1hod of claim 1. wherein
the electronic documcnl is a word processing. spreadsheet.
lCJ email or presentation document.
9. rhccompulcr-implemen!cd method of claim I. wherein
the clcc1ronic documcm includes a list of i1cms.
10. The compmcr-implcmcmcd mc1hod of claim 1.
wherein the second dir<.><::lion is opposite 1hc !irs! direction.
I I. The cmnpuler-implemcn!cd mc!hod of claim I.
wherein translating in the Jirsl dir<.><::lion prim lo reaching an
edge of the documcm has an associated spc<.-d of translation
that corresponds to a speed of movement of the object.
a scl of inslmclions ll1r pcr!lmning a llmc!ion described
above. "111c above identified modules or progrnms (i.e .. sets of
inslntctions) need not be implcmcnt<..-d as separate soliwarc
programs. procedures or modules. and thus various suhsc!s of
these modules may he combined or olhcnvisc re-arranged in l'i
various cmbodimcms. In some embodiments. memO!)' 1770
may store a subset of the modules and data structures identi-
fied above. Furthermore. memO!)' 1770 may store additional
modules and data slruclurcs no! described abo\ c.
12. The compmcr-implcmcmcd mc1hod of claim 1.
"" wherein translating in the !irs! d i n ~ c l i o n is in accordance with
a simulation of an equation of motion hm ing friclion.
'll1c ll1rcgoing description. ll1r purpose of explanation. has
b<.-cn described with rcJCrcncc to sp<.><::i!ic cmbodimcms. l low-
ever. the illustrative discussions above arc not intended to be
exhaustive or lo limit the imcnlion lo the precise J(mns dis-
closed. Many modilicalions and variations arc possible in
\ i<."\\" of the ahmc teachings. The embodiments were chosen
and dcscrib<.'d in order to best explain the principles of the
invention and its practical applications. to thereby enable
others skilled in the artlo hcsluti!izc the invention and vari-
ous embodiments with \arious modilicalions as arc suited lo
1hc particulnr usc contemplated.
What is claimed is:
1. .\ computcr-implcmcmcd method. comprising:
a1 a device with a !ouch screen display
displaying a !irs! ponion of an electronic documcm:
dc!<.><::ling a mmuncnl of an ohjccl on or ncar the !ouch
screen display:
in response lo de!<.><::ling the mmcmenL translating the
electronic document displayed on the touch screen
display in a !irs! direction to display a second portion
of the elec!ronic document. wherein the second por-
tion is dilkrcnl Jimn the Jirsl pm1ion:
in response to an edge of1hc ck><::lronic document being
reached while translating the electronic document in
1hc !irs! dir<.><::lion while 1hc object is s1ill dc1cc1cd on
or ncar the !ouch screen display
displaying an area beyond the edge of the document.
and
IJ. The cmnpuler-implemcn!cd mc!hod of claim I.
wherein 1hc area beyond the <.'dgc of the document is black.
gray. a solid color. or white.
"-' 14. The cmnpuler-implemcn!cd mc!hod of claim I.
wherein the area h'-")tmd the edge of the document is visually
distinct from the document.
15. The compmcr-implcmcmcd mc1hod of claim 1.
30
wherein 1ranslating !he document in !he S<.><::ond direction is a
damped motion.
16. The cmnpuler-implemcn!cd mc!hod of claim I.
wherein changing from 1ranslating in 1hc !irs! direction 10
translating in the second direction umil1hc area beyond the
35
edge of the document is no longer displayed makes 1hc edge
oftheelec!ronic document appear lo heclaslicaHy allached lo
an edge of the !ouch screen display or loan edgedisplayL'<i on
the 1ouch screen display
17. The compmcr-implcmcmcd mc1hod of claim 1.
40
wherein 1ranslating in !he !irs! dircc1ion prior 10 reaching the
edge of the clcc!nmic document has a lirsl associa!cd trans-
lating distance that corresponds loa distance of movement of
the objcc1 prior 10 reaching the edge of 1hc clcc1ronic docu-
ment: and wherein displaying an area beyond the edge oft he
45
clcc!nmic document comprises translating the d<.><::lronic
document in the lirsl dircclion ll1r a s<.><::tmd associa!L'<i trans-
lating distance. wherein the second associated translating
distance is less than a distance ofmovcmcm of1hcobjcc1 alicr
displaying a 1hird portion of the clcc1ronic document.
wherein the third portion is smaller than !he !irs! -;o
portion: and
reaching the edge of the ck><::lronic documcm.
18. The cmnpuler-implemcn!cd mc!hod of claim I.
wherein translating in the Jirsl dircclion prior lo reaching the
edge of the clcc!nmic document has a lirsl associa!cd trans-
lating speed !hat corresponds 10 a spc<.-d of movement of the
obj<-><::1. and wherein displaying an area beyond !he edge oft he
in response lo dc!ccling that the ohj<-><::1 is no longer on
or ncar the 10uch scr<.-cn display. translating the
ck><::lronic document in a second dircc1ion until the
area beyond the edge of the electronic document is
no longer displayed lo display a Jlnu1h portion of
the ck><::lronic document. wherein the Jlnu1h pur-
lion is dilkrcnt from the !irs! ponion.
,
5
clcc!nmic document comprises translating the d<.><::lronic
document in the lirsl dir<.><::lion a! a second associa!L'<i trans-
2. The complllcr-implemcntcd method of claim 1. wherein
1hc !irs! portion of the electronic document. 1hc second por- 60
lion of the ck><::lronic documenL the third portion oft he d<.><::-
lronic document. and the Jlmrth portion of the ck><::lnmic
document arc displayed at the same magnification.
J. The complllcr-implemcntcd method of claim 1. wherein
!he movcmcnl of !he obj<-><::1 is on the !ouch screen display
4. Thecmnpuler-implemcn!cd mc!hodofclaim I. wherein
the ohjccl is a linger.
65
lating sp<.-cd. wherein the s<.><::ond associated translming speed
is slower than the !irs! associated translating speed.
19 . .-\device. comprising.
a !ouch screen display:
one or more processors:
memory: and
one or more programs. wherein the one or more programs
arc stored in the memory and con!igurcd 10 be executed
by the one or more processors. the programs including:
inslmclions ll1rdisplaying a !irs! portion of an d<.><::lronic
document:
us 7,469,381 82
37
instmclions !l1r dclccling a mmcmcnl of<m oh_icc! on or
ncar 1hc touch screen display:
instructions for 1ranslming. the electronic document dis-
play<..'<! on the touch screen display in a !irst din.><::tion
lo display a second portion of the ckx:tronic docu- 'i
mcnl. wherein the second portion is diflCrcnl !ium the
!irst portion. in response to detecting the movement:
instructions for displaying an area beyond an edge of the
ck:clronic document and displaying a third portion of
the dcc!ronic documcnl. wherein the third portion is lCJ
smaller than the Jirsl pm1ion. in response lo the edge
of the dcctronic document being reached while trans-
lating the electronic document in the !irst din.x:tion
while the ohjccl is still detcc!cd on or ncar the touch
screen display: and l'i
instructions for trnnslating the ck>ctronic document in a
second direction until the urea beyond the edge of the
electronic document is no longer display<..'d to displuy
a !(mrth portion of the dcctronic document. wherein
the !lmrth portion is difJCrcnt from the lirst pm1ion. in ""
response to detecting thm the object is no longer on or
ncar the touch screen displuy.
20. A computer readable storage medium hming stored
therein instructions. which when cxL>cutcd by a device with a
touch screen display. cause the de\ icc to:
38
display a Jirst portion of an dcctronic document:
detect a movement of an object on or ncar the touch screen
displuy:
trnnslatc the electronic document displayed on the touch
screen display in a Hrst direction to display u second
portion of the dcctronic document. wherein the SL>cond
portion is di!krcnt !ium the Jirst pm1ion, in response to
detecting the movement
displuy an urea beyond an L'<igc oft he electronic document
and displuy u third portion of the electronic document.
wherein the third portion is smu!!cr thun the Hrst portion.
if the L'<igc of the ck>ctronic document is reached while
translating the dcctronic document in the lirst dirL>ction
while the object is still dck>ctcd on or ncar the touch
screen displuy: und
trnnslatc the ck..::tronic dtlL'umcnt in a s<.>t."ond diw::tion
until the area beyond the edge of the electronic docu-
ment is no longer displayL'<i to display a Jlnu1h portion of
the electronic document. wherein the fourth portion is
dilkrcnt from the Hrst portion. in response to detecting
thm the object is no longer on or ncar the touch screen
display.
* * * * *
UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
PATENT NO. :7,469,381 B2 Page l of l
APPLICATION NO. : ll/956969
DATED :December 23,2008
!NVENTOR(S) : Oniing
It is certified that error appears in the above-identified patent and that said Letters Patent is
hereby corrected as shown below:
Column 38, line 8, please insert-- ; --after movement.
Signed and Sealed this
Seventeenth Day of February, 2009
JOHN DOLL
Acting Director of the United Suztes Patent and Trademark Office
,
.
'
. :.
.,
.
,.
,, ' .. - '
." --'
. ' . -; ..
...
-' ..
. r, .: ljif':f,:i . i
-.'
'
1. " ' ',. ,:- < ;.- ' '
1
--------------------------------------' :- . ; -'
.. ''
,;:
' ' -
.. ;-::..;;:












































Exhibit 5
(12) United States Patent
Platzer et al.
(54) .\PPLIC\TIO:'\ PROGH..Ui:\11\"G
I\"H:H.E\CES FOH. SCH.OLLI:'\G
OPERATIO:\S
111111 1111111111111111111111111111111111111111111111111111111111111
US007844915B2
(I OJ Patent No.:
(45) Date of Patent:
7,00'),62(, B2 3 200(i
B2 200(i
7.t 17,45.1 B2 10 200(i
US 7,844,915 B2
Nov. 30,2010
-\nw"r
Dm id ct ,]
ct ,]
(75) !mentors: Andn.w J>hltzt>r, Santa Clara. CA (US):
7.t7.1,62.1 B2 2 2007 Calkins al
7,3.17,412 B2 2 Guido ct al
Scott IIcrz. S.1nta C!arn. C.\ (US)
B2 SwmV ct al
Assignee: Applt> Inc .. Cupc11ino. CA (US)
( * ) Notice: Subject to any disclaimer. the term of this
pntcnt is extended or adjusted under 35
U.S.C. l54(b) by 5R3 days.
(21) App!. No.: 111620,717
(22) Filed:
(65)
J:m. 7,2007
Prior Publication I>ata
US 2(XlR/OlOR3R4 AI
(51) Int.Cl.
<ifJ6F 3/fJfJ
<ifJ6F 3/fJJ3
G06F 31041
G06F 31048
Jul. 10. 200R
(2006.01)
(2006.01)
(2006.01)
(2006.01)
(Continu<.'d)
FORJ]GN 1'.-\TI:N'r DOCUMJ:NTS
Ll' 3 2005
(Continued)
Or! II]{ I'UBIJC..\TIONS
Toshiyuki Ylasui ct al. "l:lastk !Or l'rcdsc Data
\hnipulation", 1')<)_';, -\C\1. pp 143-t44
(Continued)
(52) u.s.(']. 7151781.715/784:715/800:
345117".
Primar_r Ewminer Xiomnra L Bmnista
(74) Attornet'. Agent. or Firm IS!akely, Sokoloii ray lor &
/_afman I I.P
(58) Fidd of (:Jassitication St>arch 7151704.
(56)
7151705. 784, 7R6. 7XX. XOO, XM. X66, 973.
715/974: 345/156.157. J(i<).173
(57) AHSTH..\CT
S<.-c npplicntion H!e for complete senrch history.
Cited
A 7 J<J9(i !Jansen d al
.'i.<Jm.<Jo2 A 5 l<J9<J <lnct al
6JWHi02
'
2 2000 ,]
Bl ll 2002 Lbillos
6.677.9(i.'i Bl l 2004 Ulmann ct al
6.74!.')')6 Bl 5 2004 ct al
B2 1 2005
6.<Jm.<Jn B2 6 2005 Anlau!T
6.<J.'i7.392 B2 lO 2005 Simistcr ct al

Bt '' to 2005 \hlsushilact al
715
AI leas! ccJ1ain embodiments of the present disclosure
include nn environment with user inter!Uce soliwnre internct-
ing with n soliware npplication . .-\ method Jix operating
through an application programming intcrliJCc (API) in this
cnvinmmcnl includes trans!Crring a sci hounce cal!. The
method li.trther includes setting at least one of maximum nnd
minimum bounce vnlues. The set bounce call cnuses a bounce
of n scrolled region in nn opposite direction of n scroll bused
on a region pas! an edge of the scrolled region being visible in
a display region at the end of the scrolL
345 17."

"""""''0""'""""'
"
,,. ... ,,..,""',_,,....""""
"''-.''"'"
""'-'''"'
"
,,._,,. -""'""'''""'"'
"'""'""-' '"""' '"-'"
'"'"""'-"'"--'""""""''"'
"

r '""""o'"""""'';;.;,_,"'l
'''""""""-''""""'""'""'
-,.-""""-'-' -
,.._.,"""'""""'"''"""'"'"''"
'"''"' '"''""''""'"""
"
'""""'"'-"'."""" ... """' ""'-'- CWC.d.w-==0>
""''""""'"'"'"'"''-" '""'''"''"'-""'- , ...
"'''
.,
21 .n Sheets
U.S. PA:t-I;N"r J)OCUMI;NI-S
7561 l.'i9 B2 7 200<) Abel ct
7576.n2 B2 g 200'! Lii
2001 0045<)4<) _\] !1 2001 Chilh;unbaram d ,]
2002 019451>9 AI 12 2002 Cristolillo ct
2003 0095096 AI 5 2003 Robbin d
200_1 012271>7
"
7 200.1 al
200_1 0132<)-':;<)
"
7 200.1 ,]
2003 0160iG2 AI g 2003 Ridgky ct a]
200_1 017414<)
"
<)200.1 htiisaki d ,]
2004 002Hi76
"
2 2004 d al
2004 002J(i91> AI 2 2004 Baldwin ct al
2004 0100479 AI 5 2004 :\akano d al
2004 02!5643
"
!0 2004 d al
2004 0222<)<)2
"
!1 2004 C.!lkins d ,]
2004 0224(>31> AI II 2004 Fadel! ct al
2005 00."7524 _\]'' .12005 l!illd,]
2005 001>1>443
"
4 2005 al
2005 01930L'i AI 9 2005 Logston ct al
200(i 003i.:796 AI 2 200(i d al
200(> 01901>33
"
:; 200(> SanGiO\anni al
200(i 0D(i263 AI to 200(i Bathkhc d al
2007 OO.'i5967 At 3 2007 Po!T<:t al
2007 0075<)6-':;
"
4 2007 lluppi d "I
2007 01742-':;7
"
7 2007 llow;ml
2007 At 2007 Vkndisd al
2007 t At t I 2007 llolkmans d al
2007
"
12 2007 Butlind"l
2oog ooo5703 At 1 2oog Radivojcvk <:1 al
2oog oot(iO% At 1 2oog Wildingdal
2001> 003402<)
"
2 2001> hmgdal
2001> _\I'' 2 2001> Tr<:nl <:1 ,]
2oog 016g395 At 7 2oog <>rding d al
2001> 0231(>10 _\I'' <) 2001> llolcllingd"l
200<) 02-':;<)<)6<) _\I'' 10 200<) p,IJako!T
!'OR!;KiN PA:t-!;N"r !)OCUM!ON'!-S
GB 2 319 59! .\ .'i
GB 23!959! .\ .'i
US 7,844,915 B2
Page 2
tn
345 173
345 tn
715
345 173
715U>3
345 1_<.7
715g33
345 173

wo
\\'()
wo
wo
\\'()

pages
()Hie<:
pages
()Hie<:
pages
W<>-2006 0(>77 t I
W< > _\I
\V< > At $
\V<
\\"(
(i 200(i
7 2001>
7 2oog
7 2oog
7 2001>
en! !!OR
Adion. L s . \ppl :--:o t I 620.723 . maik'<l
_\elion, [; s -\ppl :--:o 11 620,709. m"ik'<l
.\pi' I
-\pr I
-\clion. t: s .\ppl :--:o 11 (>20.720. maikd .hm n.
< Adion. LS .\ppl :--:o II (>20.720. maikd 23.
pages
2009. g
200<),:;
200<), 17
lg
PCT lnl<:rnalional Sc,reh Rcpm1 and Wrill<:n (>pinion for I'CT lnl<:r-
nalional _\ppln :--:o CS2001> m"ikd .lui 31. (10
pages)
PCT lnl<:rnalional Sc,reh Rcpm1 and Wrill<:n (>pinion for I'CT lnl<:r-
nalional -\ppln :--:o 00001><), maikd -\pr ti, 2001> ( 14 pag<:s)
PC 1 International S<:ardl Rcpo11 and Written< >pinion !Or PCT Inter-
national .\ppln :--:o PCT 000103. maik'<l Jun 3. (15
pages)
PC 1 International S<:ardllkpot1 and Written< >pinion !Or PCT Inter-
national Appln :--:o PCT 000069. maik'<l Vlay 2. 200g (J(i
pages)
PCT lnl<:rnalional Sc,reh Rcpm1 and Wrill<:n (>pinion for I'CT Inter-
national Appin :--:o PCT l'S200g 000060. maik'<l Apr 22. (12
pages)
OJiie<: _\elion, C s .\ppl :--:o 11 ti20. 72_1, maikd Jun s. 2010. 7
pages
< Adion. l- S Appl :--:o It (>20.709. maik'<l Jun 9. 2010. 7
pages
Final Adion. l s Appl :--:o It 620.709 maik'<l :--:ov !3.
200<), 1> pages
Fim!l Ollie<: _\elion. l s .\ppl :--:o II 620,723, mailed :--.:"' 17.
2009. 10 pages
.\dion. L S Appl :--:o t I (>20.720 maikd :--:ov 2009. 17
pages
* cited by cxmnincr
U.S. Patent Nov. 30,2010 Sheet 1 of 37 US 7,844,915 B2
100
~
RECEIVE A USER INPUT
J!l2
j.
CREATE AN EVENT OBJECT IN
RESPONSE TO THE USER INPUT
104
..
DETERMINE WHETHER THE EVENT OBJECT
INVOKES A SCROLL OR GESTURE
OPERATION
1011
..
ISSUE AT LEAST ONE SCROLL OR
GESTURE CALL BASED ON INVOKING
THE SCROLL OR GESTURE OPERATION
100
j.
RESPOND TO AT LEAST ONE SCROLL CALL,
IF ISSUED, SCROLL A WINDOW HAVING A
VIEW ASSOCIATED WITH THE EVENT OBJECT
BASED ON AN AMOUNT OF A SCROLL WITH
THE SCROLL STOPPED AT A PREDETERMINED
POSITION IN RELATION TO THE USER INPUT
110
~
RESPOND TO AT LEAST ONE GESTURE CALL,
IF ISSUED, CHANGE A VIEW ASSOCIATED
WITH THE EVENT OBJECT BASED ON
RECEIVING A PLURALITY OF INPUT POINTS
IN THE FORM OF THE USER INPUT
111
FIG. 1
U.S. Patent Nov. 30.2010 Sheet 2 of 37 US 7,844,915 B2
200
rv
TRANSFER A SET BOUNCE CELL
2P2
+
SET AT LEAST ONE OF MAXIMUM AND
MINIMUM BOUNCE VALUES
~
+
CAUSE A SLIGHT BOUNCE OF A SCROLLED
REGION IN AN OPPOSITE DIRECTION OF A
SCROLL BASED ON A REGION PAST AN EDGE
OF A CONTENT BEING VISIBLE IN A DISPLAY
REGION AT THE END OF THE SCROLL
20Ji
FIG. 2
300
rv
TRANSFER A RUBBERBAND CALL TO CAUSE
RUBBERBANDING A SCROLLED REGION
DISPLAYED WITHIN A DISPLAY REGION
302
+
TRANSFER AN EDGE RUBBERBAND CALL TO
SET DISPLACEMENT VALUES FOR AT LEAST
ONE EDGE OF THE DISPLAY REGION
3Jl4
FIG. 3
U.S. Patent Nov. 30,2010
MOVEMEN
414
T
\
WINDOW
410 ..
___/
v
GUI
DISPLAY
408
I
'
!
I
10000\\
\
'
I
I
I
I
Sheet 3 of 37
/400



INFORMATION
4J2c1
INFORMATION
4J.U
INFORMATION
4123
INFORMATION
412,4



G
FIG. 4
US 7,844,915 B2
p
0
I
I
I
I
U.S. Patent
DISTANCE
512-1
'-
TERMINUS.
514
WINDOW.
410
Nov. 30,2010 Sheet 4 of 37
GUI
DISPLAY

408
I
\
11000

\



I

412-1
- - --
VELOCITY r-
510-1
'--
- - --
I
y
v
G
FIG. SA
US 7,844,915 B2
F
c
I
U.S. Patent
1ERMINUS ----
514
WINDOW
410
-
Nov. 30,2010 Sheet 5 of 37 US 7,844,915 B2
""
GUI
DISPLAY
/400
408
'
'
'
I -
11000\.
\
c
,




INFORMATION
;12-1
- - -
/
~
l/
8
FIG. 58
U.S. Patent
DISTANCE
512-1
TERMINUS
514
WINDOW
410
Nov. 30.2010 Sheet 6 of 37 US 7,844,915 B2
f'
GUI
DISPLAY
~ 4 0 0
408
'
i
'
'
ttooo
\
0
'
\
510-2
v VELOCITY
I
iNFORMATION
I
1JH
- - - -
- - - -
I
-
r
l/
8
FIG. SC
U.S. Patent Nov. 30,2010 Sheet 7 of 37 US 7,844,915 B2
. t

r 2oo 100
PORTABLe UlfiFUNCTION D"VICE

3500.A
[ SPEAKER
1.1__1_
) r OPTICAL
_ SENSOR 1P4
[ PROXIMITY
SENSOR ill
ro
(
0
N
L]4Q2
Current Time
404
G 106
;:::
(
. }3502
3504
ro
In box
/
0
N
Marlboxes _
0
L.
3512-1 3506-2 3510-2
Bruce January 30,
Lunch meeting today.d
508

2
K. B k 35063 35103
rm roo
January 30, 2006'
3508-3
Draft agreementv
Bob Adams
I
I January 29, 2006
I
Project Orion f---.. 3514
35122
I

Darin Adler January 29, 2006
fantasy football
,,
Aaron Jones January 29. 2006
Fwd: status report
Kim Brook January 28. 2006
Re: proposal 3532
\
" j 3518
CD
w
0
9
LZJ
3020
352.1
35.2.4. 35Zll
351_Q
(MICROPHONE)
111 204
FIG. 6A
U.S. Patent Nov. 30,2010 Sheet 8 of 37 US 7,844,915 B2
2!1(; I
100

PORTABLE MULTIFUNCTION DEVICE
-
3500A
( SPEAKER
111
J l OPTICAL l PKOXIMiTY
SENSOR 164 SENSOR 166
co:
)
0
N
..::::J402
Current Time
404 IIJ> 406
-
(
ysoz


Mailboxes
In box
O'
N
-
Aaron Jones J
506
'
1
3510-1
R p . O . 3508-1
January 30,
e: rowct r1on
3534
35121 35062 35102
Bruce _ January 30,
Lunch meeting today.:l'
08
-
2
353D
K' B k 35063 I 3510-3
1m roo 1 January 30,
D I 35083 I
rat
Bob Adams
_: January 29, 2006
Project Orion
\I
I
3512
.
2
Darin Adler
fantasy football
January 29, 2006
Aaron Jones January 29, 2006
Fwd: status report
\ "f /c 3518
(j)
rn
0
9
[ZJ
)520
3522 )544
:ill.6.
3510
(MICROPHONE
ill
@
4
FIG. 68
U.S. Patent Nov. 30,2010 Sheet 9 of 37 US 7,844,915 B2
I I
100

PORTABLE MULTIFUNCTION DEV'CE
-
3500A
( SPEAKER
111
) [ OPTICAL d [ PROXIMITY J
SENSOR 164 SENSOR 1Q
rol
)
0'
N'
L:::l1DZ
Current Time
404 l:b 406
.::::
(
r502

In box
'
rol
Mailboxes

3536.
Aaron Jones -2
506
.
1
3510-1
R p . O . 35081
January 30,
e: reject
3512-1 3506-2 3510-2
Bruce January 30,
Lunch meeting today-2
508

2
353il
K' B k 35063 I 35103
Jm r00 1 January 3Q,
3508-3 1
Draft r..-3514
Bob Adams
January 29, 2006
Project Orion
\ 1
I

Adler January 29, 2006

fantasy football
\
,. J 3518
CD
GJ
0
9
[ZJ
3520
3522 3524 352(;
3.51Q

111 204
FIG. 6C
U.S. Patent Nov. 30,2010 Sheet 10 of 37 US 7,844,915 B2
PORTABLE MULTIFUNCTION DEVICE
I zoe I
100

-
3500A
[ SPEAKER
1L1
) ( OPTICAL IJ l PROX>MITY J
SENSOR 164 SENSOR 166
00>
)
a.
L::iJ!l

Current Time 4Q4 .:P 406
-
(
y5o2
(y504
51 Mailboxes
In box
N
-
Aaron Jones

3510-1
R p . I O . 35081
January 30,
e: ro)eC
3534
3512-1 3506-2 3510-2
4J Bruce January 30,
Lunch meeting

J530
Kim Brook

3510-3
January 30,
Draft agreement

Bob Adams
January 29, 2006
Project Orion
3512
-
2
Darin Adler
fantasy football
January 29, 2006
Aaron Jones January 29, 2006
Fwd: status report
\ ... __,{.3518
(jJ
rn
0
9
[2]
3520
3522 3524 3526
3510


4
FIG. 60
U.S. Patent Nov. 30.2010 Sheet 11 of 37 US 7,844,915 B2
700
~
TRANSFER A DIRECTIONAL SCROLL CALL
TO DETERMINE IF DIRECTIONAL SCROLLING
IS ENABLED
!02
~
TRANSFER A DIRECTIONAL SCROLL ANGLE
CALL TO SET A SCROLL ANGLE FOR LOCKING
THE SCROLLING IN AT LEAST ONE OF A
VERTICAL OR A HORIZONTAL DIRECTION
704
~
LOCK THE SCROLLING IN THE HORIZONTAL
DIRECTION IF A USER INPUT FORMS AN
ANGLE WITH A HORIZONTAL DIRECTION THAT
IS LESS THAN OR EQUAL TO A FIRST SCROLL
ANGLE
706
~
LOCK THE SCROLLING IN THE VERTICAL
DIRECTION IF A USER INPUT FORMS AN
ANGLE WITH A VERTICAL DIRECTION THAT
IS LESS THAN OR EQUAL TO A SECOND
SCROLL ANGLE
708
FIG. 7
U.S. Patent
---
---
Nov. 30.2010
--
sOB;- --- ..
'
'
'
'
'
'
'
'
--
Sheet 12 of 37
'
'
'
'
'
'
810'
. '
(:
' '
' '
' '
' '
' '
' '
' '
" --
---
---
---
US 7,844,915 B2
---
---
--
1\ ----- '--806
---
--
--
---
'
'
I l --.._
--
1 \ ---
1 l --
' '
' '
' '
'
'
'
'
'
1
812
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
804
FIG. 8
802
U.S. Patent Nov. 30.2010 Sheet 13 of 37 US 7,844,915 B2
900
~
TRANSFER A DECELERATION SCROLL CALL
TO SET A DECELERATION FACTOR FOR A
DRAG USER INPUT THAT I 'NOKES A SCROLL
wz

SLOW THE SCROLL TO A STOP BASED
ON THE SPEED OF THE DRAG USER INPUT
904
FIG. 9
1000
~
TRANSFER A SCROLL HYSTERESIS CALL TO
DETERMINE WHETHER THE USER INPUT
INVOKES A SCROLL
1002 .

SET THE HYSTERESIS VALUE FOR
DETERMINING WHETHER A USER INPUT
INVOKES A SCROLL
1004
FIG. 10
U.S. Patent Nov. 30.2010 Sheet 14 of 37 US 7,844,915 B2
1100
~
TRANSFER A SCROLL INDICATOR CALL TO
DETERMINE WHETHER AT LEAST ONE
SCROLL INDICATOR ATTACHES TO A SCROLL
REGION OR A WINDOW EDGE
11Q2
+
OPTIONALLY ATTACH SCROLL INDICATORS
TO A SCROLL REGION BASED ON
THE SCROLL INDICATOR CALL
1104
+
OPTIONALLY ATTACH SCROLL INDICATORS
TO A WINDOW EDGE BASED ON
THE SCROLL iNDICATOR CALL
:!JQ
FIG. 11
1200
~
TRANSFER AN INADVERTENT USER INPUT
CALL TO DETERMINE WHETHER THE USER
INPUT WAS INADVERTENT
1202
+
IGNORE THE INADVERTENT USER INPUT
BASED ON THE INADVERTENT USER INPUT
CALL
1204
FIG. 12
U.S. Patent Nov. 30.2010 Sheet 15 of 37 US 7,844,915 B2
1300
~
TRANSFER A HANDLE GESTURE EVENT CALL
1JQ2
+
TRANSFER A GESTURE CHANGE CALL IN
RESPONSE TO THE HANDLE GESTURE
EVENT CALL
1])1
FIG. 13
1400
~
TRANSFER A SCALING TRANSFORM CALL
TO DETERMINE A SCALING TRANSFORM
FORA VIEW ASSOCIATED WITH A USER
INPUT HAVING A PLURALITY OF iNPUT POINTS
llil2
+
TRANSFER A SCALING GESTURE START
CALL
1404
+
TRANSFER A SCALING GESTURE PROGRESS
CALL
.1406
+
TRANSFER A SCAL!NG GESTURE END
CALL
Hall.
FIG. 14
U.S. Patent Nov. 30,2010 Sheet 16 of 37
1502
\
~
~ ~ - - - - - - -
---. 1508
.r-J


1 5 1 ~ ' , \
\, :
'
'
'
'
/
'0::--""-_,;_.-- ~
'
,'
FIG. 15
US 7,844,915 B2
1602
)
_________ / ________ ,
1604
)
_[
1608--.l




{)
1612--,
-X
1610--.
-0
-------------------
FIG. 16A
1656
1660-
1658-
1650
)

1652
---{)
0 1664
'
'
'
'
'
'
'
1654
-x

0'
-o

'
'
'
'
'
'
'
'
'
'
'
'
'
' '
l------------------
FIG. 168

'JJ


"" ;-
=
...
:2
0

w
.o
"'
0
- 0
00
""
a
-
_,
0

w
_,

"'
__,

_..
-;,
-
'-"
s
U.S. Patent Nov. 30.2010 Sheet 18 of 37
..,.
,__
~ ,
I
b
US 7,844,915 B2
-
u.
U.S. Patent Nov. 30.2010 Sheet 19 of 37 US 7,844,915 B2
1700
rv
TRANSFER A ROTATION TRANSFORM CALL
TO DETERMINE A ROTATION TRANSFORM
FOR A VIEW ASSOCIATED WITH A USER
INPUT HAVING A PLURALITY OF INPUT POINTS
1702
~
TRANSFER A START ROTATION GESTURE
CALL
1704
~
TRANSFER ROTATION GESTURE
PROCESS CALL
lTOJl
~
TRANSFER A ROTATION GESTURE END
CALL
1708
FIG. 17
U.S. Patent Nov. 30.2010 Sheet 20 of 37 US 7,844,915 B2
NO
1800
r - - - - , ~
NO
NOTIFY
DELEGATE
181Q
FIG. 18
NO
YES CALCULATE NEXT
STATE
1806
U.S. Patent Nov. 30.2010 Sheet 21 of 37 US 7,844,915 B2
1900
rv
START AT LEAST TWO ANIMATIONS
190.2
~
DETERMINE THE PROGRESS OF EACH
ANIMATION
1904
+
UPDATE EACH OF AT LEAST TWO
ANIMATIONS BASED ON A SINGLE TIMER
liQ!i
FIG. 19
U.S. Patent Nov. 30.2010 Sheet 22 of 37 US 7,844,915 B2
2000
~
PROVIDE A SINGLE ANIMATION TIMER
2002
~
ANIMATE A PLURALITY OF ANIMATIONS
WITH THE SINGLE ANIMATION TIMER
2 0 Q ~
FIG. 20
U.S. Patent Nov. 30.2010 Sheet 23 of 37 US 7,844,915 B2
2100
rv
SET ATTRIBUTES OF VIEWS
INDEPENDENTLY WITH EACH VIEW BEING
ASSOCIATED WITH PROCESS
2102
~
TRANSFER A SYNCHRONIZATION CALL
TO SYNCHRONIZE ANIMATIONS FOR THE
MULTIPLE VIEWS OF THE DISPLAY
2104
~
TRANSFER A SYNCHRONIZATION
CONFIRMATION MESSAGE WHEN A
SYNCHRONIZATION FLAG IS ENABLED BASED
ON THE LIST OF THE PROCESS BEING
SYNCHRONIZED
2106
+
UPDATE THE ATTRIBUTES
OF THE VIEWS INDEPENDENTLY
2106
+
TRANSFER A START ANIMATION CALL TO
DRAW THE REQUESTED ANIMATIONS
2110
FIG. 21
U.S. Patent Nov. 30.2010 Sheet 24 of 37 US 7,844,915 B2
2200
2210
FIG. 22A
. 2200
2.2.10
FIG. 228
U.S. Patent Nov. 30.2010 Sheet 25 of 37 US 7,844,915 B2
2300
~
CONSTRUCT A DATA STRUCTURE HAVING
A HIERARCHY OF LAYERS WITH A
LAYER BEING ASSOCIATED WITH A
VIEW AND OWNING THE VIEW
2302
+
REMOVE THE LAYER FROM THE DATA
STRUCTURE
2304
~
SWITCH OWNERSHIP OF THE VIEW
FROM THE LAYER TO THE VIEW
<Jllo
FIG. 23
2400
~
CONSTRUCT A DATA STRUCTURE H A V I . ~ G
A HIERARCHY OF LAYERS WITH A
LAYER BEING ASSOCIATED WITH A
VIEW
2402
~
STORE THE DATA STRUCTURE IN MEMORY
2404
~
MAINTAIN A RETAINED COUNT OF THE
NUMBER OF REFERENCES TO THE VIEW
FROM OTHER OBJECTS
21l!Q
~
DEALLOCATE THE VIEW FROM MEMORY
IF THE RETAINED COUNT IS ZERO
2408
FIG. 24
U.S. Patent Nov. 30.2010 Sheet 26 of 37 US 7,844,915 B2
2508
2504 2.5.06
2512
FIG. 25A
2.5llli ~ 2512 ~
FIG. 258
U.S. Patent Nov. 30.2010 Sheet 27 of 37 US 7,844,915 B2
2600
~
CONSTRUCT A DATA STRUCTURE HAVING
A HIERARCHY OF LAYERS ASSOCIATED
WITH THE USER INTERFACE OF THE
DEVICE
2402
~
DETERMINE WHETHER EACH LAYER OFTHE
DATA STRUCTURE IS ASSOCIATED WITH
MEDIA OR NON-MEDIA CONTENT
2604
~
DETACH MEDIA CONTENT FROM THE DATA
STRUCTURE
2606
~
STORE MEDIA CONTENT IN A FIRST
MEMORY LOCATION
2608
~
STORE NON-MEDIA CONTENT IN A SECOND
MEMORY LOCATION
2610
J
COMPOSITE THE MEDIA AND NON-MEDIA
CONTENT FOR DISPLAY ON THE
DEVICE
2612
FIG. 26
U.S. Patent Nov. 30.2010 Sheet 28 of 37 US 7,844,915 B2
2712
2710
- - - ~
'
2 7 0 J ~
2120.
I I I
I I I

'
GQ
~
'
2722 2732 2742
FIG. 27
U.S. Patent Nov. 30.2010 Sheet 29 of 37 US 7,844,915 B2
2838 2832
~ 2 8 0 0
2834 \ !
\
2836
2840
2844
2846
FIG. 28
V.s. Patent
/
I
/
/
I
i
'
/
(
I
I
'
/
'
'
'
/
I
/
I
I
'
/
J--._ ___ _
- ~ ,
/
I I
/ /
' '
//
/
I
I
/
i
FIG. 29
US 7,844,91SIJ:t

U.s. Patent
308g
Nov. 30, 2010
Sheet 31 of37
-1084 30.98
!
////
// 3093
...-/
FIG. 30A
3092
)
/

--
' !
3096
<::6,
--- I,
':'J_ _________ j "-3089
{If!__________ \,_Jog1
FIG. 308
VS 7,844,915 BZ
3070
DISPLAY DEVICE
31Q6
BACKLIGHT
3106A
... .. . STORAGE
DIGITAL
PROCESSING
DEVICE
3101
D G T.
SYSTEM
ANTENNA I I ALAND/OR 3103
SYSTEM f- ANALOG RF AUDIO
31Q1 TRANSCEIVER f- MEDIA TRANSDUCER
- 31.02. PROCESSING 3.1Sl.a
SYSTEM
3109
I
I
I MICROPHONE l BATTERY
~ . 310.7
SENSOR(S) INPUT 105A
3110 DEVICE(S)
. 31Q5
3100
FIG. 31
~
~
'JJ

~
""
It
=
....
:2
~
w
.o
"'
0
- 0
00
""
a
w
"' 0
~
w
_,
~
"'
__,
~
....
-;,
-
..,
s
U.S. Patent Nov. 30.2010 Sheet 33 of 37 US 7,844,915 B2
CACHE
3208


/3200
!
3202 3204
,_;

MICROPROCESSOR
MEMORY
3206

<
BUS >
,------,1,_--....--1 -----,
DISPLAY
CONTROLLER
AND
DISPLAY
DEVICE
(OPTIONAL)
3210 3212
1/0
CONTROLLER(S)
3214
1/0 DEViCE(S)
(E.G . KEYBOARD.
CURSOR CONTROL
DEVICE. NETWORK
INTERFACE)
FIG. 32
321 6
J
SENSOR(S)
FOR USER
ACTIVITY
U.S. Patent Nov. 30.2010 Sheet 34 of 37 US 7,844,915 B2
3306
'
3302
I
3304
FIG. 33A
U.S. Patent Nov. 30.2010
I
3354
Sheet 35 of 37
FIG. 338
US 7,844,915 B2
3352
U.S. Patent Nov. 30.2010 Sheet 36 of 37 US 7,844,915 B2
-3362
\
3360
FIG. 33C
U.S. Patent Nov. 30.2010 Sheet 37 of 37 US 7,844,915 B2
3400
rv
CONSTRUCT A HIERARCHY OF VIEWS
OPERATING ON TOP OFA HIERARCHY OF
LAYERS
3402
~
PROVIDE ACCESS TO THE HIERARCHY OF
VIEWS WITHOUT PROVIDING ACCESS TO THE
HIERARCHY OF LAYERS
3404
FIG. 34
us 7,844,915 82
1
APJ>I.I<:ATIO:\ J>RO<;RHlVII:\G
1:'\TERFACES FOR SCROLLI:'\G
OPERATIO:\S
1:11:1 1)01:-nu;J)JSC!.OSURI;
"111is disclosure rclmcs to applicmion prog.r<Jmming inter-
faces thm provide scrolling opcrmions.
COMI'tri-I;R PROGRAM I IS'I-ING
A ponion of the disclosure of tills pmcnt document con-
wins mmcrial which is subject to copyright prot<..><::tion. The
copyright owner hns no objection to !Ucsimilc reproduction
by anyone oft he paten! document or the paten! disclosure. as
it appears in the l'atenl & Trademark Ollicc patent Ji!c or
n.><::ords. bllt otherwise reserves all copyright rights whmso-
cvcr.
Applicnnl has submitted herewith Computer Program I jst-
ings which arc included as Appendix A. allachcd.
BACKGROUND 0!-' TilE DISCLOSURE
2
At least certain embodiments of the present disclosure
include nn environment with user inter!Uce soliwnre internct-
ing with n soliware npplication. A method Jix operating
through an npplication progrnnuning interface (.\PI) in tills
5 environment includes trans!Crring a set bounce cal!. The
method !i.1rthcr includes setting at least one of maximum and
minimum bounce vnlues. The set bounce call en uses a bounce
of n scrolled region in nn opposite direction of n scroll bused
on a region past an edge oft he scrolled region being visible in
lCJ a display region at the end of the scm!!.
At least certain embodiments of the present disclosure
include nn environment with user inter!Uce soliwnre internct-
ing with n soliware npplication. A method Jix operating
through an application programming intcrliJCc (API) in this
15 environment includes trans!Crringa mhhcrhand call. Rubber-
handing a scrolled region within a display region occurs by a
predetermined maximum displncement when the scrolled
region exceL'ds n displny L'dge. The method further includes
trans!Crring an L"t.!gc mhhcrhand call to set displacement \a!-
"" ucs l(Jr at least one L"t.!gc of the display (e.g .. top and bottom
edges. !eli and right edges).
At lenst certnin embodiments of the present disclosure
include gesture opermions for n display of n device. The
gesture operations include pcr!(Hming a scaling transl(mJJ
"5 such as a /oom in or zoom out in response to a user input
having two or more input points. The gesture opermions nlso
include perlixming n rotntion transfonn to rotnte an image or
view in response to a user input having two or more input
An.\1'1 is n source code inter!Uce thm n compmer system or
program library prmidcs in order to support n.:qucsts !(Jr
scniccs !ium a so!lwarc application. An API is spccilicd in
termsofn progrmnming !angunge that can be imerpretmive or
compiled when nn npplication is built. rmher thnn nn explicit
low level description of how datu is laid olll in memory. The
so!iwarc that provides the li.mctionality described by an API
30
is said to be an implementation of the API.
points.
At least certain embodiments of the present disclosure
include a method for perlixming animmions for n displny of
n device. The method includes starting at lenst one animmion.
!"he method !i.1rthcr includes determining the progress o fcach
animation. "l.hc method further includes completing each ani-
Various devices such as ek>ctronic devices. computing sys-
tems. portnble devices. nnd hnndheld devices hnve soliwnre
applicmions. The .\PI inter!Uces betw. .. -cn the soliwnre nppli-
cations and user intcrbcc so!iwarc to provide a user of the
dL"\ icc with certain features and operations. A user may desire
certain operations such as scrolling. selecting. gesturing. and
animming opermions for n display of the device.
35 mat ion hasL"t.! on a single timer. rhc single timer can he hasL"t.!
on a redraw intervnl of the displny hnrdwnre.
Scrolling is the net of sliding a directional (e.g .. horizontnl
Vmious devices wllich perform one or more oft he forego-
ing methods and machine readable mL"t.!ia which. when
executed by a processing system. cause the processing system
40 to pcr!(mnthcsc methods. arc also dcscrihL"t.!.
or vc11ica!) presentation ofcontent. such as text. drawings. or
images. across a screen or display window. In a typical
grnphical user inter!Uce. scrolling is done with the help of a
scro!!bar or using keyboard shortcllls olien the nrrow keys.
Gesturing is n type of user input with two or more inplll
45
points. Animating operations include changing content
within a given time period.
Other methods. devices nnd machine rendable medin me
nlso describL'd.
ISRII:J: J)IOSC"RJI>"J"ION 01:.1.111: I)RAWINCiS
The disclosure is described by wny of example with rc!Cr-
ence to the accompnnying drnwings. wherein:
"l11c \arious types of devices may have a limited display
size. user interface. sollware. API interface and/or processing
cnpability which limits the ense of use of the devices. User
intcrbccs of dL"\iccs implement APis in order to prmidc
requested li.mctionality and !Caturcs. These user intcrliJccs
cnn have dilliculty interpreting the various types of user
inputs and providing the intendL'd li.mctionnlity associated
with the user inputs.
I'J(j. 1 is !low chart of a method for responding to n user
,
0
input of a data processing device:
I'!Ci. 2 illustrates details of an application programming
interface in How chart form nccording to certain tenchings of
the present disclosure:
I'IG. 3 i!!ustrntes details of nn applicmion programming
,
5
interliJCc in !low chartl(mJJ according to ccJ1ain teachings of
the present disclosure:
SUMMARY 01: n II; J)IOSC"RJI>"J"ION
At least certain embodiments of the presunt disclosure
include one or more applicmion progrmnming interfaces in nn 60
em ironment with user interbcc so!iwarc interacting with a
so!iwarc application. \"arious !i.mction calls or messages arc
trnns!CrrL'd vin the npplicmion progrmnming interfaces
betWL'Cn the user interface soliwnre and soliwnre applicn-
tions. Exmnple npplication progrmnming inter!Uces trnns!Cr 65
li.mction calls to impkment and animat-
ing operations !(Jr a de\ icc.
I'!Ci. 4 is a schematic diagram illustrating an embodiment
of user interface of a portable electronic device 400 having n
touch-sensitive display 408:
I'IG. 5A-5C i!!ustrme at lenst some embodiments of user
interliJCc of a portable clcctnmic dL"\ icc 400 ha\ ing a touch-
scnsiti\c display:
I'IG. 6A-6D i!!ustrnte the scrolling of n list of items to n
terminus of the list. nt which point an area beyond the lenni-
nus is displnyed and the list is then scrolled in nn opposite
din.:ction until the area bLytmd the terminus is no longer
displayed. in accordance with some embodiments:
us 7,844,915 82
3
!'Ki. 7 illustra!cs details of an application programming
intcr!Ucc in !low chnrt form according to certain teachings of
the present disclosure:
!-'IG. 8 illustrates Hrst and s<..>cond scroll angles for locking
4
FJ(j. J2 shows another example of a device in accordance
with one embodiment of the present disclosure:
a scroll of a display of a de\ icc in a hmizon!al or vertical 'i
direction according to certain teachings of the present disclo-
!-'I G. JJ.-\ is a perspective view of n device in a lirst con-
ligurmion (e.g. inn lnptop conligurmion) in accord.1ncc with
one embodiment of the present disclosure:
FJ(j. JJ!S is a pcrspccti\e \ iLw of the dL"\ iceoJTKi. JJA in
n second conliguration (e.g. a trnnsition conligurntion) in
nccordnncc with one embodiment of the present disclosure:
sure:
!-'IG. 9 illustrmcs dctni!s of an application progranuning
intcrbcc in !low chart !lmn according lo certain teachings of
the present disclosure:
I'Ki. I 0 illustrates details of an application programming
intcr!Ucc in !low churl form according to certain teachings of
the present disclosure:
FJ(j. JJC is a pcrspccti\e \ iLw of the dL"\ iceoJTKi. JJA in
lCJ a third con!iguration (e.g .. a tablet con!iguration) in accor-
dance with one embodiment of the present disclosure: and
!-'IG. J4 illustrates dcwils of nn npplication progrmnming
intcr!Ucc in How chart form nccording to certain tenchings of
the present disclosure. !'Ki. I I illustrates details of an application programming
intcrbcc in !low chart !lmn according lo certain teachings of l'i
the present disclosure:
l)!T\l!J:J) Dl:SCR!I'TION
!-'IG. 12 illustrntcs details of an applicmion prog.ranuning
inter!Ucc in !low chart form according to certain teachings of
the present disclosure:
!'Ki. B illustrates details of an application programming ""
inter!Ucc in !low chart form according to certain tcnchings of
the present disclosure:
\arious embodiments and aspects of the disclosure will he
described with rekrence to details discussed below. and the
accompanying drawings will illustrate the various embodi-
ments. The following description nnd drnwings arc illustra-
tive of the disclosure nnd arc not to be constnK'<i as limiting
the disclosure. Numerous speci!ic details are descrihL'<i to
prm ide a through understanding of \arious embodiments of
!-'IG. 14 illustrntcs details of an applicmion prog.ranuning
interbce in !low chart !lmn according to certain teachings of
the present disclosure:
!-'IG. 15 illustrntcs n display of n device having a scnling
trnnslixm ofn view
!-'IGS. 16.\ and 16B illustrntc a displny of n device with a
\ iLw ha\ ing a !irs! and a second scaling bctor:
"5 the present disclosure. However. in certnin instances. well-
known or conventional dcwils arc not described in order to
provide n concise discussion of embodiments of the present
disclosure.
!'Ki. 16(" illustrates changing a viLw Jiom a scale bctor of 30
2x to a scnle !Uctor of 1 x in m lenst some embodiments of the
Some pm1ions of the detailed descriptions which JlJI!ow
arc prcscnk-d in terms of algorithms which includcopcrmions
on dnw storL'<i within a compmcr memory .. \n nlgoritlun is
generally a sell:consistcnt sequence of operations leading to
a desired result. The operations typically require or invohc
physical manipulations of physical quantities. Usually.
present disclosure:
!-'IG. 17 illustrntcs details of an applicmion prog.ranuning
interbce in !low chart !lmn according to certain teachings of
the present disclosure:
!-'IG. 18 illustrntcs details of an applicmion prog.ranuning
inter!Ucc in !low chart form according to certain tcnchings of
the present disclosure:
!'Ki. 19 is !low chart of a method !(Jr animating \iLws
displayed on a display of a de\ ice:
!'Ki. 20 is !low chart of a method
displnycd on a display of a device:
!(H animating \ iLws
!-'IG. 21 illustrntcs details of an applicmion prog.ranuning
interbce in !low chart !lmn according to certain teachings of
the present disclosure:
!'KiS. 22A and 2 2 1 ~ illustrate synchroni/ing the resizing of
windows of a displny of a device:
!-'IG. 2J illustrntcs n method for switching ownership of a
\iLw of an application displayed on a display of a data pro-
cessing de\ ice:
!'Ki. 24 illustrates a method !(H memory management of a
view of an application displnycd on a display of a device:
!-'IGS. 25.\ nnd 258 illustrntc a datu structure having a
hierarchy oflaycrs with a layer being associa!L"d with a\ iLw:
!'Ki. 26 illustrates a method !(H cmnpositing media and
non-media content of user intcrliJCc !l1r display on a device:
!-'IG. 27 illustrmcs a dnta structure or lnyer tree hnving a
hierarchy of lnycrs:
!'Ki. 28 is a perspective\ iLw of a device in accordance with
one embodiment of the present disclosure:
!'Ki. 29 is a perspective\ iLw of a device in accordance with
one embodiment of the present disclosure:
!-'IGS. JO.-\ nnd JOB illustrmc a device J070 nccording to
one embodiment of the disclosure:
!'Ki. J1 is a block diagram of a system in which embodi-
ments of the present disclosure can he implemented:
35 though not necessarily. these quantities take the fonn ofdcc-
tricnl or magnetic signnls cnpablc of being stored. trnns!Crrl.-d.
combined. compared. and otherwise manipulated. It has
prmen comenicnt at times. principally ll1r reasons of com-
mon usngc. to refer to these signals as bits. values. dements.
"
symbols. charnctcrs. terms. numbers. or the like.
It should be borne in mind. however. thm nil of these nnd
similar tcm1s arc to he associated with the appropriate physi-
cal quantities and arc merely convenient labels applied to
these qunntitics. Unless spccilically stntcd otherwise as
45 npparcnt from the following discussion. it is app!L>ciatcd that
throughout the description. discussions milizing terms such
as "processing" or "computing" or "calculating" or "deter-
mining" or "displaying" or the like. can rdCr to the action and
processes of a datu processing system. or similar ck>ctron.ic
'0 device. thnt manipulmcs and transfonns datu represented as
physical (electronic) quantities within the system's registers
and memories into other data similarly represented as physi-
cal quantities within the system's memories or registers or
other such inli.mnmion storage. transmission or display
'5 devices.
The present disclosure c;m relate to an apparatus !l1r per-
limning one or more of the operations described herein. n1is
apparatus may be specially constmctcd ll1r the required pur-
poses. or it mny comprise n gencrnl purpose compmcr sdec-
60 tivdy nctivntL-d or reconligurL'<i by n complllcr program stored
in the computer. Such a computer program may be stored in a
machine (e.g. computer) readable storage medium. such as.
bm is not limitl.-d to. any type of disk including lloppy disks.
optical disks. CD-ROMs. and mngnetic-optical disks. rend-
65 only memories (ROMs). random access memories (R.\Ms).
erasable programmable ROMs (I ;PROMs). ck>ctrically eras-
able programmable ROMs (I;! :!'ROMs). !lash memory. mag-
us 7,844,915 82
5
netic or optical can.ls. or any type o fmcdia suitable l(Jr storing
electronic instructions. and each coupk-d to a bus.
A machine-readable medium includes any m<..>chanism Jix
storing informmion in a form readable by a machine (e.g .. a
computer). For example. a machine-readable medium
includes read only memory ("ROM"): random access
memory ("R. \M"): magnetic disk storngc media: optical stor-
age media: Hash memO!)' devices: etc.
6
The pm1ah!c media player may include a mL-dia sck>ction
device. suchns n dick wheel input device onnn iPod R or iPod
Nnno R mcdin p!nycr from Apple Compmcr. Inc. of Cuper-
tino. Cnlif .. n touch screen inpm device. pushbunon device.
5 mmahle pointing input device or other input dL"\icc. The
media sck>ction device may he used lo sclcc!thc mL-dia slorL-d
At least ccJ1ain cmhmlimcnls of the present disclosure
indudconcor application programming intcrliJccs in an cnvi- lCJ
ronmcnl with user intcrliJcc sollwarc interacting with a soli-
ware application. Vmious li.mction calls or messages arc
trans!Crr<..'d via the applicmion prog.rmnming interfaces
between the user in!erliJCc sollwarc and sollwarc applica-
tions. rrans!Cning the !imclion calls or messages may include 15
issuing. initiating. invoking or receiving the li.mction culls or
mcssngcs. Fxnmple npp!icmion prog.rnnuning imcrfnccs
1rnns!Cr li.mction culls to implemcm scrolling. gesturing. nnd
animating operations !l1r a de\ icc ha\ ing a display region. An
API may also implement !lmc!ions having paramclcrs. vari- ""
nbles. or poimcrs. An .\PI mny r<.>cdvc pmmnetcrs ns dis-
dosed or other combinmions ofpnrnmc1crs. In nddi1ion to the
AI' Is disclosed. other AI' Is individually or in cmnhinalion can
pcrllm11 similar !i.mclionality as the disclosed Al'ls.
on the storngc device nnd/or 1hc remote storngc device. The
portnblc mcdin p!nycr mny. in ntlens1 ccrwin embodiments.
include a display device which is coupled lo the media pro-
cessing system lo display titles or other indicators of mL-dia
being selec!L-d through the input device and being prescn!cd.
ci1hcr 1hrough n spcnkcr or cnrphonc(s). or on the displny
device. or on both disp!ny device nnd n spcnker or cnr-
phone(s). In some embodiments. the display de\ icc and input
device arc integra!L-d while in other embodiments the display
device nnd inpm device me scpnrmc devices. Fxnmples of n
portnblc mcdin plnycr nrc dcscribd in publishd U.S. pntcnt
npplicntion Nos. 2003/0095096 nnd 2(X)410224638. both of
which arc incorpora!L-d by rd"crcncc.
! Omhodimcnls of the disclosure described herein may he
purl of other types of dmn processing systems. such ns. for
cxnmple. cntcrtnimncnt systems or personnl digitn! nssistnms
(I'DAs). or genera! purpose computer systems. or spL>cial
purpose computer systems. or an embedded device within
"!11c display region is a llm11 of a window. A window is a
disp!ny region which mny no! hnvc n border nnd mny be the
entire disp!ny region or men of n displny. In some embodi-
ments. a display region may ha\c a! leas! one window and/or
at leas! one viLw (e.g .. web. !ext. or image conlcnl). A window
mny hnvc m lens! one view. The mc1hods. sys1cms. nnd nppn-
rntuscs disclosed cnn be implemented wi1h disp!ny regions.
windows. nnd/or views.
AI leas! ccJ1ain embodiments of the present disclosure
include scrol!ing operations !(Jr scrolling a display of a
device. The scrol!ing opcrntions include bouncing n scrolled
region in nn opposilc dir<.>clion of n scroll when n scroll
comple1cs. mbbcrbnnding n scrolled region by n predeter-
mined maximum disp!accmcnl when the scro!kd region
exceeds a display edge. and selling a scrolling angle that locks
1hc scrol! in n horizonwl or vcrticnl dircc1ion.
At lens! ccnnin embodiments of 1hc prcscm disclosure
include gesture operations ll1r a display of a device. The
gesture operations include pcrl(mning a scaling lransl(mn
such as a zoom in or /oom out in response lo a user input
hnving two or more inpm points. The gcsmrc opcrntions nlso
include pcrfonning n rotmion 1rnnsli.mn to rot me nn imngc or
\iL"\\' in response loa user input ha\ing two or more input
points.
At lens! ccnnin embodiments of 1hc prcscm disclosure
include n method Ji.1r pcrfonning nnimntions Ji.1r n disp!ny of
n device. The mc1hod includes stnrting m lens! one nnimntion.
!-he mc!hod !lu1hcr includcsdc!crmining the progress of each
animation. rhc mc!hod !i.n1hcr includes cmnplcting each ani-
mmion bused on n single 1imcr. The single timer cnn be bused
on n rcdrnw intcrvn! of the disp!ny hnrdwnrc.
AI kasl ccnain embodiments oft he disclosure may he p;u1
of a digital media player. such as a portable music and/or
\ideo media player. which may include a media processing
sys1cm 10 present 1hc mL'<iin. n s1orngc device 10 s1orc the
mcdin nnd mny limhcr include n rndio frequency (RF) trnns-
ccivcr (e.g .. an RF lranscci\cr !(Jr a cellular telephone)
coupled with an antenna system and the media processing
sys1cm. In ccrtnin embodiments. nK'<iin s1orcd on n remote
storngc device mny be trnnsmincd 10 1hc mcdin p!nycr
1hrough 1hc Rl-' 1rnnsccivcr. The mL'<iin mny be. for cxnmple.
one or more of music or other audio. still piclures. or motion
piclurcs.
"5 another device. or cellular telephones which do no! include
mcdin plnycrs. or muhi touch wble1 devices. or o1hcr muhi
touch devices. or devices which combine nspcc1s or li.mc1ions
of these de\ ices (e.g .. a mL-dia player. such as an il'odR.
combined with a I'DA. an cnlcnainmcnl system. and a cel!u-
30 lm telephone in one device). In this disclosure. ck>clronic
devices and consumer devices arc types of devices.
!-'J(j. 1 is !low chart of a method for responding to n user
input of a de\ icc. rhc method 100 includes recci\ ing a user
input a! block 102. The user input may he in the !lmn oLm
35 input key. bunon. wheel. touch. or o1hcr mcnns Ji.1r imcracting
wi1h the device. The me! hod 100 li.1r!her includes crcming an
cvcm objcc1 in response 10 1hc user inpm m block 104. The
method 100 !i.Jr!hcr includes dc!cJmining whc!hcr the even!
ohjL>cl invokes a scroll or gesture operation a! block 106. For
40 example. a single 10uch that drags a diswncc ncross n display
of the device may be intcrprck-d usn scroll operation. In one
embodiment. a two or more linger !ouch of the display may he
inlcrprc!L-d as a gesture operation. In ccnain embodiments.
dc!crmining whether the c\cnl ohjL>cl invokes a scroll or gcs-
45 lure opcrmion is basL'<i on receiving n drug user input for n
certnin time period. The method 100 limhcr includes issuing
a! kasl one scroll or gesture call based on invoking the scroll
or gesture operation a! block 108. The mc!lmd 100 !i.Jr!hcr
includes responding 10 nt lens! one scroll call. if issued. by
-;o scrolling n window hnving a view (e.g .. web. 1cx1. or imngc
content) nssocintcd wi1h the event objcc1 bused on an nmount
of a scrol! with the scrol! slopped a! a predc!cnninL-d position
in relation lo the user input a! block 110. For cxampk. an
input mny end m a ccnain posi1ion on n display of1hc device.
'-' The scrolling may continue umil reaching a prcdc1ermincd
position in relation lo the !as! input rccci\L-d li'llm the user.
The mc!hod I 00 !lu1hcr includes responding lo a! leas! one
gesture calL if issued. by changing a\ iLw associated with the
cvcm object bnsd on r<.>cciving a pluralily of input poims in
60 the form of the uscrinpm m block 112.
In ccJ1ain embodiments of the present disclosure scroll
operations include allaching scroll indicators lo a nmlcnl
edge of n display .. \hcrnativcly. the scroll indicmors cnn be
nnachcd 10 the disp!ny edge. In some embodiments. user
65 input in the form of a mouscl!ingcr down cnuscs 1hc scroll
indicators lo he displayed on the display L-dgc. content edge.
or window L-dgc of the scrolk-d region. If a mouse/linger up is
[mtu:J Tim:Jq :ll(Jl[J!!>\ ( 19-V9 "S!)!:!JO
U! (h'">J10JJS11[(! ()() pOl(PlU.JO UO!J10J:Jdo ptmq.r:Jq
-qiU ;,11.1. ;,mm 1!!!.\\ ;,:mep.Jo:x:m U! p;o.\e[dS!P
JJilUO[ ou J1[1 puo.\;,q e;o.Je J1[J [pun UO!):)..'">J!P
ue U! UJ1[J st JSH Jt[l pur, snu
!UUJJ Jl(J puo.\;,q tm JU!Od l(J!l(\\ J1' "JS!( Jl(J.JO
1' OJ tqo TIU![(lUJS Jl(J ([9-V9 "S!)I:I
:lliJ Ull :l(lJ!Sl \ JJ'i!UO[ ou )UJ)Ull:l :ll(J
.JO :lP!Slllll um'i!;o.J :lliJ )U:l)UllJ :ll(J "[]lUJS
J1JJJO p1w Jt[l JV m;,mo::> Jt[l uoti!JJ J1JJ mnoum
mnunxem J1Jl SJ!ltH! Jll[P-A JUJtuJJe[dstp J1Jl UJt[l Jt[l
m J[q!st,\ JUJJUOJ Jl!JJO ;,ilp;, Jt[lJSed uo!ilJJ e ilunwm
Jlll.JO )UJ)UOJ JJS111' .J[ Jl(J UO (h'">S1'q JJl 'u'">p 1'.JO
1' .JO :J'i!p:J 1' um'i!;,.r (h'">[(lUJS :ll(J U:lll\\

Jl[lJO UO!)JJJ!P UP- U! UO!ilJJ J1(lJO 'i!upunoq
.JO JU11mtm Jl(J Jmtm:Jpp m11m!u!m ptm
lU11lU!XlnU :ll(.!. "(h'">(qtm:J S! UO!J10JJdO J:m11oq JJliPll\\ S:lU!Ul
-JJPP zm:: )o[:lOJll [(tD ;,:m11oq :Jt].l. _)s-vs s!)!:!
U! p.'">qp::>s;op SUO!)eJJdO JJ1H10q J1[J Stu.IOJ.!Jd ()QZ p01[JJU1
JUJSJ.Jd Jl[lJO JUJOS JSeJ[ JP- U[
09 Jll[P-A mnunxem p;,mm.JJJJP.'">Jd e .\q s.Jn::>Jo 00
pOl[lJtu J1]J OJ flmp.JOJJB UO!f!JJ (h'">!\O.IJS r, flu!p1mq.J;oqqn}j
IJl' m JqTiu ptm l.PI
monoq ptm do1 J<.:J ps am JtiJ
-tpoqm;, JUlOS U[ "PO )j:lOiq J1' .\qdstp :ll(J.JO ;o'ilp.'"> :JUO JS1'JI W
s11u!tu.JJJ w ,.s:J:m11oq,
ptm JSt['iltH]]OJ:Js :ll(J .l:llJl' p:Jdttmp st sp.--.!"qo
-S!P JJOtu JO JUO Jl!JJO UO!JOtu J1[J ;,! "J-ZI S JJUeJS!p J1[J
z-ZIS Jt[l UJ1[.\\ J-O IS .(Jpop.qo ;opnwnkm
Jl!J uet[l SSJ[ ;oq ,\r,m z-O JS .\Jpop.qo ;opnJmilem J1[J "SJUJtu
-!poqm;, :Jmos u1 "tiS s11u!m.JJJ JtiJ mm.1 z-zis 1,;oq
ptm ZOIS .(Jt:lO(J\ :l.\1n[.(1,lU IZif:' J:)..'">fqo UO!J1nUJOJU!
s;,nJe\ m;,m;o::>eJdS!P OJ ue::> pueq.J;oqqtu ;,ilp;, ue ilU!J.IJJ
s;,pnpu! JJtp.mJ 00 pot[tJm JtLI. zot: :-po[q Je
Ull(Jl\\ umTiar p.'">]]OJ:Js 1' Timptmq.r;,qqtu :Jsm':J
OJ[(lD ptmq.J;oqq11.11' Tiu!J.IJ_JSUl!lJ UO!l10JJdo ptmq.J:Jq
-qtu e ilmptMl.Jd .!OJ 00 pot[lJtu JtLI. uoneJJdo pueqJ;oqqnJ
0> r, JP!Ml.Jd 01 .!Jp.!O U! UO!)P-J![dde JJe.\\lJOS e l[ll.\\ ilU!)JeJJJU!
Jl!J "tiS J1[J 1[1!.\\ UO!)JJSJJJU! Jt[l JJljP- Jtu!) r, J\"
"UO!)JJJ!P JSJJ,\JJ .(etu "1-Zlf: UO!JP-UUOJU! J1[J
"SJJ.'">!"qo p.--..\qdstp :JJOUl JO JUO JliJ ".)S . !)1:1 U!
-s1111! :JmtJ m tm J1' <.u;oz ;,q .\1'm .\JDop.\
Jl(J ;,! .\tmr JtiJ OJ JUJtuJ\om
JJe.\\lJOS J:lllJ.IJJU! JJSil 1!!!.\\ JUJtuUlU!\UJ Ue U! JJP-j
-JJJU! Tiu!mum.Rlo.rd mltllDt[dd1' Jt].l. amsops!P JU:Js;o.Jd :ll(J
.JO S'i!Ull(J1'JJ OJ UUO.J).nnp .\\0[1 U! JJ10J.l:llU!
ilu!umm.Jilo.Jd uope::>![dde ue JO S]!eJJP "!)!:!
"!\P-J JJ1H10q JJS J1]l f!UJ,\JJJJJ JO f!Upj<.lAU!
'ilmW!llll! 1lm11sst .JO :Juo J:ll(Jl:J [(lD :JJUn<.lq ps :ll(J 'ilm.r
)UJSJ.Jd :lliJ.JO ump:J:J U[
um'il;,.r JU:JJUO:J
e ;,q .(em uo!ilJJ PJ!\OJJS J1U. "90Z :jJOJq Je Jl[J.IO p1w Jl!J "tiS J1[1 1[1!.\\ JJJSJJJU! JO 1pe;o.1 "1-Zlf: J:)..'">fqo
UO!JP-UUOJU! J1[J se "SJ:)..'">fqo JJOtu JO JUO J1[J
U:lll\\ "HS "!)1:1 m "80t :ll(J Oit
\\OPU!'l"\ :ltiJ l(Jl\\ Jl':l!s.\tld p.--.soddo
J1[J 1!!!.\\ p;,Jeposse ,\Jepunoq [enutA e
Ot J1]JJe UO!f!JJ e U! J[q!S!A ilU!Jl( UO!ilJJ (h'">I\O.IJS
uo!'i!;o.J uo ]]OJ:Js 1'.JO ump:JJ!P tm m umTiar
p:l[(OJ:Js .JO :J:Jtmoq 'ilmsm'J OOZ poqpm
J1U. mdm e pJAJJ:)..'">J W1JJ.\\OptH.\\ P-JO ;,ilp.--. ;,uo JSeJJ
f:'J S Jl[l W1[J JJON "(h'">]]OJJS 'i!uwq S! Jl!J J[!l[.\\
J-OTS .\JtJO[J\ e 1[J!.\\ ilu!Mltu ;,q ,\em pue uo!ilJJ eJo
:J'i!p.--. l(Jll(.\\ SUl:lJ!JO JS!( :ll(J.JO tiS S11U!Ul.IJJ 1' lUO.I.J I -ZIS
:J:JtmJSlp ;,q I-z It p;o!"qo UO!J1ntm:JU! l(J11S "SJJ.'">!lJO
J.Jom .10 JU() ,\e[dS!P e 'ilmAet[ OOP
JJ!\JP ::>p:!OJJ:)..'">[J J]qeuod e JO J:lllJ.IJJm JJsn JO JUJ1mpoqm;,
tm ilmWJJsnm me.Jilmp e vs "D!:i e OJ oc
:JUlO:l .\[[tmO!JdO U:lliJ ptm "UO! jJ;JJtp \.'"OJ :ll(J .JO pu:J
:ll(J.l.IO JJlll10lJ JS!( :ll(J.JO pu;, JliJ
JJOtu JO JUO JU!Od tp!1j.\\ W "JS!( Jl[l JO SllU!Ul.IJJ e OJ JJ!,\Jp
e uo stuJJ! JO eJO ilm!IOJJS Jl!J .)S-\"S "SDI:i
"(UlllJ '"
-:J:J.!tp :lliJ.JO JU:lpu:Jd;opu! JUJUl:l\OUl p.--.p;,pp JO (h'">UllUJ:lPp
w 1Jl!.\\ ;,q ,\em SJil(eA JJ1moq mnm!xem pur, mnm
-m!tu JtLI. poz :jJO[q w J::>unoq mnunmm pue mnm
.JO ;,uo JS1'JI J1' Tiu!lPS s:Jp11pm OOZ poqpm Jt].J.
zoz )o[Jolq J1' l(l':J :J:Jtmoq ps s;op11pu! uo!J1o.r;odo
JJtmoq e ilmp!Ml.Jd .!OJ ooz pot[lJtu JtLI. uoneJJdo JJtmoq
r, JP!Ml.Jd 01 .!Jp.!O U! UO!)P-J![dde JJe.\\lJOS e l[ll.\\ ilU!)JeJJJU!
JJe.\\lJOS J:lllJ.IJJU! JJSil 1!!!.\\ JUJtuUlU!\UJ Ue U! JJP-j
-JJJU! Tiu!mtmo.Rlo.rd mltllDt[dd1' Jt].l. amsops!P JU:Js;o.Jd :ll(J
.JO S'i!Ull(J1'JJ OJ UUO.J).nnp .\\0[1 U! JJ10J.l:llU!
ilu!umm.Jilo.Jd uope::>![dde ue JO S]!eJJP z "!)!:!
"JJe.\\lJOS
:ltiJ.JO 'ilmJt!J:Jdo am.\\uos .\\.'"It\
;,q .\1'm .\\.'"It Jt].l. -\\JL\ .JO 1' s1' 'ilmw
-.r;,do JJ1,\\l.JOS J:J]]OJ:Js :llJ .\1'm .\\.'"It Jl(.!. \\J!.\.1Jd11s :ll(J
OJ U.\\Op pue dn f,r,Jp !\P- ;,potu JdJJJJJU!
ue nwunpoqm;, JJtpoue UJ J1Jl 01 m;,s J.Je
1' ""J"!) .rqtDS 1' OJ Ul JO JJliJO JJO[[J JO
JUO U! f:'Jf:" JUJtuJMltu OJ U! ;oq .(etu SJ:)..'">fqo p;,,\e[dS!P
Jl!JJO JUJUlJAOUl SJUJtmpoqm;, JJ1]JO U! "UO!JJJ.!!p Je[HJ!)JP-d
1' m tit JUJtuJ\om OUt JUJtmpoqm:J J(ll(.\\ mltJ
U[ arom JO 0\\J :ll( JJ:ll[l SJUJUl
-!poquw JJ1]JO m o It .\\OPU!.\\ JUO oot JUJ1mpoqm;,
J[!l\11:\ "JJeJUOJ JO nnod J1]JJO JUJtuJMltu p;,JeJJ[JJJe ilmu!tu
-JJJJP .10 ilup::>JJJP OJ ;,suods;o.J m pJWJJ(JJJe ;,q ,\em SJ:)..'">fqo
Jl(J.JO JU:llU:J.\OlU :llUOS U[ (p. !)!:! U!
:l]l'JS OJ JOU l[J!ll\\) 9IP SJl'il!p;,mm JO JUO
t[l!.\\ p;,Je!Josse JP-1JJ 1pns ,\e[dstp J1]Jl!l!.\\ peJUOJ JO JU!Od
"" u-w1p ptm d11 JSn<.lm J(ll(.\\ -\\J!.\.1Jd11s :Jl(J OJ 111du! Tit!rp
;,pmu Tit!rp uo Jd:JJJJJU! "JU:lm!poqm;, U[ \q11s
Jt[l 01 smdm ile.1p pue dn ;,snom u.\\Op
;,potu e "JUJtu!pOqtuJ JUO UJ "1l<.l!JeU!lUJJJJp S!1JJ ilm:jetu
snope.\ JJS ueJ Jd\" uv "JlJ fle.!p .10 u.\wp
>'l d11 :JS110lU tqo lllll:J :ll(J U! )11dU! .\q S! .\\.'"Ol\
-q11s .10 \\.'">!.\.1Jd11s ".\\.'"It\ ".\\opu!.\\ l[J!ll\\ JU!lUJ:Jpp OJ
e JO PIP JUJtuJMltu ilup:ntu.JJJJP JO i!U!):)..'">JJP OJ m
(h'">\Om ;,q .\1'm :ll[.!. smJJl.JO m spmp!.\
!Pll! :JJOUl JO JUO .ll:J UO!J1,lUJl:Jm P1'JUO:l OJ fll
ZIP SJJ.'">!lJO umJtmu<.:Ju! :ll(J "JU:Jmtpoqm:J .\.rqdm;,x;, tm u1 p
-ZIP Ol I-ZIP se tpns p;,,\e[dS!P
J.Jom .10 ;,uo ;,pnpm .\em OTt .\WPU!.\\ JtLI. ort .\WPU!.\\
-:)..'">U 11 pur, ilmpnpm J]dtl]lltu
r, JAP-1J.(etu .\\OpU!.\\ tpe:J JO Jjd!j]ntU flm,Wt[
UO!'i!;o.J J:JL\:lp 1' :llUOS U[
"J11du! :Jl(J Tiu! \!:JJ;tJ uo p.'">s1'lJ 1110 Tiu!
-mmv .10 m'ilmmooz .\q JJ.'">l"qo JUJ\J :ll(J liJ!.\\ .\\.'"It\
e ;,pnpu! .(em JJilJSJ!) mdm
J1JJJO uuoJ Jt[l m snnod Jndm JO .\memJd eilmAJJ:)..'">J
JJ.'"O!(JO JUJ\J :ll(J l(Jl\\ \\:l!.\ 1' .\q :lp11pm .\1'm 80P :ll(.!. "80f:' :l.\!llSU:JS-l(J110l
1' Tiu! wtl OUt :l:l!.\JP :Jm<.UJJ.'">P .JO
JUJtu!poquw tm ilU!WJJsnm me.Jilmp est p Dl:i
,: .J! "[(lD JUO JS1':lll1' O)'i!mpuods:JJ Jp11[JU!
JJillSJf! "JJilSO[JS!P JUJSJ.Jd J1]J JO SJUJUHpoqm;, U!e)JJJ U[
[[eJ pueqJ;oqqnJ Jt!J ilu!\!JJJJ .10 iluno,\m
ilu!Jetl!tH JO ;,uo JJt[l!J st ue::> pueq.J;oqqtu Jt[l iluu
"JJllSO(JSlp JU:JS;JJ\.1 Jl(J.JO SJUJtU!pOl(Ul:l U!1'UJJ U[
8
uotilJJ
PJ!\OJJS Jt!J JO ;,f,p;, .\WptH.\\ .10 ;,flp;, JUJJUOJ uo!ilJJ
Jlll 1110 SJOJ1'J!PU! Jl(J "(h'">JJ.'">Pp U:ll(J
L
zs > J6'vv8'L sn
us 7,844,915 82
9
!'!(iS. 6A-61) il!uslralc an exemplary user interlitce JSOOA
for managing an inbox in accordance with some embodi-
ments .. \n analogous user interface may be used to display
and manage other mailboxes (e.g .. dralls. sent. trnsh. per-
sonal. de.). In addition. other types of lists arc possible.
including hut no! limited lo lists of instant message conver-
sations. favorite phone numbers. contact informntion. labels.
email folders. email addresses. physical addresses. ringtoncs.
or album lklmcs.
10
lrans!Crring a directional scroll angle call lo determine if
dir<.>ctional scrolling is enabled at block 702. The method 700
li1rther includes 1rans!Crring a dir<.>clional scroll angle call1o
se1 a scroll angle for locking the scrolling in at leas1 one of a
'i vcJ1ical or a hori/Onlal direction a! block 704. The method
700 li1rther includes locking the scrolling in the horizontal
dir<.>clion if a user input Jixms an angle wi1h a horitonwl
dir<.>clion 1hat is less 1han or<.'qual to a lirs1 scroll angle at block
If the list ofcmails Jil!s more than the alloucd screen area. lCJ
706. rhe mc!hod 700 li1rthcr includes locking the scrolling in
the vcJ1ical direction if a user input l(mJJs an angle with the
vcJ1ical direction that is less than or <-'qual to a second scroll the user may scroll through the cmails using \crtically
upward and/or vertically downward swipe gestures on the
touch sen.-en. In the example of FIG. 6..\. a portion of a list of
emails is displayed in the screen area, including a lop dis-
played email3530 !ium !Smcc Walker and a boll om displayed
email 3532 from Kim Brook. A user perfonns a vertically
downwnrd swipe gesture 3514 to scroll toward the top of the
list. The vertically downward gesmre 3514 need not be
exactly \erlical: a substantially \ertical gesture is sullicicnl.
In some embodiments. a gesture within a predetermined
angle of being perfectly vertical results in vertical scrolling.
angle at block 708.
In certain cmbodimcms. a user input in 1he form of a drag
!(mns an angle with the hori/Onlal dir<.>clion that is less than
l'i the !irs! scroll angle. In this case. the user presumably intends
to scroll in the horizonwl direc1ion. The scrolling will be
locked in the horizonwl dir<.>clion unti11hc user input cxc<.-cds
the lirst scroll angle. A s<.>cond scroll angle may be used for
locking the user input in the vcJ1ical direction. The s<.>cond
"" scroll angle may he sci equal to the Jirsl scroll angle.
As a result of detecting the vertically downward geslllre
3514. in F!(j, 6!S the displayed cmails have shilled down.
such lhallheprcvious hollmn displayedcmail3532 !ium Kim
!Srook is no longer displayed, the prc\ious lop displayed "-'
email 3530 lfom Bruce Walker is now second li-O!n the top.
and the email 3534 lfom Aaron Jones. which was not dis-
played in FKi. 6A, is now displayed allhc lop of the list.
I-'IG. 8 illus1ra1es lirst and second scroll angles Jix locking
a scroll of a display of a device in a horizontal or vertical
dir<.>clion. The horizontal direction 802 and \ertical dir<.>clion
804 arc in rdCrcncc loa window or a display of a device. As
discussed in the mc!hod 701). a user input such as a drag
movement forming an angle wi1h the horitomal dircc1ion 802
less than or equal to 1hc lirs1 scrolling angle 806 or 808 will
lock the user input in the horizontal direction. In a similar
manner, a user input !(Hming an angle with the vcJ1ical dircc- In this example. the email 3534 !i-om Aaron Jones is the
lirst email in the list and thus is the terminusofthe list. Upon
reaching this email3534. in response to cominu<.-d det<.>ction
oft he vertically downward gesture 3514. an area 3536 (FIG.
6C) ahmc the lirsl email 3534 (i.e .. bcytmd the terminus of
the list) is displayed. In some embodiments. the area dis-
beyond the terminus of the list is visually indistinct
from the background oft he list. In 1-'lG. 6C. both the area 3536
and the background of the emails (e.g .. cmails 3534and3530)
arc while and thus are\ isually indistinct.
30 tion 810 less 1han or equal to 1hc second scrolling angle 810 or
812 will lock 1hc user input in the vertical direc1ion. The lirst
and second scrolling angles may be sc1 at the same angle or at
dif!Crcnl angles as well. For example. the !irs! and s<.>cond
scrolling angles may be setal 25 degrees. A user input less
35 than or <-'qtmllo 25 dcgr<.-cs wi1h respect to 1hc horizonwlor
vertical direction will lock the scrolling in the appropriate
dir<.>clion.
Once vc11ically downward gesture 3514 is complete, such
that a corresponding object is no longer detcct<.-d on or ncar 40
the touch screen display. the list is scrolled in an opposite
dir<.>clion until the area 3536 is no longer displayed. F!(i. 6])
illustrates the result of this scrolling in the opposite dir<.>clion.
the email 3534 Jimn Aaron Jones is now displayed al the lop
In some embodiments. the horizontal and \erticallocking
angles can he detcJmined in p;u1 by the aspect of the content.
For example. contcm in 1he form of a wll page may receive a
larger vertical locking angle compar<.-d to the horizomallock-
ing angle.
F!(j, 9 illustrates dc!ails of an application programming
interliJCc in !low chart l(mJJ according lo ccJ1ain teachings of
the present disclosure. The application progranuning imer-
!Uce operates in an enviromnent wi1h user imcrfacc soliwarc
of the screen area allotted to the list and the area 3536 is not 45
displayed.
In the example of !'!(iS. 6A-6D, a vc11ically downward
gesture resulted in display of an area h<-J'tmd the Jirsl item in
the list. .\s described in FIG. 3. the values for the predeter-
mined maximum displacement (e.g .. display of an area '0
beyond the lirst item in the list) arc set at block 304 for top and
hollmn edges oral block 306 !(Jr all edges of the window.
interacting with a sollwarc application in order lo prm ide a
d<.>cderalion scroll operation. '!-he me! hod 900 !(Jr prm iding
the deceleration scroll operation includes 1rans!Crring a
deceleration scroll call to set a deceleration !Uctor for a drag
user inpm at block 902. The me1hod 900 fur1hcr includes
slowing the scroll to a slop based on the spe'--d of the drag user
input and the deceleration li1clor al block 904. Similarly, a vertically upward gesture may result in display
of an area beyond the last item of the list. if the vertically
upward gcsmrc cominuesoncc the list has been scrolled to the
lasl item. rhc lasl item may be considered a lcJminus of the
list. similar lo the Jirsl item. As discussed above, the gesture
need nol be exactly vcJ1icallo result in \ertical scrolling: a
gcsmre within a pr<.-delined range of angles from perfectly
vertical is sullicient.
FKi. 7 illustrates details of an application programming
inlerbce in !low chart !lmn according lo certain teachings of
the prescm disclosure. The application programming inter-
face operates in an cnvironmcm with user imer!Uce soliware
interacting with a soliware application in order to provide a
dir<.>clional scrolling operation. rhe method 700 !(Jroperaling
through an application programming inlcrface(API) includes
In certain cmbodimcms. a user input in 1he form of a drag
'5 invokes a scroll operation for a certain time period. The user
input has a ccJ1ain speed. '!-he scroll of the scrolk-d region of
a window or a display region of a display of a device will he
slopp'--d a!ier the user input slops by applying a dccdcralion
!Uctor 10 the speed of 1he user input during the drag move-
60 ment.
F!(j, 10 illustrates dc!ails of an application programming
interliJCc in !low chart l(mJJ according lo ccJ1ain teachings of
the present disclosure. The application progranuning imer-
!Uce operates in an enviromnent wi1h user imcrfacc soliwarc
65 imeracting with a software application in order to provide a
scroll hysteresis operation. The mc!hod 1000 !(Jr prmiding
the scroll hysteresis operation includes lranskrring a scroll
us 7,844,915 82
II
hysteresis call to detcJminc whc!hcr a user input invokes a
scroll m block 1002. The method 1000 li.mhcr includes set-
ting a hysteresis value Jix determining whether a user inplll
invokes a scroll m block 1004.
I2
or display. A mask may merely pc1mi1 ccJ1ain changes while
limiting. or not pennitting. o1hcr changes. Evems of all kinds
come imo the application via a g.raphics li"amework. They arc
enqueued. collalecL'd if ncccssm)' and dispatched. If the
5 events arc system level events (e.g .. application should sus-
pend. de\ icc orientation has chanted. de) they are rou!L"d lo
the application having. an instance of a class of 1hc user
imerfacc software. If 1hc cvems are hand events based on a
In certain cmhmlimcnts. a user input in the l(mJJ or a drag
mer a certain distance across a display or window within a
display of a device invokes a scroll operation. The hysteresis
value determines the certain distance which the user inplll
must drag across the display or window prior to invoking a
scroll operation. A user input that docs no! drag the ccJ1ain lCJ
predetermined distance will not imokc a scroll operation and
may be considered a mouse up or down inplll or other type of
input.
user input. the events arc routed lo the window they occurrL'<i
over. rhc window then routes these e\enls lo the appropriate
control by calling the instance's mouse and gesture methods.
The control 1hat receives a mouse down or mouse cmcred
li.mction will continue to get allli.Jturc calls umil the hand is
!ilk-d. If a second linger is de!L><::!L-d. the gesture methods or !'Ki. 1 I illuslratcs details of all application programming
inlerbce in !low chart !lmn according lo certain teachings of
the prescm disclosure. The application prog.rmnming. inter-
face opermes in an cnvironmcm with user imer!Uce soliware
interacting. with a software application in order to anach a
scroll indicator loa scroll region edge or a window edge of a
dL'\ ice. In some embodiments. the scroll region edge is asso-
ciated with a comem edge. The window or display L'dgc may
be associated with the edg.e of a display reg. ion. The method
15 !i.mctions arc invokL-d. "!11cse !i.mctions may include start.
change. and end gcsmre calls. The comrol that R><::eives start
g.esture call will be scm allli.nure chang.e g.esture calls umil
the gcsmrc ends.
FIG. 13 illustrates details of an application programming
"" interliJCc in !low chart llm11 according lo ccJ1ain teachings of
the present disclosure. The application progranuning imer-
!Uce operates in an enviromnent wi1h user imcrfacc software
imeracting with a software application in order to provide a
gesture operation. "!.he method IJOO !l1r providing the gesture
I I 00 ll1r prm iding the scroll indicator includes lrans!Cning a
scroll indicator call to determine whether alleasl one scroll
indicator a !laches lo an edge of a scroll region or a window
edg.e at block 1102.:\ scroll indicator may be displayed on
any display edge. window edge or scroll region cdg.e. The
method I I 00 !i.u1hcr includes optionally allachingal least one
scroll indica lor lo the edge of the scroll region based on the
scroll indicator call at block 1104 .. \hcrnativdy. the method 30
"5 operation includes lranslCrring a handle gesture e\enl call al
block 1302. The method BOO li.1rther includes trans!Crring. a
g.esture change call in response to 1hc handle gesture event
call al block 131l4.
1100 further includes optionally attaching. at !east one scroll
indicator to the window edg.e of the vkw based on the scroll
indicator call al block I I 06.
In certain embodiments. a user input in the llm11 of two or
more points is rccci\L-d by a display of a dL'\ ice. A multi-touch
driver of the device receives 1he user inplll and packages the
cvem imo an cvem objL><::1. _.\window server R><::eives 1hc event
ohjL><::I and determines whether the event object is a gesture
even! objL><::1. If the window scner dctcJmincs that a gesture
In some cmhmlimenls. the operations of method I I 00 can
be alteR-d. modiHed. combined. or deleted. For example.
block 1104 can be delctL'd. Likewise. block 1106 can be
ddetL'd li"om the method 1100 .. \ltcrnativdy. the order of
block I I 04 and block I I 06 can he switched. Other methods
having \arious operations that have been disclosL'<i within the
prcsem disclosure can also be alteR-d. modi!k-d. rcarrang.ed.
collapsed. combined. or deleted.
35 cvem objcc1 has been received. then user interface software
issues or 1rans!Crs 1he handle gcsmre call at block 1302 to a
sollwarc application associated with the \iLw. The sollwarc
application conlim1s that a gesture e\enl has been rccei\L-d
and passes the handle g.esture call to a libra!)' of the user
"
In ccJ1ain embodiments of the present disclosure. transfer-
ring the scroll indicator call is either one of issuing. initialing.
imoking or receiving the scroll indicator cal!. For example.
the user interface software (e.g. .. software kit or library) may 45
R><::eive the scroll indicatorcallli"om the sollwarc application.
FKi. I2 illustrates details of an application programming
inlerbce in !low chart !lmn according lo certain teachings of
the prescm disclosure. The application programming. inter-
face operates in an cnvironmcm with user imer!Uce soliware '0
interacting. with a software application in order to dctennine
imerfacc software. The window server also associates 1hc
geslurccvcnl object with the\ iLw !hal rccci\L-d the user input.
The library responds by lranskrring a gesture change call in
response to 1hc handle gcsmrc cvem call at block 1304.
In one embodiment. a window or vkw associated wi1h the
user input rccei\es the change call in order lo pcrllmn the
gesture e\enl. The user so!lwarc that provides the view
receives a gcslllre start cvcm cal!. a g.esturc chang.cd event
cal!. a zoom to scale scning. for 1hc view. and a gcsmre end
call. The gesture calls rL><::civc an input of a gesture event
which may he base e\enl having a type such as a hand e\enl.
kLyhoard e\enl. system even!. de.:\ delegate associa!L-d with
the application R><::eivesa start g.esturccal!. gcsmre did change
cal!. and gcslllre did linish call. The user software is dynmni-
cally !inking into the application during the run lime of the
gesture process.
In some embodiments. the gesture changL-d !i.mction call
contains the following information about the gesmrc:
the number of lingers currently down:
if an inad\crtenl user input contacts a \ iLw of a display of a
dL'\ ice. The method 1200 includes lranskningan inadvcJ1cnl
user inplll call to determine whether the user inplll was inad-
vertent at block 1202. The method 1200 li.1rther includes '5
ignoring the inadvcJ1cnluscr input based on the dctennina-
tion of the inadvcJ1cnluscr input call al block I21l4. In one
embodiment. the inadvcJ1cnl user input call comprises a
1humb detcc1ion call1o de1ermine whether 1he user inplll was
the number of lingers initially down:
00 an inadvertcm 1humb.
In ccJ1ain embodiments of the present disclosure. lrans!Cr-
ring the inadvertent user input call is either one of issuing.
initiating.. invoking. or receiving. 1hc inadvertcnl user inplll
cal!.
_.\ gcslllre:\1'1 provides an interface bc1wcen an application
and user sollwarc in order lo handle gesturing. Gesturing may
include scaling. rotating. or other changes loa viLw. window.
65
the rotation of the hand:
the scale of the hand:
the 1ranslation of 1hc hand:
the posi1ion of1he im1cr and outcrmos1lingers: and
the pressure of the lirsl linger.
us 7,844,915 82
13
In other cmbmlimenls. more in!(mJJalion about each linger
down mny be included as Ji.1!lows.
14
lo represent that a larger pm1ion of content is no! being
displayed on display 1652 in FIG. 16B as a result oft he zoom
in operation.
In micas! some embodiments of the present disclosure. a
the stage of the linger Oust touch down. fully pressed. lilling
o!I etc):
the position of the linger:
the proximity of the !Inger (how hard you'n:: touching):
the orientation of the !ingcr (whm angle the ovoid is at):
5
user desires 10 chang.e a view 1670 Ji"om a scale JUc10r of 2x
loa scale liJCiorof lx as illuslra!L'd in I'Ki. 16C. :\ lirsl sci of
user inputs I672 and 1674 that mme lo the second sci of user
inputs 1676 and I678 will decrease the scale bclm Jimn 2x lo
the length of the major and minor axis.
the velocity of the !Inger: and
the ccccnlricity of the linger's moid.
1
, lx. 11 may be desirable for the user to scale Ji"om 2x to lx
wi1hout having to move !he user inputs a !arg.c diswncc across
the view 1670. In an environment with user inter!Uce soli ware
inleracling with a so!lware application. a gesture scaling
lransllmn Jlag may be sci in order lo dclermine a scaling
A gesture L'\cnl object may be a chord c\cnl object ha\ ing
a chord cmml (e.g .. mnnbcr of lingers conlac!cd the viLw or
display). a chord stnrt event. a chord change event. and a
chord end event..\ chord change event may include a scaling
or rowtion trnnsform.
l'i lransllmn !l1r a viLw associated with a user input having a
plurality of inpm points. The scaling. transform !lag scales
ci1hcr li"om a current scale !Uc10r 10 a minimum scale !Uc1or or
li"om the currcm scale factor 10 a maximum scale factor. For
example. a Jlag may be sci a! the position associated with a
"" l.Sx scale liJCior and a third sci of user inputs 1680 and 1682.
!-'IG. 14 i!lustrntcs details of an applicmion prog.nmuning
intcrbcc in !low chart !lmn according lo certain leachings of
the present disclosure. The application programming inler-
!iJCc operales in an cnvinmmcnl with user inlerbce so!iware
interacting. wi1h a soliware application in order 10 provide a "-'
scaling. lransfonn of a display reg.ion_ window. or view of a
display of a device. The method 1400 for providing the scal-
ing lrans!lmn includes lrans!Cning a scaling lrans!(mn call lo
determine a scaling lransllmJJ !l1r a \ iLw associated with a
user input having a plurality of input points a! block 1402. "!.he 30
mc1hod 1400 li.Jr!hcr includes lranslerring. a scaling gcslllre
start call a! block 1404. The me1hod 1400 Ji.Jrlher includes
:\user desiring lo change the scale bclm Jium 2x lo l x would
only have 10 move his !ingcrs.thc user inpms. from !he !irs! set
1672 and 16741o 1hc 1hird set 1680 and 1682 if the g.esture
scaling 1ransform Hag. has b<.-cn set at a scale !Uctor of 1 .5x.
I'! G. 17 illustrates dclails of an application programming
in!erliJCe in Jlow chart Jlm11 according lo ceJ1ain teachings of
the present disclosure. The application programming inler-
!Uce operates in an enviromnent wi1h user imcrfacc soliwarc
imeracting with a software applicmion in order to provide a
rotation lransllmJJ of a view. window. or display region of a
display of a device. rhe me! hod 1700 ll1r prm iding the rota-
tion lrans!(mn includes lrans!Crring a rotation lransllmn call
to determine a rotmion 1ransform for a view associmed wi1h a
1rans!Crringa scaling g.eslurc progress call a! block 1406. The
mclhod I200 Ji.Jrlhcr includes lransl"erring a scaling gesture
end call a! block I408.
35
user inpm having a p!urali1yofinput points at block 1702. The
mclhod 1700 li1rlher includes lrans!Cning a rotation gesture
star! call a! block I704. n1e mc!lmd 1700 li1r1her includes
translerring. a scaling. gcslllre prog.ress call m block 1706. The
In certain embodiments. a user input in the !lmn of two or
more input points (e.g .. lingers) moves logclher or apart lo
invoke a geslllrc cvcm !hal performs a scaling. transform on
!he view associated with !he user input. :\ scale lransfonn
40
includes a minimum and maximum scale bclm. FIG. 15
method 1700 li.mhcr includes 1rans!Crring a scaling. g.esturc
end call m block 1708.
In cenain embodiments. a user input in the Jlm11 of two or
more input points mlales lo invoke a gesture even! that per-
!(mns a rotation lrans!lmn on the \ i<.w associa!L'd with the
user input. The rowtion 1ranslixm includes a minimum and
illuslrales a display I 502 of a device ha\ ing a scaling lrans-
JlmJJ of a \iLw. The viLw I504 (e.g .. web. lex!. or image
comcm) has a !irs! scale fac1or. _.\user input (e.g .. lwo lingers
moving. a pan) associmed wi1h the view 1504 is inlcrprc1L'<i as
a gesture e\enllo zoom out from viLw I504 lo view 1508
having a second scale bel or that exceeds the maximum scale
liJclor of the\ iLw I 5I6. :\snapback Jlag dcleJmines whclher
1hc zoom out can proceL'<i pas1 the maximum scale factor to
view 1508 prior to snapping back to 1hc maximum scale
liJclor associa!L'd with view 15I 6.
45
maximum degree of rowtion for associated minimum and
maximum rotation views. The user input may temporarily
rotate a view pas! a maximum degree of rotation prim lo the
view snapping hack lo the maximum degree of rotation.
FIG. 18 illustrates dcwils of an application prog.rmnming
'0 imerfacc in How char! form according to certain 1eachings of
the present disclosure. The applicmion progranuning imer-
bce operates in an em ironment with user interliJce sollware
inleracling with a sollware application in order lo notify a
dclcg.me of at leas! one animmion associated with a display
l'Ki. 16:\ illustrates a display I604 of a dL'\ice having a
!irs! scaling bclorofa \iLw 1616.:\ user input (e.g .. two
lingers 1608 and 1610 moving 1og.e1her) associated wi1h !he
view 1614 is imcrprc!ed as a gcslllre cvcmlo zoom in from
\iLw I614 lo \iLw I664 ha\ing a second scale liJclor as
illuslra!ed in Fl(i. 161S. The dashed regions I602 and 1650
represent the total area of the conlenl with the only content
being. displayed in the display area 1604 and 1652. In per-
forming 1he scaling. 1ransform li"om FIG. 16:\ to FIG. 16B.Ihe 60
cenler of the gesture e\enl. center I612 ll1r FIG. I6:\ and
cenler I660 ll1r I'Ki. 161S. remains in the same position with
rcspec11o 1hc display 1604. The scroll indicator 1606 shrinks
'5 reg. ion. window. or view of a display of a device .. \ delay in the
animation may he specilied by the API. Also. multiple ani-
mations may he assigned priority by the :\1'1. "l11e mclhml
1800 llll notifying theddegale includes dclermining whclher
10 become scroll indicator 1654 during the 1ransfonn1o indi-
cate 1hm a smaller portion of the 1o1al content 1650 is being 65
displayed on display I 604as a result oft he zoom in operation.
"l.hedashed region I 650 is larger than the dashed region 1602
any animmion occurs at block 1802. The method 1800 Ji.Jr!her
includes checking the prog.ress of an animmion at block 1804.
If progress has occurred. then the next stale (e.g .. position.
opacity. or lransllmll) of the animation can he caleulated a!
block 1806. lfprog.rcss has complc1ed at block 1806.then at
block 1808 i1 is de!Crmined whether the view associated wi1h
the complc1ed animmion is associa1L-d wi1h a delcg.mc. If so. a
ddegale call is lransl"erred lo notify the delegate of the ani-
mation ll1r the view a! block I810. The ddegale operating
J;ot1JO tutu.] . .,.;,!.\ ;,tp OJ JO J;oqumu ;oqJ JO JUlllD
p;,umpJ 'i!U!U!l'JUWlU 00f:'Z POl1lJUl ;,t].!_
"f:'Of:'Z )jJll[lj Fl .\.iom;,m U! ;oJil)JlUJS lHL'P
J;)t[PnJ ootz ;:ltiJ. zotz :-tJo[q Je .\\;:l!A ;:lt[J t(l!.\\
w t(l!.\\ Jll
1' Tim.\1"[ amp!UJS t'll'P 1' Tiu!PlUJSmD OOPZ pm1pm
;ot].!_ 'J:l!.\:lp l'JO m1 JO . .,.;,!.\
e JO .UotuJtu .!OJ potpJm e tZ DLI
"PJS.IJ,\;:1.! .I! ;:JUO
WJO JUllll:l p;,u!l'PJ I' J.\1"(1!!1. .1..'"01 \ :ll!J. "P.'"OSJJ.\JJ JllU S! '>'.;,!.\
Jl]l.JO ;,t1JJ! .Uom:Jm tutu.] p.--.\otuaJ Jo P.'"'ll':JO[[B;op
;oq ptw OJ;oz JO JUllll:l p.--.umpJ I' ;o.\1"11[1'>\ '>'.;,!.\ ;,t].]. p:J.\OlUJJ
Sl 1\t'tJ UlliJl'Z!UlUtpu.\s I' u;,t1'>\ UO!Jl'Z
-mtmpu.\s 1' s;,pnpu! OOIZ pm1pm ;ot].L
w ue::> uonezmonpu.\s e ssJJoJd Jo UO]l
-eJ!!dde ;:lUll U[ "UOIWUl]Ue ilU!JS;:JilfJ;:JJ JJe
Jet[! Jll ;:Jt]J pue uope::>mJUJP! Jt[l
1Jl':l ;oqJ "Ju;om!poqm:J ;,uo UJ . .,.;,!.\ ;,1d11111m
;ot1J JO UO!jlml!lW 'i!UIJS:Jilfl:JJ 11"(1 :lljJ JO
09 e pue SJSSJJtUd ;:ll[l Jll UOIWZIUOR[JU.\S Jl[l JO UO]leJ!HJUJp]
ue Jo Jndu] ,\em [[eJ
:ll!J. 'tO IZ )j:l01lJ Jl' ;oqJJO S.\'\..'"01 \ ;,1d!J
-pnu :ll]J Jl:J OJ[]lD mll]l'Zmtmpu.\s
Jq [[!.\\ .\\."lJ,\ ;:ll[l OJ JJ,\e[ ;:ll[J UJO.!J J;)JU!Od ;:ll[J ;:JJJJ JJ,\e[
00 IZ poqpm ;,lJJ. puo:J.--.s
e OJ lS.!!J e mO.!J SJJnqpJW J.!lltu Jll ;:lUll S.IJJ]e UO]Jem
-me uv "JJ;:J .\)IJBdo "JZ]S r, Jpnpm ,\em .\\;:!]\ r, Jll Jll eJep ;:ll[l mO.!J p;:!AllUWJ S! J;:J,\e[ ;:ll[l UJl[/1:\ "JJ!\;:!p
Jl]l JO ;ot1J lUlUJ p.'"O\OlUaJ 'ilm;,q :ll11 l]J!.1. p.'"OJI'
!:lOSS!' .1..'"01 \ ;ot1J UO Jll:J:lll .\tml JJil):JIUJS l'll'P ;ol]J UllUJ
J;:J.(e[ Jl(l f>U!MlmJ}j eJep Jl(l mOJJ p;:!AllUWJ Jq [[!.\\
J;:J.(e[ ;:ll[l Wl[l UOIWJ!HJOU ;:lAWJ.."lJ ,\etu .\\J!\ JliJ. "JJ;:Jfqo Jl[lJO
JUno::> p:lU!!JJJJ Jt[l se fJ<"lU!JJP S! .\\;:!!\ Jt[l se tpns pJfqo ue OJ
'ilmJU!od JO J;oqmnu :ll!J. "JJ.\q Jt11 OJ . .,.;,!.\ ;,t1J tulUJ
J;ojU!lld puo:JJS V '.\'\..'"01 \ :lljJ OJ J;o.\q :ll11 UllUJ
JJUJ.!;:l,IJ.! JJJU!Od \' .\\J!A Jl(l .(q JUJJUOJ
JWJ;:JUJ'i!OJ .!Jp.!O U! .\l."l!A ;:ll[J OJ [[eJ UOIPU!1J e
. .,.;,!.\ ;oqJ l]JI'>\ J;o.\q Jt].L . .,.;,!.\ l1J!.L p:lW!:JossJ'
l'WP J;o.\q JUlOS U]
'90Z )j:lO[lj .1..'"01.\ :ll]J OJ JJ.\q
.\J.l;odOJd JO;oJnq!.IJW "J1dumx:JJO:] "ZO J z )j:l01lJ II'
l]ll'>\ Tiu!:llJ . .,.;,!.\ l1JLL .\pu:Jpu:Jd;opu! s.L.'"OL\ JO
SJJnqpJW f!U]llJS SUO]lem]Ull ;:ll]J .!OJ
0> 001 Z pllt]J;:!Ul ;:lliJ. 'JJ],\Jp eJO ll JO S.\\OpU!.\\ .!0 S.\\JIA
t(l!.\\ pJJBposse SUO]Wm]ue 01 .!Jp.!o U]
SJSS;J:Jtud JO ;o(d!J111lU l]J!.1. 1\U!J:ll'J;oJU!
;oJI,'>\JjOS ;o:JIOJ.!;oJU! J;oSI1l1JI'>\ jU:JlUUlU!.\li:J tm Ul
-J;:JJU] f.U]tutue.fcloJd Ulli)BJ!!dde ;:lliJ. ';)JilSll[JS]p JUJS;:JJd Jl[l
JO U!!Jj.l;:lJ lll f.U]p.!OJJB UUOJ petp .\\O[l U! ;:l:lllJ.!;:lJU]
JO JZ "!)!:1
"l,S.JJ.\ :l:l!.\ JO JU;oJJ'd
Jl[l UJO.!J .\\;:!]\ Jl[l Jll SJp!1[JU] J;:JljlJI11
OOfZ pot]J;:Jtu Jl[.L 'POfZ )jJO[q W ;:JJnJJIUJS eJep ;:Jt]J tull.!J JJ,\e[ Ot
Jtp'i!m \OtuaJ s:Jpnpu! J;ot1J.Il1j 00Z pm[PlU ;olJJ. 'UO!JlDI[dd!'
Jl]l.JO .H:J p.--.fqo JO ;,d.\J J:ll]JO Jo
"JXJJ "SJf.etu] "OJp]A "JU;:JJUOJ Jq .(etu S.IJ.(ej Jl[.L
zot:z )jJO[q Je .\\;:!]\ Jt[J'ilmu.\\0 pue .\\J]A e t[ll.\\ pJJeposse
'i!mJq JJ,\e[ e t]J].\\ SJJ,\e[ Jll .\tp.!eJ;:JJt] e ilm.wt[
t'll'fl s;,pnpu! OOfZ pm1pm ;olJJ. "J:J! \;op
-tud l'Jl'P l'JO .\qds1p I' Ull UlliJlDI[dd!' lWJO '>'.;,!.\
OJ Jllh!,dO lUOJJ .(J!:ll,dO :lljJ UO!ll'lU!llll
U\" "Jmll.!J eJO f!UJZ]SJJ Jpll[JU! .(Btu UO]lem]Ull ;:Jtue.!J \" m.!OJ
UO]leJtU .!0 ilU!leJS ll Jpll[JUI .(em UO]letUJUB
v um]tml!lW .\J!:ll,do tw mJt:Jsut!JJ ;,pnpu!
.\1'm ;,t]. 1. Jll:J:Jo lFJilJ.1.
[[e ]OJJUOJ .(Btu JJm]l ll "JjdUWXJ .!0:[ 'f:'OOZ )jJO[q
w JJm]l uoJwmp:m Jt[l t(l!.\\ Jll .\i!!B.!n[d
r ilu]wm]ue J;:ltJPnJ oooz potpJm JtU. zooz :-tJo[q JB
J:llU!J J11!ms l'TIU!P! \tud s:Jpnpu! OOOZ pm[PlU ;ot].].
;o:JL\Jp JO m1 . .,.;,!.\ m umTiar
r, ilupemmr, .!OJ pot[tJm eJo uetp .\\O[l oz "!)!:! e Jll d!t[SJ;:JU.\\0 ilmtpll.\\S .!OJ e Z DLI
'00 TZ pOt[l;:lUl Jll UOIWZIUOR[JU.\S ;:ll]J l[l].\\ JnJJll pue .([[eJ
-UJm;:JJJU! J!1JJO ,\etu ;:ll]J OJ ;:lJJHS JS.!!J Jl[l Ulll.!J
'ilmTiump U! ;ot].L ptw VZZ "S!)!:! m
-Sil[[! ozzz ptw OIZZ s.Lopm'>\ ;oqqo ;,t1JJO UO!Jl'Z!U
-o.!tpu.(s 00 TZ pot[tJm ;:ltiJ. '01 zz .\\OpU].\\ Jt[JJO
U! Jl[l OJ UO]UOdOJd U! ;:JZJS U] ,\em OZZZ
.w1pm.L puo:J.--.s "JlUIJ Jttws JV "HZZ !)!:!
u! o rzz .w1pu!.L ;oJJ'Js pum:Js vzz !)!:! u! o 1 zz .w1pu!.L
JS.I!J 1\u!Titmtp .\q U! JSl'!.'"OJ:JU! .\tml ;ojllljUJll'
e t]J].\\ lS.!!J e t(l!.\\ OTZZ .\\OPU].\\
e ";:JjdmeXJ .!ll:] "JJ]A;:!p e JO ,\e[dS]p e JO S.\\llpU].\\ .!0
JO Tiu!/IS;JJ ptm VZZ 'S!)I:1
.L.'"OU
Jt[l t[ll.\\ UJ;:JJJS Jt[l pue SJJepdn J."l.!l "JtmJ
'SUO]Jem]Ue .!]Jl[l ;:JAetj
;:ll]J 1!1Ul1 pJJepdn IOU S] ;:JJJl .!JpUJJ Jl[l "JJ,\J
-Wl] 1 ":J.'UJ J;o.\q ;,t1J JO .\do:J SJU;os:JJd:JJ ;o:JJJ JJPUJJ
JO JJilPIUJS l'WP puo:JJS V Jl]l JO S.Lopu!.l JO
Jl[lllJ puodS;:JJ.!OJ Wl[l S.IJ,\e[ JO ,\tp.!eJJ!l[ e SHWSJJdJJ
J;:JJJ Jo ;:JJnJJIUlS ewp JS.I!J e snwunpoqtuJ U[
'JWilJ[;:Jp Jl[l f!tH[]ll.!JUOJ Ulli)BJ!!dde Jl[l
oc uo p.."liluetp Jq .(r,m uopetmur, ue t]J].\\
JOU J:ll11() 'UlliJlmlllW ;ot1J JO u;,t1J J)l''i!;,pp
;ot].L "J)l''i!;,pp l]JI'>\ p.'"OJI'!:lOSSI' lFJI';o J;ol1Pl1'>\
JJt]UI11 .\em 0061 pot[tJm JtLL
ur::> e uodn p.."lseq Jo .\[[e::>]popJd
,:c p:lW111:lJI':l ;,q .\1'm tpto:.--. JO ;,t].!_
;,t11JO:lttWJJ JXJU Jt11 'iluunp SJ:Jump;,t11 OJ .1;op.1o
-J;oJU! .w'JP.'"OJ ;,t11 Tiu!Jnp :ll]J OJ p:Jwpdn
U! Sll[l U[ 'JJ],\Jp ;:ll]J JO ,\r,[dS]p ;:ll]J Jll Jme.!J
lXJU ll pue Jme.!J JUJJJilJ eJO ,\e[dS!p ;:ll[l UJ;:l.\\);:llj popJd ;:!Ul]l
"" lFJilJ.L(l!\JJJU! .W,.ip:JJ uo J;oUHJ J;otu!J
:ll]J amSOFJS!P :lljlJ o UWJ.l:l:l U]
'9061 )jJO]q w JJtmJ J]ilms
ll UO UOIWUl]Ue l[JBJ f!tHJ;:J[dtuOJ .!Jl[l.H1J 0061
pOl[JJm Jl[.L "f:'06 I )jJO]q Jll UO]letmur, l[JBJ Jll SSJJi!OJd Jl[l
,:1 'ilu!mtm:Jpp s:Jpnpm 0061 pm1pm Jl].!_ 'Z06l )j:lO(lj
Jl' o.LJ JSI'J1 Jl' s:Jpnpu! 0061 pm1pm ;ot].L
';)JJ,\Jp ll Jll ll Ull pJ,\B[dS]p .!0 "UO!f.JJ
r, ilupemmr, .!OJ pot[tJm eJo uetp .\\O[l 61 "!)!:!
'01 lZ )j:lO]q 11' puo:J.'"OS ;ot1J
l]J!.1. p:Jwpdn J\1"[ qJOq U:Jlj.1. fll
p:lJS:Jilfl:JJ ;ot11 .W,.!p OJ 1Jl':l 1\uu
SJpnpU] J;:Jt]UI11 001 Z pOl[JJm ;:Jt[.L 'S;:Jtu]lJUJJ;:lJJ!P W
UllUJ S;:!Jl1q].!Ue fJ<"lJepdn ;:ll[l S;:!A!J:l;:l.! .\\llp
ll! ;osJ,JJ:JUI OJ S.\'\..'"01 \ J;ol1JO
[lmO!J1ppl' ;ojJ,aJ:J .\I'm mH:JSlWJJ ;ot1J .\q :l/!S U!
'i!m:Jq .1..'"01 \ ;ot1J .\p\1Wtu;,J[V u! p:J:Jnp.u ;oq OJ
pJ;:JU ,\r,m S.\\JIA J;:Jt]Jll Jl[l JSBJ l[Jil(.\\ U] S.\\."lJ,\ JJl[lO PIUlSljll
,\em m.!OJSUB.!l Jl[l .(q U! fJ<"lSBJJJU! f.U]Jlj .\\JIA
;,t].L s.1..'"01\ Jpnpm .\tml v Jndm :ll]J -m . .,. ;ot1J "JU;om!poqm;, ;ouo UJ "80JZ )j:lO(lJ Jl' .\pu;,pu;od:Jpm
pml:l.'"OS I' OJ JS.I!J I' S.1..'"01.\ :ll11 JO
Jl[l ilupepdn sJpnpu] J;:Jt[l.H1] OOTZ ;:lliJ.
J:lllJ.!;:lJU! JJS!1 ;:ll[l f!tHJeJ;:JdOJJA.WS .\\OpU].\\ e lll
tpeJ 01 Jt[l UJt(.\\
;,q UID 'i!t'U ;,t].!_ "90JZ )j:l01lJ P.'"'1lJlm:J
91
,: l]ll'>\ p:lW!:JossJ' .1..'"01 \ ;oqJ uo mn:JstWJJ 1' suuo_p;od Jl"[l
UO!Jem]Ue Ull ;:JJe.\\ljOS U!!Jj.l;:lJ U[
"UO]l
-em1ur ;:lt[l ,\q pJmpom ilu]Jq .\\JIA Jt[l OJ m s.\\JIA
J:ll1JO J'i!tmtp UID UO!JlDI(dd!' am.LlJ OS ;ol11J 0 (lUJUO:J ;ol]J J;opun
>l
zs > J6'vv8'L sn

-ops!P JU;os;o.Jd ;oqJ m spoqpm :JI(JJO amos JSIO:.'"I( J"
OJ UeJ 008(': Jl(J "JJnSO(JSip
U[ Jl[J JO JO
OJ JJl[JO OJ JO JlldU! [OJJUOJ
I' m JU:llU:l.\OlU J;ot[Jll .Ill TIU!JI,lUitm "1\U!.IIlJS:lTI Tim
]]OJ:JS (O.IJUll:l :l11'm "Jt[TI!J1:mq
,\ejdS!P e OJ UeJ l[Jllq 09
JO 9P8Z ppgz .(J!ttHXOJd Jl[J tuOJJ
I'JI'P ;ot].l. "S:JIJ!\!PI' :ll(J Jl10l(l' .l:ll(JI'TI OJ
":lUll[" p.'"1SI1 JO "J;oqpTIOJ ;oq tmJ 9P8Z J;opmo.r;opJJI'
:ll[l p1m ppgz Josu;os .(J!UHXOJd :ll(J m<.UJ pa.!!llhJI' I'JI'P ;ot].].
,\p,\!jJJdSJJ zpgz et!UJJtHl pue OP8Z "8f8Z
JOJ ::>q ueJ euu::>nm pur,
u.Lmq-u;o.L .\uv ;,111 ;,t1J p1m ua.'"OJJs 1pm1J p1'd tpnoJ
"PI'd p1,d.\;,1 m
-Jilq "J[dtueXJ JOJ"JPil]JU! ,\r,m 9f8Z JlldU! Jll.l. "([_)"[
tm mje tpp:[.\\ JlldtH l]JilOJ r, JO 0>
OJ .\Jmqe ::>pnpm JOU tpp:[.\\ (([_)'[)
P!llh!1 1' ;,q .\1'm Pf8Z ;oJn;op .\l'[dS!P ;ot].l.
.([;o\lpadsa.J "t1Jnom ptm
e 1[1!-\\ U! p::>tl<.l!JISod .([[eJ!d.(J mq
uo uone::>O[ .\ue Je ueJ OP8Z JUOt[d
-<.UJntl p1m 8f8Z J:l1IO:.'"Ods ;ot1J paW!J;tJddl' aq ]]!!>\ J] "98Zf
;oJL\ap Jndm :ll(J .-,..opq "Zf8Z Timsnoq ;otp JO ml!pod J;o.1.0(
I' Jl' Sl OP8Z :JUOl[dOJ:J!m :ll].]. "Pf8Z ;oJn;op .\l'[dS!P
Jl[J Z8Z JO UO!UOd Ue W
OS[e 88Z "Zf8Z JO UO!j.l<.ld Ot
m' w Zf8Z Timsnoq :ll(J m<.UJ TimpuaJX:J II.Wll1S ZP8Z I'UU:JJim
:li(.J. "Zf8Z :li(JJO Ull!Ulllf .l:l">\<.111' W p.'"OUll!JISOd
-OJdaJ OJ s.(l,">\ sno!.m\ ll! Jt1TI!1 :J\Il!PPI'
m uauo) ;on]q ptm ua.uTI I(J!l(1. m ppmu :J\Il!PPI'
ue S! tp!t[.\\ e JO tuJOJ m aq ,\em
mpam-uou atl.l. f>u!wpdn JOU op W1[1
pur, "lt!JJUOJ apnpm ,\em mpam-uou
":l:l!\;op :ll(J
JO :ll(J OJ pammJs Tiu!aq OJ .lll!Jd aq .\1m1 JU:JJUOJ
:JI(.J. puo:Jas .1;od .\po.l OJ .\Ju;o_\"\.J .\]]mJimJsqns
JO ateJ e w 01 .(em
atl.l. sttwuodmo::> (JO[OJ) .\ pur,
n p1m :ll(J) JU;ouodm<.l:l a:Jtmu!mll[l'
A a1p 111!.1. ppmu .\flA 1' Jll ;,111 ll! ;oq .\1'm JU:JJUOJ
mpam atl.l. iluj.\.ieA Jat[JO Jo af!etu] o1pne u1
-iln[d oapiA oapiA apnpu] ,\em P-]pam e
Ull[l!.\\ UO]WJO[ JO "J!llll SJitjdeJf! e
umm ;,tp1D m .\.1omam Jll ;,d.\J .\1m ;,q mD UO!J
-I,Jll( .\.1om;,m :ll].l. .(1'1dS!P ;otp OJ .(1J:l.'"I.I!P JU:lJUOJ l'lfA'"IUl-UOU
pur, pur, "UOIWJO[ .\.iotuatu pUOJ.'"IS
UJOJJ J[IJJUOJ P-]p.'"Otu-UOU .'"Ol[l f.U],\.'"OUJ.'"OJ "UOIWJO[
;,111 mO.IJ JU:JJUOJ :ll(J Tiu! \;oupJ JU;oJUOJ mpam
-uou ptm :ll(J "SJU;ouupoqma amos U[
"()J9Z )jJO(l[
Jl' :l:l!.\;op ;otp Ull .H:J JU:ljllll:l l'!fA'"IlU-UllU p1m mpam
.'"Ol[l J.'"Ol[JJI11 009Z "809Z
'jJO[q te uope::>O[ .\.iomam e m Jt!JJUOJ mpam-uou
TIU!.IOJS sapnpu! ()()9Z pm[Ptu :li(.J. "909Z 1:lll1q W UO!J
-I,Jll1 .\.!om am JS.I!JI' m JU;oJum
009Z potj).'"Otu .'"011.1. "909Z 1JO[q W .'"OJilPIUJS etep .'"Ol[l tuOJj
t[J].\\ pateposse Ja.(e[ e ilmlpetap sapnpu]
009Z pot[Jatu atl.l. mpam-uou Jo mpam 1[11.\\
amp!UJS l'li'P ;ot1l.Jll s.n\1'1 ;ot1l.Jll I(JI';o J;ot1Pl1">\
9f8Z J!ldU! Jl[J pue "Zf8Z JO UOIJJOd
JJddn 1m W p::>tlll!jiSOd S! Pf8Z Jtl.l.
sz Dl:l m u.\\OtJs JoJ::>eJ mJoJ ::>tJJ moJJ
Ul.lll.J JU:JJ;,_jjlp I' :J.\1'1( .\1m1 ptm SJOSU:JS .l:l">\;,_j .Ill :l.llltu
asn .\1m1 gz "!)!:!.Jll JU;ouupoqma;,t1J Jl't1J paW!J;tJddl' aq 1[1">\ J]
"9P8Z ue pur, PP8Z .\Jitu!X<.ud e ::>pnpm
.\em mw oosz Jti.I. zrsz 1m pue
OP8Z JUOI[dOJJitu e "8f8Z e pr,d,\J'j
m' ;,q .\1'm I(Jil(.1. 9f8Z ;oJI \;op Jndm m' "Pf8Z :JJ!.\;op .\l'[dS!P
I' "Zf8Z Tiu!Sillll(l' ;opnpm .\1'm 008Z ;oJL\ap :ll(J "8Z
U[ "J[.\Js ,.Jeq-,\pue::>, e ilu!,WI[ uonemiluuoJ JUOt[d::>pJ
e U! JJIAJp e S.\\OI[S 8Z "JJilSO(JS!P JUJU!
-!poqm;, auo OJ 1\u!p.!OJJI' ()()8Z ;oJL\ap I' 8Z . !)]:1
.. ;omm.Japp OJ .r;op.ro U! p.'"OSJ:l\I!IJ Sl J."Uj .I:J.\1'( .Ill ;tlll)JIUJS l'li'P
"
atl.l. ro9Z 'jJO[q JP- mpam-uou Jo 1[11.\\ paJmJ
saJ! \;op as;ot1J
Jll .-,..;,_! 1' Jll S:J[d11mx;, :lJI,.IJSil[[! ff8Z ptm 9-p "S!)l:! saJ!.\;op
JJl[JO JO JO UO!JeU!l(tuOJ ,\ur,
ewp tpllOJ !J[ntU tpllOJ !J[ntU
ppt1ptmq \ap P(l[I'J "S:l:l!.\;op .II'Jil((JJ "S:l:l!.\;op o<:
;,[qi,UOd s.JaJndmoJ doJ1S:lp
ewp "SJJIAJp ilU!Pil[JU!
U! U!JJ
-eJedde pur, Jt[l U[
-mse s1 etep at[l JO tpea
sapnpu! 009Z pm1pm ;,t].]. zogz 1:lll]q Jl' ;oJI\ap :ll(J
JO ;oJIOJ.l;oJU! J:JSil :ll(J 111!.1. fA'"Ili'!JOSSI' s.n(I'1Jll .\tpm.!;o!l11' 1\u!
I'JI'P I' 1\mp!UJSUOJ ()()9Z pm1pm ;,t].l.
oc e uo .\ejds]p JOJ JO mpam-uou
pue potJJ-'"'tu e 9Z
aq [[!.\\ OTSZ .\\.'"11,\ ilU].!OJS
;,111 U:ll(J "OJ;oz Jll JUilOJ pau!I'PII' () JSZ J I "!)1:1
.:c m s1' JUilOJ p.'"lump.J J;o.LO(I' a.\1'1( [[L1. OJSZ .\\.'"11_\
. paMltu.'"OJ 80SZ .'"Ol[l t!Jl[l "HSZ m paWJJSil[l!
se aJilPIUJS etep at[l tuOJJ 90SZ Ja,\e[ at[l Jl
mliJID01 .\.1omam ;oJml,das 1' u! pass:JJ >'I
-<.ud ptm ampn.JJS I'JI'P ;,111 mo.!J pa.\oma.J aq ]]1">\ POLZ Ja.\q
zrsz pur,
80SZ .I:JJU!Od ;,tp Ull pas1'q O">'.JJO JUilOJ pau!I'PJ
I' VSZ "!)!:! ll! ()JSZ .\\.'"11\ ;otlJ. "Jndm J:JSill' OJ
m OJSZ .\\.'"11\ ;ot11 Tiu![(lUJs 1pns OJSZ .\\.'"1!.\ :ll(J 1[11">\ paw
SUO!jeJado [OJJUOJ .(em OfSZ [[1 "OJ SZ
.\\.'"01,\ .'"Ol[l OJ OfSZ ([[1) tuOJJ JU]Od
.\1'm ZfSZ a:Jtnl;o_pl J;oJu!od pmp V "90SZ J;o.\q :ll(J OJ OJSZ
.\\.'"II\ :ll[l tu<.UJ SJU!od z JSZ :JJtn!;op.r J;oJU!lld pmDas V "() JSZ
.\\.'"II,\ Ol90SZ tuOJj 80SZ
\' ".\\.'"0],\ papiAOJd J[IJJUOJ OJ J.'"OpJO U1
at[l OJ[[eJ e OTSZ at[l t[J].\\
90SZ J;o.(q ;,t].]. "() JSZ .\\.'"11 \ ;,t1J 1[11">\ 90SZ J;o.\w I
gosz ptm rosz zosz s.Ja.\1'1 ampn.JJs l'li'P ;,tl.l.
e 1[11.\\ iluwq e t[)l.\\ JO
1pns 1[11.\\ p::>Jeposse SJJ,\e[ .\uy
-uou Jo mp::>m 1[11.\\ JJIHJIUJS etep JO
l(JI'a .r;ot1Pl1">'. ;omm.Japp IJ!1. ()09Z poqpm ;ot].]. z JLZ JU:JJml:l
l'!fA'"IlU-UllU ;ot1J Jll :ll(J SUUll.J ".\p\I)J:JdS:JJ fll
"ZPLZ ptm "ZfLZ "ZZLZ JU:lJUOJ l'lfA'"Itu-uou 111!.1. paW!JllSSI'
JJe OPLZ pue "OfLZ "OZLZ o::>piA JOJ
::>q .(em tp!t[.\\ zr LZ 1[1!-\\
e ilm,wl[ etep e HSZ pue vsz
".')JIA.'"Op .'"Ol[JJO ,\e[tfSip .'"Ol(J Ul<.UJ
"ilm;,q .n\1'1 ;ol(Jl(JI">\ p.'"IJmJOSSI' .-,..;,!.\ ;,111 UO fA'"ISI'q JllJJO .\I'm
,: I'WP :ll1J lUO.IJ .n\1'1 ;,111 amp!UJS l'li'P
.'"Ol[l tuOJJ paAOUIJJ Sl .'"Ol[JJ] pajU.'"OUl.'"OJJ.'"Ip aq [[!.\\ .\\.'"0],\
S! ()JLZ J;o.\wj oap!\ tpns 90LZ JU:lJUOJ l1J!.L
POLZ J;o.\q ptm
1[11.\\ patepmse aq ueJ atl.l. JO .\tp.mJa!t[
e ilmAet[ Jo ewp e LZ "DI:l
.'"Ol[l OJ paJedtuOJ J.'"O.\\O[S r, W .'"Ol[l
OJ aq jU;ojUllJ l'!fA'"IlU-UllU ;ol(.J. "S.Ill10J J:ll1Jll :JJ11p
81
JO )UilOJ "80PZ 'jJO[q )!J
s1 JUno::> Jl .\.iouwm tuOJJ at[J
sapnpu! OOPZ pm1pm :JI(.J. "90PZ 1Jll[q W spa!"qo
LJ
zs > J6'vv8'L sn
us 7,844,915 82
19
!'Ki. 29 shows a device 2950 in accordance with one
embodiment of the disclosure. The device 2950 may include
a housing 2952. a display/input device 2954. a speaker 2956.
a microphone 2958 and an optional antcmla 2960 (which may
he visible on the exterior oft he housing or may be concealed -'
within the housing). The de\ icc 2950 also may include a
proximity sensor 2962 and an accdcromdcr 2964. The
device 2950 may be a cellular telephone or a device which is
an intcg.rm<..'d PI)..\ and a cellular telephone or a device which
is an integrated media player and a cellular telephone or a lCJ
tk'\ icc which is both ancnlcJ1ainmcnl system (e.g. !(Jrplaying
gmncs) and a ccllulnr telephone. or the device 2950 may be
other types of devices described herein. In one particulnr
embodiment. the device 2950 may include a cellular tele-
phone and a media player and a I'I)A, all contained within the l'i
housing 29S2. rhc de\ ice 29SO may ha\e a Jlm11 l'iJCtorwhich
is small enough thut it !its within the hund ofu normuladu!t
and is light enough thm it can be curried in one hund by un
adult. It will be appreciated that the term "pm1ab!e" means the
dL'\ice can he easily held in an adult user's hands (one or ""
both): ll1r example. a laptop computer and an il'od arc por
wble devices.
20
telephone integrated with a media player which plays Ml'3
Hies. such as Ml'3 music Hies.
Fuch oft he devices shown in FIGS. 4. SA. SB. SC. 6.\. 6B.
6C. 6D. 28. 29. JO.\ and JOB may be u wireless commtmica-
tion de\ ice, such as a cellular telephone, and may include a
plurality of components which prm ide a capability ll1r wire
less commuJlicntion. !'!G. J1 shows an embodiment of a
wireless device J070 which includes the capubility for wire-
less commtmication. The w i n ~ l e s s device J070 may he
included in any one of the de\ ices shown in FKiS. 4. SA. SB.
SC. 6A. 6JS. 6C. 6! ). 28. 29. 30A and JOB. although a!tema
tivc embodiments of those devices of FIGS. 4. S.\. SB. SC.
6.-\. 6B. 6C. 6D. 28. 29. JOA and J08 muy include more or
J(:wer components than the Wireless de\ icc 3070.
Wireless de\ ice J070 may indudeanantcnna system 3 I 01.
Wireless dcviccJ070 muy ulso indudcu digitnlandlor unulog
rndio Ji"equency (R!') trnnscciver J1 02. coupled to thcuntcnnu
system J101. to transmit und/or R><::cive voice. digiwl dmu
and/or media signals through antenna system 3101.
Wireless device J070 may also include a digital processing
system J10J to control the digitu! R!' trunsccivcr und to
manugc the voice. digitn! duw undlor mL'dia signals. Digiwl
processing system 3103 may be a genera! purpose processing
device, such as a microprocessor or controller ll1r example.
In one embodiment. the display/inplll device 29S4 muy
include a multi-point touch inplll screen in uddition to being
a display. such as an I CD. In one embodiment. the multi
point touch screen is a capaciti\e sensing medium conligurcd
to detect multiple touches (e.g .. blobs on the displuy Ji"om a
user's JUce or multiple lingers concurrently touching or
ncur!y touching the display) or ncar touches (e.g .. blobs on the
display) that occur at the same time and at distinct locations in
the plane of the touch pane! and to produce distinct signals
rcpresentmive of the locution of the touches on the p!une of
the touch panel for each of the multiple touches.
"-' l)igita! processing system JI OJ may also he a spL><::ia! purpose
processing device. such us un .\SIC (upplication spcci!ic hue-
grated circuit). FI'GA (!ield-prog.ranunuble gate array) or
DSI' (digital signa! processor). Digital processing system
3 I 03 may also include other devices, as arc known in the art,
30 to interJUce with other components of wireless device J070.
For example. digital processing system J10J muy include
unalog-to-digitu! und digita!-to-unalog converters to interJUcc
with other components of wireless device J070. J )igita! pro
cessing system 3103 may include a media processing system
In certain embodiments of the present disclosure. the
dL'\ice 2800 can be used to implement at least some of the
methods discussed in the present disclosure.
35 J109. which may ulso include u generul purpose or special
purpose processing device to munuge mcdiu. such as tiles of
uudio datu.
FKiS. JOA and JOB illustrate a device J070 according to
one embodiment of the disclosure. The device J070 may be a
cel!ulnr telephone which includes a hinge J087 that couples a 40
display housing J089 to a kLypad housing J091. The hinge
J087 allows a user to open and dose the cellular telephone so
that it c;m he placed in at least one of two dilkrcnt conligu
rutions shown in FIGS. JO.\ and JOB. In one particulnr
embodiment. the hingcJ087 muy rotmub!y couple the displuy
45
housing to the kLypad housing. In particular, a user can open
the cellular telephone to place it in the open conliguration
shown in F!Ci. JOA and can dose the cd!u!ar telephone to
pluce it in the dosed con!igurution shown in FIG. JOB. The
keypad housing J091 may include a keyp.1d J095 which '0
rccei\es inputs (e.g. telephone number inputs or other alpha
numeric inputs) Jimn a user and a microphone J097 which
R><::cives voice inplll Ji"om the user. The displuy housing J089
muy include. on its interior surJUce. a display J09J (e.g. un
I .CD) und a speaker J098 und a proximity sensor J084: on its ,
5
exterior surl'iJCc. the display housing J089 may include a
speaker J096. a temperature sensor J094. a display J088 (e.g.
another I .CD). an mnbicnt light sensor J092. und u proximity
sensor J084.\. I knee. in this embodiment. the displuy hous-
ing J089 may include a lirst proximity sensor on its interior
60
surl'i1cc and a second proximity sensor on its exterior surbce.
Wireless device J070 may also include a storage device
3 I 04, coupled to the digital processing system, to store data
und/oropernting progrmns Jixthe Wireless device J070. Stor-
uge device J 104 muy be. for example. uny type of solid-stmc
or magnetic memory dL'\ ice.
Wireless de\ icc J070 may also include one or more input
devices J 1 OS. coupled to the digital processing system 3 I 03,
to accept user inputs (e.g .. telephone numbers. nmncs.
uddrcsscs. media sek><::tions. etc.) lnplll device J 105 may be.
ll1r example. one or more of a kLypad, a touchpad, a touch
screen. a pointing device in combination with a display device
or similar input device.
Wireless device J070 muy also include m lcnst one display
device JJ 106. coupled to the digitnl processing system J 10J.
to display inllmnation such as messages. telephone call inlllr
mation. contact inllmnation. pictures. movies and/or titles or
other indicmors of mcdiu being sck><::ted via the input device
J105. Display device J106 muy be. for example. un !.CD
display de\ ice. In one embodiment, display device JI 06 and
input dL'\ice 310S may he integrated together in the same
device (e.g .. a touch screen I CJ) such as a muhitouch input
panel which is integrnted with a displuy device. such us an
!.CD display device). The displuy device J 106 may include u
backlight J 1 06A to illuminate the display dL'\ ice 3 I 06 under
certain circumstances. It will be apprccialL'd that the Wireless
device J070 may include multiple displuys.
In at !east ccJ1ain embodiments, the de\ ice J070 may con
win components which provide one or more of the Ji.mctions
of u wireless conununicution device such us a ccllulnr tele-
phone. a mediu player. an entertainment system. u I'D.\. or
other types of devices described herein. In one implemcnta
tion of an embodiment. the device J070 may he a cellular
Wireless device J070 may also include u battery J 107 to
65 supply opcrming power to components of the system includ-
ing digital RF transcci\er 3102. digital processing system
3 I 03, storage device J 104. input device JIOS, microphone
Ul '(]OJ:lS 'i!U!)jlHU! JlldU! J;osn ;oqJ UO p.'"OSI'l( JlldU! J;osn OJ 'i!mp!.\OJd jlllll[ll\\ S.\"\.'"01 \jO .\1pm.1;o!1J aqJ OJ
'ilu!pLvud sapnpu! OUt poqpm at]_]. "ZOtf )j:lO(lJ I' .ll:J UO!Jl'J;o(:l:Jap ps OJ (]lU:lS UO!Jm;op:Jap
e JOJ !d\' ue )o{JO.\\aUHUJ at]lJO .\.uuqq at]!
"Jtwunpoqma ue U] e sa)o(OAm mdu! Jasn e Jat]Jat].\\ au!m
-JaJap 01 uotFuado e ld\" ue
)jJ<.vu;,mt!l.J ;,q!Jo .\.m.!lJ!I ;,t]J "JUam!poqm;, tm UJ
umpaJ!P JO jlD!pa.\
e JO auo w U! 'ilm]]OJJS at]l iluppo] JOJ a]ilue e
01 uopeJado ]]OJJS ]euotpaJ!P e JOJ !d\" ue
-tud )jJlJ.\\;otut!l.J ;,t]lJO .\.mJql[ ;,t]l "JUam!poqm;, UJ
w SJa.(e] JO .\tpJeJa]t] e JO dot uo ilmwJado JO ,\tpJe
-Jatt] e sapnpu] Wt]J OUt pot]tam U] pawJtsnm
se !d\' atn aJe.\\lJOS at]! mt]l!.\\
JO .JO dOJ UO S.\"\.'"01 \ .JO .\l(:JJI,J;o!1]1'
-1poqm;, ;,mos UJ 1]1!-\"\ Tiu!PI'J;oJU!
I' UO ;,1\p.'"O um'i!a.J ;otjl Ual(.\"\
mmmxtmt 1' .\q uo!TiaJ
e uo pueqJaqqnJ e set] tetp UO!WJado pueqJaqqiU
e JOJ !d\" tm sap!MlJd :-jJO.\\ameJJ at]l JO .\JeJq!]
atp"Ju;ounpoqma tm UJ ;,qlJo pu;, aq1 11' utl!'ilaJ
09 aJe.\\lJOS a::>e:paJU] Jasn t]l!.\\ mammu],\Ua ue m aJeJ
-J.'))U] ilU]UHUe.riloJd UO!)P-J!]dde atl.L '.')JilSO]JS]p JUaSaJd at]l
JO S'i!U!l(:ll';ol ll!l'J.l;o:l OJ Ul.IO.JJ.m1p .\"\0[1 U! ;o:JIOJ.l;oJU!
.JO f:'f "!)J:I
JUasaJd atp m spoqpm
at]JJO amm w wama]dm] 01 pasn aq ueJ 00 aJ]\ap do1
-de] at]l masaJd atp JO umua::> U]
m ;,ll(!SL\ 'ilu!;,q um1la.1 ;o1]1JO 1m umTia11'
UO paseq r, JO UO!IJ.')J!P Ue U! UO!ilaJ pa]]OJJS e
JO a::>unoq e 01 uopeJado a::>tmoq e JOJ !d\" ue O>
'jJll.\\.')UWJJ .')l]l JO ,\JeJq!] at]) ")Uam!pOl(UW UP- U]
.\.mpa.l!P a['ilms U!
Jl"]l (":JP "jXaj "sa'i!tml! 1!;,) Sa:JJllOS.'"OJ paW! I'
os1e :pm\attmJJ atn 'ilmwmp:m pur, 'i!u!Jntsail
'ilmmJOJJad ilmmumJil<.ud
UllljlD![ddl' ;op! \tud OJ UUO.JWjd aqJ.JO S1Ul!J1Jtud aqJ 1]1!-\"\
-J;oJm at(.]_ aptn am-\\uos .JO .\.tt!ll(!] 1' 'ilmu!I'JmD
)jJO.\"\.'"IttmJJ 1' sapnpu! mJ<.:]Jqd aq1 amos UJ
'UO]ilaJ ,\r,]dS]p
e JO .\ejds]p e Jo ailpa matuoJ e 01 JOWJtpm ot
(]lU:lS aml J;ol(Pl(.\"\ ;oumuapp OJ ld\' 1j'ill10.ll[l(]lD
JOJI':l!Pll! ][OJ:lS 1\U!JJ:J_jSUl!lj Jl:J ;opnpm S1Ul!J1J
-<.ud aJom Jo auo at]l tuam]poquw ue U] e iluno,\m
tndu] atp uo mdm Jasn e JOJ JOPBJ UO]WJap::>ap e
j.')S 01 ]d\'UP- t]f!HOR]l ]jeJ ]]OJJS UO])eJa]aJap r,
;,pnpm sum.R!OJd aJotu Jo ;,uo ;,111 "Ju;ounpoqma
U] JlldU! Jasn Jal(Pl(.\"\ ;oumuapp OJ ld\'
tm t]ilmut]J]]eJ JOJ
apnpm aJom Jo auo atp watmpoqma ue U]
0[
(l'JUOZWl1]1' Jo (l':l!J.l;o\ u! TIU!J(OJ:Js ;oqJ'ilU!)j:lO(
J<.:J a['iltm wu:Js OJ JdV qTinoJqJ !Jl':l lions Jl'Ull!J
-J.')J!p e f!uUJq]SUeJJ JOJ apnpU] .')JOm
Jo auo at]l"JUam]poquw ue U] ']]OJJS e uo ailpa ,\e]dS]p
I' U0!1JaJ ;o1jl Ual(.\"\ jUama:Jl'jdS!P lUillU!Xlml ,:c
p.'"lmmJapp.<ld I' .\q UO!TiaJ I' uo P:J_]p ptmqJ;ol(lJIU
1' OJ JdV 1m qTimut]J ][l':l p1mqJ;oqqn.1 1' .H:J
apnpm sme.RloJd aJom Jo auo at]l nwunpoqma
tm U] uoJJs at]l JO pua at]JJe uotilaJ e U] ilupq
um1JaJ aqJ.JO ;o'i!p.'"O 1m JSI'd U0!1JaJ I' UO p.'"OSI'l( !!ODS I' flC
.JO UOlj:J.'"OJ!p a]!SOddo 1m U! UO!'i!.'"OJ pa(]lU:lS I'JO;o:JUilOlJ I' aSlllD
01 ]d\' Ue t]f!!llUt]l ]jeJ .')JtHlOl( e JOJ
apnpm aJom Jo auo atp watmpoqma ue U]

'09f
.\poq 1qo do1 m1 99f attmJ.J ptm Z9Z
e t]JJ.\\ aJ]\ap ta]qm atLL .) 'Dl:l m paWJlsnm se aJtAap
1a]qe1 e ilmmJOJ ]pun ZS pJeoq.\a)] e JO do1 at]J tS
,\poq at]l U!t]l!.\\ S] 8S r, ilU]UWJUOJ 9S ameJj
.JO ;,1\p.'"O UV ;o:J! \;op PllJI'l OJU! a:J!.\ap ;ol[l.JO
UO!SJa.\UO:J;o1]J !)1:1 .)f "!)1:1 p1m U!
patemnm se aJ]\ap ta]qm e OJU] paua,\UOJ aq tmJ()O aJtAap
d01de] atn '80 e pur, "90 ameJJ e po
.\poq 1' zot:f l(Jl\\ ()Off ;o:J! "''"'P do1dq 1'
Vff "!)!:! ;o1]J.JO U!l'J.l;o:J JSI';ol w OJ
'ilu!pm:J:JI' a:J!.\ap JO ;o[dumx;, Ja1]Jlllm .)Vf 'S!)!:I
masaJd atp m spot]tam
at]J JO amm 1sea1 w JtWUW]dm] 01 aq tmJ OOZ aJtAap
;,111 JUasaJd a111 JO SJU;ounpoqma umu;o:J UJ
a:l!.\ap aqJJO aJotu JO auo m JU;otujS11fp1' 1m asm':J
.\ue::>nemome uat]J .(em WZ JmsaJ<.UdtuJtm at]J
Wt]JJO J]nsaJ e sy u]aJat] s.\e.\\ Jat]JO at]l m Jo
-OJd a::>twilqpJU] ]P-P!J!JJP- tm t]ilnon]J paz.\jetm aq .(em etep
;,111 ;,]dttmx;, m:l un1;o11 Jammm aq1 u! l'll'P Jl'l(l
JO aq1 uw:pad .\1'm p1m 91Zf s.Josuas aJom Jo ;,uo
m<.UJ etep a,\]aJaJ .(em ZOZ Jossa::>oJdoJJ]m at]J ooz aJtAap
at]) JO UWIJ.')J )e Uj "JOSUaS H]'il!] )tW
-tqme ue .(J]UHXOJd e a]dmexa JOJ ilu]pnpu] upJat]
paq!J:Jsap aq1 .JO ;,uo aq .\tml 91Zf sJosuas ;o1l.L
'(.l:l.\!a:lS1mJj :nJ 1m 1\a) )j.IO\\PU I' JO )j.IO\\PU p.<H\\
e JOJ Jat]l!a tptt].\\ a::>BJJalm )]JO.\\lau e apnpm .\em tlZ
saJ]\ap tndtno;tndu] at]J aJottuat]Uil:J pued tndm tpno1 e
tpns a:J!.\ap ]OJJUO:J JosJn:J m p.11'oq.\;,1 m p1,d,\;,)j ;,pnpu!
.\I'm f:'JZf sa:J!.\;op JlldJilll/Jildm at(.]_ ;o1]J ll! l!.Wlll)jjp\\ S!
Sl' JOjptm saTip!.!lJ Sllll!.m.\ 1j'illllU1jJ J;o1jJO
tpea 01 aJom Jo auo apnpu] .(em 90Z
atl.L .(j!A!lJe J.')Sil JOJ .')J!J l]J!l].\\ 9JZ .')JOm
JO ;ouo .ll:J apL\tl.ld OJ ptm f:'JZf JlldJilll
1111dm JO.J ap!.\OJd OJ 90Zf snq atp OJ pa[dno:J
aJe n Z tndtno;tndu] aJom Jo auo "90Z
at]l t]'ilnon]J Jat]JO at]l 01 pa]dnoJ st tptt].\\ 01 Z
aJtAap pue Ja]]OJJUOJ ,\e]dstp e apnpm .(]]eUO]ldo
-t!Jado 'i!U!ll'm!1m ptm 'i!mJn]saTI Sllll!JI,.\ mn:p;od ,:1 .\tml ;o:Jn;op S!1l.L "Z()(':f ;o1jl OJ p.'"Ojdllll:l S!
tpll(.\"\ 80Zf apnpm .\j(l'utl!Jdo .\tmt OOZ a:J!.\ap ;,11.1.
'90Z snq e t]ilnoJt]l Jatpo tpea 01 pa]dno::> aJe tp]t].\\ poz
,\Jomam e pue zoz JmsaJ<.UdtuJtm se tpns e
OJ .1ap.1o m TimmumJTIOJd tm 1J'ilnoJ1]J
JO S!!!JJ UO]IJU!1J ilU!-fJ.')jSUeJJ
apnpU] .')JOm JO .')U() 'J!'Hl
;,tp .\q aq OJ pam'ilgmn ptm .\.tmu;,m a111 u!
am Jl'l(l S1Ul!JTitud aJotu Jo ;,uo 'i!U!Jm;odo ol
;,q.]_ ;,111 OJ .\.tmu;om1' p1m p1md Jndu! tm OJ
pa]dnoJ tmn e apnpm .(em aJR\\pJet]
atLL ilU]JeJado ue pur, SJtwuodmoJ aJP-.\\pJet]
mn:nqd at(.]_ 1lmw1m1m p1m 1!u!JI1JSaTI 11m
-Hons sno!.m\ sap! \tud tuJO.JW]d amos UJ
'S.\\.'"OJ,\.]0 ,\tp
-JeJatt] at]J .\\O]aq ilupeJado JO ,\tpJeJatt] atp
JHOt]l!.\"\ !d\' atp m,\ JO ,\tpJeJap:] atp t]ll.\\ pr,Jalm
.\tml Ull!JlD![ddl' UV 'f:'()tf )j:lOjl( 11' s.n\t'] JO .\t(:lJl!la!l( ;o1]J
zz
;opnpu! .\1'm OOZf a:J! \;op amsops!P aqJJO JUam!pDqm;,
tm OJ'ilmp.lo:J:JI' ;o:J! \;op a1dumx;, J;o1]Jmm s.\"\Ol(S Zf "!)!:1
JUasaJd atp m spoqpm
at]J JO amm 1sea1 w JtWUW]dm] 01 aq tmJ OLO aJtAap
at]l wasaJd atp JO swatmpoqma umua::>
Ul "VSOJf aml1jdOJ:l!Ul ;ouo JSI,;oj 11' p1m ;,mm JO
,.- ;,uo apnpm .\1'm 1]J!1]\\ "80 1 sJa:Jnps1mJJ apnpm
,\em OLO aJ]\ap .\Jaueq apup.\t] ]mam ]a'jJJU Jo
um]t]l!] a]qeailJmpaJ-uou Jo a]qeailJmpaJ e a]dumxa aq
.\em LOT .\Jauq] '901 aJ]\ap pue on (s)Josuas
"6() If m;oJK(s mpam "80I f Ja:Jnps1m.11 mpm' vo If
lZ
zs > J6'vv8'L sn
us 7,844,915 82
23
embodiment. the !ihraryofthe !ioJmewmk provides an API !(Jr
specil)ing. a scroll indicator operation to dctenninc whether
at !cnst one scroll indicmor attaches to a content <..-dgc or a
display edge of a display region.
In the foregoing spcciHcation_ the disclosure has been
5
described with rc!Crcncc to spccilic exemplary cmhmlimcnls
thcrcoL II will he cvidcnllhal \arious modi!icalions may be
made thcrclo without departing li'llm the broader spirit and
scope of the disclosure as set fonh in the following claims.
1
,
The spcci!icmion and drawings arc. accordingly. to be
rcgnrdcd in an i!lustrmivc sense rather than a restrictive sense.
What is claimed is:
I. A machine implemented method ll1r scrolling on a
louch-scnsiti\c display of a device comprising:
l'i
receiving a user input. the user input is one or more input
points applied to the touch-sensitive display that is intc- ""
g.ratcd with the device:
creating. an event obj< .. >ct in response to the user inpm:
determining whether the cvem object invokes a scroll or
g.csturc operation by distinguishing between a sing.le
input point applied lo the louch-scnsili\c display that is "-'
inlcrprc!cd as the scroll operation and two or more input
points applied lo the louch-scnsili\c display that arc
imcrprctcd as the gcsmrc operation:
issuing. at !cast one scroll or g.csturc cal! based on invoking
30
the scroll or g.csturc operation:
responding. to at least one scroll calL if issued. by scrolling
a window having a\ iLw associa!L"d with the c\cnl objL>cl
based on an amount of a scm!! with the scm!! slopped a!
a prcdc!crmincd position in relation to the user input: 35
and
responding. to at least one gcsmrc call. ifissucd. by scaling
the view associated with the event object based on
receiving. the two or more input points in the fonn of the
40
user input.
24
rL>cei\ ing a user input. the user input is one or more input
poims applied to a touch-sensitive display that is hue-
grated with the data processing. system:
creating an cvem object in response to the user input:
dctcnnining whether the event object invokes a scroll or
gcsmrc operation by disting.uishing. between a sing.!c
input point applied lo the !ouch-sensitive display !hal is
interpreted as the scm!! operation and two or more input
points applied lo the !ouch-sensitive display !hal arc
intcrprck-d as the g.csturc operation:
issuing at least one scroll or g.csturc call basL'<i on invoking
the scroll or gcsmrc operation:
responding to at least one scroll cal!. if issued. by scrolling
a window having a view associated with the event
objL>cl: and
responding loa! leas! one gesture calL if issued. by scaling
the \iLw associa!cd with the L"\cnl objecl based on
receiving. the two or more inplll poims in the form ofthc
user input.
9. The medium as in claim 8. li.Jrthcr comprising.
rubbcrbandinga scrolling. rcg.ion displayed within the win-
dow by a predetermined maximum displaccmcm when
the scrolled region cxce\.:ds a window edge based on the
scroll.
I 0. The medium as in claim 8. further comprising:
allaching scm!! indicators loa content edge of the\ iLw.
I 1. "!.he medium as in claim 8. further comprising:
allaching scroll indicators loa window edge of the view.
I2. n1e medium as in claim 8. wherein dc!ennining
whc!her the e\enl objL>cl imokcs a scroll or gesture opl."ralion
is based on receiving a drag. user input Jix a certain time
period.
lJ. The medium as in claim 8. further comprising.:
responding to at least one gcsmrc call. ifissucd. by rotating
a\ iLw associa!L"d with the e\enl objL>cl based on rL>cei\ ..
ing a plurality of input points in the J(mn of the user
input.
2. "!.he mc!hod as in claim 1. further comprising:
mbbcrbanding a scrolling region displayed within the win-
dow by a prcdc!crmincd maximum displacement when
the scrolling. reg. ion exceeds a window cdg.c basL'<i on the
scroll.
J. The method as in claim 1. further comprising.:
allaching scroll indicators to a com em cdg.c of the window
4. The method as in claim 1. further comprising.:
allaching scroll indicators to the window cdg.c.
I4. "!.he medium as in claim 8. wherein the data processing
system is one of: a data processing device. a portable device.
45 a portable da!a processing. device. a multi touch device. a
muhi !ouch portable de\ ice. a wireless device. and a cd!
phone.
I 5. An apparatus. comprising:
means !(Jr receiving. through a hardware device. a user
5. n u ~ method as in claim 1. wherein detcJmining whether
the even! objL>cl imokcs a scroll or gesture operation is based
on receiving a drag user input l(Jr a certain time period.
'0 inpm on a touch-sensitive display of the apparams. the
user input is one or more input poims applied to the
louch-sensili\e display !hal is intcgra!L"d with the appa-
6. The method as in claim 1. further comprising.:
responding. to at least one g.csturc call. if issul.-d. by rotating
a view associated with the event object basL'<i on receiv-
ing a plurality of input points in the Jixm of the user
input.
7. "!.he method as in claim 1. wherein the dL"\ice is one of:
a data processing device. a portable de\ icc. a pm1able data
processing. device. a multi touch device. a multi touch por-
table device. a wireless device. and a eel! phone.
8 . .-\ machine readable storage nK'<iium storing cxccmable 65
program inslmclions which when execu!L"d cause a data pro-
cessing system lo per!(mn a method comprising:
ralus:
means l(Jr creating an e\enl objL>cl in response lo the user
inpm:
means for determining whether the event objL'Ct invokes a
scroll or gcsmrc operation by disting.uish.ing. bctwL-cn a
sing.le input poim applied to the touch-sensitive display
that is inlcrprc!ed as !he scm!! operation and two or more
input points applied lo the !ouch-sensitive display !hal
arc imcrprctcd as the gcsmrc operation:
means for issuing. at least one scroll or gesture call based on
invoking the scroll or gcsmrc operation:
means for responding. to at least one scroll call. if issued. by
scrolling a window having a \iLw associa!cd with the
even! objL>cl: and
us 7,844,915 82
25
means !l1r responding to a! least one gestun:: call. if issued.
by scaling 1hc view nssociatcd with the event obj< .. ><::t
bas<..-d on receiving the two or more input points in the
li.mn of the user input.
16. The apparatus as in claim 15. !lu1her comprising:
means !(Jr rubhcrbamling a scrolling region displayed
within the window by a predetermined maximum dis-
placement when the scrolling region exceeds a window
edge bas<..'d on the scroll.
17. The apparatus as in claim 15. !lu1hcr comprising:
means ll1r allaching scroll indicators loa content edge of
the window.
18. The apparatus as in claim 15. Jlu1hcr comprising:
means for nttnching scroll indicntors to the window edge.
26
19. The apparatus as in claim 15. wherein dc!cnnining
whether the event obj< .. ><::t invokes u scroll or gesture operntion
is bused on receiving u drug user input Jix u certnin time
period.
20. The apparatus as in claim 15. Ji.n"lher comprising:
means ll1r responding to at !east one gesture calL ifissmxL
by rotnting u view associm<..'d with the event object bused
on receiving a plurality of input points in the li.mn oft he
user input.
21. rhc apparatus as in claim 15. wherein the apparatus is
one of: a data processing dL"\ ice. a pm"lah!c de\ icc. a pm1ah!c
dmu processing device. u multi touch device. u multi touch
portuble device. a wireless device. and u cell phone.












































Exhibit 6
(12) United States Patent
Chaudhri et al.
(54) \1ETIIOD.\:'\I>.\PP.\Il..\TUS FOR
I>ISPL\YI:'\G.\ WI\"DOW FOR.\ USEil.

(75) !mentors: Imnm ChaudhrL San Francisco. CA
(US): Ba!> Ording. San Francisco. C.\
(US)
Assignee: Applt> Inc .. Cupc11ino. CA (US)
( * ) Notice: Subject to any disclaimer. the term of this
patent is extended or adjusted under
U.S.C. 154(b) by 247 days.
(21) App!. No.: 12/012,184
(22) Filed:
(65)
Feb. 1, 2008
Prior Publication I>ata
(63)
(51)
(52)
(58)
(56)
US 2(Xl810222554 .\1 Sep. 11. 2<Xl8
Rehlted U.S. Application nata
Continuation of application No. 111635.847. Hied on
I kc. 8. 2000. which is a continuation or application
No. on Jul. 10. 2(Xl2. now Pat. No.
7.343.566.
Int. Cl.
<ifJ6F 17/f)(J
U.S.Cl.
(2006.01)
7151781.7151708:7151788:
7151795: 719/318: 3451629
Field of ClaJ>J>ification Search 7151768.
7151781. 788. 795: 7191318: 3451629
See application Jile l(Jr complete search history.
Reference!> Cited
U.S. P.\TI:NT I)OCUMI:N"rS
.'i.19Ui20 A 3 1993 l.cc
." .. 133.272
'
7 1994 C1pek cl "1
s.:nu;ts
'
ll 1991> Drc\\ s cl ,]
.'i.iG1>.311> A ll 1 <.m: l'oncr ct al
.'i.i-:92511 A 4 1999 Gelsinger ct al
.".<)29.1>."4
'
7 1<)9<) Ross
I IIIII 1111111111111111111111111111111111111111111111111111111111111
US007853891B2
(I OJ Patent No.: US 7,853,891 B2
(45) Date of Patent: Dec. 14,2010
5,940,."17
'
g 1<)<)<) Shimm<hi al
5,<)<)<),1<)1
'
12 1<)<)<) Frank al
ti,OOlU<O<J
'
12 1<)<)<) Brooks
(i.072.4g<)
'
6 2000 Gough d al
(i.24(i.407 lll 6 2001 \\'ilks ct a!
(i.24<JJQ(i Bl $ 6 2001 !'any ct al 710 ]<)
(>,252,-':;<)5 Ill 6 2001 Birminglmm ,]
(>,307,."45 Ill 10 2001 Conrad "1
ti,Bti,Ul Bl 1 2002 Wo!J(, ......................... 70<) 20.1
(>.409.603 lll 6 2002 :-.:ishino d al
(i.42<JJiiO lll g 2002 Plowctal
(i.600 . .'i00 Bl $ 7 2003 Yrunamoto .................. 7L'i 7'!5
(Continu<..'d)
l'rimwT Hxaminer Ti1deese llailu
(74) .11/orne\: Agent. or 1-i'rm Brian K. McKnight: Nmak
Dmce +Quigg I.I.P
(57) AHSTR\(T
Methods nnd nppmatuses to displny windows. In more than
one embodiments of the invention. a window is closed mno-
matically (e.g .. a!ler a timer expires. or when a condition or
criterion is met. or a system input is n ..x:ei\L-d) without user
input.ln some exmnples. the window is translucent so that the
portion ofnnother window. when present. is visible under the
window. In some examples. the image or the window is li1ded
out the window is closL-d and destroyed. In some
examples. the window does not close in response to any input
Ji"om n user input device. In some examples. the window is
repositionL'<i (or hidden) autommicnlly when nnother translu-
cent window is displayed. The degree or translucency. the
speLxf J(Jr !i!ding OU\. the discrete levels or translucency J(Jr
JUding out. the time to expire. and/or other pnrameters l\1r
controlling the displny of the window mny be set by the user
or adjusted by the system (or npplication software programs)
automatically according to system conditions or other crite-
na.
75 Claims, 21 Drawing Sheets
U.S.
Bl
6.(>70.970 Bl
6.720.<)1>2 Ill
6.:m<.<Jg<J
B2 ''
6.907.447 Bl
7.046.254 B2
7.061>.266 Bl
PA:t-I;N"r J)OCUMI;NI-S
ll 2003 Jones
12 2003 Bonura d al
4 2004 Sahguchi
US 7,853,891 B2
Page 2
7.155.72<)
7.342,-':;<)4
2002 Ot<JJ02g
2003 OOOll><J<J
2003 00Hi25.1
2003 004.1! <)7
lll
Ill
.\l
.\l
\l
\l !2 2004 Cortright ................... 715 769
(i 2005 Cooperman ct al 70<) 2o.1
2003 005J22g .\l
5 200(i Bmwndal
2003 01450(>0 .\l
(i 200(i Rudk * cited by examiner
12 200(i ct al
3 2001> Orl d ,]
12 2002 ct a!
1 2003 Pm1ancn ct al
l 20o.1 -\oki d"l
3 20o.1 Kremer cl al
3 2003 Ylaninct ct a!
7 2003 Ylanin
U.S. Patent Dec.14,2010 Sheet 1 of 21 US 7,853,891 B2
104
101
Gache
/
/103 /107 /105 /106
Volatile
Nonvolatile
Microprocessor ROM
RAM
Memory
(e.g. hard drive)
/
/102
l l l
"" ""'
""
/108
l
/109
/
Display Controller 110
& Display Device Contro!!er(s)
l
/110
1/0 Oevice(s)
(e.g. mouse, or keyboard, or
modem, or network interface, or
printer, or scanner, or video
camera)
Fig. 1
U.S. Patent Dec.14,2010 Sheet 2 of 21 US 7,853,891 B2
203
207
205
1 %:'Hm M
')1-;.h W,>lfm;. >!: to) US- M:l <::<,;p$K
M>::IWW !t> Al;lqw,
Fig. 2
Prior Art
201
209
210
U.S. Patent Dec.14,2010 Sheet 3 of21 US 7,853,891 B2
215
Fig. 3
Prior Art
U.S. Patent Dec.14,2010 Sheet 4 of 21 US 7,853,891 B2
223 225
221
227
229
Fig.4
Prior Art
U.S. Patent Dec.14,2010 Sheet 5 of21 US 7,853,891 B2
231
Fig. 5
Prior Art
U.S. Patent Dec.14,2010 Sheet 6 of 21 US 7,853,891 B2

rm
"'"it
K:&
.




?'J,
tl er
.
:&1J
_____ ,
a
"'
.
G&* t'e:.;.11>:\'l


' ., .
'-''
((<>;)
a

.
''':--
W Mm M>.'<>


.,
"tl

..
- y
Nettw<.'!k P>.r;%"flds Q'-"':-Hb.e



Jl



r <'kt.f:vrq
s. ...
rP
Fig. 6
Prior Art
U.S. Patent Dec.14,2010 Sheet 7 of21 US 7,853,891 B2
313 310
311 315
J
'
'080
Folder A
Name Date Size
303
LJ Favorite 3115/02 1:20PM
301
"
LJ Video Clips 4115/02 5:20AM
LJ Images 3/25/02 10:20 PM
L:Jremp 4/22/02 11:20 AM
Fig. 7
U.S. Patent
301
"i
Dec.14,2010
LJ Favorite
LJ Video Clips
LJ Images
Sheet 8 of 21 US 7,853,891 B2
Folder A
3115/02 1:20 PM
4/15/02 5:20AM
3125/02 10:20 PM
Fig. 8
U.S. Patent Dec.14,2010 Sheet 9 of 21 US 7,853,891 B2
88
Folder A
Name Date Size
301
303"
LJ Favorile 3/15/02 1 :20 PM
LJ Video Clips 4/15/02 5:20 AM
LJ Images 3/25/02 10:20 PM
LJ ... _. __

---- --------
Fig. 9
U.S. Patent
303"
301
Dec.14,2010
Name
LJ Favorite
LJ Video Clips
LJ Images
LJ Temp,,'
.
:--v.
'
/ ....
341
Sheet 10 of21 US 7,853,891 B2
Folder A
Dale Size
3/15/02 1:20 PM
4115/02 5:20AM
3125102 10:20 PM
.. 4l22102.1.1:20:AM'
_._._. ___ ,
---------------------'
Fig. 10
U.S. Patent Dec.14,2010 Sheet 11 of21 US 7,853,891 B2
80
Folder A
Name Date Size
LJ Favorite 3/15/02 1:20PM
301
LJ Video Clips 4/15/02 5:20AM
LJ Images 3/25/02 10:20 PM
Fig. 11
U.S. Patent Dec.14,2010 Sheet 12 of21 US 7,853,891 B2
Display a user interface window (e.g., a translucent window
v
which when displayed on top of a portion of a second
401
window allows the user to see the portion of the second
window through the translucent window)
Automatically close the user interface window {e.g., fade
out an image of the window and destroy the window)
without user input (e.g., after a timer expired, or after a
v
determination that a system status is changed or a
403
condition is met, or after receiving input that is not
associated with any user input device)
Fig. 12
U.S. Patent Dec.14,2010 Sheet 13 of21 US 7,853,891 B2
Display a first window in response to an input (e.g., an input
from a user input device, or an input that is not associated
/
with any user input device, such as an input !rigged by a
411
system event, a change in system status, ringing signals on
a phone line, or inputs initiated by the operating system)
413
Start a timer
/
Close the first window when the timer expired (e.g., fade
/
out an image of the first window and destroy the first
415
window)
Fig. 13
U.S. Patent Dec.14,2010 Sheet 14 of21
Receive an input (e.g., a user input from a user input 421
device, such as a keyboard, a mouse, a track ball, a touch
pad, a touch screen, a joy sticker, a button, or others) from
a digital processing system
Display a first translucent window on a display device (e.g.,
a LCD display, a CRT monitor, a touch screen, or others) of 423
the digital processing system (e.g., on top of a portion of a
second window), where the first window does not close in
response to any input from a user input device of the digital
processing system
425
Start a timer
~ - - - - - - - - - - 1 Process the mput
Is an input
(e.g., a user input
427
or a system input) for the first
window received "'
Restart the timer
?
429
Is a second
translucent window displayed
?
Close the first translucent window when the timer expired
(e.g., fade out an image of the first window and destroy the
first window)
Fig. 14
435
437
US 7,853,891 B2
433
431
U.S. Patent Dec.14,2010 Sheet 15 of21
441
Combine the image of a translucent window and the portion
of the image of window under the translucent window to
generate a combined image for the translucent window
and the window under the translucent window
443
Display the combined image on the screen for the
translucent window and the window under the translucent
window
445
US 7,853,891 B2
Is the translucent window
updated?
, - - - ~ - - - J .
447
Is the
wlndow under the translucent
window updated
7
Fig. 15
,., _____ ..J
U.S. Patent Dec.14,2010 Sheet 16 of21 US 7,853,891 B2
Fig ~ 6
U.S. Patent Dec.14,2010 Sheet 17 of21 US 7,853,891 B2
Fig 17
U.S. Patent Dec.14,2010 Sheet 18 of21
t -..-
__ , ,,j
'--'
Fig. 18
US 7,853,891 B2
U.S. Patent Dec.14,2010 Sheet 19 of21 US 7,853,891 B2
Flg 19
U.S. Patent Dec.14,2010 Sheet 20 of21 US 7,853,891 B2
Fig. 20
U.S. Patent Dec.14,2010 Sheet21 of21 US 7,853,891 B2
Fig 21
us 7,853,891 82
1
VIETIIOD X\!) APPARATUS FOR
DISPL\YI\"G .\WI:'\ DOW FOH. A USER
I\"H:H.E\(:E
2
The progress of the operation is indicated by progress bar 229
and an animmion showing thm the documcm is going from
one folder to another. Windows 221 and 231 in FIGS. 4 and 5
show two snap shots of the animation. A user may drag title
'!lu:: present application is a continuation or co-pending
U.S. application Scr. No.
1
v,-,_,._847. Ji!cd Dec. X. 2006. which
is a continuation of U.S. applicmion Scr. No. 10/193.573.
tiled Jul. 10.2002 now U.S. I'm. No. 7.343.566.
'i bar 223 (e.g .. pressing down and holding a bull on of a mouse
and mm ing the mouse while holding down the lmllon) lo drag
the window from one location on a screen to another: and the
user can click on the title bar to bring the window to the top
level when the window is p;u1ial!y covered by another win-
1:11;11)01'-r!!l; !N\'ION'I-!ON
"111c invention rdmcs to graphical user intcr!Uccs. and more
particularly to such intcr!Uccs with windows.
lCJ dow (e.g .. when window 241 is panially cmercd by window
243. as shown in !'!(i. 6). When the copy operation completes.
the progress window doses automatically
Traditional windows typically provide strong user imcrnc-
tions. which may causedislraclions. !'orcxample. a user wails A pm1ion of the disclosure of this palcnl document con-
tains material which is subject to copyright proh.x:lion. The
copyright owner has no objection to the facsimile reproduc-
tion by anyone of the pmcnt document or the patent disclo-
sure. as it appears in the Patent and Trademark Ollicc palenl
!ik: or records. but otherwise reserves all copyright rights
whalsoc\cr. Copyright Apple Computer. Inc.. 2(XJ2.
l'i !l1rwindow 201 lo disappear lo viLw window 210 in I'Ki. 2:
the user manipulmcs a cursor comrol device (e.g .. a mouse. a
track balL or a touch pad) to view or dismiss !lash help
window 2B in FIG. 3:and. the user interaction is provided 10
relocate the progress window or change the window display-
"" ing hierarchy lo sec the progress of window 241 in FIG. 6.
BACKGROUNI) OF Till; INVJ:N"rJON
Many digital processing systems usc window-based
grnphical user imcrfaccs for users to interact with soliwarc "-'
progrnms. Images of windows arc display<.'d on a display
dL"\ ice lo show the user the slates of the so!iware programs:
and user input dL"\ ices (e.g .. a keyboard and a mouse) arc used
SUMMARY OF Till; INVJ:NTION
Me! hods and apparatuses lo display windows arc described
here. There arc many difl"ercnt cmbodimcllls which arc
described here. Some ofthcsc embodiments arc sununarizcd
in this section.
In morel han one embodiment oft he invention. a window is
closed aulomalical!y (e.g .. alkr a timer expires. or when a
condition or criterion is met. or system inpm is received)
without user input. In some examples. the window is translu-
cent so that the ponion of another window. when present. is
visible under the window. In some examples. the image of the
window is bdcd out b d ( m ~ the window is closed and
lo accept user inputs. In addition lo user input dL-,.ices. a
digital processing system may have other devices (e.g .. sen- 30
sors) Jix accepting sys1em inputs. such as phone line status.
power supply status. storage disk usage status. cmnmunica-
tion conncclion status. execution status of so !\ware programs.
and others !hal arc not dirL><::tly relm<.'d to user inpms (signals
associak'd with user input devices).
35 dcstroyL'<i. In a further example. the level oftrnnsluccncy. the
spc<.'d for !Uding out. the discrelc levels of translucency for
bding out. the time lo expire. and/or other paramclers ll1r
controlling the display of the window may he sci by the user
!'IGS. 2-6 illuslrmc examples of traditional windows. FIG.
2 shows a typical window 210 that has title bar 203 and
lmllons 205. 207 and 209 !l1r minimizing. maximiting. and
closing the window. The title bar and the buuons on the tilk
bar can be used to manipulating the position and site of the 40
window. !'or example. tilk bar 203 may be clicked (e.g ..
pressing a bul!on of a mouse while the cursor is on the title
bar) lo bring the window lo the lop level of the window
displaying hierarchy so !hal if there arc any other windows
displayed m !he same location of window 210.!hcsc windows 45
wil! he hidden under window 210.
or adjusted by the system (or application sollwarc programs)
mnomatical!y according to system conditions or other crite-
ria.
In one embodiment of the invention. a mc!hod lo display a
user interliJCc window !(H a digital processing system
includes: displaying alirsl window in response lo rccei\ ing a
Hrst input from a user inplll device (e.g .. a keyboard. mouse.
track ball. touch pad. touch scrL-cn.joy stick. button. or others)
of the digital processing system which is capable of display-
ing a! leas! a portion of a second window under the !irs!
window: starting a timer: and closing the !irs! window in
When a user starts an application program. a window (e.g ..
window 201) may he displayed lo show the license/copyright
information while the componcms of the sollwarc program
arc being loaded. Aller the componcms of the sollwarc pro-
gram arc li.J!Iy loaded. the license/copyright window is closed
aulmnalica!ly so that the user can start lo use the so!iware
program without inter!Crcncc !i-om the license/copyright win-
dow.
'0 response to a determination that the timer expired. The Hrst
window docs not close in response 10 any inplll li"om a user
input de\ icc of the digital processing system. In one example
according lo this embodiment. the !irs! window is lransluccnl:
the portion of the second window is visible while under the
!'IG. 3 shows task bar 211 with Hash help window 2B.
When the user pauses cursor 2 I 5 a! a location of the !ask bar
'5 Hrst window: and the Hrst window is at a top level in a window
displaying hierarchy. In one example. an image of the !irs!
window is li1dcd out on the screen bcll1re the !irs! window is
destroyed lo close the !irs! window. In one example. the
ll1r a short period of time. !lash help window 213 appears. If
the user docs not move the cursor for another shon period of
time while window 2B is displayed. Hash window 2B dis-
appears. If the user moves cursor 215 sliglllly (e.g .. using a 60
mouse. a !rack halL or a !ouch pad) and pauses the cursor 215
again. !lash help window may appear again.
!'IGS. 4-6 show a window !hal displays the progress of
copying a !ilc. When a !ilc is copied li"om one location to
an01hcr location. window 221 is displayed to indicate the 65
progress. !Sullon 227 is provided ll1r canceling the copy
operation: and button 225 is ll1r closing the progress window.
second window. if displayed. doses in response 10 an input
li"om a user input device of the digital processing system: and
the !irs! window docs no! respond lo any input Jiom a user
input dcviceofthedigital processing system. In one example.
the !irs! window is rcpositiotK'<i in response to a third window
(e.g .. an alert window or a trnnsluccm window) being dis-
played: in another example. the Hrst window is hidden in
response lo a third window being displayed a! a location
where the !irs! window is displayed. In one example. the !irs!
.\.mm;,m ;,[!Jqtn-uou ;oqJ JI"[J 1 !)!:! ;o[ll(M p.nnh;oJ
J<.m t["ilnm[J(l' .\.n1m;om umput!l 1' ;,q 11!1.
,\.iom;,m :l(!JP-[0\-UOU :ll!J",\[[P-:l!d,\.J. :ll(J mOJJ
J:J\Ie eJep umwwm tp!t].\\ .\.iouwm
'S.Lopu!.l[lmO!J!Pl!llJO ;)Jl,JJS!l[[l 9-z 'S!)!:1
"UO!JU;).\Ul ;ot1J l(J!.1. ;oq .\I'm l(:lll(.1. m;oJK(S
l'WP 1'.]0 ;o[dumx;, mt,J"i!t,!p )j:lO]q I' S.Lm1s J "!)!:1
:l)jl] tp]tj.\\
m :ltJJJO :ll!J m UO!)eJ!m]] wu
pur, :ljdmex;, JO .(q
:10
JO ;,d.\J JO 11\l\(1 (!.\(] e JO :lA!Jp ]P-:l!JdO Ue JO :lA!Jp
(lD!Jdo :l!PU"ill'm 1' .10 ;o\up p.11"1 :J!puTit'm 1' .\[[tD!d.\J s1 901
.\.1om;,m ;,[!Jqtn-uou ;,t!J. .\Jom;,m ;oqJ m t'JI'fl ;,t1J umJU!tml .10
Ul !]:1!!1.\\ (11\l\ (!(!) 09
Iii!\ (I :mueu.\p se .\ue:J!d.\J :lJ!J
-l'Jll.\ ;)l(J. '601 ]!1d]lllljJ!1dU!l("illllUl11 ;)l(J OJ
am 01 1 J!ldJllll/Jndm ;oqJ .\1[!':l!d.\t J.m ;oqJ m
U.WlU)j !!;,.\'\.am l[:)ll(.1. S:JJ! \;op J;ol1JO put' o;op!.\ "KJ;oU
spJeoq.\;,1
tpltj.\\
:ltJJ UJOJJ pur, silupwJp moJJ
. Jllall,ddt' ;oq ]]!'>\ UO!JU;).\Ul ;)l(J ]0 S:JJlljl,;)j J;ol11()
' ' ' . .
;)S:ll1J llllOJl:ld OJ
;,t11 ;osm,;, sm;oJK(s "ilmss;o;,<.ud I'JI'fl uo p;,Jn;o.--.x;, u;,t1'>'.
tp!t].\\ mp;,m pue mJOJ
-J:Jd tp]tj.\\ ewp ilu]pnpu]
um:JJ;od tpll(.1. UO!JU;o\u! JU;osaJd ;,tLL
.(em tp!tj.\\ (O/i) mdmo;mdu! tpns
(l!l;ol(du;,d OJ ptm 801 ;o;,n;,p ptm J;)[(lUJUlD .\t'[dS!P
l' OJ 901 put' "SOl "L()l "f()J ;os;,t1J sp;,umnJ;oJm
OS]e pue J:lljJ:lf!OJ snope,\ S)J:JUUOJJ:JJU!
zo 1 snq 1 D!:IJO m se POI .\.iouwm 0>
OJ (h"'\jdllOJ S! 11\lHJ JO 'JUj ejOJOJOIII,J UJOJJ JmS:JJ
-tudo.Dml P!) .10 t!) 1' ;,1dmt,x;, .H:J ;,q .\1'm l(:lll1'>'. "01
m;oJK(s
(l'J!TI!p ;,tp JO \;op Jndu! J;osn 1' mOJJ P.'"'\!;oJ.'"OJ
Sl JlldU! pUOJ:JS :lljl pue :.\\OpU!.\\ :lljJ JOJ mdm pUOJ."'\S
e ilU]A];)J:JJ OJ U] J:JUHJ :ll!J ;,]dumx;,
UJ (p."'\,\e]ds!p .\\OpU].\\ pJ]tJJ e 01
m TI;,) Jndu! J;osn JllOl(Jl'>\ 1' uo p;omJIJ!Sod;oJ S! .Lop
;ot].L "9() 1 .\Jmu;,m ;o[!Jqo\-tmu 1' put' SO I
:lJ!JeJOA pur, LO 1 V\!0}1 e pue 01 e OJ
tpltj.\\ Z01 e mep eJo
1' s! tp!l(.L ro 1 m;oJK\s J;oJndm<.n ;oqJ 1 !)!:! u! u.Lm[s
"J;)J!ldmo;, l(SOJUDl'Vi[
;,1ddv tm ;,q ;,1dttmx;, .H:J .\tml 1 '!)1:1.10 J;oJndm<.n
:ll].L "UO!lU:lAU! :lljl lj)l.\\ p;,sn ;,q OS]e ,\em
:JJom Jo ;,,\etj tpltj.\\
t'JI'fl J;ol(JO put' )jJO.Lpu JI"1J p.'"OJI'
!;J;oJddl' ;oq OS(I' !!!'>\ J! 'Ullljl!;o.\ll! JU;)S:JJd ;)l(J OJ ;otmmJ;o"i! JllU
am S1!1'PP tpns ;,t11 Tiu!JJ.'"OUUO:JJ;oJm .JO J;ommm
JO Je]HJ!UL>d ,\ur, OJ JOU Sl )!
J:lJildUJOJ r, JO SHOUP-A J . !)]:J
;,1!!1'>\ JI"1J;oJON 'UO!JU;).\Ul ;)l(J l(Jl'>\ (h'"OS!l ;,q .\I'm l[:)ll(.1.
J;oJndmo;, 1lD!d.\J 1' .JO ;,[dumx;, ;,uo s.Lm1s J !)1:1
;oq .\I'm ;ouo .\1u0
mJOjJ:Jd OJ p;,mme.riloJd Jo (h"'\Uil!s:Jp :JJe tp!t].\\

-mn 1' s1' tpns (l'J!TI!P 1' l1J!.L p;,mn:p;od
;oq .\tml UO!JU;).\ll! ;oqJ .JO ;ot11 .JO .\ut'j!IJ
:lljJ
JO P!O\e OJ J:lpJO U! lOU
JO U.\\OU)j U!e)J:JJ Ul
"J;o\;o.LO] 1 um]u;o.\u! JU;osaJd ;,qqo q"ilnmm[J 1'
;opL\OJd OJ P.'"'lJ!.lJs;op ;)Jl' S1WPp ::I!J!:l;ods snoJ;omnN "UO!JU;).\Ul
ilU!jllU!] sr, p:11UJSUOJ ;,q OJ JOU pur, UO]JU:JAU! :lljJ
JO :lA]WJJSil!i! :JJe pur, f!Ul.\\OilOJ :lli.L

'UO!)
:ll!JJO JU:JUJ]poquw ;,uo OJ
.JO UJ.'"OJJS ;o[dumx;, .Lm1s IZ-91 'S!)I:1
umJu;o.\m JU;osaJd ;,t1l.JO JU;om!poqm;, ;,uo OJ "ilmpm:1:11'
.\WpU!.\\ e ,\e]dS]p OJ potjl:JUJ e S.\\OtJS S1 'DLI
'UO!)U:JAU]
jU;)lU!poqm;, ;ouo OJ "ilmp.!OJ:ll' WlPU!.1. l' (lUJUlD
OJ pm1pm I' JO um.Rlmp WltJ p;,[!l'PP I' PJ '!)]:1
'Ull!J
:ll!J JO JU:JUJ]poquw OJ .\\OpU].\\
e ;,sop OJ potjJ:JUJ e JO umJile]p .\\O[l e n 'DLI
'Ull!J
-u;o.\u! ;oqJ JO JU;om!poqm;, ;,uo OJ "ilmpm:JJI' WlPU!.1.
e OJ e JO me.rile]p .\\O[l e zr 'DLI
'UO!)
:ll!J JO JU:JUJ]poquw OJ .\\OpU].\\
1' .JO ;o[dumx;, ;)Jl,JJSI1[[! l j-L 'S!)I:1
p
-Ul'>\ JS.l!J ;oqJ ";,[dumx;, ;,uo U] .Lopu!.L JKl!J ;,t11 .1;opun p.--..\qd
U:ll].\\ .\\OpU].\\ pUOJ."'\S e JO UO!)JOd r, Jel!J ![::IllS
.\\OpU].\\ JS.I!J :ll!J pue ]eJ]il!p
Ot ;,t1JJO \;op Jndu! J;osn 1' Ulll.l.J Jndu! .\tm OJ puods:JJ J<.m s;,op
.Lopu!.L JKl!J ;ot1J TI;,) "ilmss;o;,<.ud [l'J!TI!p ;ot1l.JO ;,;,1 \;op
Jndm 1' m<.U.J Jndu! .\tm OJ u! ;,sop J<.m s;,op
.\\OpU].\\ JS.I!J :ll!J":l]dmex;, :JUO U] '(.\\OpU].\\ JS.I!J :ll!J
pur, :.\\OpU!.\\ JO ;,ileuH ue Jno ilmpeJ "'1!;,) p:lJ]dx;,
J;)lU!l ;)l(J Jl'l(J UO!li'Ulllll;)JJp I' OJ ;osuods;oJ ll! .LOpU!'>\ ;)l(J
"ilmsop ptm :J;om!ll' "ilmJ.mJs :m;oJK(s Jl'll"il!p ;oqJ.JO
:JJ!A:lp Jndm e lJl!.\\ p:lJP-]Josse JOU Jet]JJndm JSJ!! e ilu]
OJ m .\\OpU!.\\ JSJ!J e uw1
]eJ!il]p e JOJ .\\OpU].\\ e
O[ OJ pm1pm I' "UO!JU;)\Ul ;)l(J .JO jU;)lU!poqm;, l' U]
"(l'!.l;)J!.D J;)l(JO JO "p.'"O"i!tmtp I' "'"il;,) pm S!
UO!J!pmn 1' Jl"11 umwmmJ;opp 1' OJ ;osuods;oJ ll! p.--.sop
Sl .\\OpU!.\\ J:ll])JI1J e U] "UWJS.(S
JeJ]il!p JO mdm e l!Jl.\\ JOU s1 Jet]!
,:c Jndm tm OJ ;osuods;oJ ll! p;osop S! .Lopu!.L JKl!J ;oqJ ";,[dumx;,
J;)l1Jmm U] "(.LOpU!'>\ JS.l!J ;oqJ ptm WlPU!.1. ]Kl!J ;)l(J
JO ;,iletm ue Jno ilmpeJ "'1!;,) p;,sop .\\OpU].\\
U:llj.\\ Wljl OS J:lUl]l r, S]ljJ OJ
;,uo U] mdu] JHOtJJ].\\ .\\OpU!.\\
o<: "ilu!sop ptm :.Lopu!.L JKl!J ;oqJ .1;opun .wJpU!'>\ puo;o.--.s
;ot11 .JO UO!)JOd ;)l(J "'>\OpU!'>\ JS.l!J ;oqJ J;opun
-OJd ]eJ!il]p :lljJ UO .\\OpU!.\\ e JO UO]UOd
e J! Jet]! tpns .\\OpU].\\ e ilm.\e]ds!p :s;,pnpu]
Ul;)JS.(S (l'J!TI!p l' .H:J WlpU!'>\ J;)S!l l' .\qd
,:1 OJ pm1pm l' "UO!JU;).\Ul ;)l(J.JO JU;ouupoqm;, J;otjllllm U]
(m;oJK(s (l'J!TI!P ;,t1l.JO ;,;,1 \;op
mdm e mOJJ "'f!':J) .\\OpU].\\ JSJ!j JOJ Jndm pUOJ."'\S
e ilmAWJ."')J 01 U] S] J:JUJ]l :ll!J pur,
;ot11 Ull .\[[l'JUOZUOl1 (h'"OJ;oju;,;, I' "'"il;,) ;ot11 UO
ol JosJnJ tqo UO!J!Sod tqo JU;opu;,d;,pm JI"1J m;oJK(s
Jl'll"il!p ;ot11 JO .\1'1\.ls!p I' uo UlllJ!SOd l' W p;,.\t'[dS!P S! .Lopm'>\
:JUO U] "(.\\OpU!.\\ JS.I!J JUOJJ Ul .\\OpU!.\\
ilmiluuq "'1!;,) mdm pJ]tJJ e 01 U]
;oq UlD .\tpmJ;o!l1 WlPll!.1. I' m WlpU!'>\
,: ;ot1l.JO UO!JlSOd l' ptm :( WlPU!.1. ;ot11 1\u!ddmp ptm "ilmTI"ilmp
tpns ".\\OpU!.\\ :ll!J OJ jej!il]p
JO mdm e UJOJJ mdu] ue Jo .\\OpU!.\\
pm]J e Jet]! ilmJP-Jlpm mdu] ue fl;,) .\\OpU].\\ JSJ!! JOJ Jndu]
puoJ.'"OS l' OJ ;osuods;oJ ll! l' UO Sl .LOpU!'>\

Z8 !68'S<;8'L sn
us 7,853,891 82
5
a local tk'\ ice coupled directly to the res! of the components
in the dma processing system_ it will be appn.>ciatcd thm the
present invention may utilize a non-volatile memO!)' which is
remote from the system. such us a network storage device
which is coupled lo the data processing sys!cm through a
network in!crliJCc such as a modem or I ;thcrncl intcrbcc. The
bus 102 may include one or more buses conncct<..-d to each
other through various bridges. controllers and/or adapters as
is we! I known in the art. In onccmbmlimcnllhc !10 controller
109 includes a USIS (Universal Serial I Sus) adapter ll1r con-
trolling USIS peripherals. and/or an II;I;I:-IYJ4 bus adapter
for controlling IJ:J]:-1394 pcriphcrnls.
It will be apparent from this description that aspects of the
present imcnlion may be embodied, a! !cas! in p;u1, in soH-
ware. That is. the techniques may he carried out in a computer
system or o1hcr data processing sys1cm in response to its
processor. such as a microprocessor. cx<.>culing S<-'<.JUCnccs of
ins1niC1ions contained in a memory. such as ROM 107. vola-
tile RAM 105, mm-\olatik memory 106. cache 104 or a
remote storage de\ ice. In various embodiments, hardwired
circuitry may be used in combinalion with soliwarc ins1niC-
1ions to imp!cmcm the prcscm invemion. Thus. 1hc tech-
niques arc no! limited lo any speci!ic cmnhinalion of hard-
ware circuitry and sollwarc nor lo any particular source !l1r
the inslmclions ex<.>cu!cd by the data processing system. In
addition. 1hroug.holl1 !his description. various li.mc1ions and
operations arc described as being pcrli.mncd by or caused by
sollwarecodc lo simpliFy description. llowcvcr, those skilled
6
any user input. Thus, window J21 displays the message of
low bmtCI)' power 10 1hc user witholll li.1rcing 1hc user 10
provide inputs to dismiss !he message window Since window
321 is 1ransluccn1 and transient. 1hc portion of window 30J
5 that is under window J21 is still \ isihk. '!lms, the user can
nmlinuc working with window J()J (or other window) with-
om having to provide additional input1o gel message window
321 om of !he way.
In one embodiment of the present invention. lrans!uccnl
1" window J2 I is always all he lop k'\cl ofthcwindow
displaying hierarchy so that the lrans!uccnl window is always
visible when displayed. "111is eliminates the ned for the user
to change 1hc window displaying hierarchy to bring up the
lrans!uccnl window when another traditional window is
15 brought up lo the lop of the window displaying hierarchy
(e.g .. by crcming a new window or accidcmally changing the
hierarchy). In another cmbodimcm of1hc prcscm invemion.
the user can change the position of1hc translucent window in
the hierarchy so that iftheuserdcsircs the lrans!ucenl window
"" may be sen! lo a background position.
In one embodiment of 1hc present invemion. 1hc image of
window 321 is faded out when 1hc timer expires. which is
il!uslra!cd by the images of windows J21. JJI. and J41 in
I:IGS. 8, 9 and 10. :\Jlcrlhc image of window J2I is fi1dcd out,
"5 window J21 is
in the art will recognize what is mean! by such expressions is
1ha1 1hc li.mctions result from execution of the code by a 30
processor. such as the microprocessor 10J.
In an01hcr embodiment of the pre scm invemion. a translu-
cent message window swns to fade om when a swtus change
is de!<->cted. For example. a message window is
when the system dc!ecls the ringing signa! on a phone !inc.
When 1hc system dctcc1s that the ringing signal is no longer
present on !he phone line. !he image of the message window
.-\ machine readable media can be used to s1orc soliwarc
and data which when execul<-'d by a data processing sysl<-'m
causes the system lo pcrllm11 \arious mc!hods of the present
invention. Tills cxccuwble sollwarc and dma may be s1orcd in
various places including for example ROM 107. volmile
R. \M 105. non-volatile memO!)' 106 and/or cache 104 as
shown in FIG. 1. Pm1ions of this so!lwarc and/or data may be
stored in any one of these storage de\ ices.
is H1dcd out: and !he message window is dcs1roycd. Similarly.
a translucent progress window !(H showing the progress of
copying a Ilk can be bded out and destroyed a!ier the copy
35 process ends. In one example. message window J61 as shown
in 1-'lG. 11 is displayed when a new message arrives. When the
user starts 10 open an app!icmion to view 1hc new message.
message window J61 is clos<.-d automatically so that the user
does no! ha\elo provide inpullo dismiss the message window
40 orwaitli.1r 1hc message window 10 JUde out. "llllls. a madlinc readable nK'<iia includes any mcchmlism
1ha1 provides (i.e .. stores and/or 1ransmi1s) in a
llm11 accessible by a machine (e.g .. a computer. nclwork
d<.'\ice. persona! digital assistant. m;mubcluring tooL any
d<.'\ice with a sci of one or more processors. clc.). For
example. a machine rcad.1bk media includes rccordab!c/non-
r<.>cordab!c media (e.g .. read only memO!)' (ROM): random
access memory (RAM): magnclic disk storage media: optical
storage media: !lash memory devices: clc.), as well as el<.>c-
1rical. optical. acous1ica! or o1hcr fonns of signals
(e.g .. carrier waves. infrared signals. digital signals. etc.): elc. '0
In one embodiment of 1hc present invemion. 1hc image of
window J21 gradually sets in when the window is !irs! dis-
played. In another embodiment of the present imcnlion, win-
dow J21 in F!(i. 8 is automatically rclocal<.'<i or moved (e.g ..
45 in an animation !Ushion) 10 a dill'crcm !ocmion so 1ha1 the
image of window J21 docs no! obscure 1hc display of any
particular ponion of windows JOJ !l1r an extended period of
timed. For example. window J2 I may he automatically
moved across the screen horizontally from 1hc !eli hand side
of screen 301to 1hc right hand side of screen J01 (or ncar the
center of screen 321 in a circular motion). At leas! one embodiment of1hc prcscm invemion seeks to
display a window with r<.-duced dislraclions so that a user can
ll1cus on more important windows.
In one embodiment of the present imenlion, the system
1-'IGS. 7-11 i!!uS!ralc example scenarios of displaying a
window according 10 one cmbodimcm of1hc prcscm invcn- '5
lion. Traditional window JOJ is shown in FIG. 7. Window J()J
de!<. >cis (or manages) all the translucent windows so that when
a second translucent window is displayed before the Hrst
translucent window is dosed. the Hrst1ransluccll1 window is
repositioned so that the second lrans!ucenl window can he
contains control bul!ons J 1 I, J I J and J 15 !l1r closing. mini-
mizing and maximi/ing the window. Window JOJ also has
Iitle bar 310. which may be used 10 relocate 1hc window on
screen J()l. Consider a scenario where the batlel)' power of
the system is !ower than a threshold. Aller the sysl<-'m dc!ecls
such a system status change. window J21 may he displayed
ncanhc ccmcrofscrccn 301. as shown in 1-'lG. 8. Window J21
is 1ransluccn1 so !hal regular window 303 is sti!! visible under
window J2l. Once window J21 is displayed on 1hc screen. a
timer is s!;uk'd lo control the closing ofthcwindow. When the
timer expires. window J21 is automatically closed without
easily seen on the screen without inlcrl'crencc with each other.
1:or example. alkr hallery !ow window J2 I is as in
FIG. 8. the system may dc!cc1 a new message arriv<.-d Ji.1r the
60 user. Thus. 1ransluccn1 window J61 is display<.-d as in FIG. 11
lo inllmJJ!hc user about the !1'-'W message. At the same time,
window J5I is automatically mm<-xi loa position as seen in
FIG. 11 so !hal both 1ransluccn1 windows 351 and 361 can be
easily seen on the screen .. the !irs! translucent
65 window (e.g .. window 351) is hidden so !hal only the second
window (e.g .. window J61) is display<.-d. rhe timcrofthe !irs!
window is slopp<-xi while being hidden until the second win-
SUlOJj :ll(JJll Si!tOlU! :ll(J :ljlOJ:lUSi! ,\t'm -'l'l.llpU!!>\ jU:l:llljSUl!Jj :lljJ
_\'\.opm,.. p;JJ;J_Jjnq-uou 1' UJ 11 J;Jptm _\'\.Opt!!,.. ;'ll]J pmo
.\\llpU!.\\ JU;:lJil]SUlW Jl[l ,\e[dSip OJ S_\\OpU!.\\ ;:ll]J.IO SJf!P-Ul! ;:ll]J
.\WpU!.\\ ;:Jt]J pur, :.([;:JJeJed;-,s
.\WpU!.\\ Jl[l JJpun _\\OpU!.\\ J1[1 pue .\WpUp\ JUJJil[
SUlOJj JljJJll :lljJ \\OpU!\\ fA").!JJJlll( l' Ul ;,1\tmH
UJJJ:Js Jt]J Jll umuod Jt]J :lWpdn OJ p:lmn:p;,d
Jpj:- UO!JP-JJdo p;-,jepdn S! _\\OpUp\ J1[1 JJpun 09
.\\llPU!_\\ ;:Jt[J Wlll LPP UOIWJJdo JO _\\Op
-U!-"" JU:l:lll(SUlOJj :ll(J JlOl(J SPJ:- Ull!Jl,JJdO_J] "\\llPU!-""
Jt]J J:lpun_1.opm-\\ :ll(J ptm _\'\.opm-\\ JUJ:llljSUl!IJ Jt]J
U! p.")][l)jS :lUll lljjUJJ!Oddto S! J] "S_\'\.OpU!\\ jU:l:lll(SUlOJj 1(11\\ UO!J
-u;-o\m JU;-os;-oJd ;-oqJJO JU;Junpoqm;-, ;JUO J J L
.!!O[llTI:lJ uo 11m)jJO\\
;:ll]J JOJ UHlUHU]lU SP-l] .\\llpU!.\\ JUJJil[SUJJJJ
;:lljJ.IO ;:JJUJSJJd ;:ll[l"Sntl.L (Jeq ;:l]lll e .(q
,\[[tDid.\J t[J!l]\\) Sll:lllJ p.!tmq.\J)'[ Jl]J JlOl(J \\llPU!_1. :ll(J
OJ p.")p.n,_\-\.ll:J Jlldm (lllOOlj.\.")'j JljJ ")lldUl p.!!Olll(,\J)\10 JUH1SUO:l
JOU -\\opu!-"" Jt]J Jl J<.m s;-oop _\'\.opm-\\
JUJJil[SUJJJJ J1[l Jl _\\OpU].\\ JUJJil[SUJJJJ J1[1 J;:JpUn _\\OpU!.\\
;:ll]J 1[11.\\ JJJJJJJU! UlJJ J;)Sil J1[1 Je1[1 _\\OpU!.\\ JUJJil[SUJJJJ
JljJ .!:lpun jSllf Sl Jl'ljJ \\llPU!_1. JljJ .!OJ Sl JlldU! :ll(J
"WlpU!\\ Jl(J .\q fA")JdJ:J:ll' JOU ('j:l!(J l' '"11;,) jUJ\J
Jl:J :ll]J Ull Si!toml p:lU!l[lUlD :ll]J fpp Ull!Jl,J:ld()
_\\OpUp\ ;:Jt[J JJpun .\\llPU!_\\ J1[1 pur, .\WpUp\
;:llll JOJ e ;:JJeJ;:JUJil OJ .\WpUp\
:ll(J J:lpun -\\opm-\\ JO J11tom! Jt]JJO uo!uod Jt]J ptm
-\\llPU!_1. 10 JO :l1ltom! :ll]J S:lU!l[lUO:l jpp Ull!Jl,J:ld()
"UO!JlWAU! ;:llll JO jU;:lUl!pOqtuJ JUO OJ flmpJOJJ!J
.\WpU!.\\ e ,\e[dS!P OJ e s_\\Otls Sl "DL!
pJJP-]JJ JosJnJ e_11 JO:] wu _\\OpU].\\ JU;:JJH]
J1[1JI s_\\OpU].\\ [P-tuJou 01 p;-,pJe_\UOJ JJe SJndm J;:Jt]JO
mdm J;)Sil SllldU! ,\[UO
_\'\.opu!_1. JljJ ")lldU! J:lSll WlpU!\\ jUJ:lllj
"(_\\OpUp\ js.!!J J1[1 flm,\OJjS;:Jp pur, _\\OpUp\ js.!!J
:l1(JJO J11t"U! 1m jllll .\q '"11;,) S:lJ!dX:l JJUHJ Jl(J U:ll(.1.
\\llpUL1. JKl!J Jl]J S:lSll[J L.ff:' Ullljl!J:ld() "(U:lPP!lJ
,,
.J! JO) ,(r,m
f!U!U!lJlU;:lJ J1[1 "fi<")SOp Sl JO J;:ll(lUllU
l'JO :lUO UJl]ifi "_\'\.OpU!\\ jUJ:lll(SUlOJj :ll(J
-oda1 Sff:' UO!JlOJJdo \\llPU!_1.
l' W1[l S:lU!UlJJPp 6ZP Ull!JlOJJdo UJl(,'fi (p:lSS:l:Jtud Jlldm
J1[1 J;:llJP- p.")l.l!JjS;:JJ pue p.")ddOJS ,\r,m JJlU!J ;:llll .(pA]JeUJJj
-je) JlldU! J1[1 SJSSJJOJd ffp UO!JlJJ;:JdO pue :J;:JUJ]J ;:lljl S).l!JjSJJ Ot
Jfp UO!JlOJJdO 'p:l.\l:l:l.")J Sl .\'\.opu!_1. JKl!J JljJ Jl:J (JlldU!
1' .10 Jndu! .J:lsn to 11;,) Jndu! 1m Jl"Jl s;-,u!uu;,pp LZP UO!JlOJJdo
"J;:JUJ]l e SZP JeJiil]p
J1[J.IO mdu] e tu<.UJ mdu] .(ue OJ U]
JOU W.1pU!.\\ js.!!J ;:lljl ;:JJJ1(.\\ "(.\\OpU].\\ pUOJJS !JjO UOIJJOd
10 Jll do] UO TI:l) Jl'l!111p :ll(J Jll (SJ:ll(JO
j() 'U:l."U:JS 1[Jllllj l' "JOJ!UO[[J _]}L) l' (].) I l' 11;,) :l:l!.\JP
,\e[dS]p e uo _\\OpU].\\ JSJ!J e HP uo11
-eJJdo ilmssJ:JoJd JeJiil]p e tuOJJ Jo uonnq e
"J;:J'j:JIJS .\ofe 1pnoJ e ped tpnoJ e 11eq 'j:JP-JJ e ;-,snom e oc
1pns ":l:l!.\:lp Jndu! J:lsnt' mtuJ Jndm TI:l)
Jndu! tm sJ.\I:l:l.")J J zt umJt,JJdoJJUV uo!JU:l\Ul JU:ls;-oJdJt]JJO
jUJtu]pOqHW JUO OJ flmpJOJJB .\\OpU].\\ r, [OJjUOJ
01 pot]J;:Jtu e JO um.rilmp .\\OU PJI!P-JJP e PI "!)L!
( -\\opU!-"" Jl]J ptm \\llPU!_1. JKl!J :llj!Jll JTitmH ,:c
Ul' jllll 1\:l) S;JJidX:l JJUHJ JljJ UJlj\\ _\'\.opu!_1. JljJ
SIP uo!JtoJ;odo "J:lmiJ to nt umJt!J:Jd()
ilmJeJJdo ;:Jt]J .\q p;-,Jepm] smdm Jo ;-,u!l e uo
ilmiluu snJms m e "JU;:l,\;:1 e ,\q
Jndu! tm ":l:l!.\:lp Jndm .\tm t]l!-"" JOU o<:
Jl'l]J JlldU! tmJO":l:l!.\:lp Jlldm UllUJ JlldU! tm TI:l) Jlldm
ur, OJ U] .\\llpU!.\\ Js.!!J r, JJP UOIWJJd() "UO!)
-UJ,\Ul. ;:ll[l JO JU;:!Ull_poquw JUO OJ flmpJOJJ!J .\Wpm __\\
e OJ pot]J;:Jtu e JO umJile]p .\\O[l e n "DL!
"(:l:l!.\Jp Jndm JJsn ,:1
.(m, 1[11\\ J<.m Wt]JJlldm1lm\IJ:J:lJ JJ!Jl' Jo pm
UO]JipUOJ e JO r, Jel[l UO!jlJU!lUJJJ;:Jp
e JJJje Jo J;:JUJ]J e JJJje f,;-,) mdm JHOt]l!.\\
( -\\opu!-"" Jt(J ptm -\\opu!-"" :l1[1 Jll :lTitmn 1m 1110 ;optOJ
11;,) -\\opu!_1. :ll(J .\jjlO:l!JlnUOJlllO fOP UlliJ fll
-l!J:ldo pmo :( \\llPU!_1. Jl]Jl(11lllut]J _\'\.opu!_1.
;:ll[lJO UOijJOd J1[1 ;:JJS OJ ;:ll]J .\\OpU].\\ pUOJ;:JS r, JO
UOIJJOd P-JO dOJ UO p."),\JJ[dSip UJ1(.\\ 1[Jilj.\\ .\\llpU!_\\
to '"11;,) -\\opu!-"" 10 K(t,jdS!P JOt UlliJl!J:Jd() "UlliJ
U:l.\U! Jl(J Jll jUJUl!POl]UlJ :lUll OJ1IU!pJO:l:llO \\llPU!_1. ,_-
r, OJ p01[JJtu e JO me.rile]p _\\O[l e zr "DL!
"JUJJil[
-SUJJJJJOU JJP- Jel[l JOJ p.")JUJlUJ[dUH Jq tWJ UO]JUJ,\Ul
jU:lSJ.Jd :llj!Jll splll(!JlU jlOl(J Ull!JdU:JS:lp S!l]J lUlUJ J.ll' JljJ
8
U:ll(.1. 'UO!JU:l\Ul jU:lS;JJ\.1 Jlj!Jll jUJtU!pOljUl:l JUO Ul
Jndu! mOJJ Jndu! .\mo Tiu!mnsmn Jlllll(JI\\ J:lsn
;:Jt]J OJ e e .\q pJJP-]l!U] _\\OpU].\\
r, "UOIJUJ\Ul JUJS;:JJd Jl]lJOJlWtU!pOl(UlJ ;:JUO U[
J:l)lldm<.D l' U-\\llp OJ J:lSll :ll(J UJlj\\ 11;,) _\'\.OpU!\\
Jt]J ;,sop OJ TimJt!J:Jdo Jt]J 1' 1pns
s1ndm OJ [[IJS .\tml _\'\.opm-\\ :lt]J 'J:JI \JP Jndu!
e UJOJJ JlldU! .(tm OJ U] lOU SJOp _\\OpU!.\\
;:ll]J UJ1[A\ ,\[[P-Jl)P-UlOllllJ .([UO .\\llpU].\\ ;:ll[l :(_\\OpU!.\\
Jll_l ,\J'j Jll:l Wll]S l' S:l'jl'l JOU '_\'\.opu!_1. :ll(J 1\u!sop
.H:J uonnq to JWt] J<.m s;-oop _\'\.opm-\\ :lt]J TI:l) :l:l!\JP Jndu!
e UJOJJ J!ldU] .(ur, OJ U] JSO[J lOU ;:JJJAJp
mdm e 1[1].\\ Jndm ue ,\q pJJP-]l!U] _\\OpU!.\\
JUJJil[SUJJJJ Jt]l "UOIJUJ,\Ul ;:ll]J JO JU;:!Ul]pOqUW ;:JUO
U] JOU \\llpU!\\ JUllljO.\ JljJ 'Ull!JU:l.\Ul jUJS:lJd
JljlJO jU:l1U!p0l(lU:lJUO U] .(JllDijlmllljlW fA")SO(J put' ]110
Sl _\\OpU].\\ [<.UJUOJ ;:JUHl]O\ Jl]l pue J;)Ul]l ;:ll]J ";)Ul]l
JO Jmwme e Jtull[OA ;:Jt]J sdoJs
J:lSll JljJ JJJJV S! .1.0pU!\\ JUllljO.\ jUJ:llljSUl!Jj :ll(J
.H:J J:llU!l :l1(J ':llUll(ll\ .H:J ,\J'j :lljJ UllUJ JlldU!
UlO S:l.\!:l:l.")J -\\OPU!_\'\.JUllljO.\ JljJ U:ll(A\ "(UJ:lJ:lS :l1(J l'
JOUO!)P-JO[ ;:Jt[l lUOJJ f,;-,) _\\OpU].\\ ;:JUHl]O\ JU;:l:Jllj
;:ll]l UO]WJO[ r, S;:JU!lUJ;)JJp .([]P-JIWU!OJIW
(tut!!Titud Ull!)lO:l!(ddto tm .10) WlpU!\\ JljJ "Ull!JUJ\U!
JU:lS:lJd :ll(J J o JU:l1mpoqm:l ;,uo u 1 ( .10 "JSl,JJUO:J m)
]J\J[ ;:JUHl]O\ JUJJJilJ Jljl.\\01[S OJ _\\OpU!.\\ JU;:l:Jllj
r, ,\;:!)'[ UOIJJUl1J [P-]J..")dS ;:ll[l 01 U[
JO JO) ;:JUHl]O\ JSn[pr, OJ .(J'j UOIJJUl1J ]e!JJdS r,
.\tom JJsnt' ':l[dumx;, Jll:J ":l:l!.\JP Jndu! l(JI\\
Jlldm Ul' tj11ll().ll[l \\llPU!_1. :lJl'!llll! .\tml V
JU:l:l
-ll[SUJJJJ JO J1[l[<.UJUOJ .(]JJ..")J]p 01 UO]JdO Jl]l
r, Wl]l OS (f.U]SOp "')S!J, ,\;:!)'[ ;:ll]J 1[JilS
JlOIJ.")ds '"1\;,) p:lu!uu;,pp.")Jd JdJ:J:Jt' s_\'\.opm-\\
jUJS:lJd Jlj!Jll jU:llU!Plll]lUJ J\!JlOl!.l:lJ(l' JUO Ul
"(SJ:ll(JO ptm Ul J1Jutnp :lljJ 'J:llUIJl'Jll Ullljl!l!dXJ
;:ll]J 1;:1Ul JJP- U!JJUJJ U;:ll(.\\ .([]JJJ]JelUOJIW ;:JSOp
1[Jl1[.\\ OJ JS;)l]J 01 PJP]MlJd
;,q mo:J]ndm JJSn<.m ptm :1ndm J:lsn .\mo ;,umsuo:J J<.m op s_\'\.op
Ul\\ jUJ:lll(SUlOJj 'UO!JU:l.\Ul :ll(JJO jU:llU!Plll]UlJ
U[ J;:Jt]JO Jo uounq ,\of tpnoJ ped
1[JHOJ"[]eq 'j:JJJJJ e f,;-,) mdm
1[11.\\ mdm .(ue motpl.\\ e .\q p;-,Jepm] JJe
Jlot]J s_1.opm-\\ Jll J[dmtox;, mo _\'\.m[s 11 -s "S!)I:l
J:ll(JO JOfPtm '(;,sop OJ JUl!l p:lWUHJS:l m)
OJ JUl!J :l:JUlO].JOd1m JljJ OJ 1\u!p.!tl:l:ll'
S_\\OpU!.\\ J1[l JO .(ejdS]p Jl]l JO SUO!JISOd U;:J;:JJJS
;:Jt]J pur, .\][e::>pemome OJ s_\\OpU!.\\
JUJJil[SU!JJ1 J1[l .\\llpU!.\\ J1[1 "UO]JlWAU! HWSJJd
Jt]J Jll JUJtmpoqmJ JJtjl.lllJ to u1 .\[]tDIJtmJOJtm s1 _\'\.op
L
zs !68'S>8'L sn
us 7,853,891 82
9
lucent window on lop oft he other window using the image of
1hc window under it. For example. the 1ranslucent window
obwins 1hc image of the window under it alicr the window
under it draws on the lhlmc bullCr: then. the trnnsluccnt
window generales a combined image lo update the corrc- 'i
sponding portion of the screen.
10
scopeofthc invention as set ll1rth in the ll1llowingclaims. !"he
spccilicmion nnd dmwings arc. accordingly. 10 be regarded in
nn i!!us1rmivc sense rather than a rcs1ric1ivc sense.
What is claimed is:
I :\method lo display a user inlerbce window ll1r a digital
processing system. the mclhod comprising:
displnying a lirst window in response to receiving a lirst
inpm Ji"om n user inpm device of 1hc digiwl processing
system which is capable of displaying a! leas! a portion
of a second window ctmcurrent!y with the lirsl window
on a screen:
stnrting a 1imcr: nnd
closing the lirst window in response 10 n dc1cnninmion that
the timer cxpirLxl:
wherein the Jirsl window does nul close in response lo any
inpm Ji"om n user inpm device of 1hc digiwl processing
sys1cm. wherein the lirst window hns been displayed
indcpcndcmly li"om n posilion of n cursor on the screen.
2. :\ method as in claim I wherein the lirsl window is
lrans!uccnl: and the portion of the second window is visible
while under 1hc lirst window.
3. _.\ mc1hod ns in clnim 2 wherein the lirst window is at n
lop k'\d in a window displaying hierarchy.
4. :\method as in claim 2 wherein a degree of translucency
of the Jirsl window is adjustable.
5 .. \ method as in clnim 1 wherein snid closing 1hc lirst
window comprises:
li1ding out an image of the Jirsl window.
!-'IGS. 16-21 show example screen images of windows
displayed according to one embodiment of the present inven-
tion. When a user starts to adjust the volume k:vd (e.g ..
pressing on a Jimction key !(Jr increasing or decreasing vol- lCJ
umc. or selecting an item from a syslem control menu with a
cursor control device. such as a mouse or a touch pad). trans-
lucent volume window 511 appears on sen.-en 501. Since
window 51 I is lransluccnl. the portion ofwimlow 503 under
window 51 I is still visible. In one embodiment of the present 15
invention_ when window 511 is initially !ondcd. the back-
ground of volume window 511 has n high degree oftrnnspar-
cncy: and the content of window 511 hns n !ow degree of
transparency (or no transparency). the user can
easily sec the content of window 51 I when the user is sup- ""
posed to ](1cus on window 511. .\s the user provides inplll to
adjust the volume level. window 511 remains in a stmc with a
high degree of transparency ll1r the background and a low
dcgreeoflransparcncy ll1r lheeonlcnl. !'or example. when the
user decreases the \olumc levd (e.g .. pressing a Ji.mction kl:y. "5
or an army key). the volume !eve! is decreased ns indicated by
window 521 in FIG. 17. When the user Ji.1rthcr dccrcnscs the
\olumc k'\d lo mule the speakers. window 531 changes an
icon lo indicate that the speakers arc muted. as shown in l'](j.
18. When 1hc user starts 10 ndjus1 the brig!uncss of the moni-
1or. translucent brightness window 541appcars. us shown in
FIG. 19. while the volume window is hidden (or dcs1roycd. or
converted into the brightness window by redrawing the icon
and the levd bar). If the user slops providing input ll1r the
brightness window for an nmount of time (e.g .. n prcdetcr- 35
mined mnoumoflimc. a randomly selected amoum of time. a
lime period delcrmincd nccording to a system condition or
other criteria. a lime period ea!eu!atcd on the Jly. or a time
period speeilicd by a user) window 541 starts lo bde away
and be dcstroy<.'d. as shown in FIGS. 20 and 21. In one 40
embodiment of the present invention. when n 1mnsluccn1
window starts lo lit de away. the degree of transparency of the
content in the lransluccnl window is increased lo allow the
user lo see hcller the window under the translucent window.
6. :\ mclhod as in claim I wherein the second window. if
30 displayed. does close in response to an input from nuscr input
device of 1hc digitn! processing system.
as illustrm<.'d by window 551 in FIG. 20. Thus. 1hc dcgr<.'C of 45
1rnnsparcncy of the window cnn be ndjus1cd during the life
cyde of the window lo lead the Jixus point of the user.
Ftu1hcr. a user may specify the degree of transparency of the
window (e.g .. as a preference pmmnetcr). The imngc of the
window may fade out smoothly in nn nnimmion: or 1hc image '0
of1hc window mny JUde olll in a number of discrete steps. The
degree of lrans!uccncy. the spel.xl Jix li1ding out. the discrete
!e\ds of translucency ll1r li1dingoul. the time lo expire. and/or
other pammctcrs for comro!!ing the displny of the window
mny be sc1 by 1hc user or ndjus1cd by 1hc system (or app!icn- '-'
lion sollware programs) automatically according lo system
conditions orolhercrilcria. l'orcxample. the system (or appli-
cation programs) may adjust the time lo expire according lo
1hc number of translucent windows displayed concurrently
on the scR-cn: or 1hc sys1cm (or nn application program) mny 60
adjust the initial degreeoflrans!uccncy according lo the color
pallem allhe location where the lransluccnl window is dis-
plny<.'d.
In 1hc Jixcgoing spccilicmion. the invention hns been
described with rcJCrcncc to spccilic cxcmplmy embodiments 65
thcreoL II will he cvidcnllhal \arious modilicalions may be
made lherclo without departing Jimn the broader spirit and
7. _.\method as in clnim 6 wherein the lirs1 window docs not
respond lo any input Jium a user input device of the digital
processing system.
8. :\ mc1hod as in claim 1 Ji.1rthcr comprising.
rcposilioning 1hc lirs1 window in response to a third win-
dow being displayed.
9. :\ mclhod as in claim I li.1rlher comprising:
hiding the lirsl window in response loa third window being
disp!nycd at n location where the lirs1 window is dis-
p!nycd.
I 0. :\ method as in claim I li.1rthcr comprising:
repositioning the lirsl window on a display in response loa
second input Jix the Jirsl window.
11. _.\ mc1hod ns in claim 10 wherein the second input
indicmcs that n third window is disp!nycd.
12.:\ method as in claim 10 wherein the second input is
rcccivl-xl Ji-om a user input device of the digital processing
system.
B.:\ method as in claim 10 Ji.mhcr comprising:
adjusting n position of1hc lirst window inn window dis-
playing hierarchy in response loa third input.
14.:\ method as in claim I li.1rthcr comprising:
dc1cnnining a position on n display of the digiwl process-
ing system independent of a position of n cursor on the
display:
wherein the Jirsl window is displayed allhe position.
IS.:\ mclhod as in claim 14 wherein the position is cen-
tered horizontally on 1hc display
16.:\ method as in claim 1li.lf1hcr comprising:
restarting the timer in response lo rccei\ ing a second input
ll1r the lirsl window.
17. _.\ mc1hod as in claim 16 wherein 1hc second inpm is
rcceiwd Ji"om n user inplll device of 1hc digitn! processing
system.
18. :\method as in claim 16 wherein the Jirsl window is
crealLxl by a Jirsl application and the second window is ere-
us 7,853,891 82
II
atcd by a second application. wherein the !irs! application is
dillCrcnt from 1hc second applicmion.
19. A mel hod as in claim 1 wherein the user inplll device is
one of'
I2
31. A media as in claim 26 wherein the sL-ctmd window. if
displayed. docs close in response to an input from n user input
device of the digitnl processing. system.
32 . . \media as in claim 31 wherein the !irs! window docs
a) a keyboard:
h) a mouse:
.< no! respond lo any input Jimn a user input de\ icc of the digital
c) a track ball:
d) a touch pad:
c) a !ouch screen:
J) a _joy stick: and
g) a \muon.
20 . . \ method to display a user interface window Jix a
dig.itnl processing system. the method comprising:
displaying a !irs! window. the !irs! window being lranslu-
ccnl. at !cas! a portion of a second window being capable l'i
of being displayed on the digiwl processing system
under the Hrst window. the portion of the S<..>cond win-
dow. when present. being visible under the Hrst window
on a screen: and
dosing the !irs! window without user input. wherein the ""
Hrst window hns been displnycd independent from n
position ofn cursor on the screen.
21 A mc!hod as in claim 20 Ji.n"lhcr comprising:
starling a timer:
wherein said dosing the lirsl window is in response lo ""
expirmion of the timer.
22 . .-\method ns in clnim 20 li.1rther comprising.:
receiving. nn input. the inplll not nssocintcd with n user
input de\ icc of the digital processing system:
wherein said dosing the lirsl window is in response to the 30
input.
23 . .-\method ns in clnim 20 li.1rther comprising.:
determining whether or not n condition is met:
wherein said dosing the Jirsl window is in response loa
determination that the condition is mel.
35
24. A mc!hod as in claim 20 wherein said dosing the Jirsl
window comprises:
fading olll nn imnge of the !irs! window.
25.:\ mc!hod as in claim 20 wherein a degn .. -c oflranslu-
'0
ccncy of the Jirsl window is adjustable.
26. :\machine readable media containing cxL-cutahlecmn-
plllcr prog.ram instructions which when cxccli!L'd by a digitnl
processing. system cause said system to perform n method to
display a user intcrliJCc window. the mc!hod comprising:
45
displaying a lirsl window in response lo rL-cciving a lirsl
input Jimn a user input de\ ice of the digital processing
system which is cnpable of displnying. nt least n portion
of n second window concurremly with the !irst window
on a screen:
starling a timer: and
dosing the !irs! window in response loa determination that
the timer cxpirL'd:
'0
wherein the !irs! window docs not close in response 10 nny
input Jimn a user input de\ ice of the digital processing ,
5
system. wherein the lirsl window has hL-cn displayed
independently Ji-om a position of a cursor on the screen.
processing system.
33 .. \ media as in clnim 26 wherein the method Ji.Jrthcr
comprises:
repositioning the Jirsl window in response lo a third win-
dow being displayed.
34. :\ media as in claim 26 wherein the method Ji.Jr!her
comprises:
hiding. the !irs! window in response to a third window being
displayed a! a location where the lirsl window is dis-
played.
35 . . \ media as in clnim 26 wherein the method Ji.Jrthcr
comprises:
repositioning. the !irs! window on n displny in response 10 n
second input ll1r the Jirsl window.
36. A media as in claim 35 wherein the second input indi-
cates that n third window is displnycd.
37 . .-\ mcdin ns in clnim 35 wherein the second input is
received li-mn a user input device of the digital processing
system.
38. :\ media as in claim 35 wherein the method Ji.Jr!her
comprises:
adjusting. n position of the !irst window inn window dis-
playing hierarchy in response loa third input.
39. :\ media as in claim 26 wherein the method Ji.Jr!her
compnses:
detcnnining a position on n display of the dig.ital process-
ing. system independent of a position of n cursor on the
displny:
wherein the Jirsl window is displayed a! the position.
40. A media as in claim 39 wherein the position is centered
horizontally on the display.
41. .\ media as in clnim 26 wherein the method Ji.Jrthcr
comprises:
restarting the timer in response lo rL-cei\ ing a second input
ll1r the lirsl window.
42. A media as in claim 41 wherein the sL-cond input is
receiwd Ji"om n user inplll device of the digitnl processing.
system.
43. :\machine readable media as in claim 41 wherein the
Jirsl window is crca!ed by a Jirsl application and the sL-cond
window is crcntcd by a SL>cond applicmion. wherein the !irst
npplication is different from the second npplication.
44 . . \media as in claim 26 wherein the user inplll device is
one oJ":
a) a kLJ-hoan.l:
h) a mouse:
c) n track ball:
d) n touch pad:
c) n touch scrL-cn:
J) a joy slick: and
g) a bul!on.
27 . .-\ medin ns in clnim 26 wherein the !irs! window is
trnnslucem: nnd the portion of the second window is visible
while under the lirsl window.
28. A media as in claim 27 wherein the lirsl window is a! a
lop level in a window displaying hierarchy.
45 . .-\machine readnb!c medin contnining cxccuwble com-
pUler progrnm instructions which when executed by n dig.ital
60 processing system cnuse snid system to perform a method to
display a user inlerbce window. the method comprising:
29 . .-\ medin as in clnim 27 wherein a degree of trnnslu-
cency of the !irs! window is ndjustablc.
30 . . \ medin as in claim 26 wherein snid closing. the !irs! 65
window comprises:
li1ding out an image of the lirsl window.
displaying a Jirsl window. the Jirsl window being translu-
cent. at least a portion ofn second window being cnpnble
of being. displayed on the digitnl processing. system
under the !irs! window. the portion of the second win-
dow. when present. being visible under the Jirsl window
on a screen: and
us 7,853,891 82
13
dosing the !irs! window without user input. wherein the
Hrst window hns been disp!nycd independent from a
position ofn cursor on the screen.
46. A mcdin as in claim 45 wherein the method funhcr
compnscs:
starling a timer:
wherein said closing the !irs! window is in response to
cxpirmion of the timer.
47. A mcdin as in claim 45 wherein the method fun her
comprises:
rcccJnng an input. the input no! associated with a user
input de\ icc of the digital processing system:
wherein said closing the !irs! window is in response to the
input.
48. A mcdin as in claim 45 wherein the method funhcr
comprises:
detcJmining whc!hcr or no! a condition is mel:
wherein said closing the Jirsl window is in response to a
determination that the condition is meL
49 .. \ mcdin ns in dnim 45 wherein snid dosing. the !irst
window comprises:
fading olll nn imngc of the !irst window.
15
14
60. :\ digital processing system as in claim 51 !i.1rther
comprising.:
means for repositioning. the !irst window on n displny in
response to n second inplll for the !irst window
6 I.:\ digital processing system as in claim 60 wherein the
s<.>cond input indicates that a third window is displayed.
62.:\ digital processing system as in claim 60 wherein the
second inplll is received from a user inplll device ofthcdig.ital
processing system.
6J .. \ digitnl processing system ns in clnim 60 li.1rthcr
compnsmg:
means !lH adjusting a position of the !irs! window in a
window displnying. hicrnrchy in response to n third
inplll.
64 .. \ digitnl processing system ns in clnim 51 li.1rthcr
compnsmg:
means !lHdetcJmining a position on a display of the digital
processing system independent of a position of a cursor
on the display:
wherein the !irst window is displnycd m the position.
65 .. \ digitnl processing system us in dnim 64 wherein the
position is centered horizontally on the display.
50. A media as in claim 45 wherein a degree of translu-
cency of the !irs! window is adjustable.
66. :\ digital processing system as in claim 51 !i.1rther
"" compnsmg:
51 :\digital processing system to display a user interface
window. the system comprising.
means for restarting. the timer in response to receiving. n
second inplll for the !irst window.
mcnns for displnying n !irst window in response to receiv-
ing a !irst inplll from a user input device of the digitnl
processing system which is capablcofdisplayingat least
a portion of a second window concurrently with the !irs!
window on n screen:
67 .. \ digitnl processing system us in dnim 66 wherein the
s<.>cond input is received !ium a user input device of the digital
30
processing system.
mcnns for starting. n timer: and
mcnns Jix closing the !irst window in response ton deter-
mination that the timer expired:
35
wherein the !irs! window docs not close in response to any
input limn a user input de\ icc of the digital processing
system. wherein the !irst window has been displayed
independently li"om a position of a cursor on the screen.
40
52. A digital processing system as in claim 51 wherein the
!irs! window is translucent: and the portion of the second
window is visible while under the lirst window.
5J. :\ dig.ital processing. system as in claim 52 wherein the
!irst window is at n top level inn window displnyinghicrarchy 45
54 .. \ dig.ital processing system us in claim 52 wherein a
degree of translucency of the !irs! window is adjustable.
55.:\ digital processing system as in claim 51 wherein said
means !l1r closing the lirst window comprises:
mcnns for fading out nn imngc of the !irst window '0
56.:\ dig.ital processing. system as in claim 51 wherein the
S<.>cond window. if displnycd. docs dose in response to nn
input !ium a user input de\ icc of the digital processing sys-
tem.
57.:\ dig.ital processing. system as in claim 56 wherein the '
5
!irst window docs not respond to nny inplll from n user inplll
device of the dig.ital processing system.
58. :\ digital processing system as in claim 51 flll1hcr
compnsmg:
means ll1r repositioning the !irs! window in response to a
third window being displny<.'d.
59 .. \ dig.ital processing. system as in claim 51 further
comprising:
mcnns for hiding. the !irst window in response to a third 65
window being displayed at a location where the !irs!
window is displayed.
68. :\digital processing system as in claim 66 wherein the
!irst window is crcntcd by a !irst applicntion and the second
window is crcntcd by a S<.>cond applicntion. wherein the !irst
application is dif!Crcnt from the second application.
69.:\ digital processing system as in claim 51 wherein the
user input device is one of:
a) a k'-')boan.l:
b)n mouse:
c)ntrackball:
d) n touch pad:
c) n touch scr<.-en:
l) n joy stick: and
g) n button.
70. A digital processing system to display a user interbcc
window. the system comprising:
means llHdisplayinga !irs! window. the !irs! window being
trnnsluccnL at least a portion of n second window being
capable of being displny<.'d on the digital processing
system under the !irs! window. the portion of the s<.>cond
window. when present. being visible under the !irs! win-
dow on a screen: and
means for dosing. the !irst window without user input.
wherein the !irst window hus b<.-en indepen-
dent from a position of a cursor on the screen.
71. :\ digital processing system as in claim 70 !i.1rther
compnsmg:
means !l1r st;u1ing a timer:
wherein the !irs! window is closed in response to expiration
of the timer.
72 .. \ digitnl processing system ns in clnim 70 li.1rthcr
comprising.:
means ll1r r<.>cciving an input. the input not associated with
a user input de\ ice oft he digital processing system:
wherein the !irs! window is closed in response to the input.
us 7,853,891 82
15
73. A digital processing system as in claim 70 flll1hcr
compnsmg:
means for determining whether or not a condition is met:
wherein the Hrst window is dos<..-d in response to a dctcr-
minntion that the condition is met.
16
74 A digital processing system as in claim 70 wherein said
means for closing the Hrst window comprises:
means for fading out an image of the Hrst window.
75. A digital processing system as in claim 70 wherein a
'i degree of translucency of the Jirsl window is adjustable.












































Exhibit 7
(12) United States Patent
Hamel et al.
(54) CA\"TILEVEREI> PUSH HUTTO\" ILWI\"G
\1ULTIPLE CO\"T.\CTS A\"D FUL('IW\1S
(75) !mentors: Hradley .1. I lame I. Sunnyvale. CA (US):
Tang Yew Tan. San Francisco. CA (US):
Erik Wang. Redwood City. C.\ (US)
Assignee: Apple Inc .. Cupct1ino. CA (US)
( * ) Notice: Subject to any disclaimer. the term of this
pntcnt is extended or adjusted under 35
U.S.C. l54(b) by !RR days.
(21)
(22)
(65)
(60)
(51)
(52)
(5R)
(56)
Appl. No.: 12/239,102
Filed: Sep.26,2008
Prior Publication I>ata
US 2(Xl9/tnOIR57 AI Dec. 10. 2(X)9
H.clatcd U.S. Application I>ata
Provisional application No. 611059.753. tiled on Jun.
7. 200R.
Int. Cl.
//01// 2J/OO
u.s.('].
(2006.01)
200/339: 20015 R: 2001553
Field of(:Jassification Search 2001512.
20015 Hi. 517. 553. 557. 55R. 339: 34! 120 22:
345/156. !57. !61. 16R. !69. IR4
SL>c application !ile for complete search history.
References Cited
S .. 12<J.27g _\ '' 7 1<J<J4 Dombroski 341 20
111111 1111111111111111111111111111111111111111111111111111111111111
US007863533B2
(I OJ Patent No.:
(45) Date of Patent:
5,."01>,47<)
'
4 ]<)<)(;
5,."1>4.31>0
'
12 ]<)<)(;
(, 2_1<).3 <) 2 Bl -' 2001
(i.24(i.OJ<J Bl $ 6 2001
(i.274JQ(i Bl $ :; 2001
(i,%1,052 Bl 11 2005
ti,%7,300 B1 11 2005
(,,<)<)5,32(, B2 2 200(,
7.25<J.D<J Bl $ 2007
* cited by examiner
US 7,863,533 B2
Jan. 4, 2011
200 5 R
'\aitou ....................... 200 315
d al 200 5 R
d al 200 (i A
Scriz<twa d al 200 5 R
va,iri .......................... 14-' 11>4
\li;nmo a1
'""'"'"'"
200 ."12
Sugimum d a1 200 5 R
ct 200 5 A
l'riman Hxaminer Michael A Friedho!Cr
(74).11/orne\: Agent. or h'rm lkycrl aw (imup 1.1.1'
(57) AHSTH..\CT
.\cantilevered push button adapted Jix accepting. an input on
an electrical or electronic device is disclosed. The button can
include an elongated \muon lop component disposed about an
exterior surli1ce of;m clec!rical or clec!ronic dL'\ ice such that
it is accessible to a user. and having two opposing. distal ends
associated with separate user inputs .. \ !irst li.1lcrum is located
bel ween the !irs! distal end and the midpoint oft he elongated
bul!on lop component. while a second !lilcmm is located
belween the second distal end and the midpoint. A !irs! elec-
trical contact is associated with the !irs! distal end. such that
when a user presses on the !irs! distal end. the elongated
bul!on lop cmnponenl pivots about the second !lilcmm and
the !irs! clec!rical conlacl is acluated. A second clec!rical
contact is similarly associated with the second distal end and
!irst li.ilcrtlm.
20 Claims, 6 I>rawing Sheets
100
112
U.S. Patent
Jan. 4. 2011
Sheet I of 6
US 7,863,533 B2
U.S. Patent
0
0
~
Jan. 4. 2011 Sheet 2 of 6
N
N
~
US 7,863,533 B2
U.S. Patent Jan. 4. 2011 Sheet 3 of6 US 7,863,533 B2
N
N
-
-
co
.,.
C"'
-
'
(!)
i!:
U.S. Patent Jan. 4. 2011 Sheet 4 of 6 US 7,863,533 B2
U.S. Patent
0
0
~
N
0
~
Jan. 4. 2011 Sheet 5 of6 US 7,863,533 B2
U.S. Patent Jan. 4. 2011 Sheet 6 of 6 US 7,863,533 B2
aq tma Tiu!o'il;t10J a1(.!_ aa.n:J p.m-'>'.dn
1(11'>\ pap1,op1d aq .{t,m 'P!![.'>\ Jl':l!.lJXp
ad.\1 uonnq amop uea Sl:JP-JU<XJ JP-:l!Jpap pur,
lSJ!J at]l uol)]ppe U] wuawm aJq1xaum Jat]lO Jo ":J1JSJJ]d pJet]
1mam e ]epawm p1il!J .\ummmsqns e JO
aq .\1'm do] uonnq paw'iluop at]l Sllll!J1,.\ U]
u! Jo at]l utD ll"Jl
-tpal Jo uounq .n\a(1Jtm:J JaqJo apnmd 01
e S] at]l m]p.\\ pa:-pO.\\ ,\HeJauail a.wt] suo1
-1nq .\\P-S:J."")S Jo p.."")JaAajl)Ue:J suil]sap .(uem awl/\:\
"j]aSJ! UOlllll( at]lJO UO]UOd
atjl PlU)SUll:l OJ fA'"OSll aJq1X.'"O[l
Jatj]O JO (l'PlU 1' Ual(.\"\ UlD 1ndm JOJ auo
l(l!.'>'. sum.D[l1.Jltnla.J.J1P mo.1 a.\1"]tm:J s]ndu! JllOJ t]l!.'>'. uonnq
paJa,\annm::> e a]dumxa JO;] aq 01 n]'ilnot]l 1ndm uonnq
Jad UHU:J]HJ auo "JP-]IlJ])Jed U] tril]sap uaA]il e JOJ paJ1sap
aq .\1'm s1' pasn :Jq osp' .{tml ]tmO!l1PPV
J<.m mmnq a1p .JO JU!odpm1 a1p Ulll.l.J aatmJS!P Jl'nha
ue p.."")Je:JO] aq uea at]lt]ilnot]l]r Jat]Uil;] a]qeJqpJd
uonnq at]l JO )modp1tu at]l tuOJJ .\e.\\P- JllJ .\pueamtril]s
)P-tj) UO!)lJ:lOj.(Ue pua j!J)S]p r, W S] tulU:ljllJ r, JOJ UO]WJO] ]P-Ul
-!Jdo qTimll]l]V uonnq p."")J.'"I.\.'"1(1Jtm:J atp.Jo pua Jl'lS!P
J.'"lt]Jmm 11' paJtDO( :Jq tm:J pumas aq1 ptm pua Jl'lS!P
auo W fi<"")Jr,:JO] aq U!J:l UHU:ljllJ atp.(p,\]WUJa)j\" pe)UO:l
]P-JUPa]a )SJ!J at]l w pa)eJO] aq uea UHU:J]HJ puo::>as at]l Jo;pue
"Pl'JUO:l (lDUPa(.'"O puoxs fA'"Ol1':lO( aq tm:J )S.l!J
atp uonnq tjsnd p.<la.\a(!Jtm:J S1t]l.JO U]
09 stua]qll.ld pJ1,'>\Ul S! pua P.'"'11mp1' at]l aJ!l]'>\ (l':l
-ttmtpam Jat]lOtm Jo a:J]Aap at]l ,\q a:Je]d m ppt] aq 01
pua pa)enpe-uou at]ll]tuJad 01 saxau uonnq paJa,\anme::>
at]l .JO Jo amos l1'l(ltpns sp,ua11'm -'"']qt'![d Ja1po Jo ":l!l
"Jaqqtu p.--.uuo.l am suonnq p.--.Ja"-'"'l!llllD amos
sa:J!Aap J]al]l UllUJ .(JJ.\\lJ flU!AOUl Jatj)O
Jo uonnq t]ll.\\ S]t]l JOJ awsuadmo::> 01 JapJo U]
S] pua JP-]IlJ])Jed e uat].\\ a:J]Aap at]l
tu<.UJ ,\e.\W aMltu paJa,\anme::> JO sptw
O> .'"OWl] OJ .{1JlD1Plj]Sa1' SSa(l1"]'>\.'">UlOS fA'"0.\1.'"0:JJad
aq ]]1JS mD l! a]mh aq utD uat]'>\
paJa\.'"1]1JutD .{q p.--.p\ml JlmP1' at]lt]'illlm[l[V
aJt,\ap at]lJO )SaJ at]l tu<.UJ .\e.\\P- Jo pJR\\dn .\uenpe
pua Jat]lO at]l"PJJJUO:J uonnq a)enpr, 01 Jasn e .\q
a:J1\apat]l
p.m.\"\Ol pass;ud 'ilmaq uonnq paJa\."")]!llllD aJ1]Ua
atp m uea UO]Jemae pur, aatAap at]l
m pat]snd pua auo uat].\\ let]ltpns unu:J]l1J ]P-J)Ua:J e )noqe
JO.\!d .{]Jl':l!d.{J suonnq tjsnd Jo p.<la.\a(!Jtm:J tpns
qsnd P.'"'J-'"'"-'"'l!llllD atUJ .\tmm 1(11'>\
atu!l aums at]l w tpoq a]m1P1' 01 a1q1ssod J<.m
s1 l! aptml P1,JUO:J (lDUPa[a Jaum ptm puo:J.--.s
JO .\.mmpJo ilupnp Jasn e ,\q pa1enpe aq
utD (l'lS1p ptm .'"11[l.JO t]JOq sno
!.m\ U] "fA'"OJm1p1' S! Pl'JUO:l (l':l!JJ:l.'"O(:l puo:JaS .'"01(1 ptm
lSJ!J at]l moqe sJO,\td )Uauodmoa d01 uounq pa1eiluop at]l
uat]l")Uauodmoa do) uounq pa1eiluop at]lJO pua ]JJlS]p
atp uo Jasn e uat].\\ .(pewms pa)enpr, S] pe)UO:J
(lDUPa(.'"O aq1 ptm Ulll.l:l(ll.J pmnas aqJJllllq1' )0\!d UlD )U.'"OU
-odmoa do] uonnq paw'iluop a1p uat]l 1uauodmo:J do] uonnq ..
pa1eiluop at]JJO pua ]eJStp JSJ!J atp uo sassaJd e tWt]/1:\ ;c
Ot e pua Jat]lO atp uat].\\ pur, pa)enpr, J:JJJJUOJ
]JJ:l!JPap Jaum JSJ!J e pua auo tWt]/1:\ stwunq )ndu]
maJa.mp o.\\l .\]aA!)Jq[p Jaquwm Ja)no pawiluo1a
"]uama'i!ut!U1' tm tpns U] Tiu!
nwuodmo::> do) uounq pa1eiluop at]lJO pua
atp t]l].\\ aq ueJ PJJJUO:J ]P-JUPa]a at]l a]ll].\\
"]uauodm<.n do] uonnq P.'"'l1'1luo1a at]l .JO pua Jl'lS!P at]l
l]l!.\"\ aq UlD P1,]UO:l (l':l!JJ:l.'"O(:l .'"01(.!. ")U.'"OtllldlUO:l
0
,.
do) uounq paJeiluopat]lJO nnodp]tu at]l pur, pua ]eJStp
atp uaa.\\Jaq Jo pua atple p.."")Je:JO] aq ueJ unua
-]nJ puo::>as at]l a]ll].\\ nwuodmo::> do) uonnq pa)eiluop at]l
ptm amm1' 'ilmpL\OJd aJ!l]'>\
suounq o.\"\lJO J:l.'"O.J_Ja at]J'ilmp!.\OJd .JO .{1,'>\ auo
suonnq U.\\OU!f os1e t]snd paJa,\annm,)
aJe tpns
.\uem aJat].\\ aJ]\ap ]]rJaMl atp JOJ a::>tmJeadde xajdtuo:J Jo
pa1annp1' aq tm:J]ndu! Tiu!ll1mJad.n:J stmam1' qsnd
'i!U!Sll OJ :O(:l1'lJ.\"\mp aU() "J.'"OU1J!sap 1' .\q paJ!Sap ll!
pasn aq tm:JJO "P1'd k--.)j-() 1 p.moqk--.)jtqo J.md aq mD
tpns pue sqou1 tpnoJ ilu]pnpu]
.JO ]umdpm1 at]l ptm pua at]l uaa.\"\pq m pua Jl'lS!P
)S.l!J .'"01(1 W pa]lDOj aq UlD lUlU:ljll.JJS.l!J .'"01(.!. "Jasn OJ
pur, a:J]Aap :l]UOJ):l."")p JO jlJ:l!JPap tm JO a:JP-jJilS JOUa)Xa U!J
)twqe )Uauodmoa do1 uonnq pa1eiluop
at[.!_ 1ndm Jasn JS.l!J at]l Ulll.l.J ]ndu! Jasn
samwaJ t]ll.\\ Jasn e ,\q sptmmmo::> Jo uonemJOJU]
"" ilu]nndm JO aJmhaJ tWlJO tpns snoJmh]qn
1' t]l!.\"\ paw!::>oss1' ptm pua JS.l!J at]l a]1soddo pua Jl'lS!P
ptm ]ndu! 1[11'>\ P.'"'l1'!:JOSS1' pua Jl'lS!P ""
aAet] tmJ do1 uounq p.."")wiluo1a atn pr,)uoa ]P-:JUPa]a puoaas e
pue pemo::> ]JJ:l!Jl:lap e mtUJ]l11 puoaas e mtUJ]IlJ lSJ!J e
"]uauodm<.n do] uonnq P.'"'lt,Tiuop tm apnpm tm:J uounq tjsnd
p.--.Ja.\a(!ltm:J "P.'"'P!\OJd aanap :l!UOJJ:l.'"Op Jo Jl':l!Jpap
uo 1ndm tm J<.:J uonnq tjsnd p.<la.\a( "l
-]JUJJJ e uonua,\U] atp JO U]
pat]snd uounq atpJo pua tpp:].\\ .\q pau]tu
-Jalap ilmaq UHU:J]l1J at]lt]ll.\\ "lOA]d ueJ uonnq paJa,\a]
-!Jtm:J aqJt]J!l]'>\ JlllllJ1' a[d!l(l1Ul.JO asn aqJt]'ilnoJt]l
Ul W aq UlD S1l].L fA'"OljSlld fll
pawnpr, atp uat].\\ 01 at]JJO sptw pawnpe
-uou at]l aJ]nbaJ lOU op let]! Sl:JP-JUOJ ]P-JUPa]a a]dl)
]ntU 'ilmwt] p.--.Ja.\a(!llllD p11l!J as1-\\.Jat]JO Jo :J1(
Jl'PlU apL\OJd OJ UO!JU.'"O\U! al[l.JO tm S! l]
J..}J\"I"ll"lflS
flmsea]d aJotu aq os1e .(em )JJt]l S]e!Ja)etu
-sod .JO J.'"l)1,a!TI .H:J .'"1(11(.\"\ a]q1'!PJ
<:
'
.WlU am JO S.'"O:l1 \ap :l!UOJPa(.'"O JO (lDUPa(.'"O
Jatj]O ptm Jl,jlljp:l "Kl:J.{l,jd l'!fA'"Oj/IJ

J]UlUpap JO ]P-JUJJa]a
uo suonnq paJa,\anme::> JO at]l 01 .\pe(llJ])Jed aJom
ptm 01 .\j(l'Jaua'il um]ua.\m atLL
<I I'll:! !V.J!NILHL
]]!J JOJ pue .()aJ!)Ua S)l U] aJtWJqpJ ,\q UjaJat]
paJeJod!o::>m tpttj.\\ .:SVI!fHLY lfl:l C!NV S.LN.LNO,)
!!"ldi.L"IflVII !)NL\\"11 NO.LIJH! !!Sfld CHN!L\!l"II.LN\'.),
P.'"'Jl!lua xooz: "L unr P.'"'l!J "t<;C6S"OI!'J oN umJtD1[ddv
\lUd "S" [} OJ .{J!JO!Jd Stump UO!]lD1(dd1'
SNOI.LV.J! !ddV
C!:l.L\ "!!!}! O.L !UN!!}I!U!!"N ssmu
srxmtnn:I <I\:,. s.L>Y.L\:O.> :n<n.nmx
.'\:OJ .. lJlH IISJld (J:.UI:U:nl.l.'\:V:>
l
zs w;'s98'L sn
us 7,863,533 82
3
included or only partially included in any dcsin.:d combina-
tion for a given cantilevered push button.
In li.1rthcr embodiments of the present invention. an ck>c-
trical or electronic device can include a housing adapted to
contain one or more intcmal ck:clrical device components 'i
therein. one or more user interbcc regions. and a e<mtilc\crcd
push button locmcd at one of the user intcr!Ucc regions and
ad.1ptcd for accepting one or more inputs on said electrical
tk'\icc. with the cantik:n::red push bul!on being any of the
cantik:n::red push \muons described above. In the c\cnl of an lCJ
ck:clronic device. a processor located within the housing may
be included. and the user intcr!Ucc region or regions can be in
communication with the processor .. \ plurality of cnntilc-
\ered push \muons may also he used on such a device, and
may he placed at one or multiple user inlerli1ce regions. 15
Other nppnrmuses. methods. fcmures nnd ndvnntngcs of the
invention will be or will b<.>come nppnrent to one with skill in
the nrt upon cxmninmion oft he following Hgures nnd dctnilcd
description. !l is intended that all such additional systems.
methods. JCalurcs and advantages he included within this ""
description. be within the scope of the invention. nnd be
protected by the nccompnnying dnims.
IWII;I: I)I;SCRWI-ION 01'-rlll; I)RAWINGS
4
specilic embodiments of the present invention. Although
these embodiments nrc described in suflicicnt detail to ennblc
one skilled in the nrt to prnctice the invention. i1 is understood
that these examples arc not limiting. such thm other embodi-
ments may be used, and changes may be made without
dep;u1ing Jimn the spirit and scope of the imenlion.
The invemion relntes in vnrious embodiments to the imple-
mentmion nnd use of cantilevered push buHons ns input
mechanisms on dec!rical devices. Such a device can he, l(Jr
example. an elcc!ronic device. such as an il'od R media player
or il'honc R cellular telephone made by Apple Inc .. as well as
n PD.-\. handheld gnmc or video gnmc controller. radio. min-
imure television. keyboard. or nny other electricnlor elec-
lnmic de\ ice !hal uses push hullons l(Jr user input.
In \arious embodiments of the present invention. a canti-
levered push button hnving multiple contncts nnd multiple
li.1lcrums is provid<.-d. The cantilevered push button cnn be
ndapted such that dilkrent Ji.ilcrtlms arc used to pivot the
hullon depending upon which end or portion of the \muon is
prcssL'<i. In this manner, the merall motion and rcacli\e dis-
placcmcmof the button cnn be controlled. even where metal
or other rigid materinls nrc us<.-d to construct the user portion
of the hullon. The merall ellixL look and J(:el can generally
he more aesthetically pleasing lo the user.
Re!Crring lirsllo FIG. I. an exemplary canlile\ercd \muon
nlong the side of an outer housing li.1r nn nssociatcd cellular
telephone is shown in side perspective vkw. Cellular tele-
phone housing I 0 can include \arious prolmsions. ports and
!Calurcs. as will he readily appreciated. l'orcxample. a hole or
"111c included drnwings nrc for illustrntive purposes and
serve only to provide exmnples of possible structures and
arrangements !(Jr the disclosed invcnli\c apparatuses and
methods l(Jr prm iding cantilevered hullons ha\ ing multiple
contncts and Ji.llcrums. These drawings in no way limit nny
chnngcs in J(mn and detail thnt mny be made to the invention
by one skilled in the art without departing li-mn the spirit and
scope of the imenlion.
30 recess along the side of housing 10 cnn be made li.1r cantile-
vered push button 100. Such a bu1ton 100 can hnve n Hrst
distnl end 101 nnd n second distnl end 102. us well as a
midpoint or center !OJ. l:ach of distal ends 101. 102 can he
l'Ki. I illuslrales in side pcrspccli\e \i<.'\\' an exemplary
cmuilcvered button nlong the side of anomer housing for nn 35
associm<.-d cellular telephone.
adap!ed lo aclualc an associated hullon or dec!rical conlacl
within the device when they nrc dcprcss<.-d or otherwise used.
.\hhough shown on the side of a cellular telephone. there nrc
l'Ki. 2A illustrates in side de\alion and partially cutaway
\ i<.'\\' an exemplary cantilevered \muon according lo one
embodiment of the present invention.
1-'JG. 2B illustrntes in side elevmion nnd p.1rtinlly cmawny 40
view a close-up of one end of the exemplnry cantilevered
\muon ofl'Ki. 2A.
many suitable locmions Ji.1r such cantilevered buttons. nnd
numerous other types of de\ ices can JCaturc such a button or
hullons, as will he readily appnxiatcd by those skilled in the
nrt.
Moving next to FIG. 2.-\. nn exemplmy cantilevered button
according lo one embodiment of the present imenlion is
illustrates in side elevation and p;u1ially cutaway viLw. As
noted above. cantilevered push \muon I 00 can he embedded
l'Ki. 2C illustrates in side cross-sL>clional \i<.'\\' the exem-
plnry cnmilcvered bunon of 1-'JG. 2.-\.
1-'JG. J.\ illustrates in side elevmion and p.1rtinlly cmawny
view the exemplar)' cmuilcvered bun on of FIG. 2.\ with the
!eli end aclua!ed according lo one embodiment oft he present
imenlion.
1-'JG. JB illustrates in side cross-sectionnl view the exem-
plmy cantilevered button of FIG. J.\ with the !eli end actu-
ated.
I)I;'IAII 1:1) I)IOSC'RII,-r!ON
Fxcmplmy npplicntions of nppnratuses nnd methods
according lo the present invention arc descrihL'<i in this s<.>c-
tion. rhcsecxamples arc being provided solely lo add conlcxl
and aid in the understanding of the invention. It will thus be
apparent to one skilled in the an thm the prescm invention
mny be prncticed witholll some or all of these speciHc detnils.
In other instances, well known process sleps ha\e no! been
described in detail in on.lcr lo avoid unnecessarily obscuring
the present invention. Other npplicntions arc possible. such
thm the Ji.1llowing cxnmples should not be taken as limiting.
45 within a hole or other recess in housing 10 . .-\n elongnted
bun on top component 100 cnn include opposing distnl ends
I 01. 102. and a midpoint or center !OJ. Firs! distal end I 01
can he associated with a lirsl inlcmal hullon or elcc!rical
contnct 121 inside the device. while second distnl end 102 can
-;o be nssociatcd with n second intemal bulton or electricnl con-
tnct 122 inside the device. The elongnted button top compo-
nent 100 is generally outside or ahoullhe dL'\ ice and exposed
loa user, while an in lema! elongated \muon base I 04 c;m he
nnached or otherwise coupk-d to the elongmed button top
'5 component. One or more protn1sions 111 in the elongnted
hull on hasecmnponenl I 04 can he adap!ed lo help acluale the
dec!rical conlacls 121, as will he readily apprL>ciatcd.
FIG. 21S illustrates in side de\alion and partially cutaway
view n close-up of the right end of the excmplmy cantilevered
60 bunon of 1-'JG. 2.-\. .\s shown. distal end 102 is disposed
dirL>clly above internal hull on or ck>clrical con lac! 122. ln!er-
In the Ji.1llowing detailed description. references arc made 65
lo the accompanying drawings. which J(mJJ a part of the
description and in which arc shown. by way of illustration.
nal hullon or dec!rical conlacl can he, !(Jr example. a pre-
loaded dome type button or other suitable electricnl contact
device. ns will be rendily apprecim<.-d by those skilled in the
nrt. When a user presses downward on or nbolll end 102. the
hullon lop component 100 generally depresses downward al
distal end 102 and in the dirL>clion of ek>ctrical conlacl 122.
us 7,863,533 82
5 6
This !(Jrccs the portion oflmllon base 104 dircc!ly beneath
end 102 downward us well. such thnt protmsion 112 presses
on the top 132 of button or electrical contact 122. and this
s<..>cond "button" or electrical contact is thereby actuated. For
this process of actuating second \muon or contact I 22 by 'i
depressing second distal end I 02. the actual !i.J!cnnn is a! the
other end of the cantilevered button and is not shown in FIG.
2B. In the event that the other end of the cantilcvcn.-d button
present on canti!e\ered button IIHI. it is possible to depress
both ends 101. 102 to actuntc both internal buttons or electri-
cal contncts 121. 122 s.imultnn<.."'ously. In such .instances. such
s.imultnncous actuation results .in the entire cantilevered push
button being press<..-d inward toward its respecti\eelectrica! or
electnmic de\ ice. rhis kature prm ides an additional ad\<111-
tnge over trndit.ionnl or customal)' cantilever buttons . .in that
most such devices do not permit both buttons at both ends to
be actuated simultan<..ously. Accordingly. the respccti\e elec- is depressed lo acli\alc the other dcctrical conlacl (both no!
shown in F!(i. 21S). then the cantik:n::r pivots about conlacl
I 22 (i.e .. axis 141 ). t-his process is made possible due lo the
gap between button base 104 and internal structure l3J.
!-'IG. 2C illustrates in side cross-scctionnl view the cxcm-
pbry cantilevered button of !'!(i. 2A. Again. pressing at end
t" trical or electronic de\ ice can he design<..-d or programmed to
accept and act upon simultaneous inputs Jium multiple inputs
on a s.ing.!c cnntilevercd button.
I 01 actuates contact I 2 I. while pressing at end 102 actuates l'i
contnct 122. However. the li.llcrum is dill"crent for each of
Continuing now to FIGS. 3.-\ nnd 3B. the cxcmplnl)' cnn-
ti!e\ered button of!'!(i. 2A is shown with the !ell end actuat<..-d
according to one embodiment of the present invention. FKi.
3.-\ depicts a side elcvm.ion and partially cutaway view. while
FIG. 3B depicts n side cross-sectionnl view . .-\s .i!!ustrat<..-d.
d.istnl end 101 hns been pressed downwnrd such that protm-
sion 1 I 1 has activat<..-d button or electrical contact 12 I directly
these difl"crcnt actuations. In the event thm end 101 is pressed.
then the li.llcnnn is nlong. contact 122 (i.e .. nxis 141). while if
end I 02 is pressed. then the !i.J!cmm is at contact 121 (i.e .. axis
142). Since button base 104 is efli.:cti\ely separated Jium ""
intcrnnl stmcture D3. the efl"cctive mechnn.icnl contnct points
betw<..-en the elongntcd button top nnd bnse nnd the internnl
stmcture are at the internal buttons or electrical contacts I 2 I
below distal end I 01. rhe Ji.J!cmm ll1r the entire cantilevered
button 100 Jix this nctuation .is dk"'<::tively button or electrical
contnct 122 at the opposite end. with rotnt.ion g.cnernlly being
about a point atop the dome of contact 122 (i.e .. along axis
141 ). Of course. the merall device can be substantially sym-
metrical in nature. such that the same relationships and results
can be had llx depressing. distnl end 102 .
and 122. Thus. when a mechanical downward J<.ll"ceisapp!ied
at one end above one electrical contact. the ell"cctive Ji.J!crum ""
ism the other electrical contact m the other end. The result is
thnt multiple Ji.llcnnns are used with the overall cantilevered
button. such that the opposite end does not percepti\ely
extend away Jimn the device when one end is pushed inward
to nctivmc its button.
While not neccssm)' in order for the cnntilevercd push
button to dkctively hnve multiple fulcrums. the spring
loaded internal button at each end aids in prm iding more
rigidity and suppm1to the merall device. That is. when end
101 (nnd thus button orcontnct 121) is pushed downwnrd. the
spring. loaded Jixce on contnct or button 122 g.cnerally dis-
poses end 102 to rcmnin up and steady while button 122 is the
Ji.ilcrum ll1r the entire elongated button top component 100.
Although it is thought that the top of the dome of a spring
lond<..-d dome type button makes nn excellent locution for a
fulcrum. other locnt.ions nrc also possible. ns w.i!! be readily
appreciated. In bet. any location at a distal end or between
one distal end ofthecantil<..'\ered push button and its center or
midpoint could be suitable ll1r a Ji.ilcnnn when the other distal
end is being pressed or actuntcd.
.-\s w.i!! be rcnd.ily npprceintcd. the farther n li.llcnnn .is awny
from the midpoint or center of the button. the better the elli.:ct
will he as Jiu as minima! movement of the opposite end when
a part.iculnr end is pressed. Thus. an opt.imnl location for a
fulcrum can actunlly be at the opposite end of the button from
the end thnt is being pressed or nctuntcd. Conversely. n ful-
cmm location that is at or dose to the midpoint of the button
can he less desirable. although not impossible to implement in
a particular design . .-\s shown .in the cxcmplmy .i!!ustrnt.ions
provided. the Ji.1!crums used are near the distal ends of the
cantilevered push button. with the results of such locations
being bvorah!e.
Also. it should he noted that the exemplary design illus-
trntcd nnd descr.ib<..-d herein utilized "soft" li.llcnnns . .in thm
the li.llcrums do provide res.istnnce and are cfl"cct.ive .in ordi-
nary use. hut can be dd"eated if desired in a particular way.
That is. each dome loaded electrical contact prmides sulli-
eient res.istnnce to function us n Ji.llcnnn when the opposite
end .is press<..-d or actuated. but w.i!! not provide enoug.h resis-
tance to be n rigid "hard" Ji.llcnnn if force is nlso placed
generally abme it at the same time. !kcause a "hard" or
permanent centra! or midpoint li.1!cmm is not physically
.-\g.ain. because the Ji.llcnnn is moved away from the mid-
point or center of the cantilevered button. any resulting
motion rise of the opposite end is eliminated or substantially
30 reduced when either end is pressed. "111is .is true even where
the upper portion of the cantilevered button .is a rig..id materinl.
such as a metnl. hard plastic or other .inllcxib!c materinl. .-\s
will he appr<.."'<::iated. such an ability provides designers with
added Jlexihi!ity in their choices ll1r materials and appear-
35 nnces in the dcsig.n and prcsentmion of cantilevered push
buttons that are fully li.mctional and nesthctically pleasing. in
nppearnnce nnd use.
It should be not<..-d that although the Ji.J!cmms in the
examples prm id<..-d above are generally !ocal<..'d an equal dis-
40 tnnce from the midpoint of the button. tills is not nbsolutely
nccessnry. !-'or example. n pnrticulnr design might favor one
li.1!cmm being located at a distal end. and another li.1!cmm
being located halfway between the other distal end and the
45
midpoint of the button. or even closer to the midpoint.
Furthermore. nlthough the foreg.oing cxnmp!cs nll usc n
relatively simple cnntilever<..-d button with only two inputs on
opposing ends. more complex buttons with additional Ji.il-
cmms may also be used. as may be desired ll1r a gi\en design.
For example. a four-way cnntilever<..-d button hnving. Ji.1ur
-;o inputs and Ji.1ur Ji.1!crums mny be used. In such a desig.n. the
button top component may be shaped !ike a cross nnd may
ha\e a Ji.ilcnnn that cmTesponds to each of the Jlmr difl"crent
inputs at each distal end oft he cross. Altematively. the button
top could be shaped like a circle. with inputs similarly m the
'5 "north." "west." "south" nnd "cnst" coordinates of the circle.
The use of a simple single user input on such a button (e.g ..
"north") could result in a piHJting about an opposing li.1!cmm
(e.g .. "south"). similar to the two input elongated canti!ever<..-d
push button in the Ji.1reg.oing. examples. In such an cmbodi-
60 mcnt. the simultaneous nctuntion of two ndjnccnt inputs (e.g ..
north and east). could result in the simultaneous piHlling
about the two opposing li.1!cmms ll1r those inputs (e.g .. south
nnd west). Similar to the li.xcgoing two input button embodi-
ments. a cnntilever<..-d push button having Ji.1ur or more inputs
65 could also have "soli" Ji.1!crums. such that the entire button
and all inputs could he actuated at once. As will he readily
appreciated. li.1rther designs and additional inputs ll1r e\en
us 7,863,533 82
7
more complex cantilen::rcd \muons ha\ ing multiple !i.J!cnnns
could also be used within the spirit of the present invention.
Although the foregoing invention hns been described in
detail by way of illustration and example Jix purposes of
clarity and umk:rslamling. it will be rccogniA::d that the abo\c 'i
described invention may he embodied in numerous other
spcciHc variations and embodiments witholll dcpmting from
the spirit or essential charnctcristics of the invention. For
example. although many i!luslralions ha\c pointed to the usc
of metal as the material !(Jr the c;mtile\cn.::d push bul!on. a lCJ
rigid and in!lcxible plastic or other material may ahcmativcly
be used. Multiple materials may also be us<..'d to fonn the
button. Other changes and modiHcations mny be prncticcd.
and it is understood that the invention is not to be limited by
the details. hut rather is to he ddined by the scope 15
of the appended c!nims.
What is claimed is:
1 . ..\ cnnti!even.-d push button adnpt<..-d !\1r accepting one or
more inputs on nn electrical or ek>ctronic device. comprising:
a !muon top component having a !irs! distal end associated ""
with a Jirst user input and a second distal end opposite
said !irs! distal end and associatL'<i with a second user
input sepnrate from said Hrst user input. wherein snid
button top component is disposed about nn exterior sur-
bceofan ck>ctrica! or electronic de\ ice and is accessible "
5
to a user. and wherein both of said !irs! and SL>ctmd distal
ends cnn be nctumed simultaneously by a user during
ordinal)' use of said electricnl or electronic device:
a !irst li.llcnnn !ocatL'<i at snid !irst distal end or between snid
Jirst distal end and the midpoint of said butlon top com-
30
ponent:
a second li.i!crtlm located at snid SL>cond distn! end or
between snid second distn! end nnd said midpoint ofsnid
button top component:
35
a !irs! dectrica! contact associated with said !irs! distal end
of said button top component. wherein said butlon top
component pivots about said SL>cond Ji.i!crtlm and snid
Hrst electrical contnct is actuated when a user presses on
said Hrst distal end: and
"
a second ek>ctrica! contact associated with said second
distal end of said butlon top component. wherein said
button top component pivots nbollt snid Hrst Ji.llcnnn and
said SL>cond electricn! contact is nctunted when a user
presses on said second distal end.
2. n1e cantilevered push button of claim I. wherein said
!irs! Ji.J!cnnn is !ocatL-d at said SL>cond dectrica! contact.
J. The cnnti!evcred push button of claim 2. wherein snid
SL>cond Ji.llcnnn is locmed m said Hrst electricnl contnct.
45
4. The cnnti!evcred push button of claim l. wherein snid ,
0
butlon top component is comprised of a substantially rigid
materia!.
5. The cnnti!evcred push button of claim 4. wherein snid
button top component is meta!.
6. The cnnti!evcred push button of claim l. wherein snid -;5
!irs! li1!cmm is located at said !irs! distal end and said second
Ji.ilcrum is located at said second distal end.
7. The cnnti!evcred push button of claim l. wherein snid
!irst and second electrical contncts comprise dome button
type electrical contncts. 60
8. n1e cantilevered push button of claim 7. wherein said
dome butlon type dectrica! contacts are prdoaded with a
positive upwnrd force.
9. The cnnti!evcred push button of claim l. wherein snid
simultaneous nctuation results in the entire cnnti!even.-d push 65
butlon being pressed inward toward said dectrica! or ek>c-
tronic device.
8
I 0. The cantilevered push button of claim 1. wherein said
button top component is adnpt<..-d to be disposed through an
opening of nnd not contact n housing of snid electricnl or
electronic device.
I 1. An ek>ctrica! dL"\ ice. comprising:
a housing adapted to contain one or more intema! dectrica!
device components therein:
one or more user interJUce regions: nnd
a cantilevered push button located at one of said one or
more user interli1ce regions and adapted l\1r accepting
one or more inputs on said dectrica! device. wherein
said cantilevered push button includes
an elongat<..-d button top component having a !irst distal
end associated with a Jirst user input and a SL>cond
distal end opposite said Jirst distal end and associated
with n second user inplll separnte from said Hrst user
input. wherein said elongated button top component is
disposL'<i about nn exterior surJUce of an electricn! or
dectnmic device and is accessible to a user. and
wherein both of said !irs! and second distal ends can
be nctumed simultnnL'ously by n user during ordinmy
use of said electricnl device.
a Jirst Ji.J!cnnn located at said !irs! distal end or between
said !irs! distal end and the midpoint of said elongated
button top component.
a second li.i!crtlm located at said second distnl end or
between snid second distnl end and snid midpoint of
said dongatL-d button top component:
a Jirst ek>ctrica! contact associated with said Jirst distal
end of snid elongat<..-d button top component. wherein
said dongnted button top component pivots about
said second li.i!crtlm nnd snid !irst electrical contnct is
actuatL-d when a user presses on said !irs! distal end.
and
a second electricn! contnct nssocinted with said second
distn! end of snid elongmed button top component.
wherein said elongmed button top component pivots
about said !irs! Ji.ilcrum and said second dectrica!
contact is actuatL-d when a user presses on said SL>cond
distn! end.
12. The electricn! device of claim 11. wherein said electri-
cal dL"\ ice comprises an ek>ctronic device.
I 3. The electrical device of claim 11. wherein said Jirst
!i.1!cmm is located at said second ek>ctrica! contact.
14. The ek>ctrica! device of c!nim IJ. wherein snid second
Ji.1!crum is locnted at said !irst electrical contnct.
15. '!-he ek>ctrica! device of claim I 1. wherein said don-
gated button top component is comprised of a substantially
rigid materin!.
16. The electrical device ofc!nim 11. wherein said !irst nnd
second ek>ctrical contacts comprise dome button type electri-
cal contacts.
17. The ek>ctrica! device of claim I I. wherein said simu!-
tnnL'OUS nctumion results in the entire cnntilevcred push but-
ton being pressed inward toward said electricnl device.
18. The ek>ctrica! device of claim I 1. wherein said don-
gated button top component is disposed through an opening in
said housing and docs not contact said housing.
19. An ek>ctronic device. comprising:
a housing ad.1pted to contnin one or more internnl electrical
de\ ice components therein:
a processor located within said housing:
one or more user interface regions having one or more user
interface components in conununicntion with snid pro-
cessor: nnd
one or more cantilevered push buttons located at at !cast
one of said one or more user interli1ce regions and
us 7,863,533 82
9
adapted !(Jr accepting a plurality or on said dec-
Ironic device. wherein each of said one or more cantilc-
push buttons includes
an elongated button top component having a Hrst distal
end associntcd with a !irst user input and a S<..>cond -'
distal end opposite said !irs! distal end <md associated
with a second user input separate ![om said !irs! user
input. wherein said dongntcd button top component is
disposed through an opening in said hm1sing. docs not
conlacl said housing. and is acccssihk: to a user.
a Jirsl fu!cmm and a second lli!cmm about which said
elongated bul!on lop pivots when aclmJtcd by a user.
a !irs! dcctrical conlacl associated with said Jirsl distal
end of said elongated button top component. wherein
10
said elongated \muon top component piHJls about
said second li.ilcrtlm nnd snid !irst elcctricnl contnct is
nctunt<..-d when n user presses on snid !irst distnl end.
nnd
a second electrical contact associat<..-d with said second
distal end of said elongated bu!lon top component.
wherein said elong.mcd button top component pivots
nbom snid !irst fulcrum and snid second electrical
contact is actuated when a user presses on said second
distal end.
20. "!.he clectnmic d<..'\ ice of claim 19. wherein both of said
!irst nnd second diswl ends cnn be actuated simultaneously by
n user during ordinnry usc of said electronic device.












































Exhibit 8
(12) United States Design Patent
Chaudhri
(54) GH..\PIIIC\L USEil. FOH..\
I>ISPL\Y SCREE\" OR PORTIO\" TIIEH.EOF
(75) !mentor Imnm ChaudhrL San Francisco. CA
(US)
(73) Assignee: .\pplc Inc .. Cupcnino. C.\ (US)
(**) Term: 14 Years
(21) App!. No.: 29/281,656
(22) Filed: Aug. 20, 2007
H.clatcd U.S. Application I>ata
(63) Continuation of npplication No. 291281.695. Hied on
Jun. 28. 2(Xl7. which is n continumion-in-part ofnppli-
cation No. 291281.507. !ik:d on Jun. 25. 2007. now Pal.
No. Des. which is a continuation-in-pari of
applicmion No. 291281.460. tiled on Jun. 23. 2(Xl7.
now I'm. No. Des. 604.305.
(51)
(52)
(58)
(56)
LOC (9) Cl.
U.S.Cl.
14-04
1>14/486
Field of(:JaJ>J>ification Search 1)14/485 95:
DIR/24 33: Dl9/6.52.9.10: D20111: D21/324 33:
7151700 867. 973 77
Sec application Jilc l(Jr complete search history.
Cited
U.S. P.\TJ:NT J)OCUMJ:N"rS
4.736.199
'
4 191>1> Chadwick ,] 715 71>1
D29.".1>77 s 5 191>1> d a!
.'i.2!4.7.'i6 A 5 l<J<J3 Franklin ct
.'i.34.'i5.'i2 A <) l<J<J4 Bmwn ........................ 7!571>4
.".5."." .. 164
'
<) 1<)<)(,
.".ti42.490
'
6 1<)97 \1organ a! 715 762
.'i.7!2.<J<J.'i A I J<J<jg Cohn ......................... 7!57'!2
D.190.541> s 2 1<)91> \hckmv" a!
!).192.266 s .11<)91> Snydcrcl ,]
.'i.<J<J.'i.tm A II l<J<J<J Ashe
s l 2000 ct
D420.<J9.1 s 2 2000 Decker
Jl'
(I OJ Patent No.:
(45) Date of Patent:
D4.11>.21.1 s 2 2001
])441.76.1 s "2001
6.22<J . .".12 B1
"2001
])445.421> s 7 2001
])462.076 s g 2002
us 0627,790 s
** Nov. 23,2010

Kahn
Fujii .......................... 7!." 7.1.1
Pallcndcn
Robbin
(Continu<..'d)
I:ORIO!CiN I'Al"ION"I"I)OC"UMI;N"l"S
Dl235!27 4 2005
Or!!!]{ PUB!JC..\T!ONS
..PD.\s \\ilh a buill-in !..\'\.using '\ursing Suppm1
(a,io .Vc"H'.\' Re!ea.\e. hllp """ ""sio co jp release 2003
nns htmL 21>. 2003
(Continu<..'d)
l'rimwT Hxaminer Melanic I I Tung
(74) Attomel'. Agent. or 1-i'rm S!crnc. Kessler. Goldstein &
Fox P!.!.C
(57) CI.AI\'1
The ornmnenwl desig.n for a graphical user interface for a
display screen or portion !hereo[ as shown and described.
])ES<:RIPTI():\
rhc I'!CiURI: is a front \iL"\\" of a graphical user inlcrbcc l(Jr
a display screen or portion thereof showing my nL-v. design.
The broken lines of the display screen or portion thereof and
other dements J(mn no p;u1 of the claimed design.
1 ("]aim, 1 ])rawing Sheet
US D627,790 S
Page 2
U.S. PA:t-I;N"r J)OCUMI;NI-S
Bl <) 2002 Go ken 715 716
Bl 12 2002 Bmwndal 715762
!)472.244 s .1200.1 Wasko
Bl 3 2003 Kido 715767
6.nt.3t6 B2 5 2004 I krigstad ct al
Bl'' <) 2004 Del Vecchio d al _1(il <)7
6.1>07.5."7 Bl'' !0 2004 '\m-,cs cl al 70<) 201
!)507577 s 7 2005 Toucn d al
6.<JlG.424 Bl 1 200(> Dull a
7JJ20.1W< B2 ''
.1200(> Roscn/wcig cl al 7151>."5
7.051>.902 B2 (i 200(i lwcma ct al 715100
7.075512 Bl 7 200(i Fabrcct al 345 J.'i(i
7.107.54<) B2 <) 200(> Dcalond al
7.11>-'Ug<J Bl'' 2 2007 'li!im" 7151>!0
!)540.340 s 4 2007 Cummins
1)544.1>75 s 6 2007 Wang cl ,]
D541U3<J s :; 2007 Rimas-Ribik<!uskas cl al
D.'i.'i2.(i20 s to 2007 Sato ct al
D.'i.'i.'i.(i60 s II 2007 '\ovidlo d al
D5S<J.261 s 1 2001> Jung cl ,]
!)564530 s 3 2oog Kimctal
D56.'i5l>(i s 4 2oog Shin ct al
D56.".51>1> s 4 2001>
D561>.<)00 s 5 2001> al
D569.31>3 s 5 200i-: Jung et al
D570.35i-: s (i 200i-: Anastasopoulos et al
D574.00<) s 7 2001>
D574 .. 11>1> s :; 2001> ,]
D576.171 s 9 200i-: Annendarit et al
D579.4.'i(i s lO 200i-: Chen ct al
D579.<J4ti s 11 2001> I d al
D5l>O.<J4<) s 11 2001>
D51>1.424 s 11 200i-: !long
D5K'i.075 s I 2009 Flynt ct al
7 .41>7 .467 B1'' 2 200<) Kmv,hamd al 7151>10
D51>7.720 s .1200<) '\miello d al
7552.401 B2 (i 2009 Guido eta! 715i-:31>
7574.(i(i6 B2 g 2009 Vliyrunoto 715769
7 .51>7 .til>O B2 ''
<) 200<) Wad.a 7151>04
7.(>00.192 B1'' 10 200<) ]],shimolodal 7151>02
7.M3.036 B1 I 2010 Rees 345 594
7.(i53.i-:1>2 B2 I 2010 llorentrup et al 715 i-:21
OT!IER PUBLIC.\TIONS
l- S .\ppl '\o 29 21>1.460. Annues et al. Icons. Graphkal Lser
and ](,-a Disph!y
or Portion fikd Jun n. 2007
l- S .\ppl '\o 29 2l>L'i07. Annues et al. Icons. Graphkal Lser
Interface. and Graphical Lser Interfaces for a Display
or Portion fikd Jun 2.". 2007
l- S .\ppl '\o 29 21>1.695. Annues et al. Icons. Graphical Lser
Interface. and Graphical Lser Interfaces for a Display
or Portion fikd Jun 21>. 2007
1: S -\ppl '\o 29 21>1. 747. al "! . Icons.
Interface. and Graphical Lser Interfaces for a Display
or !'onion Thereof. fik-.1 .\ug 10. 2007
1: S -\ppl '\o 29 21>2.1>35. "!. Icons.
and ](,-a Disph!y
or !'onion Thereof. fik-.1 Jul 30. 2007
1: S -\ppl '\o 29 21>2.1>37. al "!. Icons.
and ](,-a Disph!y
or !'onion Thereof. fik-.1 Jul 30. 2007
\. S .\ppl '\o 29 2i-:2.1>3K .\ntures ct al. Icons. Graphicall-scr
;md -\nim,led Gmphica! l."ser for" Display
Screen or 1'o11ion filed Jul 30.2007
\. S .\ppl '\o 29 2i-:2.1>39 .. \ntures at al. Icons. Graphicall-scr
;md -\nim,led Gmphica! l."ser for" Display
or P011ion filed Jul .10. 2007
\. S .\ppl '\o 29 2i-:2.1>40 .. \ntures ct al. Icons. Graphicall-scr
lntel'!ilce. and .\nimated Graphical l'scr lntel'!ilces !Or a Display
or P011ion filed Jul .10. 2007
\. S .\ppl '\o 29 2i-:2.1>4!. .\ntures ct al. Icons. Graphicall-scr
lntel'!ilce. and .\nimated Graphical l'scr lntel'!ilces !Or a Display
or P011ion filed Jul .10. 2007
1; S -\ppl '\o 29 21>2.1>42. -\nn1res "' ,]. Icons. Gmphica! l."ser
lntel'!ilce. and .\nimated Graphical l'scr lntel'!ilces !Or a Display
or P011ion filed Jul .10. 2007
1; S -\ppl '\o 29 21>2.1>43. -\nn1res d ,]. Icons. Gmphica! l."ser
lntel'!ilce. and .\nimated Graphical l'scr lntel'!ilces !Or a Display
Screen or 1'o11ion filed Jul 30.2007
1; S .\ppl '\o 2<) 21>4.264.1 emay ala!. Interlaces
](,-a Disph!y or P011ion filed Sep 4. 2007
\. S .\ppl '\o 29 302.019. Chaudhri ct al . Image !Or A
Portion of Disph!y Jikd .hm 1>. 2001>
1; S .\ppl '\o 29 .102.06.1. -\n/ures d a!. Gmphica! Interlaces
for a Display Screen or l'o11ion filed Jan 9. 2001>
\. S Appl '\o 29 302.217 .. \n;-uresct al.Graphical Lser Interfaces
](,-a Disph!y or P011ion filed Jan 11. 2001>
\. S .\ppl '\o 29 302.429. Chaudhri at al . Image !Or A
!'onion of Display Screen. fik-.1 Jan 16. 200i-:
1; S .\ppl '\o 29 .104 . ."1>4. -\n/ures d a!. Gmphica! Interlaces
](,-a Disph!y or P011ion filed \1ar 4. 2001>
\. S Appl '\o 29 317.755 .. \n;-uresct al.Graphical Lser Interfaces
for a Display Screen or l'o11ion filed Vlay (i. 2001>
1; S .\ppl '\o 2<) d "!. Gmphic;!ll."ser Interlaces
for a Display Screen or l'o11ion filed Vlay (i. 2001>
\. S .\ppl '\o 29 3 !i-:.44(i Chaudhri et al . Graphicall-scr Interfaces
](,-a Disph!y or P011ion filed \1ay 20. 2001>
1; S .\ppl '\o 29 .11 <).31><). -\n/ures d a1. Interlaces
for a Display Screen or l'o11ion filed Jun i-:. 2001>
1; S .\ppl '\o 29 .11 <).397. -\n/ures d a1. Interlaces
](,-a Disph!y or P011ion filed Jun <J. 2001>
\. S Appl '\o 29 319.434 .. \n;-uresct al.Graphical Lser Interfaces
for a Display Screen or l'o11ion filed Jun 9. 2001>
1; S .\pp1 '\o 2<) .12 1..174. Chaudhri . Gmphicall:serlnlerJaces
](,-a Disph!y or P011ion filed Jul 1.". 2001>
\. S .\ppl '\o 29 324.194. Evans et al . Graphical l'scr Interfaces
](,-a Disph!y or P011ion filed Sep 1>. 2001>
1; S .\pp1 '\o 2<) .124.1<).". Chaudhri . Gmphicall:ser Interlaces
for a Display Screen or l'o11ion filed Sep 1>. 200i-:
\. S Appl '\o 29 324.196. ('haudhri eta! . Graphicall:ser Interfaces
;md Icons fora Display or P011ion filed Sep 1>. 2001>
\. S Appl '\o 29 324.935 .. \n;-uresct al.Graphical Lser Interfaces
and Icons for a Display Screen or !'onion I hereof. Scp 23.
2001>
1; S -\pp1 '\o 29 32(>.3%. Christie d ,] . Icon](,-" Display
or !'onion Thereof. fik-.1 Oct 16. 200i-:
\. S Appl '\o 29 333.233 Graphicall-scr lmel'!ilces and Icons for a
Disph!y or P011ion filed \1ar 4. 200<)
\. S Appl '\o 29 333 .. 'i59 .. \nimaK-.1 Image. Graphicall-scr Inter
faces and Icons !Or a Display Screen or !'onion I fik-.1 Vlar II.
2009
1; S .\ppl '\o 29 B.1.l>ti7. lim1ge. Graphicall."ser Inter-
faces and Icon for a Display Scrwn or !'onion I hereof. fik-.1 Vlar l(i.
2009
1; S .\ppl '\o 2<) 3.1.".21>2. and Icon for" Display
Screen or l'o11ion filed Apr 13. 2009
* cited by examiner
U.S. Patent Nov. 23,2010 US D627,790 S

' '
/ --------------------------. '
, ,...-:.------------------------- ' '
I 1/ " '\ \
I 'I \ \
I I, \\ I
I
' I' I, .. -----, 'I 'I
'-------'
' 'I
I I, i i
i! I I
' ' - c '"""' = II
I I t .... I 1
'' tl
liODDDii
I
' I I'
' II
, I '
! i I
I]DDDDii i I I'
i . I\ '
I, i I I
i! l i i
I, i I
I
I I I I
' II
'I ,
I, i I
I
I! I I
I I
, I '
I, i I
I
I! I 1
11
'I '
I, i I
liDDDOii
'' II
I
I , '
11
'I '
I, /--..._ /1
i ! t' -r-r \ i i 1
\ t I, \ / J} j
''\ ' / t'J
\ \ -..... __ .. /I;
' ' ........ .. .. /
'' --------------------------/
.............. _-=--_ -::_-_ -:._-_-=. -_ -=----=----=--_ -=----=:_-=--_ -=:_-=--_ / ..
FIGURE












































Exhibit 9
(12) United States Design Patent
Andre et al.
(54) ELECTH.O\"IC I>EVICE
(75) Inventors: Hartley K..\ndrc. Menlo l'nrk. C.\
(US): Daniel J. Coster. San Francisco.
CA (US): J):mide J)c Iuliis, San
Francisco, CA (US): Richard P.
Howarth. San !-'rnncisco. C.\ (US):
Jonathan P. Iw. S.1n !-'rnncisco. C.\
(US): Stcvt> .Jobs. Palo Alto. CA (US):
J)uncan Robcrt Kerr. San Francisco.
C.\ (US): Shin \"ishibori. Portola Valley.
C.\ (US): \1atthcw Dean Rohrbach.
San !-'rnncisco. C.\ (US): Peter
Russdi-Ciarkc. San Francisco. CA
(US): Douglas H. Satzgcr. Menlo Park.
C.\ (US): Christopher J. Stringer.
Woodside. C.\ (US): Eugene .\ntony
\Vh:mg. San Francisco. CA (US): Rico
Zorkcndorfcr. San Francisco. CA (US)
(73) Assignee: .\pplc Inc .. Cupenino. C.\ (US)
(**)
(21)
(22)
(63)
Term: 14
.-\pp!. No.: 29/J19,J77
Filed: .Jun. 6, 2008
J.tclatcd U.S. Application I>ata
Continuation-in-part of applicmion No. 291306.334.
tiled on .\pr. 7. 2008. and a continumion-in-pan of
application No. 2()/306.950. tiled on Apr. IX. 2(XJX.
which is a continuation ofapplication No. 2()/306.334.
Jik:d on Apr 7. 200X.
(51) LOC(9)CI.
(52) u.s. Cl.
14-02
DI4/J41
(I OJ Patent No.: us 0602,016 s
** Oct. 13, 2009 (45) Date of Patent:
(58) Field of("Jassification Search ])141341 347.
(56)
])14/420. 426.427.432.439 441.448.496.
])14/125. 129 130. 137. 138. 147. 156.218.
Dl4/247 24X. 250. 3X9: DI0/65. 104: DlX/6 7:
I )2! 1329. 6X6: D6/596. 601. 605: 455190.3.
45516.1. 556.2. 575.1. 575.3. 575.4: 3791433.01.
379/433.04.433.06.433.07: 361/814: 341/22:
345/169: 3461173
Sec application lik: I(Jr compk:lc search history.
J.tcfcrcnccs Cited
U.S. 1'.-\TI:N"r DOCUMJ:NTS
7 19'!3
2 2000 \1omles
.'i 2000
DD7 . .'i(i<J S
])420.354 s
D424 . .'i3.'i S
(i.2<J7.7<J.'i Bl $
])4<)1>.754 s
])."04.1>1><) s
D.'iO.'i.<J.'iO S
])."14 . ."<)0 s
D."l<J.l Hi S
D.'i20.020 S
D.'i27.37<J S $
])."21> .. '42 s
])."21> . ."61 s
D.'i2<J.04.'i S
D.".12.7<)1 S
])."34.!4.1 s
D.'i3.'i.2i:t S
D.'i3(i.6<J I S
D.".16.%2 S
D.'i4l.2<Jl> S
D.'i4l.2<J<J S $
])."46.31.1 s
])."41>.7.12 s
D.'i41>.747 S
10 2001 Kmo d .................. 34.'i 61>4
1! 2004 Blyth
." 2005 d
6 2005 Summit d
2 2006
4 2006 d
.'i 200(i d
1>200(i
9 2006 l.uminosu d ,]
9 2006 e1 ,]
9 200(i Shin
1!2006 Kim
12 2006
I 2007 Yang
2 2007 Pm-k
2 2007
4 2007 .\mired
4 2007 .\mired
7 2007
:; 2007 ,1
1>2007
D.'i.'i(i.2Jl S 112007
])."57.2.11> S 12 2007 Kim
D.'i.'i7.2.'i<J S $ 12 2007 llirs<:h
])14 241>
])14 4%
])14 217
US D602,016 S
Page 2
s 1 2oog Yu eta!
D5."1>.7Sti s 1 2001> _\n<lrc d ,]
D.'i.'il>.7.'i7 s 1 2oog Andre ct al
D.'i.'il>.7.'ig s 1 2oog Andre ct al
s 1 2001> Chigira
D5S<J.220 s 1 2001> Icc d al
1)560.192 s 1 2oog Lccd al
D560.MO s 1 2oog Lee
D560.M<ti s 1 2001> Kim
D56l.L'i3 s 2 2oog llongd al
1)561.204 s 2 2oog Toh
D56L71>2 s 2 2001> Kim
!)562.21>5 s 2 2001> lim
1)563.432 s 3 2oog Kim
D563.<J2<J s .12001> l'"rk
D5MUO<J s 5 2001> Ccbc cl al D14 .141
D57.'i.2.'i<J s g 2oog Kimctal
1)571>500 s to 2oog Lccd al D14 Dl> G
D57<J.1Nl> s 11 2001> Icc d al Dl4 BlHi
IW<0 . 11>7 s 11 2001> _\n<lrc d ,]
D.'il>l.31>4 s 11 2oog Kimctal D14 131> .\D
IWH.<J22 s 12 2001> _\n<lrc d ,]
IWG .. 145 s 12 2001> Kimd"l
D5M.272 s I 200'! Chung ct al D14 131> G
D5M.303 s I 200'! Andre ct al Dl4 341
D5M.ti4ti s 1 200<) Sanguinclli 1)1065
D5M.711 s I 200'! Kimctal
D.'iM.ng s I 200'! Kimctal Dl4 4%
1)51>.".411 s 1 200<) F<1ton Dt4
JW<g.14ti s .1200<) Tsuboi ])14 474
D5<Jl.2.'i2 s 4 2009 Kimctal Dt4 nn;
D592.ti2g s 5 2009 Kimctal Dt4 ng .\D
200_1 012507<) _\]'' 7 200_1 l'"rk <:tal 455 5."(,
2004 OD2499 AI 7 2004 Abc
2004 016ti<J07 AI g 2004 Yajima
2004 022.1004
"
!1 2004 I inch<:! al
2007 oognJg
"
4 2007 Yoon<:l al
200i-: 00040K'i AI I 200i-: Jung ct a!
!-'ORJ:JGN 1'.\TJ:NT J)OCUMI:nrs
Jl' Dl2S04g7 <) 200-'
KR 12 200.'i
OTIIER PUBLIC.\TIONS
SmTISung F700. announc<:d hb 2007. [online[. fr<:tric,cd on Feb g.
2007] R<:tric,cd Jfom Intern<:! .<l."R!_ hup wwwgsm,rcnacom>
Philips S900. Jun 200ti. on Feb 20.
2007 J Retrieved from lntcmd .<l-RI. http v,v./V/ gsmarcna com>
J _(i KJX<;O l'mda. ;mnounccd Jan 2007. fonlin<: ]. on hb
20. 2007[ from lntcmct .<LRL hup v.wwgsmarcna
com>
-\pplc il'hon<:. ;mnounccd .h!n 2007. fonlin<: ]. fr<:tri<:\<:d on \hr !2.
2007] R<:tric,cd Jfom Intern<:! .<l."R!_ hup wwwgsm,rcnacom>
linnos PDA. \1ay 19. 2006.[on!inc].[rdrievcd on .\ug 22.
2007] R<:tri<:\<:d Jfom lnt<:rn<:t .<l."RJ _ hup www nmbik'\vhack
com>
\1ciftl \1K posted Jan. 29. 2007. [online[. on Scp D.
2007] Retrieved from lntcmd .<l-RL http wv,wcngadgd com>
Jlilon I G D\1B \11'.1 F\1.1-'.p"stcd Jun 26. 2006. fonlinc].[r<:tric\cd
on D<:c 2ti. 200g1 R<:tric,cd Jfom lnt<:rn<:L <l."RJ _ hllp hilon com
en amobak agi-:JOOO.'i549>
\1ag;uin<:. '\o !2 (2004). p 60
\1ag;uin<:. '\o !1<) (Jun L 2005). p 45
cFashion \1aga;-inc. '\o 172 (Apr 2006). p 26
l- S .\pp! '\o 29 I. Amlrcd a! . Lkdronic Device. fikd Ju!
_10. 2007
l- S .\pp! '\o 29 Amlrcd a!. Lkdronic Device. fikdJu!
2007
1: S -\ppl '\o 2<) 2g2.g33 _\ndr<: <:! al. Electronic Dc,icc. filed Jut
_10. 2007
l- S .\ppl '\o 29 2i-:4.1i-:7 And red a! . Fkcnonic Device. fik-.t Aug
2007
{. S App! '\o 29 2i-:4.1 gg Andree! al .l:!cctmnic Device. fikdAug
3!. 2007
{. S .\pp! '\o 29 2i-:4.2(i9 .\ndrccta!. F!cctmnic Device. fik-.l Scp
4. 2007
1; S -\ppl '\o 2<) 2g4,27!. -\ndr<: <:tal. Ekctronic Dc,icc. Jikd Scp
4. 2007
{. S .\pp! '\o 29 2i-:4.272 .\ndrccta!. F!cctmnic Device. fik-.l Scp
.'i. 2007
1; S -\ppl '\o 2<) 2g4,276. -\ndr<: <:tal. Ekctronic Dc,icc. Jikd Scp
.'i. 2007
{. S .\pp! '\o 29 2i-:4.2i-:g .\ndrccta!. F!cctmnic Device. fik-.l Scp
-' 2007
1; S -\ppl '\o 2<) 2g4,_10g. -\ndr<: <:tal. Ekctronic Dc,icc. Jikd Scp
.'i. 2007
1; S -\ppl '\o 2<) 2g4,_110. -\ndr<: <:tal. Ekctronic Dc,icc. Jikd Scp
-' 2007
{. S .\pp! '\o 29 .\ndrccta!. F!cctmnic Device. fik-.l Scp
.'i. 2007
1; S -\ppl '\o 2<) _106 .. 134. -\mired,] . Electronic D<:\ icc. Jilcd .\pr
7. 2oog
{. S .\ppl '\o 29 9.D9 Andree\ a! . Lkctmnic Device. fikd Jun
-' 2oog
1; S -\ppl '\o 29 3! <JAB. _\ndr<: <:tal . Fkctronic D<:\ ic<:. filed Jun
9. 200i-:
l. S .\pp! '\o 29 .. \ndrcd al . Fkcnonic Device. fikd Apr
]g. 2oog
* cited by cxmnincr
l'riman Hxaminer Calhron C ISnmks
Assistant Hxaminer ISarhara l'ox
(74) Allome:r. A?,ent, or Firm Sterne. Kessler. Goldstein &
Fox P.LLC
(57) CI.AI\-1
The ornamental dcsig,n Jix an electronic device. as shown and
described.
I>ES<:RIPTIO:'\
FIG. 1 is a front view of an electronic device
showing our new design:
FIG. 2 is a rear view thereof:
FIG. J is a front view thereof:
FIG. 4 is a rear view thereof:
FIG. 5 is a side view thereof:
FIG. 6 is another side view thcRof:
FIG. 7 is a top view thcRof:
FIG. 8 is a bottom view thereof:
FIG. 9 is a front perspective vi<..'\v of another embodiment of
an ck>ctronic device showing our n<..'\V design:
FIG. 10 is a rear perspective view thereof:
FIG. 11 is a !font view thereof:
FIG. 12 is a rear vi<..'\v thereof:
FIG. lJ is a side vi<..'\v thereof:
FIG. 14 is another side view thereof:
FIG. 15 is a top view thereof: and.
FIG. 16 is a bottom view thereof.
The !Ucc. back and side sur!Uccs of the electronic device
shown in the FIGS. 1 8 arc illustrated with color dcsig,na-
tions. The face and the back surface arc black. The side
smbccs arc silver. gn:y or chrome.
The bee. hack and side surbccs of the dcc!nmic device
shown in the nc;s. 9 16 arc illustrated with color dcsigna-
US D602,016 S
Page 3
lions. The liJCe and the back surbcc arc white. '!lu:: side
surfaces arc silver. grey or chrome.
The broken lines in the Figures represent unclaimed portions
of the electronic device andb form no p.1rt of the clnimcd
design. The shade lines in the Figures show contour and not
sur!Ucc omamcntmion.
1 Claim, 4 Drawing Shcch
U.S. Patent Oct. 13, 2009 Sheet I of 4 US D602,016 S
FIG. 1
FIG. 2
U.S. Patent Oct. 13,2009 Sheet 2 of 4
FIG. 3
FIG. 5 FIG. 6
FIG. 4
FIG. 7
FIG. 8
US D602,016 S
,,
,,
'
,,
U.S. Patent Oct. 13, 2009
FIG. 9
FIG. 10
Sheet 3 of 4

: ~ . - '
US D602,016 S
U.S. Patent Oct. 13, 2009
r--------------------,
. '
' '
' '
' '
...
FIG. 11
"
H
r!
"
u
"
''
FIG. 13
,,
'
FIG. 14
Sheet 4 of 4 US D602,016 S

,.)
:1
FIG. 12
FIG. 15
.,,
,,,
"
"

"'
-- --
FIG. 16












































Exhibit 10
(12) United States Design Patent
Andre et al.
(54)
(75)

( *)
(**)
(2!)
(22)
ELECTH.O\"IC I>EVICE
Inventors: Hartley K..\ndrc. Menlo l'nrk. C.\
(US): Daniel J. CoJ>tcr. San Francisco.
CA (US): J):mide J)c Iuliis, San
Francisco, CA (US): Richard P.
Howarth. Sun !-'rnncisco. C.\ (US):
Jonathan P. Iw. S.1n !-'rnncisco. C.\
(US): Stew Jobs. Palo .\Ito. C.\ (US):
J)uncan Robcrt Kerr. San Francisco.
CA (US): Shin :\ishiborL Portola \"alky.
C.\ (US): \1atthcw Dean Rohrbach.
Sun !-'rnncisco. C.\ (US): I>oughts It
Satzgcr. Menlo Park. CA (US): CalYin
Q. Scid. Palo Alto. CA (US):
Christophcr .1. Stringer. Woodside. CA
(US): Eugene Antony Whang. S.1n
Francisco. C.\ (US): Rico
Zorkcndorfcr. San Francisco. CA (US)
Assignee: Applc Inc .. Cupc11ino. CA (US)
Notice:
Term:
This paten! is subject lo a lcnnina! dis-
claimer.
14 Years
App!. No.: 29/328,018
Filed: :'100\'. 18, 2008
Rehlted U.S. Application nata
(60) Di\ision of application No. !!led on Jul.
2(XJ7. now Pal. No. Des. 58! .022. which is a con-
tinuation of application No. 29/270.888. H!cd on Jan.
5. 2007. now Pat. No. Des. 558.758.
(51) LOC(9)Cl. 14-02
(52) u.s. c1. D !4/248: D 1412m.7
(58) Field of Classification Search Dl41341.
!)141342. 343.344.345.346.347.420.426.
D !41427. 432. 440. 44 L 448. 4%. !25.
Dl4!!37. 12'>. no. ns. 250. 147. 2!8.
Dl4/247. 248.!56: DI0/65. 104: nn/168:
DlR/6. 7: D21/329. 686: 455190.3.556.1.
455/556.2.575.1. 575.3. 575.4: 37<)/433.01.
3701433.04, 36!1814: 34!122:
3451160, !73: 1)615<)6, 60!. 605
Sec application !lie ll1r comp!clc search history.
(I OJ Patent No.: us 0618,677 s
(45) Date of Patent: ** *Jun.29,20l0
(56) H.eferem:es Cited
U.S.I'Al"ION'I-IJOC"UMI;N"rS
5 Jn7 Gemmell ct
(Conlinm:d)
FOR!]GN PAT!:nr DOCUMI:NTS
F\1 OOO."ti<Jl57-0005 5 200(,
(Continu<.'d)
Or! I!]{ PUB!JC\TIONS
t; S .\ppl '\o 2<) . likd .ltil

(Conlinm:d)
l'riman Hxaminer Cathron C ISrooks
Assislanl Ewmim:r Angela J L<.'C
(74) Allome:r. A?,elll, or Firm Sterne. Kessler. Goldstein &
Fox PI LC
(57) CI.AI\-1
The ornamental design of an ck>ctronic device. as shown and
described.
nEs<:n.wno:"
FIG. 1 is a Jiont perspective view of an electronic device in
accordance with the present invention:
1:](;. 2 is a rear pcrspccli\c \ i<.w thcrcoL
FJ(j_ J is a front \i<.w thereof:
1:](;. 4 is a rear\ i<.w thereof:
FJ(j_ 5 is a lop vi<."\'. thereof:
FJ(j. 6 is boll om \ i<."\'. thereof:
FJ(j_ 7 is a !di side view thereof: and,
FIG. 8 is a right side view thereof.
The claimed surJUcc oft he electronic device is i !!ustratcd with
the color designation ll1r the color black.
rhc elcc!ronic device is no! !imi!<."d lo the scale shown herein.
As indicated in the title. the art ide of manubclurc lo which
the ornamental design has been app!kd is an ck>ctronic
device. media player (e.g .. music. video and/or game player).
media storage de\ icc, a persona! digital assistant. a commu-
nication device (e.g .. cd!ubr phone), a novelty item or loy.
I Claim, 2 Drawing Sheets
!)337569 s
D4l0.440 S
D424.535 S
D4.'i.'i.4D S
D4.'i6.023 S
S
D46<J.4U S
s
JWW.7."4 S
!)499.42_1 s
D502.l73 S
D504.iHI9 S
D50S.<JSO S
!)507.00.1 s
D.'il4.l2l S
!)5!4.590 s
D5!9.1 lti S
D519523 S
!)520.020 s
7JJ42.7!2 B2 ''
S
S
D529JJ45 S
!)532.791 s
D534.l43 S
S
D536.ti91 S
!)536.962 s
D53iUQ2 S
D54L291> S
D54L29<J S
!)541.71>5 s
D546 .. 1U S
D54i<.732 S
!)541>.747 s
D.'i.'i4.on s
!)5."6.2!1 s
!)5."7.231> s
7.30H24 B2
!)5."1>.460 s
D5."1>.7Sti S
D.'i.'il>.7.'i7 S
D.'i.'il>.7.'ig S
D5Si<.792 S
D.'i.'i9.220 S
D560.l92 S
D560.MG S
D560.M<ti S
D56l.l.'i3 S
!)561.204 s
D56L71>2 S
!)562.21>5 s
!)563.432 s
!)563.929 s
D5MUO<J S
!)573.14_1 s
!)574.015 s
7.409.059 B2
D579.<J30 S
JW<0 .. 11>7 S
D.'il>l.922 S
D.'iM.ng S
D.'i&i.l>OO S
J))g<J.<J7<) s
2004 OD2499 At
20040J6(i<)07 At
2004 022.1004 _\ l
2005 01307!-':; _\]
200(i AI
2007 AI
7 1993 Kando
(i 1999 Carnell
5 2000 l'dtob!
4 2002 Alviar ct a!
4 2002 Andre ct al
6 2002 p,]m d ,]
1200.1 Tod"l
5 2004 !luang
11 2004 Blyth
12 2004 B"hroochad al
2 2005 Jung ct a!
5 2005 Andre ct al
6 2005 Summit d ,]
7 2005 l'"i d al
I 200(i Johnson
2 2006 '\"mki
4 2006 '!imaka d ,]
4 200(i Chill Cl a]
5 200(i Scnda ct al
5 2006 Ghosh cl ,]
9 200(i Luminosu ct a!
9 200(i Ka-Wci ct a!
<) 2006 Shin
112006 Kim
12 200(i
I 2007 Yang
2 2007 l'"rk
2 2007 -L1!UlCr
3 2007 Andre ct al
4 2007 _\n<lrc d ,]
4 2007 _\n<lrc d ,]
5 2007 ]]wang ct al
7 2007 I hccm
2007 Ccbc cl al
2007 Andre ct al
10 2007 Lee
11 2007 1 !ow;ml
12 2007 Kim
12 2007 Tu ct al
1 y, cl ,]
1 _\n<lrc d ,]
I Andre ct al
I Andre ct al
1 Chigim
I Lee ct al
I Lee ct al
1 Icc
1 Kim
2 !long ct al
2 Toh
2 Kim
lim
3 Kim
3 Park
5 Ccbc cl al
7 l'"rk cl al
7 V!asui
Fujisawa
11 \hsk<!tia
11 _\n<lrc d ,]
12 Andre ct al
I 2009 Kim ct al
2 200<) _\n<lrc d ,]
4 200<) _\n<lrc d ,]
7 2004 Abc
2004 Yajima
11 2004 I inckc d al
6 2005 htiisawa
12 200(i /.uo ct al
4 2007 Yoon ct a!
US D618,677 S
Page 2
])]4 34(i
])]4 441
D14 .14.1
D14 126
.16167<)0.1
D14 .141
D14 .141
])]4 4%
379 4D 13
D14 .141
])]4 4%
D14 .141
D14 4%
.\1 $
2oog -\1
I Ohki ct al
I .l!!ng d ,]
Jl'
Jl'
IP
KR
KR
I:ORIOICiN I'AriON'I-IJOC"UMI;N"rS


2004 2902."6
30-0394921
30-0422221
12 2002
9 2005
10 2005
12 2005
200(i
Or! II]{ I'UBIJC..\TIONS
4.'i.'i .'i(i(i
[; S .\ppl '\o 2<) 2g2,K12, -\mire cl ,] , Electronic De\ icc, lilcd .l!!l
30. 2007
L S Appl '\o 29 .. \ndrc ct al . Fk-ctronk Dcvkc. Jul
30. 2007
[; S .\ppl '\o 2<) g7 _\ndrc d ,] , Electronic De\ icc. filcd_\!!g
31. 2007
[, S Appl '\o 29 Andrcd al .l:ledmnk Dcvkc. filed Aug
31.2007
L S .\pp! :--:o 29 .\ndrcctal. Flcdmnk Dcvkc. fik-.1 Scp
4. 2007
[; S -\ppl :--:o 2<) -\mire cl al. Electronic Dc,icc.lilcd Scp
-' 2007
L S .\pp! :--:o 29 .\ndrcctal. Flcdmnk Dcvkc. fik-.1 Scp
.'i. 2007
[; S -\ppl :--:o 2<) -\mire cl al. Electronic Dc,icc.lilcd Scp
.'i. 2007
L S .\pp! :--:o 29 .\ndrcctal. Flcdmnk Dcvkc. fik-.1 Scp
-' 2007
[; S -\ppl :--:o 2<) -\mire cl al. Electronic Dc,icc.lilcd Scp
.'i. 2007
l!ilon IG D\1B \11'.1 F\HS. posted on Jun 26.2006, [online],
[rdric,cd on Dec 26, Rctric,cd Jfom lnlcmcl. <CRL hup
hilon com en autobak
cl-ashion V!aga;-inc. 2004. vol :--:o 12. p 60. China
cFashion \1aga/inc . .l!!n !. 2005, \ol :--:o 119, p 45. China
cFashion \1aga/inc .. \pr 2006. \o] '\o 142. p 26, China
:--:otification and !Or Invalidation of Chinese Patent
i'l 2007.10J4g767 X. !'"len! Reexamination Bo;ml of the State of
lntdlcclu"l Properly O!licc of the PRC and English lrmdation,
maik-.1 Dec II. 19 pages
:--:otification and !Or Invalidation of Chinese Patent
i'I2007.10J4g75! <J. Palen! RccxmTiim1lion Board of the St,lc of
Intellectual Propeny Ollicc of the PRC and Lnglish translation.
maik-.1 Dec II. 19 pages
'\otilic"lion and Request for !m-,]i<hilion of Chinese Palen!
i'I2007.10J4g71<JO. Palen! RccxmTiim1lion Board of the St,lc of
Intellectual Propeny Ollicc of the PRC and Lnglish translation.
maik-.1 Dec II. 19 pages
[; S -\ppl '\o 2<) .106 .. 134. -\mired,] . Electronic De\ icc.lilcd .\pr
7.
L S .\pp! :--:o 29 306.950 .\mlrcctal. Electronic Device. filed Apr
2oog
[; S -\ppl '\o 2<) 31 <J.n<J. _\n<lrc cl al , Fk-clronic De\ icc, filed .l!!n
.'i.
L S .\ppl :--:o 29 319.433 Andree\ a! . Lk-ctmnk Device. filed Jun
<J.
[; S -\ppl :--:o 2<) 324,1.10. -\mire cl al. Electronic Dc,icc.lilcd Scp
.'i.
L S .\pp! :--:o 29 324.137 .\ndrcctal. Flcdmnic Dcvkc. fik-.1 Scp
6.
[; S -\ppl :--:o 2<) 324,262. -\mire cl al. Electronic Dc,icc.lilcd Scp
9.
L S .\pp! :--:o 29 .\ndrcctal. Flcdmnic Dcvkc. fik-.1 Scp
4. 2007
L S .\pp! :--:o 29 .\ndrcctal. Flcdmnic Dcvkc. fik-.1 Scp
.'i. 2007
[; S -\ppl '\o 2<) 31 <J.377, _\n<lrc cl al , Fk-clronic De\ icc, filed .l!!n
6.
Samsung 1700. announced lcb on lcb
2007 f fmm Internet. <l:RJ. http W'V/V/ gsmarcna com>
US D618,677 S
Page 3
Philips S900. Jun 200(i. on Feb 20.
2007] Rdric,cd Jfom lnlcrnd, <-CRI hllp """ gsm,rcn" com>
In D!X'iO l'mda. annotuKcd Jan 2007. [online]. on Feb
20. 20071 Rdri<:vcd from lntcmct. <LRL hup v.v.wgsmm-cna
com>
-\pplc iPhonc, ;mnolinccd .h!n 2007, ]online]. [rdric\cd on \hr 12.
2007 J Retrieved from !mcmd. <l-RL http v.v.w gsmm-cna wm>
Tinnos I'D.\. posted V!ay 19. 2006. [online]. on Aug 22.
2007] Rdric,cd from lnlcmd. <l.'RI. hllp """ mobikv.h,ck
com>
Vkiill posted Jan 29. 2007. [online]. on Scp 13.
2007] Rdric\cd from lnlcmcl. <UU hllp wwwmgmlgd com>
* cited by examiner
U.S. Patent Jun. 29,2010 Sheet I of 2 US D618,677 S
FIG. 1
FIG. 2
U.S. Patent
'
'
Jun. 29,2010
,
'
'
'
I

FIG. 3
'
,,
' '
'
'-,
'
-'
'
"
"
I''
< '
"
'l '
' '
"
Ut
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
'
FIG. 7
,,
'
' '
'
'
}----I
' '
' '
'
'
'
'
' '
' ' '-<..
FIG. 8
Sheet 2 of 2 US D618,677 S

' '
' '
{ ,,.;1-------------- ',
\ "
'
'
' '

' '
' '
'
'
'
'
'
'
'
'
'
'
,
---------------
'
'
t:
.. ,
' '
' '
'
'
'
' ,
'
----------------------
/
FIG. 4
----------------------- . ,
(--- _)
--------'
FIG. 5
FIG. 6
Ex. 11
lnt. Cl.: 9
Prior U.S. Cis.: 2J, 23, 26, 36 nnd 38
United States Patent and Trademark Offi ce
Reg. No. 3,470,983
.July 22, 20011
TRADEMARK
PRINCIPAL REGISTER
APPLC INC. (CALIPORNIA C'OiillORI\TION)
I INFlNITB LOOP
CUI'ERTINO. CA ?SOI4
J?OR: Hi\Nl)HllLo MOOILB DJOITAL ELECTRO
NTC Dl!VICilS COMPIUSilO Or A MOOJLB PHONB,
DIOITAl. AlJI)IO ANI'l VII)HO Pt.AVHR. HAND
HOLD COM!'UTJ!I(, PI!I(SONAL mOn'AL AliSl);
TAN'T, flLflCTROI\.IC PllRSONAl. OROANIZI!R.
POCI<rrr COMPtiTI1R I"'OR NOTI:.TAKINO, !!l.I!C.
TRONJC CALUNDAR. CALCUL.ATOR. AND CAM
f!RA, AND CAPADLD OfT PROVIDINO ACCl!SS TO
TIH"! IN"rf! RNIIT AND SllNI)INO AND RliCillVINO
I!LI!CTRONIC MAll.. OIOI'rAt. AUDIO, VIDEO.
Tt:!XT, IMAOI!S, OK/\l'HI<'S ANI> MULTlMiiDII\
PILES, IN CLASS 9 (U.S. CLS. 21. 23.26, 36 ANl) 3!!).
FIRST USB 6-292007; IN <'OMMf!RCI! 6-29.20()7.
NO CLAJM IS MAOIJ TO Till! I!XCLUSIVll
RlOHT TO US I! "SMS". APART PROM Tllll MARK
AS SHOWN
TH!l COLOI((S) DLACK. IJLUH. UROWN, DROWN
ORA V, ORAV..(JRP,RN, ORIIHN, ORANUB, IUiO,
SILVBR, Ti\N, WHITIJ ANI) YULLOW ISj i\Rll
CLAIM DO AS A f't!ATURB 017 THll MART<.
'l'Htl MARK <.:ONSISTS OF THfl CONI'IIOURA
ilON OF A RBCI'ANOULAR IIANDII BLD MOD!I.J!
OIOITAL l!Lil<"I'RONlC UUVICI:l WITH ROUNOBD
SILVI!R llUOI!S, A 9Li\CK PACH, AJ'O AN ARRAY
Of' 16 SQIJARI!ICONS WITH KOUNOI10 !!DOllS.
THU TOP 12 ICONS APPI'.AR ON A Ol.ACK OACK
OROUND. AND Ti ll! DO'ITOM 4 APPBI\R ON II
Sli..VBR DACKOROUNO. Til!! f71RST ICON Dl!
PICTS THI" L.UI11iltS "SMS" IN Olm UN INSfDE A
WHITt! SPI'I'C:H IJUDOLU ON A ORUI!N OACK
OROUNI); THII St CONO ICON IS WHITH WITH A
TtlLN RLD STRIN! ICON
D6PIC7S A SUNPLOWI!R WITH YI!LLOW PIITAL.'I,
i\ DROWN t'CNTIJR. AND A ORlli!N :m!M IN
FRONT 01' A ULUH SKY; Tlill fOURTH ICON
DL1PICTS A C'AM BRA LeNS WITH 1\ OLi\CK llAR
RP.L ANU GLASS ON i\ SlLVllR 01\CK
GROUND; Til l! Pl f7TH ICON D!PICTS A TAN
Tl'!LDVISION CONSOLR WITH DROWN KNOllS
AND A URA Y.ORI!f!N SCRBBN; THE SIXTH ICON
J)l:li'J< . ."i'S A WH!Tl! ORArll UNI! ON A OLUI!
OACKOKOUNU; THH SEVENTH I<.:ON Dlii'ICI'S A
MAP WITH YULLOW AND ORANOE ROADS, A Plf'l
WITH A Kl!l) Hl:AD. ANI) A IUlD.J\ND DLUI!
ROAD SION WITH TH!l NUMI!RAL "2H0" If'!
WHIT11; ntr niOHTH ICON OPPICI'S AN 0KAN01'
SUN 01\ A DLUU 0/\CKGROUND. WITH 1 Hll
TEMPI'lRATURB IN WHIT!!; Till! NINTH IC'ON
Dfll'ltiS A WIIITI! CLOCK Wrn.t DLACK ANO
Rim HANI)S AND NUMBRALS ON A DLAC'K
IJACKOROUNO: THH TBNHI ICON DI!PICTS
THRIJil OROWNOitAY C' IRCt.llS ANn ONfl OR
ANOB CIRCLU ON A BLACK BACKGROUND
WITH 1\ WHITH WITH 'fHI1 MA'IHUMA
TICAL SYMOOL.') f'OR A l)OIT{ON. Sl iBTJlACI.'.ION.
MULTII'UCATION. AND Till! IJQUAL S!ON ms.
Pl..AYlJO IN WII ITIJ ON Til l! CIRCU!S; THI! Bl.H
VJJN'l'li ICON OIJI'tc:rs A PORTION Of' A YIJLLOW
NO'flii'AU Wrlll ULUIJ AND RIID RUUNO, WITII
IIROWN OINIJINCJ AT THH TOP: THB iWULP'fli
lmN OPPIC'iS THKllH SlLVHR GI!ARS OVHR A
IILACK.AND-SILVI!R IIACK
OROUND; THU 'n IIRTr.f!NT14 ICON DIJPICTS A
WliJTI! TULBPtiOND RDCI!JVDR AOAINST A
ORJ:DN DACKOROUND; ntB FOURTI!DNnt ICON
l)liPIM'S A WHIT II UNVIlLOI' B OVBR A DLUIJ SKY
WITH WHITE CLOUDS; THr FIFTJ:.:bNTH lCO!'\ SI:::C, 2(1-').
DEPlCI'S A WHl'fl:.! COMPASS WlTli A WWTl;:-
AND-RrD NBI'l1lc OVER A BLUE MAl'' THE
SlXT.GENTf/ ICON DEPICTS THB UtSTil'CTIVE SER. !\0. 77-303,2S:;, PlLBD 10-12-201.17.
CONFJGUHA'l'lON Of<' APPLICANT'S MBDIA
PLAYER DE'.Vl(:Jl JN WH.lTIJ OYER ORANGf:
RACKGROUND. SKYEJ YOUNG. A'ITOJ<NEY
Ex. 12
Int. CI.: 9
Prior U.S. Cis.: 21, 23, 26, 36 and 38
United States Patent and Trademark Office
Reg. No. 3,457,218
July lj 2008
TRADEMARK
PRINCIJ>AL REGlS'U:R
APPLE IKC. (CALIFORNIA CORPORATION)
!INFINITE LOOP. MS 3TM
C\JPERTTNO. CA 95014
FOR: HAN!JHELD MOBILE DIGITAL ELLGTJ\0
NIC })(,VICES COMPIUShD OF A MOB!Lh PHON!:.
DIGITAL AUDIO AND VtrmO f'tA YER, HAND-
HELD COMPUTER, PERSONAL DIGITAL ASSIS-
TANT, ELECTROl\IC PERSONAL ORGANIZER.
POCKET C'OMPllTHR FOR NOTbTAKING, I::LHC
TROt<lC CA!.UI\1)AI\. CAJ.CULAIOR. ANl) CAM
ERA, AND CAPABLE OF PROVIDING ACCESS TO
THE INTERNET ANn SBNDlNG AND RECEIVIKG
ELECTRONIC MAlL. DIGlTi\L t\UDIO, Y!DEO,
TEXT, IMAGI:!S, GRAPHICS AND MULTIMEDIA
FILES, IN CLASS 9 (U.S. CLS. 21, 23, 26, 36 AND 3:S).
FIRST USE 19200'/; IN COMMERCE 629200'1.
THE MARK CONSISTS OF THO CONriGliRA
TION OF A RECTANGULAR HAND HeLl> MOBILe
DJO!TAL DEVlC:E W(TH ROIJNDI\D
CORNERS. THE MATTER SHOWN IN BROKEN
LINES IS NOT PART OF THE MARK
2(r).
SKYE YOUJ\"G. nXAMlNJNG i\TTORNDY
Ex. 13
Int. Cl.: 9
Prior U.S. C1s.: 21, 23, 26, 36 and 38
United States Patent and Trademark Office
Reg. No. 3,475,327
July 29, 2008
TRADEMARK
PRINCIPAL IlliGISTER
APPLE. INC. (CALIFORNIA CORPORATION)
I INFINITE LOOP
CUPERTINO. CA 95014
FOR: HANDIIDW D!Gll'AL ELECTRO
NIC DEVICES (YJMPRISED OF A MOD!LC: PHON C..
DIGITAL AUDIO AND VIDEO HAND-
HELD COMPUTER, PbRSONAL lJIGITAL ASSIS
Ti\NT. f.LECTRON.IC PERSONAL ORGANIZER.
POCKET COMPUTER FOR NOTETAKING, EJ..Ji(".
'!'RONIC CALENDAR. CALCULATOR, AND CAM
URA, A\\TJ) CAPAUL.E t-'RO'VlDlNG ACCESS TO
THE JNTF.:RNBT AND SENDING AND RECEIVINO
l::lECTRONIC' MAIL DIGITAL AUDIO, VIDEO,
'I'EXT, IM/\UES, URAPHK'S AND MULTIMEDIA
PILES, IN CLASS 9(\I.S. Ct."l. 21, 21. 26,16 AND 38).
FlRST lJSl.l 1-9-2()07; IN COMMbRCb 629200'1,
THE COLCR(S) GRAY, SILVER AND BLACK lSi
THE MARK CONSISTS OF THJ.:: L'ONfiiGURA
TlON OF A HANDHEl.D MOlllLL DIGITAL ELEC
Tl\ONIC DEVICE. THE MATERIAL SHOWN IN
DOTTED LINES. NAMELY. THE BUTTONS AND
OPEI<!NGS ON THE DEVICE SHOW THE POSITION
Of THE MARK IN RELATION TO Tl-W DEVICI.!
ANL> ARE NOT C:ONSIDElUm A PART or- TJ:U!
MARK. IHb COLOR ORA Y APPEARS AS A REC
Ti\NGLI':! AT TJJI: PRON1, CENTER OF THE DE
V1CP. THE COT.OR RLACK APPEARS ON THE
FRONT OF THE DEV1CE !\DOVE AND BilLOW
THH GRAY IU!CTANGL.t.i AND ON THH CURVED
CORNERS OF THE DEVICE. THE COLOR Sll.Vt<R
APPEARS !IS THE OUTER BORDER AND SIJ)IlS OF
THE lJEVJCE THE COLOR WHITD IS SHOWN
SOLf!LY TO IDENTIJ-IV PLAc;EMENi' OF TH!i
MAl{K AND IS NOT CLAIMED AS A PART 01'
THE MARK.
S8C.liF)
8ER. NO. 77103.049, FILED IOll2007.
CLAIMhD AS A Fl:.ATLIRL OP THIJ MARK. SKYE: VOlJNO, EXAMlNlNO ATTORNEY
Ex. 14
RcJt. No. 3,886, 196
Registered Dec. 7, 2010
Int. 1.: 9
TRAI>t::MARK
PRt NCJilAI. RF.GISTF.t"l
II I'I>U! INC (CALIFORNIA COJU'Oitii110N)
I INFJNm> LOOP
<.:UI'EJWNO, <.:11 95014
ThLlii'IIONY W/\JU!, NIIMI!LY, COMI'U11cR SOI
1
1WIIIU; JOK MAKINO,
MIINIIOINO.AND RRCmVINU l'IIONBCAJ.J.'i,ANil rOJt IU VII WINO, MANI\OINO,AND
I'I.IIY1Nl1 Fl.llCTRONIC VOICP MliS.'IAOil.'!, IN CI.AI\S 9 (II S Cl.'i lb, '() Mffi
'K)
1-li(S I USI' t>-2?-2UC0, IN {'llMMIItCI' ().21).2007
OWNI-:1{ OF U 8 R!!O. NO
1111\ COLOR(S) ORI'JlN, 1.101 IT ORRRN, 01\RK ORIhN 1\Nl> wiiJ II l:l/1\Rl! t'l.IIIMilll
Nlll lt.:ATURI! OI'TIW 1\JARK.
'11fT' MARK CONSISTS 01' /1 RFC11\NOLI Wl'lll ROUNI>I'Il ('(lRNIIRil I >I!J'IC'I'INO II
IITYI.Ii'J\11 Wlll'l1i "11il.li riiONI! IUiCIOIVJ :R 1101\INST II 1\TIUI'IU) OHI'HN liND DI\RK
OllPHN RM:J(OilOI JNI) i1 SlJAJ)P or 1.101 f1 cJilrllN I'OVHIIS '11 111 tll'l
1
1!lt 111\1,1' 01''11 II'
RI!CII\NOJ.I: D"SION.
ANDREW Rl HM. hXJ\MJNIN<.i 111' l'ORNIJY
Ex. 15
Reg. No. 3,889,642 AI'I'I.R INC. (CAI.IIIORNV\ coRPORATION)
I INlINl l1t I,O<W
Rcgistcr('d l>l.'C. I 4, 20 I 0 cuPiiR11NO, cA 9SOI4
Int. CJ.: 9 toR nn<T 11Nn Mln.nYJmlll Ml!l\!WHNo sui IWIIRI,, NIIMI'I v. soi71WAtm rok
I'ROCE.'lSINO UIU\I'IIIt'S ANI> 'll!X'I , IN tl ASS 9 (IJS Cl .'i ll . 2l, 26, .16 A Nil
1R)

PRINCIPAl. RECISTER
HRS1' 1JSI' (,.l9-l007, IN CUMMP.Rt'l' f.-l9-l!Ml7
OWNI* OF U.S. IillO NO .IA7U,?8J
1111. COI .OR(S) WHITE, (JIU\I!N, 0/\RK CIRI
1
1
1
N AND !.lOin JIN IS/AIU' CI.AIMI' D
AS 1\ H
1
A1111ll' <W 11 II' M/\ RK
1'111; MARK CONSIS'I':I 01' 1\ IUIC'T,\NOI.I: Willi llOlJNI)III) COIINIIUII>HI'IC'IlN<I/\
S I VI.IZH) SPHHCH RlliHI.h ON A lli/\(IONAI, S I 1(11
1
1
1
1> tJAt'KOI{OIINI) II IH C()l,(>lt
Wlllll! 1\1'1'1'.1\RS IN Till! Sl'r!I\CII ll\Jillli.P 1)11SION, 1111 COI.OIIS (llti'I!N AND DARK
UIU'.I!N AI' I'I'AR IN lllf 01/\00N/\1, STRII'HS IN 1'111' U/\C.:KUIIOUNO 01' l llll ltl!(.;'l
II Ntii,P fli'.SiON; ANI) 'niP. C:OI.OR Lill i II' l illlli'N 1\ l'l'HAI<II IN 'II II! 111\I ,Jl OF
'11m llHCTANOI .I! I>HSION
Slm NO R5-ll l R,959, PII.Jol) 4-l llOIU
ANIJRI\W Ill liM, /\TIUliNI
1
V
Ex. 16
Reg. No. 3,886,200
Registered Dec. 7, 2010
Int. Cl.: 9
TRADEMARK
PRINCIPAL
AI'PI .I! IN(' {CA!.IFORNlA CORP<)Rr\TION)
I JNI
1
JNITI
1
1.001'
t'III' I' I{J'IN(l, t:A ?501-1
JiOit t'O'MJ'U I lilt ljOJTWAilli I'OR 01\UANJI.INO, !fiUIUNO. SIIAIUNO. ANU VIUWINO
IMAillt'!, SOLI) AS A IIJ!II'J'URI! 01' <:OMI>t!ll'li'l AND IIANI)III/1.1) MOIIIII llltliiAI
I!I.E<.''JRONJ<.: (.'OMI'KJSI!IJOF MOllii.J! l'liONI!.'l, IJIOIJAI.AUJ)JOI\NU VIJ)J!Q
1'1.1\YI!RS, IIANOlllll.D COMI'UTr!RS, PI IIOONAI . DIOI'Ii\1 ASiil!lli\N l'!i, ANill'l.l:c.
I KONIC l'EKSONAL OJWANIZI
1
RS,JN CI.ASS 9 (US 11, 23,ll, )(;ANI) 1HJ
US!! 6-292007: IN COMMI'.RCP 6-2920117
OWNLR 01
1
U.S. RI!O NC. 3,4'10,983
IIlli CI)WI<(N) YEI.LOW Jji)Jl' tiKill! N. UROWN Ul.At'K tlltAY ANU WIU llllS/AJU!
C'l.AJMUD AS A fi:>A11JJU: 01111 J[ MARK
1111. MARK CONS ISIS 01
1
A tlllAY, WIIIJII, AND III,OII IU'(' II\ NUl 1
1
Wll II KOIINtWII
CORNT:RS f)flJIICTINCIA IN1111iCOI,OilSC11U!BN, Ylii.IAIW, IIROWN,
ULAI.:K. WIUJI:ANlHiRAY
JAMES MACJIARl.ANH, UXAMININCI/\ 1 J'()flNI!Y
Ex. 17
Reg. No. 3,889,685
APPLE INC. (CALIFORNIA COI(l'ORAflON)
I INFJNJ I 1: LOOP
Registered Dec. 14,2010 Clll'Fl<TINO, cA '11014
Int. Cl.: 9
TRADEMARK
PRINCIPAL REGISTER
JIOJ{: t:Otvl1
1
(J'l.HR l:'OH. lJSI.:R SV1-r!"J:M Sl'l"J'IN<lS/INI l l'll:I:J:J.:R-
l.NCI;S Sf)[_ I) AS A l;l.AllJI?.F OF COMPIJTI:RS fiND lJAND-ll.ELD DIUITI\L
J)JNJCES COMPRISI:D OF MOHJLJ! Dlt!UAL i\UDlU AND Yll>h") l'l.JWHRS,
C()l\,jJ
1
,JTnHS, J)JUI'IA!. AND T\J.I':CTRONJ{'
OIHlANJ:t.I:RS, IN CI,,,S, 21, 23, /..(>, )(1 /\ND
01
1
1 l S Rl:u. NOS .. M70,I)8111.Nl>
THE COLOR(::;) UH.J\Y, WI U i\ND HL/\CK IS/ARI' Cl.AIMELl .1\S 1\ FEAJURF
OJ. l'JJl: MARK
I JILMARK CONSHWS 01' I'ARTJ.AL I!YIA(lLSOI TJ !I{J:J:(n<:A.RS S) JOWN IN (lR!\ Y, Wlllll:
ANIJ SILVI'Jt ON A IIACKOR01JND 01' Ol<AY W1T1J l.ll.ACK ])()18 ALL CON'IAINii])
'WI I J liN A I{Ji.C I.AN\lllLAl< (Jl{I!Y ANI) Will l I\ J'lo!.AMJ. WIT! l RO!JNJlld l CORNJ:_tu.;
FRI D CARL. l::X!.MINING lll'TORNIY
Ex. 18
Reg. No. 3,886, 169
Al'PLlliNC. (CALIFORNIA CORI'PRi\'llON)
I I NHN11'1
1
I,CX)I>
Rcgistccd Dec. 7, 20 I 0 CUI'WIINo. cA
Jnt. Cl.: 9 FOil COMJ)lJ'I1'R SOJo1iVAJUl JIOR COMI'OSIN<l, IU'AI>IN<J, SI'ARCIIINU, SIIAIUNO
ANIHIYNI 'JNU 'I l:X I' l'llli.'l.IN ')(US l'I.S 21.l1. 26, lltANl> lij)
TIV\ OV.MARK
PRI NCIPAL REGISTER OWNllR 01 US llliO NO 3.470,983
nm COI.OR(S)YI!LI.OW,DitOWN AND GRAY IS/AIU CI.AIMI'l)ASA I'I!Al1JRI
1
01' '1111.
MAIU<
11 IE MARK CONSIS'I S 01 A RECTANOI.I Willi ROUNI)Jof) t'OitNl,lt.'i l>l!l'IC'I1NO A
Sl YL!Zhl) 01' A J'A<Jii Oil Nctl HIIOOK I'AI'I II 1111: ('ol,OR Yl'l.l,OW
Al'PI!ARS IN 'Ill! NOilillOOK PIII'I!R. Till! C:OI.Oii IIIIOWN AI'I'I!AIIS AI 1111! '101'
AUOVI! Till! NOillUOOK I'AI'I'R ANU IN '1111 Vl' lniCAL l.INUI ON I Ill! 1 SIOil Of'
I In Nm'P.HOOK PIIPP.R. ANI)TI rr. COLOR OR/\ Y 1\Jli'I!/\RS IN'I1 Ill I IORI/.ON II\ I I.INP.S
/\t'I<OSS ' I Ill! NO rL!IJUOK
!UIR NO liS-1119,1%,111.1\D 421211111
IUUM, EXAMININU A'rt'llltNI(Y
Ex. 19
Reg. No. 3,886, 197
1\1'1'1.1! INC. (CI\t.IFORNIA C'ORJ'ORJ\'nON)
1 llX)J>
Ucc. 7, 2010 CUI'I'tmNo.cA 95014
Int. Cl.: 9 ttoR coM!'tiTP.R soFT\\'AR!l R>R coN lAC 1 fNI oRMAtiON MANAUHMEN IWWAS
PRINCIPAL REGISTER
1\ HOI\TURilOFCOMI'tln:JU!I\NDIII\NDIIIll D MOilll f1 Dl<llll\1, H.I\CTRONIC r>l!VICt:s
CllMPRillrOOF Mf>Jlll P.PIIoNJl"'\ Ol(illl\1 AIIT>IOI\NI> VIIWO l'l.i\ YLitS. lli\Nl>llliLP
COMl'IIrERS, PI!RSONI\l, DICII1AI. AND I.I.I:C""ffi()NIC I'I'RSONI\1 , OR
01\NI)'J,RS,INCI.I\SS>I(US n.S 21,ll,ll.llii\N1>1M)
llSI! 6-f>l.21J()Q, IN COMMt:RCl! (lo f92009
OWNI H 01' U.S. RI!G. NCl'\ 3,462,199, ,,462,224,1\NI> 1,4711,'/M,
I Ill t'()LUI<(I:i) !JROWN, WJlrJ IJ 1\NU ORI\Y lll/1\lti' CLAIMHD /\:! 1\ t'IJ/\1 Ulill OF lilt
MARK
n m MI\RKCONSIST!H,II'A BROWN IU!" I'IINill.I
1
WI II II{(lllNilt:J)('OI<NI'IlS I)JII'It'"I1Ntl
A STYI.I/,1(1) W!Rfi I10\IND BOOK WITII'Il IH SII ,IIOIJETI h 01
1
1\ MAN IN1111! MII)I)J,R
rill; WlltH BINIJIN(i

IN Wlll11! ANI) 01!1\Y. l'llfiRh ARE IIIIOWN !MIS ON

Till\ RIOIIT OJ111 m llOOK Wl'I1 1'111P.t.rrm tiS "1\ll(:llHfi" IN <liMY A Nil WI IIIII.
lil!li NO KSOI'>,KOV, JIILHil
Mi\CI'ARLANIJ, fiX/\MlNINO 1\'JTORNI!Y
Ex.20
Latest Stotus Info
l'ogc I of"3
This page wns gcncmtcd by the TAlUl. system on 20 II 04 15 12.59.061 I
Scrinl Number: 85041463 A;;signmcnt lnform!!t jon fwd)mnrk ()ocumcnt Rctneval
Number: (NOT A VAJLABLE)
Mnrk
Stnndurd Character clnim: No
Current Status: Rcv1cw priur It) puhlu,:tttiun completed,
l>atc of Stntus: 20 I 1 03 16
Filing Dntc: 20 I 0-05- 18
The Inrormntion will bc/wus published in the OffiCial Gazett e on 20 11 -04- 19
Trnnsformcd intu 11 Nntional Application: No
Rcglstrntion l>ntc: (DATE NOT AVAILABLE)
Register: Principal
Lsnv Orti ce Assigned: I A W OII ICI IOJ
Attoml.ly Assigned:
RI I IM ANDREW
Current Locntinu: 650 l'ublicotion And Issue Section
Date In Locntion: 2011-03- 1 (>
LAST AIPl..lCANT(S)/OWNER(S) OF
I. Appl e Inc.
Address:
Ap!liC Inc.
I lnfini.c l.oop
scriul&.cnlry 85041463 4/ 15/2011
l,al<.lsl Stutus Info
C up<.lrtinu. C 1\ 950 14
United Statclo
Legal Enti ty Type: Corporati on
State or Country oflncoqwration: Califomio
Ioternutionul Class: 009
Class Status: /\ctiv<.l
GOODS AND/OR SEH. VJC.I!:S
Page 2 of3
Computer !.Oilwure li1r u-.c in browsing. n:vkwing. 'inmpling. playing, purchasing. and
downlou(ling pre-recorded uudi o and video content
Dnsis: I (u)
First Usc Ontc: 20011 0600
Fll'st tlsc In Commerce 011t c: 2008-06-00
ADDITIONAL INFORMATI,ON
Color(11) Claimed : ' I he color(s) purrlc, white and light purple is/nrc cluimcd us u fbuturc of the mork.
Description of Mnrk: I he murl-. of a rcctunglc with rounded corner'> depicting u
musical note. 1\ circle -.urrnunds the musiculnotc. The color purple uppcur., in the lower portion of the
rcctun!llc; the color white appcnrs in the note und in the ci rcle .,urrounding the mu!> iculnotc: und
the color li ght purple appear'> in the upper J'lOrtion of the recumglc.
Design Senrch Codc(s):
24.17. 13 Clef symbol (musical): Mu-. ical note .. : Musical ::.tuff; Musil.:u l symbob. including treble and
has!> clef' symhols. shurp nnd nut symbols and notes
26.01.21 that nrc totally or partially shaded.
26. 11.21 - Rcctunglcs thut urc completely or partially shaded
Registration Nnmbcr(s):
2935038
1470983
MADRID PROTOCOL INFOil MATION
lJSPTO Reference Number : /\0022114
lntcmalional Rcgistrnlion Number: I 057<.>5!1
lnlcrllallonal Registration Date: 2() I 0- 1 12
Original Filing Olltc with USPTO: 20 I 0- 1 1 12
lntcnaallonul Rcgistrution Shlh111: Application For IR Rcgi::.tcrcd fly IB
Oate ort Registration Status: 20 I 0- 12-09
lntcanationnl Rcgistrntion ltcncwal Date: 2020 11 12
l rrcguhtrlty Reply by J>ntc: (DATE NOT AVAILABLE)
Mudrid lllstory:
12-0<>-20 I 0 21:01 1 Ftlr IR Ill
11 - 13-20 I 0 - 21:02: n - 1R Certified /\nd Sent To 18
11 - 13-20 I 0 - I: I 9 New Application For IR Received
85041463
4/15/2011
Latest Status Info
Page 3 of3
111320 I () - OS:51: 19 - Automoticully Certifi ed
PROSECUTION HISTORY
NOTE: To view uny document referenced below, click on the link to "Trndcmnrk Document
Rctrievul" shown ncar the top of this pugc.
201 1-03-30 - Notice of publicotion
20 11 -0J- 16 - l.aw Otlicc P1hliclition Review Completed
20 I 1-0J- 1 ( Assigned l'o I IE
2011 -02-27 - Approved lor Pub - Principul Register (lnitinl exam)
20 11 02-03 - 'l'cus/Enuli I Entered
20 11-02-03- Communication received nom applicant
20 - rt :AS Rcsptmsc to 011icc Al-Li on Rccch ell
20 I 0-0K-03 - Non- final act inn moiled
20 I 0-08-03 Non-1 inul Acti on Written
20 I 0-08-03 - Assigned l'o Lxamincr
20 I 0-05-25 - Notice or Design Seun:h Code Mailed
20 I 005-22 - New Appli cation Ol1icc Sup!)l icd Dat {l Fntcrccl l n r rum
20 I 0-05-2 1 - New A pplicmion In Trnm
Allorncy of Record
l.isn G. Widup
Correspondent
I.JS/\ Cl. WID\ JP
INC.
INFORMATION
I INFINI'I F I .OC>P tl MSHM
ClJ PFRTINO. CA 95014-2083
http://tnrr.uspto.gov/scrvl ct/ tarr?regscr scrial&ent ry'RS04 1463
4/ 15120 II
Ex.21
Int. Cl.: 9
Prior U.S. Cis.: 21, 23, 26, 36 and 38
United States Patent and Trademark Office
TRADEMARK
PRINCIPAL REGISTER
Reg. No. 2,935,038
Regb;tered Mar, 22, 2005
COMf'lJ7ER. INC (("ALH"
1
0HNlA FlRST USE lN COMMERCE
PO RATION)
I INfiNITU LOOP
CUPHRTINO, CA 95014 THE MARK CONSISTS OF A DESIGN OF A
POR: COMPUTER SOFTWARE FOR USE IN
PURCHASING, AUTHORING, DOWNLOADING,
COMPACT DISC WITH TWO MUSICAL NOTES.
TRANSMITTING, RECElV!I\0, EX- SER. NO. 7B.JE2,E67, PILHJJ.Jl-2004.
TRACTING. f:NCOJ)]NG. DOCODINO, PLA \'ING,
STORTNO AND ORGANJZil\"G AUDIO DATA . IN
CLASS 9 (U.S. CLS. ;.':!, 23, 26. AND 38). HENRYS. ZAK. EXA.'MIN1NG ATTORI\EY