3

279683B2

a2 United States Patent (10) Patent No.: US 10,279,683 B2

Torii 45) Date of Patent: May 7, 2019
(54) VEHICLE DISPLAY SYSTEM AND METHOD 2009/00507; GO7C 5/006; GO7C 5/08;
OF CONTROLLING VEHICLE DISPLAY GO7C 5/0808; GO7C 5/085; GO7TC
SYSTEM 9/00309; GOSG 1/202
USPC ... 3407438, 901-903, 425.5, 426.1, 426.34,
(71) Applicant: SUBARU CORPORATION, Tokyo 340/426.28, 905, 427, 426.22, 435-436,
340/455, 463, 468, 525, 534-535, 539.22
ap) See application file for complete search history.
(72) Inventor: Takeshi Torii, Tokyo (JP) (56) References Cited

73) Assignee: Subaru Corporation, Tokyo (IP) U.S. PATENT DOCUMENTS

) Notice: Subject to any disclaimer, the term of this 2003/0098211 Al* 52003 Saito B60G 17/0185
patent is extended or adjusted under 35 19173
US.C. 154(b) by 0 days. 2007/0038344 Al* 212007 Oota B6OR 25/2009
701/49
@n Appl. No.: 15/898,269 (Continued)
2 Filed: Feb. 16, 2018 FOREIGN PATENT DOCUMENTS
(65) Prior Publication Data P
P
2016-136199
2016-136199 A
* 12015
72016
US 2018/0264945 Al Sep. 20, 2018 (Continued)
30) Foreign Application Priority Data OTHER PUBLICATIONS
Mar. 15,2017 (IP) worrrsrssnnnns 2017-050349 Japanese Office Action issued in corresponding Japanese Patent
Application No. 2017-050349, dated Sep. 11, 2018, with English
(51) Int. CL Translation.
B60Q 1/00 (2006.01)
B6OK 35/00 (2006.01) Primary Examiner — Daniel Previl
(Continued) (74) Attorney, Agent, or Firm — McDermott Will &
(52) US.Cl Emery LLP
CcpC .. B60K 35/00 (2013.01); GO2B 27/01 7 ABSTRACT
(2013.01); GO2B 27/0101 (2013.01); GO6K
900302 (2013.01); GO6K 9/00369 (2013.01): A vehicle display system includes a display device, a deter-
GOGK 9/00791 (2013.01); GOSG 1/0962 ‘miner, and a display controller. The display device provides
(2013.01); GOSG 1/096716 (2013.01); GOSG a display on a window of a vehicle. The determiner deter-
1/096775 (2013.01); GO8G /166 (2013.01); ‘mines whether a driver of the vehicle comes close to the
(Continued) vehicle. The display controller controls the display device to
provide the display for an outside of the vehicle, when the
(58) Field of Classification Search determiner determines that the driver of the vehicle comes
CPC . GOTC 5/008; GOTC 5/00; GOTC 5/02; GOTC close to the vehicle.
5/06; GO7C 5/0816; GOTC 9/00111;
GOTC 1/10; GO7C 1/24; GOTC 15 Claims, 11 Drawing Sheets

-
o
N
|[EWIROWEN T ORATION
500

0
200
DRIVER SENSOR JOUIRING WIT
]

-
WIRELESS KEY
I
 US 10,279,683 B2
Page 2

(51) Int. CL
GOGK 9/00 (2006.01)
GO8G V16 (2006.01)
G02B 27/01 (2006.01)
GOSG 1/0962 (2006.01)
GOSG 1/0967 (2006.01)
(52) US.CL
SIS B6OK 2350/1096 (2013.01); B6OK
2350/965 (2013.01); B6OR 2300205
(2013.01); GO2B 2027/0138 (2013.01); G02B
2027/0141 (2013.01); GOSK 9/00532
(2013.01)
(56) References Cited
U.S. PATENT DOCUMENTS

2007/0040705 Al* 2/2007 Yoshioka GO1C 21/00

340/988
2010/0253594 AL* 10/2010 Szezerba GO1S 13/723
3457
2011/0083075 AL* 42011 MacNeille B60K 37/06
715/728
2012/0268264 AL* 10/2012 Son GOSB 13/1427
340/457
2014/0309806 Al* 10/2014 Ricci B60Q 1100
701/1
2016/0066127 Al* 3/2016 Choi HO4W 4/008
455/41.2
2016/0189435 Al* 6/2016 Beaurepaire GOSD 1/021
705/13

FOREIGN PATENT DOCUMENTS

P 2016136199 * 72016
P 2017-177883 A 1012017
* cited by examiner
U.S. Patent May 7, 2019 Sheet 1 of 11 US 10,279,683 B2

400
100 / 500
VERTCLE-EXTERIOR
SENSOR CONTROLLER HUD DEVICE
402
200 ENVIRONMENT [NFORMATION 600
DRIVER SENSOR ACQUIRING UNIT SPEAKER
404
300 DRIVER INFORMATION 800
SEATING SENSOR ACQUIRING UNIT ANTENNA
407
350 ENVIRONMENT STATE
DOOR SENSOR DETERMINER
412
700 DRIVER STATE 1100
COMMUNIGATOR DETERHINER WIRELESS KEY
414
HUD
<;:i:::::;5_“1300 CONTROLLER
416
1200 ABNORMALITY
CLOUD SERVER DETERHINER
418
DRIVER BOARDING
DETERMINER
1000

F1G.1
U.S. Patent May 7, 2019 Sheet 2 of 11 US 10,279,683 B2

510

F1G.3
U.S. Patent May 7, 2019 Sheet 3 of 11 US 10,279,683 B2

= =] [O] &
TIRE PRESSURE 1S LOWERED

~| Z 0] [2
@3A3woJ eI 3auee3saq IAIT

F1G.5

= (& [0
TIRE PRESSURE IS LOWERED
US 10,279,683 B2

L7914
SHIN NOILYRYOINT O144v¥L
ol ol
0S 0§
Sheet 4 of 11
May 7, 2019

1SY03404 YIHLYIM AVQHIYIE 40 INJWIONNONNY

AVOS¥NHL 'S¢ "bnv
g00Vr
iAVQH1Y1g AddVH
U.S. Patent

HLIM ANOTD
U.S. Patent May 7, 2019 Sheet 5 of 11 US 10,279,683 B2

D~ams 7]
> AN
A ]
U.S. Patent May 7, 2019 Sheet 6 of 11 US 10,279,683 B2
U.S. Patent May 7, 2019 Sheet 7 of 11 US 10,279,683 B2

1914
U.S. Patent May 7, 2019 Sheet 8 of 11 US 10,279,683 B2

100 100 100 100

\ I‘f
] l\
) \

10 50 800 800 800 800 800

U.S. Patent May 7, 2019 Sheet 9 of 11 US 10,279,683 B2

70

100 100 100 100

\n M) )
A=
"[’ I'Y(
U.S. Patent May 7, 2019 Sheet 10 of 11 US 10,279,683 B2

HAPPINESS SURPRISE

NO EMOTION

ANGER SADNESS

F1G. 16
U.S. Patent May 7, 2019 Sheet 11 of 11 US 10,279,683 B2

START

S10
DRIVER
GOMES CLOSE TO VEHICLE NO

YES
$12

S 26
W_LOCATED NEAI
oo

NFORMATION FO
E

DRIVER
ENTERS VEHICLE
?

CLE, AR DISPLAY AFTER

PRESET PERIOD

END
F1G. 17
 US 10,279,683 B2
1 2
VEHICLE DISPLAY SYSTEM AND METHOD to determine whether a driver of the vehicle comes close to
OF CONTROLLING VEHICLE DISPLAY the vehicle, and control the display device to provide the
SYSTEM display for an outside of the vehicle, when the driver is
determined as coming close to the vehicle.
CROSS-REFERENCE TO RELATED H An aspect of the technology provides a vehicle display
APPLICATIONS system that includes: a display device configured to provide
a display on a window of a vehicle; and circuitry configured
The present application claims priority from Japanese to determine whether a driver boards the vehicle, and control
Patent Application No. 2017-050349 filed on Mar. 15, 2017, the display device to provide a display for an inside of the
the entire contents of which are hereby incorporated by vehicle when the driver is determined as boarding the
reference. vehicle, and to provide the display for an outside of the
vehicle when the driver is determined as exiting the vehicle.
BACKGROUND
BRIEF DESCRIPTION OF THE DRAWINGS
The technology relates to a vehicle display system and a
method of controlling the vehicle display system. FIG. 1 schematically illustrates an example of a configu-
Japanese Unexamined Patent Application Publication (JP-
A) No. 2016-136199 discloses an exemplary technique ration of a vehicle display system according to one imple-
associated with displaying of a head-up display (HUD). In mentation of the technology.
this technique, when an accessory (ACC) power supply of a FIG. 2 schematically illustrates an example of a head-up
vehicle is on, the HUD provides a regular-mode display display (HUD) device that includes a self-luminous inter-
within a driver’s front field of view. layer film.
In the above technique, when the ACC power supply is FIG. 3 schematically illustrates an example of a state
off, the HUD provides a security-mode display on a wind- where a driver sensor captures an image of a driver of a
shield of the vehicle so that a person outside the vehicle is vehicle.
able to view this display. FIG. 4 schematically illustrates an example of a display
that the HUD device provides on a windshield of the vehicle.
SUMMARY FIG. 5 schematically illustrates another example of the
display that the HUD device provides on the windshield of
An aspect of the technology provides a vehicle display the vehicle.
system that includes: a display device configured to provide FIG. 6 schematically illustrates further another example
a display on a window of a vehicle; a determiner configured of the display that the HUD device provides on the wind-
to determine whether a driver of the vehicle comes close to
shield of the vehicle.
the vehicle; and a display controller configured to control the FIG. 7 schematically illustrates examples of a display to
display device to provide the display for an outside of the be provided in a case where no abnormalities occur in the
vehicle, when the determiner determines that the driver of
vehicle when the driver boards the vehicle.
the vehicle comes close to the vehicle.
An aspect of the technology provides a vehicle display FIG. 8 schematically illustrates an example of a display to
system that includes: a display device configured to provide be provided for an inside of the vehicle when an entry ofthe
a display on a window of a vehicle; a driver boarding driver in the vehicle is detected while the display of FIG. 7
determiner configured to determine whether a driver boards is provided.
the vehicle; and a display controller configured to control the FIG. 9 schematically illustrates an example of a display
display device to provide a display for an inside of the on the windshield of the vehicle which is to be provided for
vehicle when the driver boarding determiner determines that the driver inside the vehicle when the driver turns off an
the driver boards the vehicle, and to provide the display for ignition switch.
an outside of the vehicle when the driver boarding deter- FIG. 10 schematically illustrates an example of a manner
miner determines that the driver exits the vehicle. in which a display for the inside of the vehicle is switched
An aspect of the technology provides a method of con- to a display for the outside of the vehicle when an exit of the
trolling a vehicle display system. The method includes: driver from the vehicle is detected while the display of FIG:
determining whether a driver of a vehicle comes close to the 5 9 is provided.
vehicle, in which the vehicle includes a display device that FIG. 11 schematically illustrates examples in which a
provides a display on a window of the vehicle; and control- window on which a display is to be provided is switched to
ling the display device to provide the display for an outside another window, depending on a position of the driver
of the vehicle, when the driver is determined as coming moving around the vehicle.
close to the vehicle. FIG. 12 schematically illustrates an example of a method
An aspect of the technology provides a method of con- of detecting the driver present outside the vehicle.
trolling a vehicle display system. The method includes: FIG. 13 schematically illustrates another example of the
determining whether a driver boards a vehicle, in which the method of detecting the driver present outside the vehicle.
vehicle includes a display device that provides a display on FIG. 14 schematically illustrates an example of a manner
a window of the vehicle; and controlling the display device in which a driver state determiner determines condition of
to provide a display for an inside of the vehicle when the the driver.
driver is determined as boarding the vehicle, and to provide FIG. 15 schematically illustrates an example of a method
the display for an outside of the vehicle when the driver is in which the driver state determiner determines a facial
determined as exiting the vehicle. expression of the driver.
An aspect of the technology provides a vehicle display s FIG. 16 schematically illustrates an example of a warning
system that includes: a display device configured to provide display to be provided when an abnormality occurs in the
a display on a window of a vehicle; and circuitry configured driver.
 US 10,279,683 B2
3 4
FIG. 17 is a flowchart of an example of a process sensor 100 may be a stereo camera that includes a pair of
according to one implementation. right and left cameras each having an imaging device such
as, but not limited to, a charge-coupled device (CCD) sensor
DETAILED DESCRIPTION or a complementary metal-oxide semiconductor (CMOS)
sensor. Then, the stereo camera may capture an image of an
Atechnique disclosed in JP-A No. 2016-136199 is utterly environment outside the vehicle and transmit information on
silent as to allow for displaying useful information for a the captured image to the controller 400. In an example
driver when the driver comes close to a vehicle in order to implementation, the stereo camera may be a color camera
drive the vehicle. disposed on an upper portion of a windshield of the vehicle
It is desirable to provide a vehicle display system and a and be able to acquire color information.
method of controlling the vehicle display system that both
make it possible to display useful information for a driver The HUD device 500 may be a display device that
when the driver comes close to a vehicle in order to drive the
displays information directly in a human visual field. The
vehicle. HUD device 500 may display a real image on a window such
In the following, some non-limiting implementations of as, but not limited to, the windshield or a rear window of the
the technology are described in detail with reference to the vehicle. Although a known HUD device typically displays a
accompanying drawings. Note that the following description virtual image, the HUD device 500 in this implementation
is directed to illustrative examples of the disclosure and not ‘may be a display device that displays a real image. The HUD
to be construed as limiting the technology. Factors includ- device 500 may display a real image at a viewing angle of
ing, without limitation, numerical values, shapes, materials, about 360 degrees, thereby allowing both persons inside and
components, positions of the components, and how the outside the vehicle to view the image. In an alternative
components are coupled to each other are illustrative only implementation, however, the HUD device 500 may display
and not to be construed as limiting the technology. Further, a virtual image.
elements in the following example implementations which In one specific but non-limiting implementation, the HUD
are not recited in a most-generic independent claim of the device 500 may be a device that includes a self-luminous
disclosure are optional and may be provided on an as-needed interlayer film 510, as illustrated in FIG. 2. In such an
basis. The drawings are schematic and are not intended to be implementation, the self-luminous interlayer film 510 may
drawn to scale. Throughout the present specification and the be disposed between two sheets of glass 520 in the window
drawings, elements having substantially the same function of the vehicle such as, but not limited to, the windshield or
and configuration are denoted with the same reference
numerals to avoid any redundant description.
the rear window. The self-luminous interlayer film 510 may
contain a luminescence material. When a portion of the
A technique disclosed in JP-A No. 2016-136199 is utterly
self-luminous interlayer film 510 is irradiated with laser
silent as to allow for displaying useful information for a
driver when the driver comes close to a vehicle in order to
light from a projector 530 mounted in the vehicle, the
drive the vehicle. In addition, this technique is also utterly irradiated portion may emit light, displaying objects, includ-
silent as to allow for changing a display on the vehicle, ing characters and images. The objects may have visibility
depending on a position of the driver relative to the vehicle. at all angles and may be thus viewable not only from the
Therefore, there may arise a concern that, when the driver driver seat but also from the other seats as well as any
moves away from the display outside the vehicle, he/she has location outside the vehicle. In an alternative implementa-
difficulty visually recognize a content of the display. tion, the HUD device 500 may have a configuration in which
One implementation of the technology provides a vehicle a self-luminous device is disposed at the window of the
display system and a method of controlling the vehicle vehicle. In such an alternative implementation, the HUD
display system that both make it possible to display useful device 500 may have a transparent screen that contains an
information for the driver when the driver comes close to the organic electroluminescence (EL) element, a transmissive
vehicle in order to drive the vehicle. 4 liquid crystal device, or any other member that allows for
FIG. 1 schematically illustrates an example of a configu- transmissive displaying. In a further alternative implemen-
ration of a vehicle display sysiem 1000 according to one tation, any device other than the HUD device 500 may be
implementation of the technology. The vehicle display sys- used as the display device. In such an alternative implemen-
tem 1000 may be basically mounted in a vehicle. The vehicle tation, a large-sized liquid crystal display device, a light-
may be, for example but not limited to, an automobile. s emitting diode (LED) display device, or any display device
Referring to FIG. 1, the vehicle display system 1000 may provided on a member such as, but not limited to, an
include a vehicle-exterior sensor 100, a driver sensor 200, a instrument panel may be used as the display device. In the
seating sensor 300, a door sensor 350, a controller 400, a following, a description is given of one implementation in
HUD device 500, a speaker 600, a communicator 700, and which the HUD device 500 provides a display on the
an antenna 800. window, such as the windshield or the rear window, of the
Provided around the vehicle display system 1000 vehicle. In one implementation, the HUD device 500 may
mounted in the vehicle may be a wireless key 1100 and a serve as a “display device”. The display device according to
cloud server 1200. Both of the wireless key 1100 and the any implementation of the technology, however, is not
cloud server 1200 may be able to communicate with the limited to the HUD device 500. The term “HUD device” or
vehicle display system 1000. 6 “display device” according to any implementation of the
The vehicle-exterior sensor 100 may be a stereo camera, technology encompasses any and all display devices pro-
a monocular camera, millimeter wave radar, an infrared vided inside and/or outside a vehicle. In an example imple-
sensor, or any other sensing device. The vehicle-exterior mentation, the display device may be provided at any
sensor 100 may measure a parameter such as, but not limited external site of the vehicle other than the window, such as a
to, a position and speed of an object. Non-limiting examples body of the vehicle. In an example implementation, the
of the object may include a vehicle and a person around the display device may be provided at any internal site of the
vehicle. In an example implementation, the vehicle-exterior vehicle other than the window, such as the instrument panel
 US 10,279,683 B2
5 6
and a seat. The display devices disposed inside and outside distance information subjected to the grouping process with
the vehicle may be provided integrally with or separately preset three-dimensional data such as, but not limited to,
from each other. three-dimensional object data. By performing the processes
‘The driver sensor 200 may be implemented by a camera, in this manner, the controller 400 may be able o recognize
a line-of-sight sensor, a motion sensor, or any other sensing various entities, including a person, another vehicle, a stop
device. This driver sensor 200 may measure movements of sign, a stop line, and an electronic toll collection (ETC) gate.
the driver’s arms and head, a direction of his/her line of On the basis of the information on the distance to a factor
sight, and any other parameters related to the driver. In an generated by the principle of triangulation, the environment
example implementation where the driver sensor 200 is a information acquiring unit 402 may also calculate a varia-
camera, the driver sensor 200 may acquire the movements of tion in the distance to the factor and/or a speed relative to the
the arms and head, the direction of the line of sight, and any factor. Non-limiting examples of the factor may include a
other parameters by subjecting an image captured by the person and another vehicle. The variation in the distance
camera to image processing. In another example implemen- may be determined by integrating distances to the factor in
tation where the driver sensor 200 is a line-of-sight sensor, frame images. The frame images may be acquired at regular
the driver sensor 200 may detect the line of sight by using time intervals. The relative speed may be determined by
a corneal reflection method or any other similar method. The dividing the distances acquired at regular time intervals by
seating sensors 300 may be provided in the respective seats the unit time.
of the vehicle, and each of the seating sensors 300 may In the above manner, the environment information acquir-
determine whether a person sits on a corresponding seat. The ing unit 402 may acquire the image information regarding an
door sensors 350 may sense the opening and closing of the environment outside the vehicle, from the vehicle-exterior
respective doors. The speakers 600 may emit a warning sensors 100. In addition, the environment information
sound toward the inside and outside of the vehicle when the acquiring unit 402 may analyze the acquired image infor-
HUD device 500 displays a warning to persons inside and mation through an image analyzing process. As a result of
outside of the vehicle. analyzing the image information, the environment informa-
The controller 400 may control the displaying performed tion acquiring unit 402 may acquire environment informa-
by the HUD device 500 on the basis of the information tion regarding the environment outside the vehicle.
detected by the vehicle-exterior sensors 100, the driver The antenna 800 may communicate with the wireless key
sensor 200, and other sensors. The controller 400 may 1100. The wireless key 1100 may transmit a radio wave to
include an environment information acquiring unit 402, a the antenna 800. When the antenna 800 receives the radio
driver information acquiring unit 404, an environment state wave from the wireless key 1100, the vehicle may lock or
determiner 407, a driver state determiner 412, an HUD unlock the doors. Furthermore, when the antenna 800
controller 414, an abnormality determiner 416, and a driver receives the radio wave from the wireless key 1100, the
boarding determiner 418. In one implementation, the HUD environment state determiner 407 may determine a position
controller 414 may serve as a “display controller”. In one of the driver outside the vehicle and a distance from the
implementation, the environment state determiner 407 may vehicle to the driver, on the basis of an intensity of the
serve as a “determiner”. Each of the components of the received radio wave. Details of this operation are described
controller 400 may be implemented in hardware or in later.
software. In an example implementation where one compo- The communicator 700 may communicate with any exter-
nent of the controller 400 is implemented in hardware, this nal device outside the vehicle. The cloud server 1200 may be
component may be circuitry. In another example implemen- coupled to the communicator 700 over a network 1300 such
tation where one component of the controller 400 is imple- as, but not limited to, the Internet. The cloud server 1200
mented in software, this component may be a program that ‘may transmit general information to the communicator 700.
causes a central processing unit (CPU) or other processor to Non-limiting examples of the general information may
perform its function. include a weather forecast, traffic information, news, and
The environment information acquiring unit 402 may routing information. The information transmitted to the
acquire information on a distance to a target, namely, a ‘communicator 700 may be used for a regular display that the
subject in this case. This information may be called distance HUD controller 414 causes the HUD device 500 to provide.
information. In an example implementation, the environ- Details of the regular display are described later.
ment information acquiring unit 402 may generate the FIG. 3 schematically illustrates an example of a state
distance information, on the basis of an amount of shift s where the driver sensor 200 captures an image of the driver
between corresponding locations in a pair of right and left if the driver sensor 200 is implemented by a camera. As
stereo images and by means of principle of triangulation. illustrated in FIG. 3, as an example, the driver sensor 200
‘The pair of right and left stereo images may be captured by ‘may be mounted on an upper portion ofa steering column.
a set of right and left cameras of the stereo camera which In an example implementation where the driver sensor
constitutes each vehicle-exterior sensor 100 in one imple- 5 200 is a camera, the image captured by the driver sensor 200
mentation. Together with the acquisition of the distance may be supplied to the controller 400. Then, the driver
information, the environment information acquiring unit 402 information acquiring unit 404 may acquire information on
may also acquire information on a location of the subject a face and body of the driver from the received image
from the information on the images captured by the set of through an image processing technique such as, but not
right and left cameras of the stereo camera which constitutes limited to, an edge and face detection technique. The infor-
each vehicle-exterior sensor 100. This information may be ‘mation on the face and body may be regions of the face and
called image information. Further, the environment infor- body, for example.
mation acquiring unit 402 may perform a known grouping The environment state determiner 407 in the controller
process on the distance information generated by the prin- 400 may determine an environment state outside the vehicle
ciple of triangulation. Then, the environment information on the basis of the environment information acquired by the
acquiring unit 402 may obtain data on a three-dimensional environment information acquiring unit 402. Further, the
object, a lane line, and other objects by comparing the environment state determiner 407 may set targets to persons
 US 10,279,683 B2
7 8
outside the vehicle and may determine conditions and posi- FIGS. 5 and 6 schematically illustrate examples in which
tions of the persons, and individual distances between the the door sensor 350 detects an entry of the driver in the
vehicle and the persons and between the persons. Further- vehicle 10 while the HUD device 500 provides the display
more, the environment state determiner 407 may determine of FIG. 4, and a display for the inside of the vehicle 10 is
positions of the targets outside the vehicle and distances H performed accordingly. FIG. 5 illustrates the display as seen
between the targets and vehicle, on the basis of the intensity from the outside of the vehicle 10. FIG. 6 illustrates the
of the radio wave that the antenna 800 has received from the display as seen from the inside of the vehicle 10. In
wireless key 1100. examples of FIGS. 5 and 6. the displays on the windshield
The driver state determiner 412 may determine condition 50 may be equivalent to an inversion of the display of FIG.
of the driver on the basis of the environment information 4. As in these examples, when the driver boards the vehicle
acquired by the environment information acquiring unit 402. 10, the HUD device 500 may invert the display of FIG. 4,
An example of the determination method made by the driver cause the display to blink, or may perform any other method
state determiner 412 is described later. that calls the driver’s attention, thereby enabling the driver
The driver boarding determiner 418 may determine to continuously recognize the warning in the display even
whether the driver boards the vehicle on the basis of the after he/she has entered the vehicle 10. In an alternative
information on the driver acquired by the driver information implementation, when the driver boards the vehicle 10, the
acquiring unit 404 and the information acquired by the door HUD device 500 may switch the display for the outside of
sensors 350. When the information acquired from the door the vehicle 10 to another display. For example, when the
sensors 350 indicates that at least one of the doors is opened driver boarding determiner 418 determines that the driver
and closed and the information acquired from the driver boards the vehicle 10, the HUD controller 414 may control
information acquiring unit 404 indicates that the driver is at the HUD device 500 to switch the display for the outside of
the driver seat, the driver boarding determiner 418 may the vehicle 10 to the display for the inside of the vehicle 10.
determine that the driver boards the vehicle. As described above, when an abnormality occurs in the
The HUD controller 414 may control the displaying vehicle 10, the HUD device 500 may display a warning
performed by the HUD device 500. In an example imple- toward the outside of the vehicle 10 before the driver enters
‘mentation, the HUD controller 414 may control the display- the vehicle 10. This configuration makes it possible to
ing of the HUD device 500 on the basis of the determination promptly inform the driver of the occurrence of the abnor-
results of the environment state determiner 407 and the mality. The example of FIG. 4 helps the driver to view the
driver state determiner 412. warning and check the air pressure before entering the
The abnormality determiner 416 may determine whether vehicle 10. As a result, the driver is able to reliably recognize
an abnormality occurs in the vehicle. In an example imple- the occurrence of the abnormality in the vehicle 10 at an
mentation, the abnormality determiner 416 may determine early stage before driving the vehicle 10. In other words, the
whether an abnormality occurs in the vehicle, on the basis of driver is able to acquire useful information for use in driving
various pieces of sensor information acquired over a con- the vehicle 10,
troller area network (CAN) mounted in the vehicle. Non- FIG. 7 schematically illustrates examples of a display that
limiting example of the abnormality in the vehicle deter- the HUD device 500 provides on the windshield 50 of the
mined by the abnormality determiner 416 may include vehicle 10. One example illustrated in FIG. 7 is based on the
lowering of an air pressure, an oil temperature, an oil level, premise that an abnormality or other issue does not occur in
and a battery voltage. the vehicle 10 upon boarding of the driver. As illustrated in
In one implementation, when the driver comes close to the FIG. 7. when an abnormality or other issue does not occur
vehicle in order to drive the vehicle, the HUD device 500 in the vehicle 10, the HUD controller 414 may control the
may provide a display on the window of the vehicle. FIGS HUD device 500 to provide the regular display. The regular
410 6 schematically illustrate examples of a display that the display may indicate that the vehicle 10 is in normal
HUD device 500 provides on a windshield 50 of a vehicle condifion. The regular display may also indicate general
10. These examples are based on the premise that an information that contains, for example but not limited to, an
abnormality or other issue occurs in the vehicle 10 upon announcement of a birthday, a photograph, a painting, a
boarding of the driver. In the example of FIG. 4, the HUD weather forecast, traffic information, routing information,
device 500 provides the display for the driver outside the news, or any other information in general.
vehicle 10, thereby giving a warning to the driver that a tire FIG. 8 schematically illustrates an example in which the
pressure is lowered. As an alternative example, instead of s door sensor 350 detects the entry of the driver in the vehicle
the warning that the tire pressure is lowered, the HUD 10 while the HUD device 500 provides the display of FIG.
device 500 may display other warning information, such as 7, and a display for the inside of the vehicle 10 is performed
information indicating that the oil temperature, the oil level, accordingly. In one example of FIG. 8, the HUD device 500
or the battery voltage is lowered. For example, when the may display routing information 60 on the windshield 50
environment state determiner 407 determines that the driver s Even after the driver has entered the vehicle 10, the HUD
comes close to the vehicle 10 and the abnormality deter- device 500 may continue to provide the display of FIG. 7. In
miner 416 determines that an abnormality occurs in the an altemative implementation, the HUD device 500 may
vehicle 10, the HUD controller 414 may control the dis- provide another display after the driver has entered the
playing of the HUD device 500 on the basis of the deter- vehicle 10. In an example implementation where the display
mination results of the environment state determiner 407 and of FIG. 7 is continued to be provided after the driver has
the abnormality determiner 416. In this way, the HUD entered the vehicle 10, the HUD device 500 may invert front
controller 414 may control a display as illustrated in FIG. 4 and back of the display.
In one specific but non-limiting implementation, the envi- FIG. 9 schematically illustrates an example of a display
ronment state determiner 407 may determine that the driver for the driver inside the vehicle 10 that the HUD device 500
comes close to the vehicle 10 when a distance between the provides on the windshield 50 of the vehicle 10 when the
driver and the vehicle 10 becomes equal to or less than a driver turns off an ignition switch (IG_SW). FIG. 10 sche-
predetermined threshold. ‘matically illustrates an example of a manner in which the
 US 10,279,683 B2
9 10
HUD device 500 switches the display for the inside of the of the vehicle 10 on the window located near the driver 70.
vehicle 10 to the display for the outside of the vehicle 10, In alternative implementation, the HUD device 500 may
where the door sensor 350 detects an exit of a driver 70 from provide a display for the outside of the vehicle 10 on the
the vehicle 10 while the HUD device 500 provides the window located nearest the driver 70.
display of FIG. 9. Tn one example of FIG. 10, the HUD FIG. 13 schematically illustrates an example of a method
device 500 may provide a display for the outside on a of detecting the driver 70 by means of the wireless key 1100.
window 52 located on the driver seat side. When the driver As illustrated in FIG. 13, the antennas 800 may be installed
boarding determiner 418 determines that the driver 70 exits in the front, rear, right, and left portions of the vehicle 10.
from the vehicle 10, the HUD controller 414 may control the These antennas 800 may receive the radio wave emitted
HUD device 500 to switch the display for the inside of the from the wireless key 1100. Then, the environment state
vehicle 10 to the display for the outside of the vehicle 10. By determiner 407 may determine whether an intensity of the
switching the displays in response to the exit of the driver 70 radio wave that the antennas 800 have received from the
from the vehicle 10, it is possible to continue sending a wireless key 1100 is equal to or higher than a predetermined
message to the driver 70. The HUD device 500 may clear the threshold. When the intensity of the received radio wave is
display for the outside of the vehicle 10 after a predeter- equal to or higher than the predetermined threshold, the
mined period has passed. environment state determiner 407 may determine that the
FIG. 11 schematically illustrates examples in which, when driver 70 is present near the vehicle 10. In response, the
the driver 70 moves around the vehicle 10, the HUD device HUD device 500 may provide a display for the outside of the
500 switches a window on which a display is to be provided vehicle 10.
to another window, depending on a position of the driver 70. FIG. 14 schematically illustrates an example of a manner
As in the left example in FIG. 11, when the driver 70 is in which the driver state determiner 412 determines condi-
present near the driver seat, the HUD device 500 may tion of the driver 70. As described above, the driver state
provide the display on a window 52 located on the driver determiner 412 may determine condition of the driver 70 on
seat side. As in the right example in FIG. 11, when the driver the basis of the environment information acquired by the
70 moves to the rear of the vehicle 10, the HUD device 500 environment information acquiring unit 402. One example
may provide the display on a rear window 54 of the vehicle illustrated in FIG. 14 is based on the premise that the driver
10. Likewise, when the driver 70 moves to the front of the 70 comes close to the vehicle 10 with a stagger along a
vehicle 10, the HUD device 500 may provide the display on trajectory T. The environment information acquiring unit
the windshield 50 of the vehicle 10. In an example case 402 may acquire positions of the driver 70 and distances
where the driver 70 comes close to the front of the vehicle between the driver 70 and the vehicle 10 by means of the
10 and then enters the vehicle 10 from a door located on the image recognition, thereby determining the trajectory T.
driver seat side, the HUD device 500 may first provide the Then, the driver state determiner 412 may determine con-
display on the windshield 50, and in turn may provide the dition, or a behavior, of the driver 70 on the basis of the
display on the window 52 on the driver seat side in accor- determined trajectory T.
dance with the movement of the driver 70 to the driver seat. ‘When a lateral stagger of the driver 70 is larger than a
In which case, the driver 70 is able to reliably view the predetermined threshold amount, the driver state determiner
display for the outside of the vehicle 10. For example, the 412 may determine that the driver 70 is in abnormal con-
environment state determiner 407 may determine a position dition. For example, the driver state determiner 412 may
of a person outside the vehicle, and then the HUD controller determine that the driver 70 is in abnormal condition when
414 may control the displaying the HUD device 500 on the a lateral displacement of the driver 70 over a preset period
basis of the determination result of the environment state is larger than a predetermined threshold amount. In an
determiner 407. In this way, the HUD controller 414 may alternative implementation, when a speed at which the
control a display as illustrated in FIG. 11 driver 70 moves to the vehicle 10 is higher than a prede-
FIGS. 12 and 13 each schematically illustrate an example termined threshold speed, the driver state determiner 412
of'a method of detecting the driver 70 outside the vehicle 10. may determine that the driver 70 is in abnormal condition.
FIG. 12 schematically illustrates an example of a method of The predetermined threshold amount or the predetermined
detecting the driver 70 by means of the vehicle-exterior threshold speed may be based on an average value obtained
sensors 100. As illustrated in FIG. 12, the vehicle-exterior in advance. For example, when the lateral displacement of
sensors 100 may be provided in the vehicle 10 at a plurality the driver 70 over the preset period is larger than the
of sites. The environment information acquiring unit 402 5 predetermined threshold amount or when the moving speed
may recognize the driver 70 outside the vehicle 10 and of the driver 70 is higher than the predetermined threshold
measure a distance to the driver 70. For example, the speed, the driver state determiner 412 may determine that
environment information acquiting unit 402 may recognize the driver 70 is fatigued and that the driver 70 is in abnormal
the driver 70 outside the vehicle by means of the above- condition accordingly.
described image recognition technique. Then, the environ- s Ifonly a specific person drives the vehicle 10 as the driver
ment state determiner 407 may determine whether the 70, the predetermined threshold amount or the predeter-
distance to the driver 70 is equal to or less than the mined threshold speed may be set on the basis of a normal
predetermined threshold. When the distance to the driver 70 value. The normal value may be determined from the
is equal to or less than the predetermined threshold, the specific driver 70 being in regular condition. For example,
HUD controller 414 may control the HUD device 500 to the predetermined threshold amount or the predetermined
provide a display for the outside of the vehicle 10 threshold speed may be set to a value that is based on a
The environment state determiner 407 may determine a product of the normal value of the specific driver 70 and a
position of the driver 70 on the basis of the position preset coefficient. The normal value of the specific driver 70
information on the driver 70 that the environment informa- may be determined through learning. When the lateral
tion acquiring unit 402 has recognized by means of the displacement of the specific driver 70 over the preset period
image recognition. On the basis of this determination result, is larger than the predetermined threshold amount based on
the HUD device 500 may provide a display for the outside the normal value or when the moving speed of the specific
 US 10,279,683 B2
11 12
driver 70 is higher than the predetermined threshold speed YES), the flow may proceed to step S20. In step S20, the
based on the normal value, the driver state determiner 412 HUD controller 414 may control the HUD device 500 to
may determine that the specific driver 70 is fatigued and that switch the display for the outside of the vehicle 10 to the
the driver 70 is in abnormal condition accordingly. display for the inside of the vehicle 10.
FIG. 15 schematically illustrates an example of a method s In step S22, the driver boarding determiner 418 may
in which the driver state determiner 412 determines a facial determine whether the driver 70 turns off the ignition switch
expression of the driver 70. As illustrated in FIG. 15, some and exits from the vehicle 10. When it is determined that the
pieces of image information on a facial expression may be driver 70 turns off the ignition switch and exits from the
stored in advance in relation to happiness, surprise, anger, vehicle 10 (S22: YES), the flow may proceed to step S24. In
sadness, and other feelings. The driver state determiner 412 step S24, the HUD controller 414 may control the HUD
may compare image information on the face of the driver 70 device 500 to switch the display for the inside of the vehicle
which the environment information acquiring unit 402 has 10 to the display for the outside of the vehicle 10, and may
acquired and the pieces of image information as illustrated clear the display after the preset period has passed. After the
in FIG. 15 stored in advance. The driver state determiner 412 step $24 has been performed, the process may be concluded
may select, from the pieces of image information illustrated (END).
in FIG. 15, any image information that is high in similarly When it is determined in step S14 that an abnormality
to the image information on the face of the driver 70. Then, does not occur in both of the vehicle 10 and the driver 70
the driver state determiner 412 may determine condition, or (S14: NO), the flow may proceed o step S26. In step S26,
a feeling, of the driver 70 on the basis of the selected image the HUD controller 414 may control the HUD device 500 to
information. When the driver 70 is very angry, very sad, or provide the regular display on a window of the vehicle 10.
very sleepy, for example, the driver state determiner 412 Affer the step $26 has been performed, the flow may
may determine that the driver 70 is in abnormal condition. proceed 1o step S18.
In such an implementation, as illustrated in FIG. 16, the ‘When it is determined in step S18 that the driver 70 does
HUD device 500 may display a message saying “DRIVE not enter the vehicle 10 (S18: NO), the flow may proceed to
CAREFULLY” toward the outside of the vehicle 10, thereby 25 step S28. In step S28, the abnormality determiner 416 may
giving a warning to the driver 70. Accordingly, the driver determine whether the abnormality is eliminated. When it is.
state determiner 412 may determine whether the driver 70 is determined in step S28 that the abnormality is eliminated
in abnormal condition. Then, the HUD controller 414 may (S28: YES), the flow may proceed to step $26 in which the
control the displaying of the HUD device 500 on the basis HUD controller 414 may control the HUD device 500 to
of the determination result of the driver state determiner
412. 30 provide the regular display on a window of the vehicle 10.
In this way, the vehicle display system 1000 may provide a When it is determined in step S28 that the abnormality is not
display as illustrated in FIG. 16. eliminated (S28: NO), the flow may proceed to step $30. In
FIG. 17 is a flowchart of an example of a process step S30, the HUD controller 414 may control the HUD
according to one implementation. In step S10, the environ- device 500 to clear the display afier the preset period has
ment state determiner 407 may determine whether the driver passed. After the step S30 has been performed, the process
70 comes close to the vehicle 10, on the basis of the may be concluded (END). Note that the process may also be
information acquired from the vehicle-exterior sensors 100 concluded (END) when the environment state determiner
or the intensity of the radio wave received from the wireless 407 determines in step S10 that the driver 70 does not come
key 1100. When the environment state determiner 407 close to the vehicle 10 (S10: NO).
determines that the driver 70 comes close to the vehicle 10 According to one implementation, as described above, it
(S10: YES), the flow may proceed to step S12. In step S12, is possible for the vehicle display system 1000 to display a
the environment state determiner 407 may determine which message in such a way that persons inside and outside a
of the front, rear, right, and left portions of the vehicle 10 the vehicle are able to reliably identify for which target this
driver 70 is near, on the basis of the information acquired ‘message is intended. For example, when the driver 70 comes
from the vehicle-exterior sensors 100 or the intensity of the a5 close to the vehicle 10, the vehicle display system 1000
radio wave received from the wireless key 1100. In an displays information on a window of the vehicle 10, thereby
alternative implementation, the environment state deter- enabling the driver 70 o recognize various pieces of infor-
miner 407 may determine which of the front, rear, right, and ‘mation before driving. Moreover, the vehicle display system
left portions of the vehicle 10 the driver 70 is nearest. 1000 may switch a window on which the information is to
In step S14, the abnormality determiner 416 may deter- s be displayed to another window, depending on movement of
mine whether an abnormality occurs in the vehicle 10, and the driver 70 outside the vehicle, thereby enabling the driver
the driver state determiner 412 may determine whether an 70 to reliably recognize the information even when he/she
abnormality occurs in the driver 70. When it is determined moves
that an abnormality occurs in one or both of the vehicle 10 The controller 400 illustrated in FIG. 1 is implementable
and the driver 70 (S14: YES), the flow may proceed to step 55 by circuitry including at least one semiconductor integrated
S16. In step S16, the HUD controller 414 may control the circuit such as at least one processor (e.g.. a central pro-
HUD device 500 to display. on a window of the vehicle 10, cessing unit (CPU)), at least one application specific inte-
information for the outside of the vehicle 10 which indicates grated circuit (ASIC), and/or at least one field programmable
the occurrence of the abnormality. In such an implementa- gate array (FPGA). At least one processor is configurable, by
tion, the HUD controller 414 may control the HUD device 6 reading instructions from at least one machine readable
500 to display the information on the window located near non-transitory tangible medium, to perform all or a part of
the driver 70. In an alternative implementation, the HUD functions of the controller 400. Such a medium may take
controller 414 may control the HUD device 500 to display many forms, including, but not limited to, any type of
the information on the window located nearest the driver 70. ‘magnetic medium such as a hard disk, any type of optical
In step S18, the driver boarding determiner 418 may deter- 6: medium such as a CD and a DVD, any type of semicon-
mine whether the driver 70 enters the vehicle 10. When it is ductor memory (i.e., semiconductor circuit) such as a vola-
determined that the driver 70 enters the vehicle 10 (S18: tile memory and a non-volatile memory. The volatile
 US 10,279,683 B2
13 14
memory may include a DRAM and a SRAM, and the 8. The vehicle display system according to claim 7,
nonvolatile memory may include a ROM and a NVRAM. wherein the controller controls the display device to provide
The ASIC is an integrated circuit (IC) customized to per- the first image towards inside the vehicle on the first window
form, and the FPGA is an integrated circuit designed to be by inverting front and back of the first image provided
configured after manufacturing in order to perform, all or a towards outside the vehicle.
part of the functions of the controller 400 illustrated in FIG. 9. The vehicle display system according to claim 1,
1 wherein the controller is configured to detect an abnormality
Although some implementations of the technology have of the driver,
been described in the foregoing with reference to the accom- wherein the controller controls, in response to detecting
panying drawings, the technology is by no means limited to the abnormality of the vehicle, the display device to
the implementations described above. It should be appreci-
provide a third image based on the abnormality of the
ated that modifications and alterations may be made by
persons skilled in the art without departing from the scope driver, and
as defined by the appended claims. The technology is wherein the controller controls the display device to
intended to include such modifications and alterations in so provide the third image on the first window.
far as they fall within the scope of the appended claims or 10. A vehicle display system comprising:
the equivalents thereof. a display device configured to provide a first image on
The invention claimed is: windows of a vehicle; and
1. A vehicle display system comprising: a controller configured to:
a display device configured to provide a first image on determine that a driver is inside the vehicle;
windows of a vehicle; and control, in response to determining that the driver is
a controller configured to: inside the vehicle, the display device to provide a
detect, outside the vehicle, a driver of the vehicle first image towards inside the vehicle;
within a threshold distance of the vehicle; determine that the driver exists the vehicle;
determine a position of the driver outside the vehicle; determine a position of the driver outside the vehicle;
and and
control, in response to determining the position of the control, in response to determining the position of the
driver, the display device to provide the first image driver, the display device to provide the first image
towards outside the vehicle on a first window of the towards outside the vehicle on a first window of the
windows, wherein the first window is located closer windows, wherein the first window is located closer
to the position of the driver than other windows of to the position of the driver than other windows of
the vehicle. the vehicle.
2. The vehicle display system according to claim 1, 11. The vehicle display system according to claim 10,
wherein when the controller detects a change in the position wherein when the controller detects a change in the position
of the driver outside the vehicle, the controller controls the 35 of the driver outside the vehicle, the controller controls the
display device to provide the first image towards outside the display device to provide the first image towards outside the
vehicle on a second window of the windows, vehicle on a second window of the windows,
wherein the second window is located closer to the wherein the second window is located closer to the
changed position of the driver than other windows of changed position of the driver than other windows of
the vehicle. the windows of the vehicle.
3. The vehicle display system according to claim 1, 12. A method of controlling a vehicle display system, the
wherein the controller is configured to detect an abnormality method comprising:
of the vehicle, detecting, outside a vehicle, a driver of the vehicle within
wherein the controller controls, in response to detecting a threshold distance of the vehicle, the vehicle includ-
the abnormality of the vehicle, the display device to a5 ing a display device that provides an image on windows
provide, on the first window, a second image indicating of the vehicle;
that the abnormality of the vehicle. determining a position of the driver outside the vehicle;
4. The vehicle display system according to claim 1, and
wherein the controller controls the display device to provide controlling, in response to determining the position of the
the first image for the outside of the vehicle, when the 50 driver, the display device to provide the image towards
abnormality of the vehicle is not detected. outside the vehicle on a first window of the windows,
5. The vehicle display system according to claim 4, wherein the first window is located closer to the posi-
wherein the first image comprises one or more of a message tion of the driver than other windows of the vehicle.
intended for the driver, a weather forecast, news, traffic 13. A method of controlling a vehicle display system, the
information, and routing information. method comprising:
6. The vehicle display system according to claim 4, determining that a driver is inside a vehicle, the vehicle
further comprising a communicator configured to commu- including a display device that provides an image on
nicate with an external server, windows of the vehicle; and
wherein information regarding the first image is acquired controlling, in response to determining that the driver is
by the communicator from the external server. 60 inside the vehicle, the display device to provide the
7. The vehicle display system according to claim 1, image towards inside the vehicle;
wherein the controller is configured to determine that the determining that the driver exits the vehicle; and
driver is inside the vehicle, determining a position of the driver outside the vehicle;
wherein the controller controls, in response to determin- and
ing that the driver is inside the vehicle, the display controlling, in response to determining the position of the
device to provide the first image towards inside the driver, the display device to provide the image towards
vehicle on the first window. outside the vehicle on a first window of the windows,
 US 10,279,683 B2
15 16
wherein the first window is located closer to the posi- 15. A vehicle display system comprising:
tion of the driver than other windows of the vehicle, a display device configured to provide an image on
14. A vehicle display system comprising: windows of a vehicle; and
circuitry configured to
a display device configured to provide an image on determine that a driver is inside the vehicle;
windows of a vehicle; and control, in response to determining that the driver is
circuitry configured to: inside the vehicle, the display device to provide the
detect, outside the vehicle, a driver of the vehicle image towards inside the vehicle;
within a threshold distance of the vehicle; determine that the driver exists the vehicle; and
determine a position of the driver outside the vehicle;
determine a position of the driver outside the vehicle; and
and control, in response to determining the position of the
control, in response to determining the position of the driver, the display device to provide the image
driver, the display device to provide the image towards outside the vehicle on a first window of the
towards outside the vehicle on a first window of the windows, wherein the first window is located closer
15 to the position of the driver than other windows of
windows, wherein the first window is located closer
to the position of the driver than other windows of the vehicle.
the windows of the vehicle.