US010522188B2

United States Patent ( 10 ) Patent No.: US 10,522,188 B2

Rothschild et al. (45 ) Date of Patent: * Dec . 31 , 2019
( 54 ) SYSTEM AND METHOD FOR USING , (52) U.S. CI.
PROCESSING , AND DISPLAYING CPC GIIB 27/10 ( 2013.01); G06K 9/00892
BIOMETRIC DATA (2013.01 ); GIIB 27/031 (2013.01); GIIB
27/102 ( 2013.01) ; G16H 40/63 ( 2018.01 );
(71) Applicants :Richard A Rothschild , London (GB ); H04N 5/76 (2013.01); H04N 9/8205
Dan Macklin , Stafford (GB ); Robin S ( 2013.01 ); G06K 2009/00939 ( 2013.01 )
Slomkowski, Eugene , OR (US ); Taska (58) Field of Classification Search
Harnischfeger , Eugene, OR (US) CPC G11B 27/10 ; G11B 27/031; H04N 5/772;
GOOK 9/00885; G06F 19/3418
(72 ) Inventors: Richard A Rothschild , London (GB ); USPC 386/248
Dan Macklin , Stafford (GB ); Robin S See application file for complete search history .
Slomkowski, Eugene, OR (US ); Taska
Harnischfeger , Eugene , OR (US) (56 ) References Cited
( * ) Notice : Subject to any disclaimer, the term of this U.S. PATENT DOCUMENTS
patent is extended or adjusted under 35 2014/0214568 A1 * 7/2014 Argue G07G 3/003
U.S.C. 154(b ) by 0 days . 705/18
This patent is subject to a terminal dis 2014/0316713 A1 * 10/2014 Chmiel G06F 13/4239
702/19
claimer .
* cited by examiner
( 21) Appl. No.: 16 /273,141
Primary Examiner Thai Q Tran
( 22 ) Filed : Feb. 11 , 2019 Assistant Examiner - Jose M Mesa
Prior Publication Data (74 ) Attorney, Agent, or Firm Fitzsimmons IP Law
(65)
US 2019/0325914 A1 Oct. 24 , 2019 (57) ABSTRACT
A method is provided for identifying and displaying video
Related U.S. Application Data data of a user, either alone or together ( in synchronization )
(63 ) Continuation of application No. 15/ 495,485, filed on with other data , such as biometric data acquired during a
Apr. 24 , 2017 , now Pat. No. 10,242,713 , which is a time that the video data was captured/received . The method
continuation of application No. 15 / 293,211, filed on includes storing biometric data separately from the video
Oct. 13 , 2016 , now abandoned . data, allowing the biometric data to be search quickly to
identify at least one value ( e.g., a value corresponding to at
(60 ) Provisional application No. 62/240,783 , filed on Oct. least one biometric event). At least one biometric time
13, 2015 . stamp (e.g. , a time, a sample rate, a position within a
plurality of values , etc.) linked to the identified value can
(51) Int. Cl. then be used to identify a corresponding video time- frame,
GIIB 27/10 (2006.01) which can then be used to play the video data , either alone
GIIB 27/031 ( 2006.01 ) or together with biometric data, starting at a particular time
G16H 40/63 ( 2018.01) (e.g., at a time that the event occurred , shortly before the
H04N 5/76 (2006.01) event occurred , etc.).
G06K 900 (2006.01)
H04N 9/82 ( 2006.01) 30 Claims, 18 Drawing Sheets

1 310

320
0 76 90

330
????? 340
U.S. Patent Dec. 31, 2019 Sheet 1 of 18 US 10,522,188 B2

104
+

102

100

110
108
106

CL L
Figure 1

108 110
106

Figure 2A
U.S. Patent Dec. 31, 2019 Sheet 2 of 18 US 10,522,188 B2

jengolation
230 200

204
210 202 206

Core hjem
Module 214
208 we

Centras Algorithms
212 Sharing
216
Networked 220
218
250 240
3 Party
A nenucales Services

Figure 2B
U.S. Patent Dec. 31 , 2019 Sheet 3 of 18 US 10,522,188 B2

310

320
0 76 90

330

14 340

Figure 3

Transceiver
Processor Memory

Keyboard Sensor Display

Figure 4
400
U.S. Patent Dec. 31, 2019 Sheet 4 of 18 US 10,522,188 B2

500

Transceiver
Processor
-

.
Memory Sensor Display

1
1

{
}

Transceiver( 1) Processor Memory

Transceiver( 2 ) Keyboard Display

Figure 5
600
U.S. Patent Dec. 31, 2019 Sheet 5 of 18 US 10,522,188 B2

Start 700

Receive Biometric 702

Data w / Time Stamp

Receive Audio /Video 704

Data wl Time Stamp

Sync Biometric Data 706

W / Audio /Video Data

Search For
Play Audio /Video Option ? Biometric Data

708 712
710

Stop 714

Figure 6
U.S. Patent Dec. 31 , 2019 Sheet 6 of 18 US 10,522,188 B2

710

720
0
20 76 90

730

??????? 740

Figure 7

802
Video
804
800
806

Figure 8
U.S. Patent Dec. 31, 2019 Sheet 7 of 18 US 10,522,188 B2

802

Session ID Activity ID
902 904
Figure 9
1000

802
Biometric ( 1 ) Biometric ( 2 ) ID

1002 1004 804

T

1006
S

Figure 10
1100
802
Self-Realization ( 1 )
1102 1104
M

Figure 11
U.S. Patent Dec. 31, 2019 Sheet 8 of 18 US 10,522,188 B2

100
90
80

TH1 T+2 T+3

Figure 12
100
90
80

T+ 1 T+2 T+3

Figure 13

Video 1402
SR 1406
1404 B ( 1) B (2 ) 1408
+
T+ 1 T +2 T+3 T+ 4 T +5

Figure 14
U.S. Patent Dec. 31 , 2019 Sheet 9 of 18 US 10,522,188 B2

VidiMetrics
1500

Ready to work out?

Sign in

Login with Facebook

Figure 15
U.S. Patent Dec. 31 , 2019 Sheet 10 of 18 US 10,522,188 B2

a VIRGIN

Create Session 1600

1602

Session not started: 00:00:00 1606

Start session X Cancel session

Bluetooth Heart Rate Sensor 2752

Chenge to a different sensoor

85 bpre 1604

New
+ 2013
Set-up

Figure 16
U.S. Patent Dec. 31, 2019 Sheet 11 of 18 US 10,522,188 B2

1700
Figure 17

ter your ses ion

Please enter your session name:
Exomple : Summer Biking in Berlin

22
Please enter a description for your session:
Maximum 20.aas ODIO SIT
amet, consectetur adipiscing elit. Integer a
viverra arcu. Aliquam ullamcorper tincidunt
ex quismaximus. Donec justo .

Add on Audio Recording

92
Figure 18
1800
U.S. Patent Dec. 31 , 2019 Sheet 12 of 18 US 10,522,188 B2

1904 1902 1906 1908 1900

Figure 19
Session started : 1:04:57

Pouse Session

102 .Review Session Summer Biking in Berlin

H Show Graph Options

Figure 20
2000 2002
U.S. Patent Dec. 31, 2019 Sheet 13 of 18 US 10,522,188 B2

Figure 21 2100

Graph Display Options

Hide all graphis
? Heart Rate
Heart Rate Vorrance
Speed
Temperature

Apply

Review Session : Summer Biking in Berlin

K « >> Show Graph Options

n Q Search HR volue

2206
Figure 22
2200 2202 2204
U.S. Patent Dec. 31, 2019 Sheet 14 of 18 US 10,522,188 B2

Figure 23 2300

Auch GPS Data on Google Map x

ad

Review Session: Summer Biking in Berlin

MEROS DETAILS ? Save

Mix HR: 79.1 bpm Nome Summer Biking in Berlin

Avg HR 94.8 bpm
Max HR 107.4 bpm Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Integer a viverra arcu. Aliquam
My Colories 658 Kcal ullamcorper tincidunt ex quismaximus.
Donecotum justo .
Audio Recording New Audio Recording Delete
Figure 24
2400
U.S. Patent Dec. 31, 2019 Sheet 15 of 18 US 10,522,188 B2

2500
Figure 25

Enter a target HR value and find it in your session :

100 bpm + 0.5 bpm

< Search HR value: 100 bpm * 0.5

100

QO 20:44
* 100 46 >> Next

2608 Figure 26
2600
U.S. Patent Dec. 31, 2019 Sheet 16 of 18 US 10,522,188 B2

Search HR value: 100 bpm 0.5 X

100048 > Next

2708 Figure 27
2700
U.S. Patent Dec. 31 , 2019 Sheet 17 of 18 US 10,522,188 B2

X
Refine Search
2800

Date: FROM 03 November 2015

TO 10 August 2016

Length : 12 + 2
55 + 20
ve Avg HR: 100 bpm + 0.5
e Min HR : 80 bpm +1.0
Max HR : 120 bpm + 0.7
Calories: 300 kcal - † 50
Apply Filters

Figure 28
U.S. Patent Dec. 31, 2019 Sheet 18 of 18 US 10,522,188 B2

VIRGIN

< Session List 2900

Q Search
Filtered list . 8 Sessions found

Snowboard in Patagonia
9 Aug 2016 , 5:05 pm • 12:55

Summer Biking in Berlin n

15 Jul 2016 , 10:05 am • 04:37

Mountain Bicycle in Tuscany

2 Jul 2016 , 3:21 pm • 09:22

RED BULL AIR RACE 201

Kayaking with Julia A
17 Jun 2016 , 7:05 am • 02:34
+
Now View List Set-up

Figure 29
 US 10,522,188 B2
1 2
SYSTEM AND METHOD FOR USING , Such a system and/or method could be used by an
PROCESSING , AND DISPLAYING individual (e.g., athlete , etc.) or their trainer, coach , etc., to
BIOMETRIC DATA visualize the individual during the performance of an ath
letic event (e.g., jogging, biking, weightlifting, playing
BACKGROUND OF THE INVENTION 5 soccer , etc.) in real - time ( live ) or afterwards, together with
the individuals concurrently measured biometric data (e.g.,
1. Field of the Invention heart rate, etc. ), and /or concurrently gathered “ self -realiza
tion data,” or subject- generated experiential data, where the
The present invention relates to the reception and use of individual inputs their own subjective physical or mental
biometric data , and more particularly, to a system and 10 states during their exercise , fitness or sports activity /training
method for displaying at least one biometric of a user along (e.g., feeling the onset of an adrenaline “ rush ” or endorphins
with video of the user at a time that the at least one biometric in the system , feeling tired , “ getting a second wind,” etc. ).
is being measured and /or received . This would allow a person (e.g. , the individual, the indi
vidual's trainer , a third party , etc.) to monitor/ observe physi
2. Description of Related Art 15 ological and /or subjective psychological characteristics of
an individual while watching or reviewing the individual in
Recently , devices have been developed that are capable of the performance of an athletic event, or other physical
measuring, sensing , or estimating in a convenient form activity . Such inputting of the self-realization data, ca be
factor at least one or more metric related to physiological achieved by various methods,including automatically , time
characteristics, commonly referred to as biometric data . For 20 stamped -in - the-system voice notes, short-form or abbrevia
example, devices that resemble watches have been devel tion key commands on a smart phone , smart watch , enabled
oped which are capable of measuring an individual's heart fitness band , or any other system -linked inputmethod which
rate or pulse, and , using that data together with other is convenient for the individual to utilize so as not to impede
information (e.g., the individual's age, weight, etc.), to (or as little as possible ) the flow and practice by the
calculate a resultant, such as the total calories burned by the 25 individual of the activity in progress.
individual in a given day . Similar devices have been devel Such a system and /or method would also facilitate, for
oped for measuring , sensing, or estimating other kinds of example , remote observation and diagnosis in telemedicine
metrics, such as blood pressure , breathing patterns, breath applications, where there is a need for the medical staff, or
composition , sleep patterns, and blood -alcohol level , to monitoring party or parent, to have clear and rapid confir
name a few . These devices are generically referred to as 30 mation of the identity of the patient or infant, as well as their
biometric devices or biosensor metrics devices . visible physical condition , together with their concurrently
While the types of biometric devices continue to grow , the generated biometric and /or self -realization data .
way in which biometric data is used remains relatively static . Furthermore , the system and /or method should also pro
For example , heart rate data is typically used to give an vide the subject, or monitoring party, with a way of using
individual information on their pulse and calories burned . 35 video indexing to efficiently and intuitively benchmark ,map
By way of another example , blood -alcohol data is typically and evaluate the subject's data , both against the subject's
used to give an individual information on their blood own biometric history and /or against other subjects ' data
alcohol level, and to inform the individual on whether or not samples, or demographic comparables, independently of
they can safely or legally operate a motor vehicle. By way whichever operating platforms or applications have been
of yet another example, an individual's breathing pattern 40 used to generate the biometric and video information . By
(measurable for example either by loudness level in deci being able to filter/ search for particular events ( e.g., biomet
bels , or by variations in decibel level over a time interval) ric events, self -realization events , physical events, etc.), the
may be monitored by a doctor, nurse, or medical technician acquired data can be reduced down or edited (e.g., to create
to determine whether the individual suffers from sleep a “ highlight reel,” etc.) while maintaining synchronization
apnea . 45 between individual video segments and measured and /or
While biometric data is useful in and of itself, such data gathered data (e.g., biometric data, self-realization data ,
would be more informative or dynamic if it could be GPS data, etc. ). Such comprehensive indexing of the events ,
combined with other data ( e.g., video data , etc. ), provided and with it the ability to perform structured aggregation of
(e.g., wirelessly, over a network , etc.) to a remote device , the related data (video and other ) with (or without) data from
and /or searchable ( e.g., allowing certain conditions, such as 50 other individuals or other relevant sources, can also be
an elevated heart rate , to be quickly identified ) and /or utilized to provide richer levels of information using meth
cross - searchable ( e.g., using biometric data to identify a ods of “ Big Data” analysis and “ Machine Learning," and
video section illustrating a specific characteristic , or vice adding artificial intelligence (“ AI” ) for the implementation
versa ). Thus, a need exists for an efficient system and of recommendations and calls to action .
method capable of achieving at least some, or indeed all, of 55
the foregoing advantages , and capable also of merging the SUMMARY OF THE INVENTION
data generated in either automatic or manual form by the
various devices , which are often using operating systems or The present invention provides a system and method for
technologies ( e.g., hardware platforms, protocols , data using , processing, indexing , benchmarking , ranking, com
types , etc.) that are incompatible with one another. 60 paring and displaying biometric data, or a resultant thereof,
In certain embodiments of the present invention , the either alone or together ( e.g., in synchronization ) with other
system and /or method is configured to receive, manage , and data (e.g., video data , etc.). Preferred embodiments of the
filter the quantity of information on a timely and cost present invention operate in accordance with a computing
effective basis, and could also be of further value through the device (e.g. , a smart phone, etc.) in communication with at
accurate measurement, visualization ( e.g. , synchronized 65 least one external device ( e.g. , a biometric device for acquir
visualization, etc.), and rapid notification of data points ing biometric data , a video device for acquiring video data ,
which are outside (or within ) a defined or predefined range . etc. ). In a first embodiment of the present invention , video
 US 10,522,188 B2
3 4
data, which may include audio data , and non -video data , the exemplary computing device , and a processor for oper
such as biometric data , are stored separately on the com ating and /or driving the transceiver, memory, sensor, and
puting device and linked to other data , which allows search display. The exemplary computing device includes a trans
ing and synchronization of the video and non - video data . ceiver ( 1) for receiving biometric data from the exemplary
In one embodiment of the present invention , an applica- 5 biometric device , a memory for storing the biometric data ,
tion (e.g., running on the computing device , etc.) includes a a display for interfacing with the user and displaying various
plurality of modules for performing a plurality of functions. information (e.g. , biometric data , set-up data , operation data,
For example , the application may include a video capture such as start , stop , and pause , input in -session comments or
module for receiving video data from an internal and /or add voice notes, etc.), a keyboard (or other user input) for
external camera , and a biometric capture module for receiv- 10 receiving user input data, a transceiver(2 ) for providing the
ing biometric data from an internal and /or externalbiometric biometric data to the host computing device via the Internet,
device . The client platform may also include a user interface and a processor for operating and / or driving the transceiver
module , allowing a user to interact with the platform , a video ( 1 ), transceiver (2 ), keyboard , display , and memory.
editingmodule for editing video data , a file handling module The keyboard (or other input device ) in the computing
for managing data , a database and sync module for repli- 15 device , or alternatively the keyboard ( or other input device )
cating data, an algorithm module for processing received in the biometric device, may be used to enter self-realization
data, a sharing module for sharing and /or storing data , and data, or data on how the user is feeling at a particular time.
a central login and ID module for interfacing with third party For example , if the user is feeling tired , the user may enter
social media websites, such as FacebookTM the “ T ” on the keyboard . If the user is feeling their endor
These modules can be used , for example , to start a new 20 phins kick in , the user may enter the “ E ” on the keyboard .
session , receive video data for the session ( i.e., via the video And if the user is getting their second wind , the user may
capture module ) and receive biometric data for the session enter the “ S ” on the keyboard . Alternatively, to further
( i.e., via the biometric capture module ). This data can be facilitate operation during the exercise , or sporting activity ,
stored in local storage, in a local database , and / or on a short-code key buttons such as “ T,” “ E ,” and “ S ” can be
remote storage device (e.g., in the company cloud or a 25 preassigned , like speed -dial telephone numbers for fre
third -party cloud service , such as DropboxTM , etc. ). In a quently called contacts on a smart phone, etc., which can be
preferred embodiment, the data is stored so that it is linked selected manually or using voice recognition . This data (e.g.,
to information that (i) identifies the session and (ii) enables the entry or its representation ) is then stored and linked to
synchronization. either a sample rate (like biometric data ) or time- stamp data ,
For example, video data is preferably linked to at least a 30 which may be a time or an offset to the start time that each
start time (e.g., a start time of the session ) and an identifier. button was pressed . This would allow the self-realization
The identifier may be a single number uniquely identifying data to be synchronized to the video data . It would also allow
the ion, or a plurality of numbers ( e.g., a plurality of the self-realization data , like biometric data , be searched
global or universal unique identifiers (GUIDs/UUIDs)), or filtered ( e.g., in order to find video corresponding to a
where a first number uniquely identifying the session and a 35 particular event, such as when the user started to feel tired ,
second number uniquely identifies an activity within the etc.).
session , allowing a session to include a plurality of activi In an alternate embodiment of the present invention , the
ties . The identifier may also include a session name and/or computing device (e.g. , a smart phone , etc.) is also in
a session description . Other information about the video data communication with a host computing device via a wide
(e.g., video length , video source, etc. ) ( i.e., “ video meta- 40 area network (“ WAN ” ), such as the Internet . This embodi
data ” ) can also be stored and linked to the video data . ment allows the computing device to download the appli
Biometric data is preferably linked to at least the start time cation from the host computing device , offload at least some
( e.g., the same start time linked to the video data ), the of the above -identified functions to the host computing
identifier ( e.g., the same identifier linked to the video data ), device, and store data on the host computing device (e.g.,
and a sample rate, which identifies the rate at which bio- 45 allowing video data , alone or synchronized to non - video
metric data is received and /or stored . data , such as biometric data and self-realization data , to be
Once the video and biometric data is stored and linked , viewed by another networked device ). For example, the
algorithms can be used to display the data together . For software operating on the computing device (e.g., the appli
example , if biometric data is stored at a sample rate of 30 cation , program , etc.) may allow the user to play the video
samples per minute (spm ), algorithms can be used to display 50 and /or audio data , but not to synchronize the video and/ or
a first biometric value (e.g., below the video data , superim audio data to the biometric data . This may be because the
posed over the video data , etc.) at the start of the video clip , host computing device is used to store data critical to
a second biometric value two seconds later (two seconds synchronization ( time- stamp index , metadata , biometric
into the video clip ), a third biometric value two seconds later data , sample rate, etc.) and /or software operating on the host
( four seconds into the video clip), etc. In alternate embodi- 55 computing device is necessary for synchronization. By way
ments of the present invention , non -video data (e.g., bio of another example , the software operating on the comput
metric data , self -realization data , etc.) can be stored with a ing device may allow the user to play the video and/or audio
plurality of time- stamps (e.g., individual stamps or offsets data , either alone or synchronized with the biometric data ,
for each stored value , or individual sample rates for each butmay not allow the computing device (or may limit the
data type ), which can be used together with the start time to 60 computing device's ability ) to search or otherwise extrapo
synchronize non - video data to video data . late from , or process the biometric data to identify relevant
In one embodiment of the present invention , the biometric portions (e.g., which may be used to create a “ highlight reel ”
device may include a sensor for sensing biometric data , a of the synchronized video /audio /biometric data ) or to rank
display for interfacing with the user and displaying various the biometric and /or video data. This may be because the
information (e.g., biometric data , set -up data , operation data , 65 host computing device is used to store data critical to search
such as start, stop , and pause, etc. ), a memory for storing the and/or to rank the biometric data (biometric data , biometric
sensed biometric data , a transceiver for communicating with metadata , etc.), and/or software necessary for searching (or
 US 10,522,188 B2
5 6
performing advanced searching of) and /or ranking (or per A more complete understanding of a system and method
forming advanced ranking of) the biometric data. for using processing , and displaying biometric data , or a
In one embodiment of the present invention , the video resultant thereof, will be afforded to those skilled in the art,
data, which may also include audio data, starts at a time“ T ” aswell as a realization of additional advantages and objects
and continues for a duration of “ n.” The video data is 5 thereof,by a consideration of the following detailed descrip
preferably stored in memory (locally and /or remotely ) and tion of the preferred embodiment. Reference will bemade to
linked to other data, such as an identifier, start time, and the appended sheets of drawings, which will first be
duration. Such data ties the video data to at least a particular described briefly.
session , a particular start time, and identifies the duration of
the video included therein . In one embodiment of the present 10 BRIEF DESCRIPTION OF THE DRAWINGS
invention , each session can include different activities. For
example , a trip to Berlin on a particular day (session ) may FIG . 1 illustrates a system for using , processing , and
involve a bike ride through the city ( first activity ) and a walk displaying biometric data , and for synchronizing biometric
through a park ( second activity ). Thus , the identifier may data with other data (e.g., video data, audio data , etc.) in
include both a session identifier, uniquely identifying the 15 accordance with one embodiment of the present invention ;
session via a globally unique identifier (GUID ), and an FIG . 2A illustrates a system for using, processing, and
activity identifier, uniquely identifying the activity via a displaying biometric data , and for synchronizing biometric
globally unique identifier (GUID ), where the session / activ data with other data (e.g. , video data, audio data , etc. ) in
ity relationship is that of a parent/ child . accordance with another embodiment of the present inven
In one embodiment of the present invention , the biometric 20 tion ;
data is stored in memory and linked to the identifier and a FIG . 2B illustrates a system for using, processing, and
sample rate “ m .” This allows the biometric data to be linked displaying biometric data , and for synchronizing biometric
to video data upon playback . For example , if identifier is data with other data (e.g. , video data , audio data , etc. ) in
one , start time is 1:00 PM , video duration is one minute, and accordance with yet another embodiment of the present
the sample rate is 30 spm , then the playing of the video at 25 invention ;
2:00 PM would result in the first biometric value to be FIG . 3 illustrates an exemplary display of video data
displayed ( e.g.,below the video , over the video , etc.) at 2:00 synchronized with biometric data in accordance with one
PM , the second biometric value to be displayed (e.g., below embodiment of the present invention ;
the video , over the video , etc.) two seconds later, and so on FIG . 4 illustrates a block diagram for using , processing,
until the video ends at 2:01 PM . While self-realization data 30 and displaying biometric data , and for synchronizing bio
can be stored like biometric data (e.g., linked to a sample metric data with other data (e.g., video data , audio data , etc.)
rate), if such data is only received periodically, it may be in accordance with one embodiment of the present inven
more advantageous to store this data linked to the identifier
and a time-stamp, where “ m ” is either the time that the FIG . 5 illustrates a block diagram for using, processing,
self-realization data was received or an offset between this 35 and displaying biometric data, and for synchronizing bio
time and the start time (e.g., ten minutes and four seconds metric data with other data (e.g., video data , audio data , etc.)
after the start time, etc.). By storing video and non -video in accordance with another embodiment of the present
data separately from one another, data can be easily search invention ;
and synchronized . FIG . 6 illustrates a method for synchronizing video data
With respect to linking data to an identifier,which may be 40 with biometric data , operating the video data, and searching
linked to other data (e.g., start time, sample rate , etc.), if the the biometric data , in accordance with one embodiment of
data is received in real-time, the data can be linked to the the present invention ;
identifier(s) for the current session (and /or activity ). How FIG . 7 illustrates an exemplary display of video data
ever,when data is received after the fact (e.g., after a session synchronized with biometric data in accordance with
has ended ), there are several ways in which the data can be 45 another embodiment of the present invention ;
linked to a particular session and /or activity (or identifier(s) FIG . 8 illustrates exemplary video data, which is prefer
associated therewith ). The data can bemanually linked (e.g., ably linked to an identifier ( ID ), a start time ( T ),and a finish
by the user )or automatically linked via the application . With time or duration (n );
respect to the latter, this can be accomplished , for example, FIG.9 illustrates an exemplary identifier (ID ), comprising
by comparing the duration of the received data (e.g., the 50 a session identifier and an activity identifier;
video length ) with the duration of the session and/or activity, FIG . 10 illustrates exemplary biometric data , which is
by assuming that the received data is related to the most preferably linked to an identifier ( ID ), a start time ( T ), and
recent session and /or activity, or by analyzing data included a sample rate ( S );
within the received data . For example, in one embodiment, FIG . 11 illustrates exemplary self -realization data , which
data included with the received data (e.g., metadata ) may 55 is preferably linked to an identifier ( ID ) and a time (m );
identify a time and/or location associated with the data , FIG . 12 illustrates how sampled biometric data points can
which can then be used to link the received data to the be used to extrapolate other biometric data point in accor
session and/or activity . In another embodiment, the comput dance with one embodiment of the present invention ;
ing device could display data (e.g., a barcode, such as a QR FIG . 13 illustrates how sampled biometric data points can
code , etc.) that identifies the session and /or activity. An 60 be used to extrapolate other biometric data points in accor
external video recorder could record the identifying data (as dance with another embodiment of the present invention;
displayed by the computing device ) along with (e.g., before, FIG . 14 illustrates an example of how a start time and data
after, or during ) the user and /or his/her surroundings. The related thereto (e.g., sample rate, etc.) can be used to
application could then search the video data for identifying synchronized biometric data and self-realization data to
data, and use this data to link the video data to a session 65 video data ;
and /or activity. The identifying portion of the video data FIG . 15 depicts an exemplary " sign in ” screen shot for an
could then be deleted by the application if desired . application that allows a user to capture at least video and
 US 10,522,188 B2
7 8
biometric data of the user performing an athletic event (e.g., sure ), speed (walking speed , running speed ), location and
bike riding , etc.) and to display the video data together (or distance travelled , breathing rate , heart rate variance (HRV ),
in synchronization ) with the biometric data ; EKG data , perspiration levels , calories consumed and/or
FIG . 16 depict an exemplary “ create session ” screen shot burnt, ketones, waste discharge content and /or levels , hor
for the application depicted in FIG . 15 , allowing the user to 5
mone levels, blood content, saliva content, audible levels
create a new session ; (e.g., snoring , etc.), mood levels and changes , galvanic skin
FIG . 17 depicts an exemplary “ session name” screen shot response , brain waves and /or activity or other neurological
for the application depicted in FIG . 15 , allowing the user to measurements, sleep patterns, physical characteristics (e.g.,
enter a name for the session ;
FIG . 18 depicts an exemplary “ session description " 10 height etc. )
, weight, eye color, hair color, iris data , fingerprints,
or responses (e.g. , facial changes, iris (or pupal)
screen shot for the application depicted in FIG . 15 , allowing changes , voice (or tone ) changes, etc.), or any combination
the user to enter a description for the session ; or resultant thereof.
FIG . 19 depicts an exemplary " session started ” screen As shown in FIG . 1, a biometric device 110 may be in
shot for the application depicted in FIG . 15 , showing the 15 communication with a computing device 108 , such as a
video and biometric data received in real-time;
FIG . 20 depicts an exemplary “ review session ” screen smart phone, which , in turn , is in communication with at
shot for the application depicted in FIG . 15 , allowing the least one computing device (102, 104 , 106 ) via a wide area
user to playback the session at a later time; network (“ WAN ” ) 100 , such as the Internet. The computing
FIG . 21 depicts an exemplary " graph display option " devices can be of different types , such as a PC , laptop, tablet,
29

screen shot for the application depicted in FIG . 15, allowing 20 smart phone, smart watch etc. , using one or different oper
the user to select data (e.g., heart rate data, etc.) to be ating systems or platforms. In one embodiment of the
displayed along with the video data ; present invention , the biometric device 110 is configured to
FIG . 22 depicts an exemplary “ review session ” screen acquire ( e.g., measure , sense , estimate , etc.) an individual's
shot for the application depicted in FIG . 15 , where the video heart rate ( e.g. , biometric data ). The biometric data is then
data is displayed together (or in synchronization ) with the 25 provided to the computing device 108 , which includes a
biometric data ; video and /or audio recorder ( not shown ).
FIG . 23 depicts an exemplary “ map ” screen shot for the In a first embodiment of the present invention , the video
application depicted in FIG . 15 , showing GPS data dis and / or audio data are provided along with the heart rate data
played on a Google map , to a host computing device 106 via the network 100.
FIG . 24 depicts an exemplary “ summary ” screen shot for 30 Because the concurrent video and /or audio data and the heart
the application depicted in FIG . 15 , showing a summary of rate data are provided to the host computing device 106 , a
the session; host application operating thereon (not shown ) can be used
FIG . 25 depicts an exemplary " biometric search ” screen to synchronize the video data , audio data , and /or heart rate
shot for the application depicted in FIG . 15 , allowing a user data , thereby allowing a user (e.g. , via the user computing
to search the biometric data for particular biometric event 35 devices 102 , 104) to view the video data and /or listen to the
(e.g., a particular value , a particular range , etc.); audio data (either in real-time or time delayed ) while view
FIG . 26 depicts an exemplary “ first result " screen shot for ing the biometric data . For example , as shown in FIG . 3 , the
the application depicted in FIG . 15 , showing a first result for host application may use a time-stamp 320, or other
the biometric event shown in FIG . 25 , together with corre sequencing method using metadata , to synchronize the video
sponding video ; 40
data 310 with the biometric data 330 , allowing a user to
FIG . 27 depicts an exemplary “ second result” screen shot view , for example , an individual ( e.g., patient in a hospital,
for the application depicted in FIG . 15 , showing a second baby in a crib , etc.) at a particular time 340 ( e.g. , 76 seconds
result for the biometric event shown in FIG . 25 , together past the start time) and biometric data associated with the
with corresponding video ; individual at that particular time 340 ( e.g., 76 seconds past
FIG . 28 depicts an exemplary “ session search ” screen 45 the start time).
shot for the application depicted in FIG . 15 , allowing a user It should be appreciated that the host application may
to search for sessions thatmeet certain criteria ; and further be configured to perform other functions , such as
FIG . 29 depicts an exemplary “ list” screen shot for the search for a particular activity in video data , audio data,
application depicted in FIG . 15 , showing a result for the biometric data and /or metadata , and /or ranking video data ,
criteria shown in FIG . 28 . 50
audio data , and /or biometric data. For example, the host
application may allow the user to search for a particular
DETAILED DESCRIPTION OF THE biometric event, such as a heart rate that has exceeded a
PREFERRED EMBODIMENT particular threshold or value , a heart rate that has dropped
below a particular threshold or value, a particular heart rate
The present invention provides a system and method for 55 (or range) for a minimum period of time, etc. By way of
using , processing, indexing, benchmarking, ranking , com another example , the host application may rank video data ,
paring and displaying biometric data, or a resultant thereof, audio data, biometric data , or a plurality of synchronized
either alone or together ( e.g., in synchronization ) with other clips (e.g., highlight reels) chronologically , by biometric
data ( e.g., video data , etc.). It should be appreciated that magnitude (highest to lowest, lowest to highest, etc.), by
while the invention is described herein in terms of certain 60 review (best to worst, worst to best, etc. ), or by views (most
biometric data (e.g., heart rate, breathing patterns, blood to least , least to most, etc.). It should further be appreciated
alcohol level, etc.), the invention is not so limited , and can that such functions as the ranking, searching, and analysis of
be used in conjunction with any biometric and /or physical data is not limited to a user's individual session , but can be
data, including , butnot limited to oxygen levels, CO2 levels , performed across any number of individual sessions of the
oxygen saturation , blood pressure , blood glucose , lung func- 65 user, as well as the session or number of sessions ofmultiple
tion , eye pressure, body and ambient conditions ( tempera users. One use of this collection of all the various informa
ture, humidity, light levels, altitude , and barometric pres tion (video , biometric and other) is to be able to generate
 US 10,522,188 B2
9 10
sufficient data points for Big Data analysis and Machine synchronizing data , searching/ filtering data, creating a high
Learning of the purposes of generating AI inferences and light reel, etc.), a sharing module 220 for sharing and /or
recommendations. storing data (e.g., video data , highlight reel , etc. ) relating
By way of example , machine learning algorithms could either to a single session or multiple sessions , and a central
be used to search through video data automatically , looking 5 login and ID module 218 for interfacing with third party
for themost compelling content which would subsequently social media websites, such as FacebookTM
be stitched together into a short “ highlight reel.” The neural With respect to FIG . 2B , the computing device 200 , which
network could be trained using a plurality of sports videos , may be a smart phone , a tablet, or any other computing
along with ratings from users of their level of interest as the device , may be configured to download the client platform
videos progress . The input nodes to the network could be a 10 from the host computing device 240. Once the client plat
sample of change in intensity of pixels between frames form is running on the computing device 200, the platform
along with the median excitement rating of the current can be used to start a new session , receive video data for the
frame. The machine learning algorithms could also be used , session ( i.e., via the video capture module 210 ) and receive
in conjunction with a multi -layer convolutional neural net biometric data for the session ( i.e., via the biometric capture
work , to automatically classify video content ( e.g., what 15 module 212 ). This data can be stored in local storage, in a
sport is in the video ). Once the content is identified , either localdatabase , and /or on a remote storage device (e.g. , in the
automatically or manually , algorithms can be used to com company cloud or a third -party cloud , such as DropboxTM ,
pare the user's activity to an idealized activity. For example, etc.). In a preferred embodiment, the data is stored so that it
the system could compare a video recording of the user's is linked to information that (i) identifies the session and (ii)
golf swing to that of a professional golfer. The system could 20 enables synchronization .
then provide incremental tips to the user on how the user For example, video data is preferably linked to at least a
could improve their swing. Algorithmscould also be used to start time (e.g., a start time of the session) and an identifier.
predict fitness levels for users (e.g., if they maintain their The identifier may be a single number uniquely identifying
program , giving them an incentive to continue working out), the session , or a plurality of numbers (e.g. , a plurality of
match users to other users or practitioners having similar 25 globally (or universally) unique identifiers (GUIDs/UUIDs),
fitness levels, and /or create routines optimized for each user. where a first number uniquely identifying the session and a
It should also be appreciated , as shown in FIG . 2A , that second number uniquely identifies an activity within the
the biometric data may be provided to the host computing session , allowing a session ( e.g., a trip to or an itinerary in
device 106 directly, without going through the computing a destination , such as Berlin ) to include a plurality of
device 108. For example, the computing device 108 and the 30 activities (e.g., a bike ride, a walk , etc.). By way of example
biometric device 110 may communicate independently with only , an activity (or session ) identifier may be a 128 bit
the host computing device , either directly or via the network identifier that has a high probability of uniqueness, such as
100. It should further be appreciated that the video the 8bf25512 - f17a -4e9e -b49a -7c3f59ecle85 ) . The identifier
audio data , and /or the biometric data need not be provided may also include a session name and /or a session descrip
to the host computing device 106 in real-time. For example, 35 tion . Other information about the video data (e.g., video
video data could be provided at a later time as long as the length , video source , etc.) ( i.e., “ video metadata” ) can also
data can be identified , or tied to a particular session . If the be stored and linked to the video data . Biometric data is
video data can be identified , it can then be synchronized to preferably linked to at least the start time (e.g., the same start
other data (e.g., biometric data ) received in real- time. time linked to the video data ), the identifier ( e.g., the same
In one embodiment of the present invention , as shown in 40 identifier linked to the video data ), and a sample rate, which
FIG . 2B , the system includes a computing device 200 , such identifies the rate at which biometric data is received and / or
as a smart phone , in communication with a plurality of stored . For example , heart rate data may be received and
devices, including a host computing device 240 via a WAN stored at a rate of thirty samples per minute ( 30 spm ), i.e.,
( see, e.g., FIG . 1 at 100 ), third party devices 250 via the once every two seconds, or some other predetermined time
WAN (see , e.g., FIG . 1 at 100 ), and local devices 230 ( e.g., 45 interval sample.
via wireless or wired connections). In a preferred embodi In some cases, the sample rate used by the platform may
ment, the computing device 200 downloads a program or be the sample rate of the biometric device ( i.e., the rate at
application ( i.e., client platform ) from the host computing which data is provided by the biometric device ). In other
device 240 (e.g., company cloud ). The client platform cases , the sample rate used by the platform may be inde
includes a plurality of modules that are configured to per- 50 pendent from the rate at which data is received ( e.g., a fixed
form a plurality of functions. rate , a configurable rate , etc.). For example , if the biometric
For example , the client platform may include a video device is configured to provide biometric data at a rate of
capture module 210 for receiving video data from an internal sixty samples per minute (60 spm ), the platform may still
and /or external camera, and a biometric capture module 212 store the data at a rate of 30 spm . In other words, with a
for receiving biometric data from an internal and/or external 55 sample rate of 30 spm , the platform will have stored five
biometric device. The client platform may also include a values after ten seconds, the first value being the second
user interface module 202 , allowing a user to interact with value transmitted by the biometric device, the second value
the platform , a video editing module 204 for editing video being the fourth value transmitted by the biometric device ,
data, a file handling module 206 for managing (e.g., storing, and so on . Alternatively, if the biometric device is configured
linking , etc.) data (e.g., video data, biometric data, identi- 60 to provide biometric data only when the biometric data
fication data , start time data , duration data , sample rate data , changes, the platform may still store the data at a rate of 30
self-realization data , time- stamp data , etc.), a database and spm . In this case, the first value stored by the platform may
sync module 214 for replicating data (e.g. , copying data be the first value transmitted by the biometric device, the
stored on the computing device 200 to the host computing second value stored may be the first value transmitted by the
device 240 and /or copying user data stored on the host 65 biometric device if at the time of storage no new value has
computing device 240 to the computing device 200 ), an been transmitted by the biometric device , the third value
algorithm module 216 for processing received data ( e.g., stored may be the second value transmitted by the biometric
 US 10,522,188 B2
11 12
device if at the time of storage a new value is being for storing the biometric data , a display for interfacing with
transmitted by the biometric device , and so on. the user and displaying various information ( e.g. , biometric
Once the video and biometric data is stored and linked , data , set-up data , operation data , such as start , stop , and
algorithms can be used to display the data together. For pause, input in -session comments or add voice notes, etc.),
example , if biometric data is stored at a sample rate of 30 5 a keyboard for receiving user input data , a transceiver ( 2 ) for
spm , which may be fixed or configurable , algorithms (e.g., providing the biometric data to the host computing device
216 ) can be used to display a first biometric value ( e.g., via the Internet (e.g., using any of telemetry, any WiFi
below the video data, superimposed over the video data, standard , DNLA , Apple AirPlay, Bluetooth , near field com
etc. ) at the start of the video clip , a second biometric value munication (NFC ), RFID , ZigBee , Z -Wave, Thread , Cellu
two seconds later (two seconds into the video clip ), a third 10 lar, a wired connection , infrared or other method of data
biometric value two seconds later ( four seconds into the transmission , datacasting or streaming, etc.), and a processor
video clip ), etc. In alternate embodiments of the present for operating and /or driving the transceiver( 1), transceiver
invention , non - video data (e.g. , biometric data , self- realiza (2 ), keyboard , display, and memory .
tion data , etc.) can be stored with a plurality of time-stamps The keyboard in the computing device 600, or alterna
(e.g., individual stamps or offsets for each stored value ), 15 tively the keyboard in biometric device 500 , may be used to
which can be used together with the start time to synchronize enter self -realization data , or data on how the user is feeling
non - video data to video data . at a particular time. For example , if the user is feeling tired ,
It should be appreciated that while the client platform can the user may hit the “ T ” button on the keyboard . If the user
be configured to function autonomously (i.e., independent of is feeling their endorphins kick in , the user may hit the “ E ”
the host network device 240 ), in one embodiment of the 20 button on the keyboard . And if the user is getting their
present invention , certain functions of the client platform are second wind , the user may hit the “ S ” button on the
performed by the host network device 240, and can only be keyboard . This data is then stored and linked to either a
performed when the computing device 200 is in communi sample rate (like biometric data ) or time- stamp data , which
cation with the host computing device 240. Such an embodi may be a time or an offset to the start time that each button
ment is advantageous in that it not only offloads certain 25 was pressed . This would allow the self -realization data, in
functions to the host computing device 240 , but it ensures the same way as the biometric data , to be synchronized to the
that these functions can only be performed by the host video data . It would also allow the self-realization data, like
computing device 240 (e.g., requiring a user to subscribe to the biometric data , to be searched or filtered (e.g., in order
a cloud service in order to perform certain functions ). to find video corresponding to a particular event, such as
Functions offloaded to the cloud may include functions that 30 when the user started to feel tired , etc. ).
are necessary to display non - video data together with video It should be appreciated that the present invention is not
data (e.g., the linking of information to video data , the limited to the block diagrams shown in FIG . 5 , and a
linking of information to non - video data , synchronizing biometric device and /or a computing device that includes
non- video data to video data , etc.), or may include more fewer ormore components is within the spirit and scope of
advanced functions, such as generating and /or sharing a 35 the present invention . For example , a biometric device that
“ highlight reel.” In alternate embodiments, the computing does not include a display, or includes a camera and /or
device 200 is configured to perform the foregoing functions microphone is within the spirit and scope of the present
as long as certain criteria has been met. This criteria may invention , as are other data -entry devices or methods beyond
include the computing device 200 being in communication a keyboard, such as a touch screen , digital pen , voice /audible
with the host computing device 240 , or the computing 40 recognition device , gesture recognition device, so -called
device 200 previously being in communication with the host “ wearable ,” or any other recognition device generally
computing device 240 and the period of time since the last known to those skilled in the art. Similarly , a computing
communication being equal to or less than a predetermined device that only includes one transceiver, further includes a
amountof time. Technology known to those skilled in the art camera (for capturing video ) and/or microphone ( for cap
(e.g., using a keyed hash -based method authentication code 45 turing audio or for performing spatial analytics through
(HMAC ), a stored time of said last communication (allow recording or measurement of sound and how it travels ), or
ing said computing device to determine whether said delta is further includes a sensor (see FIG . 4 ) is within the spirit and
less than a predetermined amount of time), etc.) can be used scope of the present invention . It should also be appreciated
to ensure that this criteria is met before allowing the per that self -realization data is not limited to how a user feels,
formance of certain functions . 50 but could also include an event that the user or the appli
Block diagramsof an exemplary computing device and an cation desires to memorialize . For example, the user may
exemplary biometric device are shown in FIG . 5. In par want to record (or time-stamp) the user biking past wildlife,
ticular, the exemplary biometric device 500 includes a or a particular architectural structure , or the application may
sensor for sensing biometric data , a display for interfacing want to record (or time-stamp) a patient pressing a “ request
with the user and displaying various information (e.g., 55 nurse ” button , or any other sensed non -biometric activity of
biometric data , set-up data, operation data , such as start, the user.
stop , and pause , etc. ), a memory for storing the sensed Referring back to FIG . 1 , as discussed above in conjunc
biometric data , a transceiver for communicating with the tion with FIG . 2B , the host application (or client platform )
exemplary computing device 600 , and a processor for oper may operate on the computing device 108. In this embodi
ating and /or driving the transceiver, memory , sensor, and 60 ment, the computing device 108 ( e.g., a smart phone) may
display . The exemplary computing device 600 includes a be configured to receive biometric data from the biometric
transceiver( 1) for receiving biometric data from the exem device 110 (either in real- time, or at a later stage , with a
plary biometric device 500 ( e.g., using any of telemetry, any time- stamp corresponding to the occurrence of the biometric
WiFi standard , DNLA , Apple AirPlay, Bluetooth , near field data ), and to synchronize the biometric data with the video
communication (NFC ), RFID , ZigBee , Z -Wave, Thread , 65 data and/or the audio data recorded by the computing device
Cellular, a wired connection, infrared or other method of 108 ( or a camera and/or microphone operating thereon ). It
data transmission , datacasting or streaming, etc.), a memory should be appreciated that in this embodiment of the present
 US 10,522,188 B2
13 14
invention , other than the host application being run locally example , at least video of an athlete , or a sportsman or
( e.g., on the computing device 108), the host application (or woman (e.g., a football player , a soccer player, a racing
client platform ) operates as previously discussed . driver , etc.) to be shown in action (e.g. , playing football,
Again , with reference to FIG . 1, in another embodiment playing soccer, motor racing, etc.) along with biometric data
of the present invention , the computing device 108 further 5 of the athlete in action (see , e.g., FIG . 7 ). By way of example
includes a sensor for sensing biometric data . In this embodi only , this would allow a user to view a soccer player's heart
ment of the present invention , the host application (or client rate 730 as the soccer player dribbles a ball , kicks the ball,
platform ) operates as previously discussed ( locally on the heads the ball, etc. This can be accomplished using a time
computing device 108 ), and functions to at least synchronize stamp 720 (e.g., start time, etc.), or other sequencingmethod
the video , audio , and /or biometric data , and allow the 10
synchronized data to be played or presented to a user (e.g., using metadata ( e.g., sample rate , etc. ), to synchronize the
via a display portion , via a display device connected directly user to data
video
view
710 with the biometric data 730 , allowing the
the soccer player at a particular time 740 ( e.g.,
to the computing device, via a user computing device 76 seconds) and biometric data associated with the athlete at
connected to the computing device (e.g., directly , via the that particular time 340 (e.g., 76 seconds). Similar technol
network , etc.), etc.). 15
It should be appreciated that the present invention , in any ogy can be used to display biometric data on other athletes ,
embodiment, is not limited to the computing devices (num cardWhere players , actors , online gamers, etc.
it is desirable to monitor or watch more than one
ber or type ) shown in FIGS. 1 and and may include any
of a computing, sensing, digital recording , GPS or otherwise individual from a camera view , for example , patients in a
location -enabled device ( for example , using WiFi Position- 20 hospital ward being observed from a remote nursing station
ing Systems“ WPS ” , or other formsof deriving geographical or, during a televised broadcast of a sporting event such as
location , such as through network triangulation ), generally a football game, with multiple players on the sports field , the
known to those skilled in the art, such as a personal system can be so configured , by the subjects using Bluetooth
computer, a server, a laptop , a tablet, a smart phone , a or other wearable or NFC sensors (in some cases with their
cellular phone , a smart watch , an activity band , a heart-rate 25 sensing capability also being location -enabled in order to
strap , a mattress sensor, a shoe sole sensor, a digital camera , identify which specific individual to track ) capable of trans
a near field sensor or sensing device , etc. It should also be mitting their biometrics over practicable distances , in con
appreciated that the present invention is not limited to any junction with relays or beacons if necessary, such that the
particular biometric device , and includes biometric devices viewer can switch the selection of which of one or multiple
that are configured to be worn on the wrist (e.g. , like a 30 individuals ' biometric data to track , alongside the video or
watch ), worn on the skin ( e.g., like a skin patch ) or scalp , or broadcast, and , if wanted and where possible within the
incorporated into computing devices (e.g. , smart phones , limitations of the video capture field of the camera used , also
etc.), either integrated in , or added to items such as bedding , to concentrate the view of the video camera on a reduced
wearable devices such as clothing, footwear, helmets orhats, group or on a specific individual. In an alternate embodiment
or ear phones, or athletic equipment such as rackets , golf 35 of the present invention , selection of biometric data is
clubs, or bicycles, where other kinds of data, including automatically accomplished, for example, based on the
physical performance metrics such as racket or club head individual's location in the video frame (e.g., center of the
speed , or pedal rotation /second , or footwear recording such frame), rate of movement ( e.g.,moving quicker than other
things as impact zones, gait or shear, can also be measured individuals ), or proximity to a sensor (e.g., being worn by
synchronously with biometrics, and synchronized to video . 40 the individual, embedded in the ball being carried by the
Other data can also be measured synchronously with video individual, etc.), which may be previously activate or acti
data, including biometrics on animals (e.g., a bull's accel vated by a remote radio frequency signal . Activation of the
eration or pivot or buck in a bull riding event, a horse's sensor may result in biometric data of the individual being
acceleration matched to heart rate in a horse race, etc.), and transmitted to a receiver, or may allow the receiver to
physical performance metrics of inanimate objects, such a 45 identified biometric data of the individual amongst other
revolutions/minute (e.g., in a vehicle , such as an automobile , data being transmitted (e.g., biometric data from other
a motorcycle , etc. ), miles/hour (or the like ) (e.g., in a individuals ).
vehicle , such as an automobile , a motorcycle , etc., a bicycle , In the context of fitness or sports tracking, it should be
etc.), or G - forces (e.g., experienced by the user, an animal, appreciated that the capturing of an individual's activity on
and inanimate object, etc.). All of this data (collectively 50 video is not dependent on the presence of a third party to do
“ non -video data ,” which may include metadata , or data on this , but various methods of self-videoing can be envisaged ,
non - video data ) can be synchronized to video data using a such as a video capture device mounted on the subject's
sample rate and /or at least one time- stamp , as discussed wrist or a body harness, or on a selfie attachment or a gimbal,
above . or fixed to an object (e.g., sports equipment such as bicycle
It should further be appreciated that the present invention 55 handlebars , objects found in sporting environments such as
need not operate in conjunction with a network , such as the a basketball or tennis net, a football goal post, a ceiling, etc.,
Internet. For example , as shown in FIG . 2A , the biometric a drone-borne camera following the individual, a tripod ,
device 110 , which may be, for example, be a wireless etc.). It should be further noted that such video capture
activity band for sensing heart rate, and the computing devices can include more than one camera lens, such that not
device 108 , which may be, for example, a digital video 60 only the individual's activity may be videoed , but also
recorder , may be connected directly to the host computing simultaneously a different view , such as what the individual
device 106 running the host application (not shown), where is watching or sees in front of them ( i.e., the user's sur
the host application functions as previously discussed . In roundings ). The video capture device could also be fitted
this embodiment, the video , audio , and /or biometric data can with a convex mirror lens, or have a convex mirror added as
be provided to the host application either (i) in real time, or 65 an attachment on the front of the lens, or be a full 360 degree
(ii) at a later time, since the data is synchronized with a camera , or multiple 360 cameras linked together, such that
sample rate and /or time-stamp. This would allow , for either with or without the use of specialized software known
 US 10,522,188 B2
15 16
in the art, a 360 degree all-around or surround view can be It should be appreciated that biometric data does not need
generated , or a 360 global view in all axes can be generated . to be linked to physical movement or sporting activity , but
In the context of augmented or virtual reality, where the may instead be combined with video of an individual at a
individual is wearing suitably equipped augmented reality fixed location ( e.g., where the individual is being monitored
(“ AR ” ) or virtual reality (“ VR ” ) glasses, goggles, headset or 5 remotely or recorded for subsequent review ), for example,
is equipped with another type of viewing display capable of as shown in FIG . 3, for health reasons or a medical condi
rendering AR , VR , or other synthesized or real 3D imagery , tion , such as in their home or in hospital, or a senior citizen
the biometric data such as heart rate from the sensor, in an assisted - living environment, or a sleeping infant being
together with other data such as, for example , work -out run monitored by parents whilst in another room or location .
or speed , from a suitably equipped sensor, such as an 10 theAlternatively , the individual might be driving past or in
proximity of a park or a shopping mall, with their
accelerometer capable of measuring motion and velocity, location being recorded , typically by geo -stamping, or addi
could be viewable by the individual , superimposed on their tional information
viewing field . Additionally an avatar of the individual in altitude or weatherbeing at
added by geo -tagging , such as the
the specific location , together with
motion could be superimposed in front of the individual's 15 what the information or content
viewing field , such that they could monitor or improve their acted with by the individual (e.g. , aisparticular
, being viewed or inter
exercise performance , or otherwise enhance the experience a movie trailer, a dating profile, etc.) on theadvertisement ,
Internet or a
of the activity by viewing themselves or their own avatar, smart/enabled television , or on any other networked device
together ( e.g., synchronized ) with their performance (e.g., incorporating a screen , and their interaction with that infor
biometric data, etc.). Optionally , the biometric data also of 20 mation or content, being viewable or recorded by video, in
their avatar, or the competing avatar, could be simultane conjunction with their biometric data , with all these sources
ously displayed in the viewing field . In addition (or alter of data being able to be synchronized for review , by virtue
natively ), at least one additional training or competing of each of these individual sources being time-stamped or
avatar can be superimposed on the individual's view , which the like (e.g., sampled , etc.). This would allow a third party
may show the competing avatar(s) in relation to the indi- 25 (e.g., a service provider, an advertiser , a provider of adver
vidual ( e.g., showing them superimposed in front of the tisements, a movie production company/ promoter, a poster
individual, showing them superimposed to the side of the of a dating profile , a dating site, etc.) to acquire for analysis
user , showing them behind the individual (e.g., in a rear of their response , the biometric data associated with the
view -mirror portion of the display, etc.), and /or showing viewing of certain data by the viewer, where either the
them in relation to the individual ( e.g., as blips on a 30 viewer or their profile could optionally be identifiable by the
radar - screen portion of the display, etc. ), etc. Competing third party's system , or where only the identity of the
avatar(s ), either of real people such as their friends or viewer's interacting device is known , or can be acquired
training acquaintances, can be used to motivate the user to from the biometric sending party's GPS , or otherwise loca
improve or correct their performance and/or to make their tion - enabled , device .
exercise routine more interesting (e.g., by allowing the 35 For example, an advertiser or an advertisement provider
individual to “ compete” in the AR , VR , or Mixed Reality could see how people are responding to an advertisement, or
("MR " ) environment while exercising, or training , or virtu a movie production company /promoter could evaluate how
ally “ gamifying” their activity through the visualization of people are responding to a movie trailer, or a poster of a
virtual destinations or locations, imagined or real, such as dating profile or the dating site itself , could see how people
historical sites, scanned or synthetically created through 40 are responding to the dating profile .Alternatively , viewers of
computer modeling). online players of an online gaming or eSports broadcast
Additionally , any multimedia sources to which the user is service such as twitch.tv, or of a televised or streamed online
being exposed whilst engaging in the activity which is being poker game, could view the active participants ' biometric
tracked and recorded, should similarly be able to be recorded data simultaneously with the primary video source as well as
with the time stamp, for analysis and /or correlation of the 45 the participants ' visible reactions or performance . As with
individual's biometric response . An example of an applica video /audio , this can either be synchronized in real -time, or
tion of this could be in the selection of specific music tracks synchronized later using the embedded time-stamp or the
for when someone is carrying out a training activity , where like (e.g., sample rate , etc.). Additionally , where facial
the correlation of the individual's past response, based, for expression analysis is being generated from the source
example , on heart rate (and how well they achieved specific 50 video , for example in the context of measuring an individu
performance levels or objectives ) to music type ( e.g., the al's response to advertising messages , since the video is
specific music track (s ), a track ( s) similar to the specific already time-stamped (e.g., with a start time), the facial
track (s ), a track (s ) recommended or selected by others who expression data can be synchronized and correlated to the
have listened to or liked the specific track ( s), etc.) is used to physical biometric data of the individual , which has simi
develop a personalized algorithm , in order to optimize 55 larly been time-stamped and/ or sampled ,
automated music selection to either enhance the physical As previously discussed, the host application may be
effort, or to maximize recovery during and after exertion . configured to perform a plurality of functions. For example ,
The individual could further specify that they wished for the the host application may be configured to synchronize video
specific track or music type, based upon the personalized and /or audio data with biometric data . This would allow , for
selection algorithm , to be played based upon their geo- 60 example , an individual watching a sporting event (e.g., on a
graphical location; an example of this would be someone TV, computer screen , etc.) to watch how each player's
who frequently or regularly uses a particular circuit for biometric data changes during play of the sporting event, or
training or recreational purposes . Alternatively, tracks or also to map those biometric data changes to other players or
types of music could be selected through recording or other comparison models . Similarly , a doctor, nurse , or
correlation of past biometric response in conjunction with 65 medical technician could record a person's sleep habits , and
self-realization inputting when particular tracks were being watch , search or later review , the recording (e.g., on a TV ,
listened to . computer screen , etc.) while monitoring the person's bio
 US 10,522,188 B2
17 18
metric data. The system could also use machine learning to The concern that a monitored party would not wish to be
build a profile for each patient, identifying certain charac permanently monitored ( e.g., a senior citizen not wanting to
teristics of the patient ( e.g., their heart rate rhythm , their have their every move and action continuously videoed )
breathing pattern , etc. ) and notify a doctor, a nurse, or could be mitigated by the incorporation of various additional
medical technician or trigger an alarm if the measured 5 nently featuresrecording
. In one embodiment , the video would be perma
in a loop system which uses a reserved
characteristics appear abnormal or irregular.
The host application could also be configured to provide memory space , recording for a predetermined time period ,
biometric data to a remote user via a network , such as the and then , automatically erasing the video , where n repre
Internet. For example , a biometric device (e.g., a smart sents the selected minutes in the loop and E is the event
phone with a blood - alcohol sensor) could be used to mea 10 and which prevents the recorded loop of n minutes being erased ,
sure a person's blood -alcohol level (e.g., while the person is state triggers both the real time transmission of the visible
talking to the remote user via the smart phone ), and to party, as well asof the
or actions the monitored person to the monitoring
ability to rewind , in order for the
provide the person's blood -alcohol level to the remote user. monitoring party to be able to review the physical manifes
By placing the sensor near , or incorporating it in the micro- 15 tation leading up to E. The trigger mechanism for E could be ,
phone , such a system would allow a parent to determine for example , the occurrence of biometric data outside the
whether their child has been drinking alcohol by participat predefined range , or the notification of another ano nomaly
ing in a telephone or video call with their child . Different such as a fall alert, activated by movement or location
sensors known in the art could be used to sense different sensors such as a gyroscope , accelerometer ormagnetometer
chemicals in the person's breath , or detect people's breath- 20 within the health band device worn by, say the senior citizen ,
ing patterns through analysis of sound and speed variations, or on their mobile phone or other networked motion -sensing
allowing the monitoring party to determine whether the device in their proximity. The monitoring party would be
subject has been using alcohol or other controlled substances able not only to view the physical state of the monitored
or to conduct breath analysis for other diagnostic reasons. party after E , whilst getting a simultaneous read -out of their
The system could also be adapted with a so -called “ lab on 25 relevant biometric data , but also to review the events and
a chip ” (LOC ) integrated in the device itself, or with a biometric data immediately leading up to the event trigger
suitable attachment added to it , for the remote testing for notification. Alternatively, it could be further calibrated so
example , of blood samples where the smart-phone is either that although video is recorded , as before, in the n loop , no
used for the collection and sending of the sample to a testing video from the n loop will actually be transmitted to a
laboratory for analysis , or is used to carry out the sample 30 monitoring party until the occurrence of E. The advantages
collection and analysis within the device itself. In either case of this system include the respect of the privacy of the
the system is adapted such that the identity of the subject and individual, where only the critical event and the time pre
their blood sample are cross -authenticated for the purposes ceding the event would be available to a third party , resulting
of sample and analysis integrity as well as patient identity also in a desired optimization of both the necessary trans
certainty , through the simultaneous recording of the time- 35 mission bandwidth and the data storage requirements . It
stamped video and time and /or location (or GPS ) stamping should be appreciated that the foregoing system could also
of the sample at the point of collection and /or submission of be configured such that the E notification for remote senior,
the sample . This confirmation of identity is particularly infant or patient monitoring is further adapted to include
important for regulatory, record keeping and health insur facial tracking and/or expression recognition features.
ance reasons in the context of telemedicine , since the 40 Privacy could be further improved for the user if their
individual will increasingly be performing functions which , video data and biometric data are stored by themselves , on
till now , have been carried out typically on -site at the their own device , or on their own external, or own secure
relevant facility, by qualified and regulated medical or third -party “ cloud ” storage, but with the index metadata of
laboratory staff , rather than by the subject using a networked the source material, which enables the sequencing , extrapo
device, either for upload to the central analysis facility , or for 45 lation , searching and general processing of the source data ,
remote analysis on the device itself . remaining at a central server, such as, in the case ofmedical
This , or the collection of other biometric data such as records for example , at a doctor's office or other healthcare
heart rate or blood pressure, could also be applied in facility . Such a system would enable themonitoring party to
situations where it is critical for safety reasons, to check , via have access to the video and other data at the time of
regular remote video monitoring in real time, whether say a 50 consultation , but with the video etc. remaining in the pos
pilot of a plane, a truck or train driver, are in fit and sound session of the subject. A further advantage of separating the
condition to be in control of their vehicle or vessel or hosting of the storage of the video and biometric source data
whether for example they are experiencing a sudden inca from the treatment of the data , beyond enhancing the user's
pacity or heart attack etc. Because the monitored person is privacy and their data security , is that by virtue of its storage
being videoed at the same time as providing time-stamped , 55 locally with the subject, not having to upload it to the
geo -stamped and /or sampled biometric data, there is less computational server results both in reduced cost and
possibility for the monitored person or a third party, to increased efficiency of storage and data bandwidth . This
" trick " , " spoof" or bypass the system . In a patient/doctor would be of benefit also where such kind of remote upload
remote consultation setting , the system could be used for of tests for review by qualified medical staff at a different
secure video consults where also , from a regulatory or health 60 location from the subject are occurring in areas of lower
insurance perspective , the consultation and its occurrence is bandwidth network coverage. A choice can also be made to
validated through the time and/or geo stamp validation . lower the frame rate of the video material, provided that this
Furthermore, where there is a requirement for a higher level is made consistentwith sampling rate to confirm the correct
of authentication , the system could further be adapted to use time stamp, as previously described .
facial recognition or biometric algorithms, to ensure that the 65 It should be appreciated that with information being
correct person is being monitored , or facial expression stored at the central server (or the host device ), various
analysis could be used for behavioral pattern assessment. techniques known in the art can be implemented to secure
 US 10,522,188 B2
19 20
the information , and prevent unauthorized individuals or otherwise extrapolate from , or process the biometric data to
entities from accessing the information . Thus, for example , identify relevant portions (e.g., which may be used to create
a user may be provided (or allowed to create ) a user name, a " highlight reel” of the synchronized video /audio /biometric
password , and /or any other identifying (or authenticating ) data ) or to rank the biometric and /or video data . This may be
information (e.g., a user biometric, a key fob , etc. ), and the 5 because the central server is used to store data critical to
host device may be configured to use the identifying (or search and /or rank the biometric data (biometric data , bio
authenticating ) information to grant access to the informa metric metadata , etc.), and/ or software necessary for search
tion (or a portion thereof). Similar security procedures can ing (or performing advanced searching of) and /or ranking
be implemented for third parties, such asmedical providers , (or performing advanced ranking of) the biometric data .
insurance companies , etc., to ensure that the information is 10 In any or all of the above embodiments , the system could
only accessible by authorized individuals or entities. In be further adapted to include password or other forms of
certain embodiments , the authentication may allow access to authentication to enable secured access (or deny unauthor
all the stored data , or to only a portion of the stored data ized access ) to the data in either of one or both directions,
( e.g., a user authentication may allow access to personal such that the user requires permission to access the host, or
information as well as stored video and/ or biometric data , 15 the host to access the user's data . Where interaction between
whereas a third party authentication may only allow access the user and the monitoring party or host is occurring in real
to stored video and /or biometric data ). In other embodi time such as in a secure video consult between patient and
ments, the authentication is used to determine what services their medical practitioner or other medical staff , data could
are available to an individual or entity logging into the host be exchanged and viewed through the establishment of a
device , or the website . For example , visitors to the website 20 Virtual Private Network ( VPN ). The actual data (biometric ,
(or non -subscribers )may only be able to synchronize video / video , metadata index , etc.) can alternatively or further be
audio data to biometric data and /or perform rudimentary encrypted both at the data source, for example at the
searching or other processing, whereas a subscriber may be individual's storage, whether local or cloud -based , and /or at
able to synchronize video /audio data to biometric data the monitoring reviewing party , for example at patient
and /or perform more detailed searching or other processing 25 records at the medical facility , or at the host administration
( e.g., to create a highlight reel, etc.). level.
It should further be appreciated that while there are In the context of very young infant monitoring, a critical
advantages to keeping just the index metadata at the central and often unexplained problem is Sudden Infant Death
server in the interests of storage and data upload efficiency Syndrome (SIDS). Whilst the incidences of SIDS are often
as well as so providing a common platform for the interop- 30 unexplained , various devices attempt to prevent its occur
erability of the different data types and storing the video rence . However, by combining the elements of the current
and /or audio data on the user's own device (e.g., iCloudTM , system to include sensor devices in or near the baby's crib
DropBoxTM , OneDriveTM , etc.), the present invention is not measure relevant biometric data including heart rate , sleep
so limited . Thus, in certain embodiments, where feasible , it pattern , breath analyzer,and othermeasures such as ambient
may be beneficial to ( 1) store data ( e.g., video , audio , 35 temperature , together with a recording device to capture
biometric data , and metadata ) on the user's device (e.g., movement, audible breathing , or lack thereof (i.e., silence)
allowing the user device to operate independent of the host over a predefined period of time, the various parameters
device ), (2 ) store data (e.g., video , audio , biometric data, and could be set in conjunction with the time-stamped video
metadata ) on the central server (e.g., host device ) (e.g., record , by the parent or other monitoring party , to provide a
allowing the user to access the data from any network- 40 more comprehensive alert, to initiate a more timely action or
enabled device ), or (3) store a first portion ( e.g., video and intervention by the user, or indeed to decide that no action
audio data ) on the user's device and store a second portion response would in fact be necessary. Additionally, in the
(e.g., biometric data and metadata ) on the central server case, for example , of a crib monitoring situation , the system
(e.g., host device ) (e.g., allowing the user to only view the could be so configured to develop from previous observa
synchronized video/audio /biometric data when the user 45 tion, with or without input from a monitoring party, a
device is in communication with the host device , allowing learning algorithm to help in discerning what is “ normal,”
the user to only search the biometric data (e.g., to create a what is false positive , or what might constitute an anomaly ,
" highlight reel” ) or rank the biometric data ( to identify and therefore a call to action .
and /or list data chronologically ,magnitude (highest to low The host application could also be configured to play
est), magnitude (lowest to highest ), best reviewed , worst 50 video data that has been synchronized to biometric data, or
reviewed , most viewed , least viewed , etc.) when the user search for the existence of certain biometric data . For
device is in communication with the host device , etc.). example , as previously discussed , by video recording with
In another embodiment of the present invention , the sound a person sleeping , and synchronizing the recording
functionality of the system is further (or alternatively ) with biometric data ( e.g., sleep patterns, brain activity ,
limited by the software operating on the user device and /or 55 snoring, breathing patterns, etc.), the biometric data can be
the host device . For example, the software operating on the searched to identify where certain measures such as sound
user device may allow the user to play the video and/or levels, as measured for example in decibels, or periods of
audio data , but not to synchronize the video and /or audio silences, exceed or drop below a threshold value , allowing
data to the biometric data . This may be because the central the doctor, nurse , or medical technician to view the corre
server is used to store data critical to synchronization 60 sponding video portion without having to watch the entire
(time- stamp index , metadata , biometric data , sample rate , video of the person sleeping.
etc. ) and /or software operating on the host device is neces Such a method is shown in FIG . 6 , starting at step 700 ,
sary for synchronization . By way of another example, the where biometric data and time stamp data ( e.g., start time,
software operating on the user device may allow the user to sample rate ) is received (or linked ) at step 702. Audio /video
play the video and/or audio data , either alone or synchro- 65 data and time stamp data ( e.g., start time, etc.) is then
nized with the biometric data, but may not allow the user received (or linked ) at step 704. The time stamp data ( from
device (or may limit the user device's ability ) to search or steps 702 and 704) is then used to synchronize the biometric
 US 10,522,188 B2
21 22
data with the audio /video data . The user is then allowed to this time and the start time 804 (e.g., ten minutes and four
operate the audio /video at step 708. If the user selects play, seconds after the start time, etc.).
then the audio /video is played at step 710. If the user selects This can be seen , for example , in FIG . 14 , where video
search , then the user is allowed to search the biometric data data starts at time T, biometric data is sampled every two
at step 712. Finally , if the user selects stop , then the video 5 seconds (30 spm ), and self-realization data is received at
is stopped at step 714 . time T + 3 (or three units past the start time). While the video
It should be appreciated that the present invention is not 1402 is playing, a first biometric value 1404 is displayed at
limited to the steps shown in FIG . 6. For example , a method time T + 1, first self -realization data 1406 is displayed at time
that allows a user to search for biometric data thatmeets at T + 2, and a second biometric value 1406 is displayed at time
least one condition , play the corresponding portion of the 10 video
T +4. Bydatastoring
can bedata in thisseparately
stored fashion,fromboth onevideoanother
and non
and
video (or a portion just before the condition ), and stop the synchronized in real -time, or at the time the video is being
video from playing after the biometric data no longer meets played
the at least one condition (or justafter the biometric data non of data. may
It should be appreciated that while separate storage
longer meets the condition ) is within the spirit and scope 15 memory and /be
of or
advantageous for devices having minimal
processing power, the client platform may
the present invention . By way of another example, if the be configured to create new video data, or data that includes
method involves interacting between the user device and the both video and non - video data displayed synchronously .
host device to synchronize the video /audio data and the Such a feature may advantageous in creating a highlight reel,
biometric data and/ or search the biometric data , then the which can then be shared using socialmedia websites , such
method may further involve the steps of uploading the 20 as FacebookTM or YoutubeTM , and played using standard
biometric data and /or metadata to the host device (e.g., in playback software , such as QuicktimeTM . As discussed in
this embodiment the video /audio data may be stored on the greater detail below , a highlight reel may include various
user device ), and using the biometric data and / or metadata portions (or clips ) of video data (e.g., when certain activity
to create a time- stamp index for synchronization and /or to takes place, etc.) along with corresponding biometric data .
search the biometric data for relevant or meaningful data 25 When sampled data is subsequently displayed , the client
(e.g., data that exceeds a threshold , etc. ). By way of yet platform can be configured to display this data using certain
another example, the method may not require step 706 if the extrapolation techniques. For example, in one embodiment
audio / video data and the biometric data are played together of the present invention , as shown in FIG . 12 , where a first
(synchronized ) in real- time, or at the time the data is being biometric value 1202 is displayed at T + 1, a second biometric
played ( e.g., at step 710 ). 30 value 1204 is displayed at T + 2 , and a third biometric value
In one embodiment of the present invention , as shown in 1206 is displayed at T + 3 , biometric data can be displayed at
FIG . 8 , the video data 800 , which may also include audio non -sampled times using known extrapolation techniques ,
data , arts at a time " T " and continues for a duration of “ n ." including linear and non - linear interpolation and all other
The video data is preferably stored in memory ( locally extrapolation and / or interpolation techniques generally
and/or remotely ) and linked to other data , such as an 35 known to those skilled in the art. In another embodiment of
identifier 802 , start time 804 , and duration 806. Such data the present invention , as shown in FIG . 13 , the first biomet
ties the video data to at least a particular session , a particular ric value 1202 remains on the display until the second
start time, and identifies the duration of the video included biometric value 1204 is displayed , the second biometric
therein . In one embodiment of the present invention , each value 1204 remains on the display until the third biometric
session can include different activities. For example , a trip to 40 value 1206 is displayed , and so on .
a destination in Berlin , or following a specific itinerary on a With respect to linking data to an identifier, which may be
particular day (session ) may involve a bike ride through the linked to other data (e.g., start time, sample rate , etc.), if the
city ( first activity ) and a walk through a park ( second data is received in real-time, the data can be linked to the
activity ) . Thus, as shown in FIG . 9 , the identifier 802 may identifier(s) for the current session ( and /or activity ). How
include both a session identifier 902 , uniquely identifying 45 ever, when data is received after the fact ( e.g., after a session
the session via a globally unique identifier (GUID ), and an has ended ), there are several ways in which the data can be
activity identifier 904 , uniquely identifying the activity via linked to a particular session and /or activity (or identifier(s )
a globally unique identifier (GUID ), where the session / associated therewith ). The data can bemanually linked ( e.g.,
activity relationship is that of a parent/child. by the user) or automatically linked via the application . With
In one embodiment of the present invention , as shown in 50 respect to the latter , this can be accomplished , for example,
FIG . 10 , the biometric data 1000 is stored in memory and by comparing the duration of the received data (e.g. , the
linked to the identifier 802 and a sample rate “ m ” 1104. This video length ) with the duration of the session and/or activity ,
allows the biometric data to be linked to video data upon by assuming that the received data is related to the most
playback . For example , if identifier 802 is one, start time 804 recent session and/or activity, or by analyzing data included
is 1:00 PM , video duration is one minute , and the sample 55 within the received data. For example , in one embodiment,
rate 1104 is 30 spm , then the playing of the video at 2:00 PM data included with the received data (e.g., metadata ) may
would result in the first biometric value (biometric ( 1)) to be identify a time and/or location associated with the data ,
displayed ( e.g., below the video , over the video , etc.) at 2:00 which can then be used to link the received data to the
PM , the second biometric value (biometric (2 )) to be dis session and/ or activity . In another embodiment, the comput
played (e.g. , below the video , over the video , etc.) two 60 ing device could display or play data (e.g., a barcode, such
seconds later, and so on until the video ends at 2:01 PM . as a QR code, a sound , such as a repeating sequence ofnotes ,
While self-realization data can be stored like biometric data etc.) that identifies the session and /or activity . An external
(e.g. , linked to a sample rate ), if such data is only received video /audio recorder could record the identifying data (as
periodically, itmay be more advantageous to store this data displayed or played by the computing device ) along with
110 as shown in FIG . 11, i.e., linked to the identifier 802 and 65 ( e.g. , before , after , or during ) the user and /or his/her sur
a time-stamp 1104 , where " m ” is either the time that the roundings . The application could then search the video /
self- realization data 1100 was received or an offset between audio data for identifying data , and use this data to link the
 US 10,522,188 B2
23 24
video /audio data to a session and /or activity. The identifying length of the session , but it triggers a start time that is stored
portion of the video /audio data could then be deleted by the in memory and linked to a globally unique identifier (GUID )
application if desired . In an alternate embodiment, a barcode for the session . By linking the video and biometric data to
(e.g., a QR code ) could be printed on a physical device ( e.g., the GUID , and linking the GUID to the start time, the video
a medical testing module, which may allow communication 5 and biometric data is also (by definition ) linked to the start
ofmedical data over a network ( e.g., via a smart phone )) and time. Other data , such as sample rate, can also be linked to
used (as previously described ) to synchronize video of the the biometric data , either by linking the data to the biometric
user using the device to data provided by the device . In the data, or linking the data to the GUID , which is in turn linked
case of a medical testing module, the barcode printed on the to the biometric data .
module could be used to synchronize video of the testing to 10 Either before the session is started , or after the session is
the test result provided by the module. In yet another over , the user may be allowed to enter a session name via an
embodiment, both the computing device and the external exemplary " session name” screen 1700, as shown in FIG .
video / audio recorder are used to record video and/or audio 17. Similarly , the user may also be allowed to enter a session
of the user ( e.g., the user stating “ begin Berlin biking description via an exemplary “ session description " screen
session ,” etc.) and to use the user -provided data to link the 15 1800, as shown in FIG . 18 .
video /audio data to a session and /or activity. For example, FIG . 19 shows an exemplary “ session started ” screen
the computing device may be configured to link the user 1900 , which is a screen that the user might see while the
provided data with a particular session and /or activity ( e.g., session is running . On this screen , the user may see the video
one that is started , one that is about to start, one that just data 1902 (if provided in real- time), the biometric data 1904
ended , etc.), and to use the user-provided data in the video / 20 ( if provided in real- time), and the current running time of the
audio data to link the video /audio data to the particular session 1906. If the user wishes to pause the session, the user
session and/or activity . can press the “ pause session " button 1908 , or if the user
In one embodiment of the present invention , the client wishes to stop the session , the user can press the “ stop
platform (or application ) is configured to operate on a smart session ” button (not shown). By pressing the “ stop session "
phone or a tablet. The platform (either alone or together with 25 button (not shown ), the session is ended , and a stop time is
software operating on the host device )may be configured to stored in memory and linked to the session GUID . Alterna
create a session , receive video and non - video data during the tively , by pressing the “ pause session ” button 1908 , a pause
session , and playback video data together (synchronized ) time (first pause time) is stored in memory and linked to the
with non -video data . The platform may also allow a user to session GUID . Once paused , the session can then be
search for a session , search for certain video and /or non- 30 resumed (e.g., by pressing the “ resume session ” button , not
video events, and/or create a highlight reel. FIGS. 15-29 shown ), which will result in a resume time ( first resume
show exemplary screen shots of such a platform . time) to be stored in memory and linked to the session
For example, FIG . 15 shows an exemplary “ sign in " GUID . Regardless of whether a session is started and
screen 1500 , allowing a user to sign into the application and stopped ( i.e. , resulting in a single continuous video ), or
have access to application -related , user -specific data , as 35 started , paused (any number of times), resumed (any number
stored on the computing device and / or the host computing of times ), and stopped ( i.e. , resulting in a plurality of video
device. The login may involve a user ID and password clips ), for each start/pause time stored in memory, there
unique to the application , the company cloud , or a social should be a corresponding stop /resume time stored in
service website , such as FacebookTM . memory .
Once the user is signed in , the user may be allowed to 40 Once a session has been stopped , it can be reviewed via
create a session via an exemplary “ create session ” screen an exemplary “ review session ” screen 2000 , as shown in
1600, as shown in FIG . 16. In creating a session , the user FIG . 20. In its simplest form , the review screen may
may be allowed to select a camera (e.g., internal to the playback video data linked to the session ( e.g., either a
computing device, external to the computing device (e.g., single continuous video if the session does not include at
accessible via the Internet, connected to the computing 45 least one pause/resume,multiple video clips played one after
device via a wired or wireless connection ), etc.) that will be another if the session includes at least one pause /resume, or
providing video data . Once a camera is selected , video data multiple video clips played together if the multiple video
1602 from the camera may be displayed on the screen . The clips are related to one another (e.g., two videos (e.g., from
user may also be allowed to select a biometric device ( e.g., different vantage points ) of the user performing a particular
internal to the computing device , external to the computing 50 activity , a first video of the user performing a particular
device (e.g., accessible via the Internet, connected to the activity while viewing a second video , such as a training
computing device via a wired or wireless connection ), etc.) video ). If the user wants to see non -video data displayed
that will be providing biometric data . Once a biometric along with the video data , the user can press the “ show graph
device is selected , biometric data 1604 from the biometric options” button 2022. By pressing this button , the user is
device may be displayed on the screen . The user can then 55 presented with an exemplary “ graph display option ” screen
start the session by clicking the “ start session ” button 1608 . 2100, as shown in FIG . 21.Here, the user can select data that
While the selection process is preferably performed before he/ she would like to see along with the video data, such as
the session is started , the user may defer selection of the biometric data (e.g., heart rate , heart rate variance, user
camera and/ or biometric device until after the session is speed, etc.), environmental data (e.g., temperature, altitude,
over . This allows the application to receive data that is not 60 GPS , etc.), or self -realization data (e.g. , how the user felt
available in real-time, or is being provided by a device that during the session ). FIG . 22 shows an exemplary “ review
is not yet connected to the computing device (e.g., an session ” screen 2000 that includes both video data 2202 and
external camera that will be plugged into the computing biometric data , which may be shown in graph form 2204 or
device once the session is over). written form 2206. If more than one individual can be seen
It should be appreciated that in a preferred embodiment of 65 in the video , the application may be configured to show
the present invention , clicking the “ start session ” button biometric data on each individual, either at one time, or as
1608 not only starts a timer 1606 that indicates a current selected by the user (e.g., allowing the user to view biomet
 US 10,522,188 B2
25 26
ric data on a first individual by selecting the first individual, said start time and a sample rate to link individual ones of
allowing the user to view biometric data on a second said plurality of values and individual ones of said plurality
individual by selecting the second individual, etc.). of other values to individual times within said period of
FIG . 23 shows an exemplary “ map ” screen 2300 , which time; determining a threshold decibel level associated with
may be used to show GPS data to the user . Alternatively , 5 said biometric event, said threshold decibel level being a
GPS data can be presented together with the video data (e.g., level above a normal decibel level, wherein at least one
below the video data , over the video data , etc.). An exem program is used to monitor said audio data to determine said
plary “ summary ” screen 2400 of the session may also be normal decibel level and determine said threshold decibel
presented to the user (see FIG . 24 ), displaying session
information such as session name, session description, vari- 10 level
at
therefrom ; using said at least one program to identify
least one of said plurality of values corresponding to one
ous metrics , etc.
By storing video and non - video data separately, the data said at leastand
of reaching exceeding said threshold decibel level; using
can easily be searched . For example, FIG . 25 shows an rate to identify atprogram
one
least
, said start time, and said sample
one time within said period of time
exemplary " biometric search ” screen 2500 , where a user can
search for a particular biometric value or range ( i.e., a 15 and
corresponding to said at least one of said plurality ofvalues;
displaying at least said video data during said at least
biometric event). By way of example , the user may want to
jump to a point in the session where their heart rate is one time along with at least one of said plurality of values
between 95 and 105 beats -per-minute (bpm ). FIG . 26 shows received during said at least one time, wherein said at least
an exemplary “ first result” screen 2600 where the user's one time is used to show said at least one of said plurality
heart rate is at 100.46 bmp twenty minutes and forty -two 20 of other values in synchronization with a portion of said
seconds into the session ( see, e.g., 2608 ). FIG . 27 shows an video data that shows at least said user during said one of
exemplary " second result ” screen 2700 where the user's said reaching and exceeding said threshold decibel level.
heart rate is at 100.48 bmp twenty - three minutes and forty 2. The method of claim 1, wherein said step of receiving
eight seconds into the session (see, e.g., 2708 ). It should be said biometric data from said at least one sensor further
appreciated that other events can be searched for in a 25 comprises receiving heart rate data from a heart rate monitor.
session , including video events and self -realization events. 3. The method of claim 1, wherein said step of receiving
Not only can data within a session be searched ,but so too said biometric data from said at least one sensor further
can data from multiple sessions. For example, FIG . 28 comprises receiving oxygen saturation data from a pulse
shows an exemplary “ session search ” screen 2800 , where a oximeter.
user can enter particular search criteria , including session 30 4. The method of claim 1, wherein said step of receiving
date , session length , biometric events , video event, self said biometric data from said at least one sensor further
realization event, etc. FIG . 29 shows an exemplary " list" comprises receiving heart activity data on said user from an
screen 2900 , showing sessions that me the entered criteria . EKG device .
The foregoing description of a system and method for 5. The method of claim 1, wherein said step of receiving
using , processing, and displaying biometric data , or a resul- 35 said biometric data from said at least one sensor further
tant thereof, has been presented for the purposes of illustra comprises receiving brain activity data on said user.
tion and description . It is not intended to be exhaustive or to 6. The method of claim 1, wherein said step of receiving
limit the invention to the precise forms disclosed , and many said video data from said camera and said audio data from
modifications and variations are possible in light of the said microphone further comprises receiving said video and
above teachings. Those skilled in the art will appreciate that 40 audio data from a camera feature and a microphone feature,
there are a number of ways to implement the foregoing respectively , on a mobile computing device .
features, and that the present invention it not limited to any 7. The method of claim 6 , wherein said step of using said
particular way of implementing these features. The inven at least one program to identify at least one of said plurality
tion is solely defined by the following claims. of values further comprises using a program operating on a
What is claimed is : 45 computing device to identify at least one of said plurality of
1. A method for identifying video corresponding to a values corresponding to one of reaching and exceeding said
biometric event of a user to detect a medical condition of threshold decibel level, wherein said computing device is in
said user, said video being displayed along with at least one communication with said mobile computing device via a
biometric of said user during said biometric event, compris wide area network (WAN ).
ing: assigning a session number to a session , said session 50 8. The method of claim 7 , wherein said step of displaying
starting at a start time; receiving video data from a camera at least said video data during said at least one time further
and audio data from a microphone, said video data including comprises displaying at least said video data during said at
video of at least said user during a period of time, said audio least one time on at least one of said computing device and
data including audio of at least said user during said period a remote device in communication with said computing
of time, said period of time starting at said start time; storing 55 device via said WAN .
a plurality of values on an audible breathing level of said 9. The method of claim 1 , wherein said step of displaying
user during said period of time, wherein said plurality of at least said video data during said at least one time further
values are acquired from said audio data ; receiving biomet comprises skipping over at least portions of said video data
ric data from at least one sensor, said at least one sensor between said start time and said at least one time.
being attached to said user and used to acquire said biomet- 60 10. The method of claim 9 , wherein said step of display
ric data during said period of time; storing a plurality of ing at least said video data during said at least one time
other values on at leastone biometric of said user during said further comprises displaying at least said video data imme
period of time, wherein said plurality of other values are diately before said video data during said at least one time,
acquired from said biometric data ; using at least said session thereby showing said user immediately before said one of
number to link said plurality of values on said audible 65 reaching and exceeding said threshold decibel level and
breathing level and said plurality of other values on said at during said one of reaching and exceeding said threshold
least one biometric to at least said video data ; using at least decibel level.
 US 10,522,188 B2
27 28
11. The method of claim 1 , wherein said step of receiving least one of said plurality of other values in synchronization
said biometric data from said at least one sensor is per with a portion of said video data that shows at least said user
formed either in real- time, during said period of time, or during said one of said reaching and exceeding said thresh
indirectly and after said video and audio data has been old decibel level.
received , after said period of time. 5 17. The system of claim 16 , wherein said step of receiving
12. The method of claim 6 , wherein said step of using said biometric data from said at least one sensor comprises
at least one program to identify at least one of said plurality receiving heart rate data from a heart rate monitor.
of values further comprises using artificial intelligence (AI) 18. The system of claim 16 , wherein said step ofreceiving
operating on said mobile computing device to automatically biometric data from said at least one sensor comprises
identify at least one of said plurality of values corresponding 10 receiving data on oxygen levels of said user from a pulse
to one of reaching and exceeding said threshold decibel oximeter.
level. 19. The system ofclaim 16 , wherein said step ofreceiving
13. The method of claim 1 , wherein said video corre biometric data from said at least one sensor comprises
sponding to said biometric event is used to detect a sleeping receiving heart activity data from an EKG device .
disorder . 15 20. The system of claim 16 , wherein said step ofreceiving
14. The method of claim 7, further comprising the step of biometric data from said at least one sensor comprises
streaming at least said audio and video data to said com receiving brain activity data from an EEG device .
puting device via said WAN in real- time, during said period 21. The system of claim 16 , wherein at least a portion of
of time. said second set of machine readable instructions are pro
15. The method of claim 12 , wherein said step of dis- 20 vided from said at least one server to said computing device
playing at least said video data during said at least one time via said WAN .
comprises displaying said video data during said at least one 22. The system of claim 16 , wherein said second set of
time on a display portion of said mobile computing device. machine readable instructions are operating on one of said at
16. A system for identifying video corresponding to a least one server and a second computing device in commu
biometric event of a user to detect a medical condition of 25 nication with said at least one server.
said user, said video being displayed along with at least one 23. The system of claim 16 , wherein at least a portion of
biometric of said user during said biometric event, compris said second set of machine readable Instructions are pro
ing : at least one server in communication with a wide area vided from said at least one server to a second computing
network (WAN ); and at least onememory device for storing device via said WAN , thereby allowing said second com
machine readable instructions, at least a first set of machine 30 puting device to perform at least said step of displaying at
readable instructions being provided to a computing device least said video data during said at least one time along with
via said at least one server and said WAN , said first set of said at least one of said plurality of other values received
machine readable instructions being adapted operate on during said at least one time.
said computing device and perform the steps of; receiving 24. The system of claim 16 , wherein said second set of
video data from a camera and audio data from a microphone , 35 machine readable instructions are further adapted to skip at
said video data including video of, and said audio data least portions of said video data between said start time and
including audio of, at least said user during a period of time, said at least one time.
said period of time beginning at start time; storing a 25. The system of claim 16 , wherein said step of receiving
plurality of values on an audible breathing level of said user said biometric data from said at least one sensor is per
during said period of time, wherein said plurality of values 40 formed in real -time, during said period of time.
are acquired from said audio data; receiving biometric data 26. The system of claim 16 , wherein said step ofreceiving
from at least one sensor, said at least one sensor being said biometric data from said at least one sensor is per
connected to said user and used to acquire said biometric formed indirectly , via an intermediary device , and after said
data during said period of time; and storing a plurality of period of time.
other values on at least one biometric of said user during said 45 27. The system of claim 16 , wherein said video corre
period of time, wherein said plurality of other values are sponding to said biometric event is used to detect a sleep
acquired from said biometric data; wherein a second set of apnea condition .
machine readable instructions are adapted to perform the 28. The system of claim 23 , further comprising the step of
steps of: using at least a session identifier to link said providing at least said video data to said second computing
plurality of values on said audible breathing level and said 50 device via said WAN during said period of time, allowing
plurality of other values on said at least one biometric to at said video data to be remotely monitored in substantially
least said video data ; using at least said start time and a real- time.
sample rate to link individual ones ofsaid plurality of values 29. The system of claim 16 , wherein at least said steps of
and individual ones of said plurality of other values to (i) monitoring said audio data to determine a normal decibel
individual times within said period of time; monitoring said 55 level, (ii) determining a threshold decibel level associated
audio data to determine a normal decibel level; determining with said biometric even , (iii) identifying at least one of said
a threshold decibel level associated with said biometric plurality of values corresponding to one of reaching and
event, said threshold decibel level being a level above said exceeding said threshold decibel level, (iv ) using at least
normal decibel level; identifying at least one of said plurality said start time and said sample rate to identify at least one
of values corresponding to one of reaching and exceeding 60 time within said period of time corresponding to said at least
said threshold decibel level; using at least said start time and one of said plurality of values, and (v ) displaying at least
said sample rate to identify at least one time within said said video data during said at least one time along with at
period of time corresponding to said at least one of said least one of said plurality of values received during said at
plurality of values; and displaying at least said video data least one time, are performed automatically using artificial
during said at least one time along with at least one of said 65 intelligence ( AI).
plurality of other values received during said at least one 30. A system for displaying video in synchronization with
time, wherein said at least one time is used to show said at at least one biometric of a subject for medical diagnosis,
 US 10,522,188 B2
29 30
comprising : at least one server in communication with a one biometric , and said video data ; using at least said start
wide area network (WAN ) ; and at least one memory device time and a sample rate to link individual ones of said
for storing machine readable instructions, at least a first set plurality of values and individual ones of said plurality of
other values to individual times within said period of time;
ofmachine readable instructions being provided to a mobile monitoring said audio data to determine a normal decibel
device via said at least one server and said WAN , said first 5 level; determining
set ofmachine readable instructions being adapted to per said biometric event a threshold decibel level associated with
form the steps of; receiving video data from a camera and level above said normal , said threshold decibel level being a
audio data from a microphone , said video data including one of said plurality of decibel level ; identifying at least
values corresponding to one of
video of said subject during a period of time and said audio 10 reaching and exceeding said threshold decibel level ; using at
data including audio of at least said subject during said least said start time and said sample rate to identify at least
period of time; extracting a plurality of values from said one time within said period of time corresponding to said at
audio data , said plurality of values corresponding to an least one of said plurality of values, and displaying on a
audible breathing level of said subject; and receiving bio computing device in communication with said at least one
metric data from at least one sensor during said period of 15 server via said WAN at least said video data during said at
time, said biometric data including a plurality ofother values least one time along with at least one of said plurality of
on at least one biometric of said user during said period of other values received during said at least one time, wherein
time; wherein a second set ofmachine readable instructions at least portions of said video data between said start time
are adapted to perform the steps of: using a session identifier and said at least one time are skipped over prior to display
to link at least said plurality of values on said audible ing said video data during said at least one time.
breathing level, said plurality of other values on said at least *