Proceedings of the 53rd Hawaii International Conference on System Sciences | 2020

A Digital Twin City Model for Age-Friendly Communities: Capturing

Environmental Distress from Multimodal Sensory Data
Changbum R. Ahn Youngjib Ham Jinwoo Kim Jaeyoon Kim
Texas A&M University Texas A&M University Texas A&M University Texas A&M University
ryanahn@tamu.edu yham@tamu.edu jwkim@tamu.edu skybluego@tamu.edu

Abstract beyond the home; traditional urban planning/design

practices do not adequately consider the special needs
As the worldwide population is aging, the demands of older adults experiencing substantial physical and
of aging-in-place are also increasing and require cognitive limitations [4], and overall maintenance
smarter and more connected cities to keep mobility conditions of America’s aging urban infrastructure
independence of older adults. However, today’s aging (e.g., streets, transit systems, buildings) continue to
built environment often poses great environmental worsen [5]. Such an urban built environment often
demands to older adults’ mobility and causes their presents great physical and/or cognitive demands to
distresses. To better understand and help mitigating older adults’ mobility and causes older adults’
older adults’ distress in their daily trips, this paper physical and emotional distress, thereby inhibiting
proposes constructing the digital twin city (DTC) their mobility. Therefore, assessing urban built
model that integrates multimodal data (i.e., environment from older adults’ perspectives would be
physiological sensing, visual sensing) on the critical first step in fostering age-friendly
environmental demands in urban communities, so that communities that protect mobility, well-being, and
such environmental demands can be considered in health of older adults.
mobility planning of older adults. Specifically, this One promising approach is to exploit new sources
paper examines how data acquired from various of urban data emerging from technological, social, and
modalities (i.e., electrodermal activity, gait patterns, business innovations. For example, wearable sensing
visual sensing) can portray environmental demands data crowdsourced from urban residents’ wearable
associated with older adults’ mobility. In addition, it devices could provide direct observations on how
discusses the challenges and opportunities of residents feel and respond to the environmental
multimodal data fusion in capturing environmental demand of urban communities [6]. In addition, visual
distresses in urban communities. sensing data on urban built environment (e.g., Google
Street view, crowdsourced photos) could contain key
information on environmental stressors in urban
1. Introduction communities. Such multimodal data on environmental
distress associated with older adults’ mobility can be
Population aging is a global phenomenon. There then integrated into the Digital Twin City (DTC)
will be more elderly people than young children in the model, in order to provide spatiotemporal patterns of
global population in 2035, and older adults in the US environmental distress associated with older adults’
will be over 20% of the entire US population in 2050 mobility. In this context, this paper aims to examine
[1]. A majority of these elderly people want to live in how data acquired from diverse modalities can be
their present communities as long as possible [2]. Such analyzed to represent environmental demands
aging-in-place goals enable older adults to maintain associated with older adults’ mobility and can be
their independence, autonomy, and connection to integrated into a virtual city model. In addition, this
social support. It offers significant social and paper will discuss challenges and opportunities in
economic advantages (e.g., a low rate of clinical fusing such multimodal data, by using experimental
depression and obesity) at the individual and societal data collected in the Bryan downtown, Texas.
levels [3]. These goals demand smarter and more
connected cities for their independent mobility and 2. Research background
healthy aging.
However, there is a growing concern about the 2.1 Mobility of older adults
inclusiveness and quality of urban built environment

URI: https://hdl.handle.net/10125/63945
978-0-9981331-3-3 Page 1675
(CC BY-NC-ND 4.0)
 Older adults’ mobility plays an important role in real-time. Similarly, the DTC models of Singapore
their physical well-being, health behaviors, mental have been used by Singapore’s security forces to
wellness, and satisfaction, and has an impact on public simulate terrorists’ attack at a sports stadium in order
health [7]. Many studies indicate that older adults’ to mitigate potential risk [17]. The DTC model could
mobility is closely related to their physical activity [7], be combined with healthcare systems to monitor,
social engagement [8], mental disorders [9], nutrition diagnose, and predict the health of older adults by
(e.g., consumption of fruit and vegetables) [10], and integrating the medical physical and virtual spaces; it
access to medical services [11]. Protecting and can provide diverse application scenarios such as real-
enhancing older adults’ mobility is, therefore, the time supervision, resource optimization and accurate
critical first step to promote the quality of their lives. crisis warning systems [18]. As such, the real-time
The mobility of an older adult is mainly affected data obtained from monitoring sensors in cities has
by his/her physical capacity and surrounding brought significant potential to improve data-driven
environmental conditions including distance, ambient decision-making for achieving safe and healthy urban
conditions (e.g., weather conditions), terrain built environments.
characteristics (e.g., slope, stairs, and uneven
surfaces), physical disorders (e.g., vandalized building 2.3 Urban sensing data and environmental distress
and litter in the streets), objects or people density in
walking path, and speeding cars [3, 5, 6, 7]. Various New sources of urban data—including
attempts have been made to assess environmental infrastructure-based sensors [19], user-generated data
demands associated with older adults’ mobility. Most [20], and administrative data [21]—are emerging from
of them rely on surveys and interviews from actual technological, social, institutional, and business
users (i.e., older adults) and/or trained inspectors [12]. innovations [22]. These new datasets have stimulated
However, opinion surveys and visual audits are not empirical data-driven research towards bottom-up
free from subjectivity concerns (i.e., inter-rater sensing of the city [23]. Evaluating urban
reliability issues) and cannot provide continuous environments using new urban data to promote
assessment for urban communities where many community residents’ quality of life is one of the areas.
dynamic individual factors (e.g., time of day, traffic, Extensive efforts have already been made using
weather, maintenance conditions) interplay [13]. passively collected location data from a user’s mobile
Therefore, a novel breakthrough by leveraging new phone [24], crowdsourced self-reports on
sources of urban data in understanding the interaction environmental issues [25], administrative data (e.g.,
between older adults and surrounding urban built New York City’s 311 complaint data) [21], and video
environment is necessary to enable the continuous and image data (e.g., Google Street view) [26].
assessment and create smarter and more connected One of the promising data sources to better
cities for age-friendly communities. understand human experience in the urban built
environment would be physiological responses (e.g.,
2.2 Digital twin city (DTC) electrodermal activity (EDA), gait patterns, blood
volume pulse) captured from wearable devices of
The term ‘Digital Twin’ can be defined as a older adults. Such physiological responses could
dynamic digital representation which mirrors and provide information regarding how individuals feel
simulates a physical system to help organize and share and respond to environmental demands, including
data for informing better decision-making in the fight-or-flight responses to threatening stimuli.
system [14]. To facilitate decision-making regarding Chrisinger and King [25], for example, attempted to
the complex systems of urban built environments, capture the stress experiences of pedestrians in urban
many cities have created and leveraged a DTC model built environments using EDA signals. This research
based on real-time data from the diverse Internet of investigated the utility of physiological signals by
Things (IoT) sensors to bridge the gap between real examining the relationships between subjective
and virtual world [15]. For example, the ‘Virtual evaluation forms (e.g., walkability survey) and
Rennes’ project by the city of Rennes has created the physiological signals [26]. Duchowny et al. [27]
data-rich DTC model to support urban planning and examines the usefulness of exploiting gait speed and
management in the context of the growing population, stride length to identify the influence of an
energy consumption, and environmental issues [16]. environmental demands for mobility. They found
In particular, for achieving safe and healthy urban variability in accordance with physiological responses
built environment, ‘Virtual Singapore’ project has in high-demand environments (e.g., absence of traffic
created the DTC model informed by IoT sensor-based signals and sidewalk defects). Also, our previous study
dynamic data to analyze noise and pollution level in [6] highlighted that collective levels of gait stability

Page 1676
and relative heart rates captured from pedestrians in signal segmentation; (2) extraction of physiological
naturalistic ambulatory settings can be indicative of features; (3) calculation of physiological saliency cue
adverse built environmental features that harm (PSC); and (4) PSC aggregation across individuals.
neighborhood walkability. Segmentation is the first step to extract sub-
There have been few recent studies on vision- features from physiological signals. Using fixed-
based methods to measure urban perception from length segmentation might not be efficient because
images for analyzing changes in the physical signal changes might occur within the analysis
appearances in scenes [28,29]. These studies built on segment [33]. Specifically, different physiological
the prior works dedicated to analyzing the aesthetic signals have different latencies, and each
aspects of visual data using generic image features environmental distress has different magnitude and
such as color, texture, and SIFT [30-32]. To identify duration of physiological responses [33]. To address
the correlation between visual features and the these challenges, a non-fixed-length approach is
perceived safety of a Google Street View image, Naik needed, so physiological signals are divided by
et al. [29] proposed a scene understanding algorithm segments in a data-driven way. Specifically, a bottom-
to rate a perceived safety score, called Streetscore, up segmentation is used. It starts with identifying
using training data collected from an online survey many possible change points and continuously
with the contributions of participants. removes less significant ones [38]. The entire signal is
While these studies highlight opportunities to partitioned into smaller sub-signals, and then near
leverage new urban data to capture environmental segments are successively combined by computing
distress, it has not been fully examined how such data similarities between segments. Fig. 1 indicates an
can identify environmental distress to a specific example of the bottom-up segmentation of the EDA
population (e.g., older adults). More importantly, signal captured in naturalistic ambulatory settings. It
research into whether and how the fusion of shows the usefulness of bottom-up segmentation
multimodal data on urban communities generates which captures the signal’s appropriate change points
added value in capturing environmental distress to to determine distinct segments. The 9 distinct parts
older adults is uncharted territory. were partitioned in the EDA signal using bottom-up
segmentation. Each part is visually different from their
near neighbors. For example, the first segment in Fig.
3. Capturing environmental distress from 1 has low EDA signal values compared to the higher
urban sensing data EDA signal values in the second segment. Thus, the
bottom-up segmentation was used for both EDA and
gait patterns.
This section discusses a methodology to capture
distress (or perceptual distress) from multimodal data
sources. In particular, this study focuses on (1)
physiological response data captured from older
adults’ wearable devices, including EDA and gait
patterns, and (2) visual sensing data available from
various sources (e.g., crowdsourced, Google Street
view).

3.1 Capturing environmental distress from

wearable sensing data

Although pedestrian’s physiological signals have a Figure 1. An example of bottom-up

great potential to understand the effect of the urban segmentation on Electrodermal activity
built environment, it cannot be directly used in raw
data level due to many technical and technological Secondly, features from each physiological signal
challenges (e.g., motion artifacts and electrode contact (i.e. EDA, gait patterns) were extracted within each
noise) [33-36]. Thus, this section will describe an segment which is identified by bottom-up
alternative data processing method to effectively segmentation. EDA’s features include average skin
capture environmental distress from physiological conductance response (SCR) amplitude, SCR
signals. Additionally, the data processing way was frequency, and mean skin conductance levels (SCL)
introduced in the author’s previous research in detail which are commonly used to portrait emotional
[37]. The proposed method includes (1) physiological arousal [39]. Mean SCL is measured as the mean level

Page 1677
of the EDA signal [39]. Mean SCR amplitude With these backgrounds, negative skewness is
calculates the SCRs’ mean amplitude in the time exploited to compute collective PSC values across
window, and SCR frequency is the number of SCRs multiple subjects in one location.
partitioned by the length of the time segment [39]. To
assess gait patterns, spatiotemporal stride time (ST) is
used as a feature. ST is the time span for one heel-
strike event and is widely used in ambulatory gait
analysis [40]. Specifically, ST is calculated by
determining the standard deviation in each segment’s
gait cycle duration to identify the intensity of the
stride-to-stride fluctuations [40].
Thirdly, ‘Physiological Salience Cue’ (PSC) was
proposed and used to calculate the distinctiveness of
one segment in compared to others. The PSC
calculation is motivated by an image’s contrast
calculation in the technique of computer vision [41].
The PSC of segment 𝑗 for participant 𝑖 is described by Figure 2. Histogram showing the distribution
the signal comparison between all other segments of of physiological saliency cues across all
participant 𝑖 as follows: participants; (a) negative skewness value in
collective EDA PSC equal to - 1.58; and (b) a
negative skewness value in collective EDA
PSC equal to + 0.51
(1)
3.2 Assessing perceptual distress from visual
where u is physiological for segment 𝑗, 𝑇 is the time sensing data
duration of the entire signal for 𝑖 (𝑖=1,…,N), 𝑡 is the
time duration of segment 𝑗, and a is the number of To assess the human perception on scenes that may
segments for participant 𝑖. Our previous research cause physical and emotional distress, images are
highlights the detailed operation process of the PSC ranked through a pairwise comparison. In this paper,
equation [37, 42]. The calculated PSC values are participants compare a pair of street-level images to
normalized in each participant, and then PSC values determine which scene looks more stressful or
are computed and aggregated to extract each sub- uncomfortable to walk. The outcomes of the pairwise
feature. The aggregation of EDA PSC values is comparison are then converted into a ranked score for
presented by equation 2, and PSC values for gait each image through the Bayesian graphical model
patterns will be denoted as 𝑐 [43]. The ranked score does not have a universal unit
as every entity is given a unitless number, and thus that
is only effective to compare with others relatively, not
deterministically quantify the value [44]. For example,
(2) the Microsoft Trueskill used this concept for rating
players competing in online games [45, 46] In this
Lastly, since the physiological response of an paper, building on the Bayesian inference, visual
individual can be affected by their momentary actions sensing data (i.e., image) is the entity to assess the rank
and/or physiological reactivity, collective analysis of compared with others for understanding the human
physiological signals across many individuals is perception on given scenes in terms of physical and
important to reliably capture environmental distress in emotional distress, and ultimately predicting that of
each location. For that, histograms of the PSC values unexplored paths. In order to rank images through
are composed for all participants to afford summative pairwise comparison matches, the visual appearance
information. The diverse histograms indicate different of streetscapes related with a behavior and health of
concentration patterns. Specifically, a certain location pedestrians can be collected by crowdsourcing in large
with high PSC values illustrates a highly right-skewed city scale [28, 29]. Among diverse matches such as
distribution with negative skewness (See Fig. 2). Fig. Free-for-All and team games with the different
2-b indicates that many individuals presented high number of players, our experiments were conducted
PSC values in the location, so its histogram shows on 1 vs 1 match (i.e., pairwise comparison). Thus, the
skewed towards the right. Therefore, it has a higher
negative skewness value (0.51) then Fig. 2a (-1.58).

Page 1678
algorithm was able to be simplified based on the
relationship between two-players. Here, the function 𝑣(𝜃) and 𝑤(𝜃) are defined by the
To cope with the uncertainty in the pairwise Normal distribution function 𝑁(𝜃) and Cumulative
comparison, the score of each image is modeled in the distribution function Ф(𝜃).
form of a Gaussian distribution (i.e., a normal
distribution) composed of mean μ (i.e., peak point) and
standard deviation σ rather than a single fixed score 4. Integrating environmental distress into
for each image. In this paper, μ represents the average digital twin city: case study
score of the image; σ represents the degree of
uncertainty in the image score. Since the gaussian Environmental distress captured from multimodal
distribution is characterized that 99.7% of the data are data sources can be integrated into a DTC model, so
within three times of σ, it was assumed that the initial that older adults consider such environmental demand
μ is three times greater than the initial σ in the ranking in their mobility planning. A case study was conducted
system. During multiple two-image matches, μ and σ to demonstrate the proposed approach to capture
value are iteratively updated: σ value is going to keep environmental distress and integrate into a DTC
lower in conducting every pairwise comparison; the μ model.
of selected image (the winner) is increased, and the μ
of not-selected image (the loser) is decreased at every 4.1 Experiment and data collection
match. Once the pairwise comparison between image
A and B is finished, the μ, σ is updated building on
Field experiments were performed in the Bryan
[45], for example, in which A wins against B:
downtown in Texas to collect data in ambulatory
settings. The experiment was performed from June 1st
σ μ −μ 𝜖 to 2, 2019 between 8 am to 11 am, and the average
μ ←μ + ∙𝑣 , (3)
𝑐 𝑐 𝑐 temperature was 82.99 Fahrenheit (28.33 Celsius).
σ μ −μ 𝜖 Nine older adults (over 65 years old) were recruited,
μ ←μ − ∙𝑣 , (4)
𝑐 𝑐 𝑐 and EDA and IMU data from the participants were
𝑐 = 2𝛽 + σ +σ (5) collected while they walked on the pre-defined path
with the preferred walking speeds. Wristband-type
Where μ is the mean value of selected image, μ is sensors (Empatica E4) and commercial IMUs (Opal,
the mean value of not-selected image, σ is the APDM Inc.) were used for data collection. In addition,
standard deviation of selected image, σ is the Global Positioning System (GPS) data was collected
standard deviation of not-selected image, 𝛽 is the by a smartphone across all participants. EDA and IMU
uncertainty due to the performance variation, 𝜖 is the data were sampled at the 4Hz, 125Hz, respectively,
draw margin. The configurable constant β denotes an and were coordinated with GPS. The Bateaman low-
uncertainty due to the performance variation, which pass filter of 24 sample lengths was used to smooth the
indicates that 𝛽 is the additional score above other EDA signal, and Butterworth low-pass filter with a
images to identify an 80% probability of win against cut-off 4Hz was used to remove high-frequency noise
the others [43]. ε is an empirically estimated value in the IMU data.
representing the size of draw margin which depends The total distance of the experiment was 1,322.88
on the probability of draw obtained from empirical ft (403.21 m). Fig. 3 provides visual evidence of
tests. The draw margin is a range where the terrain rendering of the experiment. In order to
performance of images is assumed to be equivalent recognize the correlation between (1) distress
even though their values are slightly different. By continually captured from physiological responses
equation (3) and (4), every image has its own mean during the experiment and (2) perceptual distress
value based on the outcome of the pairwise sparsely measured from visual sensing data (30
comparison. Accordingly, the standard deviation of images) at a few specific locations, the entire walking
each image is changed as follows: path to experiment was divided into 18 POIs
(segments) of equal length. Thus, some POIs may
contain multiple images; an image which has the
σ μ −μ 𝜖
σ ←σ ∙ 1− ∙𝑤 , (6) lowest score among images was selected. All POIs
𝑐 𝑐 𝑐 were used to align participants’ PSC values in one POI
σ μ −μ 𝜖 and image scores. Therefore, these POIs which
σ ←σ ∙ 1− ∙𝑤 , (7) indicate physical locations are different from the
𝑐 𝑐 𝑐
segments by bottom up segmentation.

Page 1679
 Several environmental stressors were identified by PSC values of EDA and gait patterns were
trained inspectors’ assessment and the interviews of calculated using physiological sensing data. Fig. 4
participants and included: partially broken wall (POI shows raw physiological signals and PSC values of
5), dumpster (POI 6 and 16), uneven sidewalk (POI 8) one subject. As shown in Fig. 4a, the bottom-up
(d) dead animal (POI 11), (e) blocked sidewalk by a segmentation effectively captures the change points of
car (POI 13), and (f) dead branches and leaves EDA signals, and the resulting PSC values portrait the
overhanging the sidewalk (POI 14). prominent local patterns. Specifically, the locations
which present prominent PSC values include many
environmental stressors, such as dumpster (112 to 125
seconds), uneven sidewalk (140 to 150 seconds), and
blocked sidewalk by a car (197 to 208 seconds). With
respect to gait patterns, PSC values clearly portray
distinct patterns around gait cycles 1–118 and 150 to
200, as shown in Fig. 4b. These locations coincide
with the POIs containing dead animals, blocked
sidewalk by a car, and dead branches and leaves
overhanging the sidewalk.

Figure 3. Terrain rendering and

environmental distress; (a) partially broken
wall, (b) dumpster, (c) uneven sidewalk, (d)
dead animals, (e) blocked sidewalk by a car,
and (f) dead branches and leaves
overhanging the sidewalk

While conducting field experiments to collect

EDA and IMU data, a total of 18 geocoded photos
were collected along the pre-defined path by using a
smartphone (Samsung Galaxy J7). For the pairwise
comparison experiments of the collected images, eight
participants in 20s to 30s who did not participate in
field experiments were instructed to be aware of the
question: “which place looks more stressful or Figure 4. Comparison graph of raw
uncomfortable to walk?”. To converge the image physiological signal and physiological
score for ranking photos, the participants conducted saliency cues; (a) EDA, (b) gait patterns
totally 1740 pairwise comparisons including all
possible pairs. Each participant did 217 or 218 In addition, perceptual distress of photos taken in
pairwise comparisons respectively to converge the the walking path was measured through the pairwise
image score for understanding a level of perceptual comparisons, as shown in Fig. 5a. In our experiment
distress of each image. The computation time to as shown in Fig. 5b, the pairwise comparisons were
determine the rank was 0.739s on a computer with the conducted by the eight participants described in
configuration of the Intel i7-8750H CPU and the GTX section 4.1; the μ of each image was eventually
1070 8GB GPU. In the proposed approach, the latest converged around 80 comparisons per image
score of each image will be only ranked so that the old approximately.
one will be replaced with the updated one. Here, high PSC values of each segment and ranks of visual
rank of an image indicates that the scene looks safe sensing data were mapped to POIs using GPS
and comfortable to walk. coordinates, and PSC values of EDA and gait patterns
across all the subjects were aggregated in each POI
4.2 Assessment results of environmental distress using the skewness measure. Fig. 6a and 6b indicates
using physiological signals and visual sensing data that collective PSC values of EDA and gait patterns
are correlated with the locations where various

Page 1680
environmental stressors exist. For example, high PSC sidewalk, and dumpster) captured by pedestrians in a
values of EDA are found in the locations with DTC model. In addition, Fig. 6e illustrates the result
dumpsters (POI 6), blocked sidewalk by a car (POI of EDA, gait pattern, and the image pairwise
13), and dead branches and leaves overhanging the comparison at each segment in the DTC model. Using
sidewalk (POI 14), and dead branches and leaves GPS coordinates of the center of each POI, (1)
overhanging the sidewalk (POI 14), and collective normalized PSC values of EDA, gait patterns and (2)
PSC values of gait patterns (see Fig. 6b) also present normalized image score were visualized by using
high skewness values in several POIs with broken wall different color and size of spheres along with the
(POI 5), uneven sidewalk (POI 8), and dead branches predefined path. The size of the spheres represents the
and leaves overhanging the sidewalk (POI 14). value of each parameter (i.e., normalized PSC values
and image scores) at each POI to help users recognize
the correlation between distress captured from
physiological responses and perceptual distress
measured from visual sensing data. By leveraging
smartphone-based physiological signals and
crowdsourced visual sensing data reported by
pedestrians in the future research, DTC model
effectively enables older adults to plan their daily trips
based on potential environmental distress in a virtual
environment.

5. Opportunities and challenges of

multimodal data fusion
The case study results highlight the opportunities
to leverage multimodal data in capturing and assessing
environmental distress associated with older adults’
Figure 5. The pairwise image comparisons mobility. Firstly, the results indicate that the effects of
for ranking score using Bayesian graphical several environmental stressors could be manifested in
model; (a) an example of pairwise different modality data sources. For example, the
comparison of scenes, (b) results of pairwise segments, including blocked sidewalk by a car (POI
comparisons and convergence to a 13), and dead branches and leaves overhanging the
consistent score after 80 clicks in our sidewalk (POI 14), created consistent effects on
experiments. feature values from all the three sources (i.e., EDA,
gait pattern, images). Uneven sidewalk (POI 8)
Perceptual distresses captured from visual sensing contributed to high distress captured from gait patterns
data are also correlated with the locations having and perceptual distress on visual sensing data. This
environmental stressors, including uneven sidewalk conforming results from different modalities highlight
(POI 8), dead animal (POI 11), blocked sidewalk by a an opportunity of predicting environmental distress on
car (POI 13), dead branches and leaves overhanging the locations where a single modality data exist (e.g.,
the sidewalk (POI 14), and dumpster (POI 16). only image data is available).
On the other hand, the results indicate that data
4.3 Physiological signals and visual sensing data in obtained from multimodalities could be
3D virtual city model complementary, which would add value in capturing
environmental distress. For example, the highest PSC
Visual sensing data and physiological responses value of EDA was reported at dumpsters (POI 6),
obtained from experiments were fed into a 3D virtual while those dumpsters didn’t create much effects on
city model as shown in Fig. 6e, toward DTC modeling. the PSC value of gait patterns or perceptual distress on
Objects/areas potentially having a negative impact on their image. The odors of dumpsters might have
older adults’ mobility were also imported into the 3D impacted the EDA of the participants, while such
virtual city model through the geospatial localization odors were not visible in their images and did not
building upon [46]. trigger the change of gait patterns (e.g.,
Fig. 6e presents objects that may cause physical
and emotional distress (e.g., dead animal, uneven

Page 1681
 Figure 6. (a) collective PSC for EDA at each POI, (b) collective PSC for gait patterns at each POI,
(c) visual sensing data at each POI, (d) photos corresponding high values of POIs, (e) an
example of reporting environmental distress by multimodal sensing-based geospatial
localization in digital twin city model
avoidance/hesitation actions). These results highlight data captured from various modalities, including
the potential diversity gained from multimodality. physiological sensing data collected from older adults’
Such diversity potentially enhances the overall wearable devices during their daily trips and
performance of capturing stresses and/or the perceptual distress captured from images on as-is
development of the personalized model, compared to environmental conditions, could identify
the approach using a single modality [47]. environmental distress associated with older adults’
However, how to resolve conflicts between mobility.
different modalities still requires further research. For The applications of the proposed DTC model are
example, the image of a dead animal body (POI 11) thus expected to greatly help older adults’ mobility
generated the highest perceptual distress, but it did not planning in a way to identify less-demanding paths
create much effect to PSC values of EDA and gait considering potential environmental distress in a
patterns; it was found that most the subjects were not spatial dimension. However, further efforts are
able to spot the stressor during their experiments. necessary in fusion data obtained from
Likewise, data from different modalities may report on multimodalities, in particular resolving contradicting
different aspects of the urban built environment, results from different modalities.
although those data are collected in one location; this
may be viewed as no commensurability [44], one of 7. Acknowledgements
the key challenges in data fusion. Also, there is still a
great chance of contradicting results from different
modalities, even if data from different modalities. This study was partially supported by the National
deliver observations on the same aspect of the built Science Foundation–United States (#1800310). Any
environment. Future research would be necessary for opinions, findings, conclusions, or recommendations
resolving these issues in leveraging multimodal expressed in this article are those of the authors and do
sensory data. not necessarily reflect the views of the National
Science Foundation.
6. Conclusions
8. References
A concept of the DTC model that captures and
integrates environmental distress associated with older [1] Ortman, J.M., V.A. Velkoff, and H. Hogan, An aging
adults’ mobility is proposed by leveraging multimodal nation: the older population in the United States, United
States Census Bureau, Economics and Statistics
data sources data. A case study to demonstrate the Administration, US Department of Commerce, 2014.
concept of the DTC for age-friendly communities was
conducted using the experimental data collected in
Bryan downtown, Texas. The results highlight that

Page 1682
[2] Keenan, T.A., Home and community preferences of the [14] “Virtual Singapore”, 2019.
45+ population, AARP Research & Strategic Analysis, https://www.nrf.gov.sg/programmes/virtual-singapore
2010.
[15] Mohammadi, N., and J.E. Taylor, “Smart city digital
[3] Wiles, J.L., A. Leibing, N. Guberman, J. Reeve, and twins”, 2017 IEEE Symposium Series on Computational
R.E.S. Allen, “The Meaning of ‘Aging in Place’ to Older Intelligence (SSCI), IEEE (2017), 1–5.
People”, The Gerontologist 52(3), 2012, pp. 357–366.
[16] Coletta, C., L. Evans, L. Heaphy, and R. Kitchin,
[4] King, A.C., and J.M. Guralnik, “Maximizing the Creating Smart Cities, Routledge, 2018.
potential of an aging population”, JAMA 304(17), 2010, pp.
1944–1945. [17] Liu, Y., L. Zhang, Y. Yang, et al., “A novel cloud-
based framework for the elderly healthcare services using
[5] BS, D.A., P.L. PhD, J.P. PhD, T.B. PhD, and I.H.C. digital twin”, IEEE Access 7, 2019, pp. 49088–49101.
MA, “Creating Elder-Friendly Communities”, Journal of
Gerontological Social Work 49(1–2), 2007, pp. 1–18. [18] Liu, J., Y. Li, M. Chen, W. Dong, and D. Jin,
“Software-defined internet of things for smart urban
sensing”, IEEE communications magazine 53(9), 2015, pp.
[6] Kim, J., C.R. Ahn, and Y. Nam, “The influence of built 55–63.
environment features on crowdsourced physiological
responses of pedestrians in neighborhoods”, Computers, [19] Zheng, Y., F. Liu, and H.-P. Hsieh, “U-air: When
Environment and Urban Systems 75, 2019, pp. 161–169. urban air quality inference meets big data”, Proceedings of
the 19th ACM SIGKDD international conference on
[7] Clarke, P., and N.A. Gallagher, “Optimizing Mobility in Knowledge discovery and data mining, ACM (2013),
Later Life: The Role of the Urban Built Environment for 1436–1444
Older Adults Aging in Place”, Journal of Urban Health
90(6), 2013, pp. 997–1009. [20] Tasse, D., and J.I. Hong, “Using user-generated
content to understand cities”, In Seeing Cities Through Big
[8] Rosso, A.L., J.A. Taylor, L.P. Tabb, and Y.L. Michael, Data. Springer, 2017, 49–64.
“Mobility, Disability, and Social Engagement in Older
Adults”, Journal of Aging and Health 25(4), 2013, pp. [21] Zheng, Y., T. Liu, Y. Wang, Y. Zhu, Y. Liu, and E.
617–637. Chang, “Diagnosing New York city’s noises with
ubiquitous data”, Proceedings of the 2014 ACM
International Joint Conference on Pervasive and
[9] Lampinen, P., and E. Heikkinen, “Reduced mobility Ubiquitous Computing, ACM (2014), 715–725.
and physical activity as predictors of depressive symptoms
among community-dwelling older adults: An eight-year [22] Wang, J., C. Wang, X. Song, and V. Raghavan,
follow-up study”, Aging Clinical and Experimental “Automatic intersection and traffic rule detection by
Research 15(3), 2003, pp. 205–211. mining motor-vehicle GPS trajectories”, Computers,
Environment and Urban Systems 64, 2017, pp. 19–29.
[10] Wylie, C., J. Copeman, and S.F.L. Kirk, “Health and
social factors affecting the food choice and nutritional [23] Lathia, N., V. Pejovic, K.K. Rachuri, C. Mascolo, M.
intake of elderly people with restricted mobility”, Journal Musolesi, and P.J. Rentfrow, “Smartphones for large-scale
of Human Nutrition and Dietetics 12(5), 1999, pp. 375– behavior change interventions”, IEEE Pervasive
380. Computing 12(3), 2013, pp. 66–73.

[11] Glass, T.A., and J.L. Balfour, “Neighborhoods, aging, [24] Fan, Z., T. Pei, T. Ma, et al., “Estimation of urban
and functional limitations”, Neighborhoods and health 1, crowd flux based on mobile phone location data: A case
2003, pp. 303–334. study of Beijing, China”, Computers, Environment and
Urban Systems 69, 2018, pp. 114–123
[12] Cunningham, G.O., Y.L. Michael, S.A. Farquhar, and [25] King, A.C., S.J. Winter, J.L. Sheats, et al.,
J. Lapidus, “Developing a Reliable Senior Walking “Leveraging citizen science and information technology for
Environmental Assessment Tool”, American Journal of population physical activity promotion”, Translational
Preventive Medicine 29(3), 2005, pp. 215–217. Journal of the American College of Sports Medicine 1(4),
2016, pp. 30.
[13] “Urban Planning in 3D: How Creating a Digital Twin
Leads to Smarter Cities”, Meeting of the Minds, 2018. [26] Wan, L., S. Gao, C. Wu, Y. Jin, M. Mao, and L. Yang,
https://meetingoftheminds.org/urban-planning-3d-creating- “Big data and urban system model-substitutes or
digital-twin-leads-smarter-cities-25212 complements? a case study of modelling commuting
patterns in beijing”, Computers, Environment and Urban
Systems 68, 2018, pp. 64–77.

Page 1683
 [38] Yang, K., C.R. Ahn, and H. Kim, “Validating
[27] Duchowny, K., P. Clarke, N.A. Gallagher, R. Adams, ambulatory gait assessment technique for hazard sensing in
A.L. Rosso, and N.B. Alexander, “Using Mobile, construction environments”, Automation in Construction
Wearable, Technology to Understand the Role of Built 98, 2019, pp. 302–309.
Environment Demand for Outdoor Mobility”, Environment
and Behavior, 2018, pp. 0013916517749256. [39] Chaspari, T., A. Tsiartas, L.I.S. Duker, S.A. Cermak,
and S.S. Narayanan, “EDA-gram: Designing electrodermal
activity fingerprints for visualization and feature
[28] Dubey, A., N. Naik, D. Parikh, R. Raskar, and C.A. extraction”, 2016 38th Annual International Conference of
Hidalgo, “Deep learning the city: Quantifying urban the IEEE Engineering in Medicine and Biology Society
perception at a global scale”, European conference on (EMBC), IEEE (2016), 403–406.
computer vision, Springer (2016), 196–212.
[40] Hausdorff, J.M., “Gait variability: methods, modeling
[29] Naik, N., J. Philipoom, R. Raskar, and C. Hidalgo, and meaning”, Journal of NeuroEngineering and
“Streetscore-predicting the perceived safety of one million Rehabilitation 2(1), 2005, pp. 19.
streetscapes”, Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition Workshops,
(2014), 779–785. [41] Itti, L., C. Koch, and E. Niebur, “A model of saliency-
based visual attention for rapid scene analysis”, IEEE
[30] Datta, R., D. Joshi, J. Li, and J.Z. Wang, “Studying Transactions on Pattern Analysis & Machine
aesthetics in photographic images using a computational Intelligence(11), 1998, pp. 1254–1259.
approach”, European conference on computer vision,
Springer (2006), 288–301. [42] Yadav, M., T. Chaspari, J. Kim, and C.R. Ahn,
“Capturing and quantifying emotional distress in the built
[31] Machajdik, J., and A. Hanbury, “Affective image environment”, Proceedings of the Workshop on Human-
classification using features inspired by psychology and art Habitat for Health (H3): Human-Habitat Multimodal
theory”, Proceedings of the 18th ACM international Interaction for Promoting Health and Well-Being in the
conference on Multimedia, ACM (2010), 83–92. Internet of Things Era, ACM (2018).

[32] Marchesotti, L., F. Perronnin, D. Larlus, and G. [43] Herbrich, R., T. Minka, and T. Graepel,
Csurka, “Assessing the aesthetic quality of photographs “TrueSkillTM : A Bayesian Skill Rating System”, In B.
using generic image descriptors”, 2011 International Schölkopf, J.C. Platt and T. Hoffman, eds., Advances in
Conference on Computer Vision, IEEE (2011), 1784–1791. Neural Information Processing Systems 19. MIT Press,
2007, 569–576.
[33] Truong, C., L. Oudre, and N. Vayatis, “Selective
review of offline change point detection methods”, [44] Lahat, D., T. Adali, and C. Jutten, “Multimodal data
arXiv:1801.00718 [cs, stat], 2018. fusion: an overview of methods, challenges, and
prospects”, Proceedings of the IEEE 103(9), 2015, pp.
[34] Choi, B., H. Jebelli, and S. Lee, “Feasibility analysis 1449–1477.
of electrodermal activity (EDA) acquired from wearable
sensors to assess construction workers’ perceived risk”, [45] Moser, J., “Computing Your Skill”, 2010.
Safety science 115, 2019, pp. 110–120 http://www.moserware.com/2010/03/computing-your-
skill.html
[35] Jebelli, H., B. Choi, H. Kim, and S. Lee, “Feasibility
study of a wristband-type wearable sensor to understand [46] Kim, J., H. Kim, and Y. Ham, “Mapping Local
construction workers’ physical and mental status”, Vulnerabilities into a 3D City Model through Social
Construction Research Congress, (2018), 367–377. Sensing and the CAVE System toward Digital Twin City”,
Computing in Civil Engineering (2019), 451–458.
[36] Lee, G., B. Choi, H. Jebelli, C.R. Ahn, and S. Lee,
“Reference Signal-Based Method to Remove Respiration [47] Van Mechelen, I., and A.K. Smilde, “A generic
Noise in Electrodermal Activity (EDA) Collected from the linked-mode decomposition model for data fusion”,
Field”, In Computing in Civil Engineering 2019: Data, Chemometrics and Intelligent Laboratory Systems 104(1),
Sensing, and Analytics. American Society of Civil 2010, pp. 83–94.
Engineers Reston, VA, 2019, 17–25.

[37] Kim, J., M. Yadav, C.R. Ahn, and T. Chaspari,

“Saliency Detection Analysis of Pedestrians’ Physiological
Responses to Assess Adverse Built Environment Features”,
The 2019 ASCE International Conference on Computing in
Civil Engineering, ASCE (2019).

Page 1684