Multisensor Data Fusion for Physical Activity Assessment

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 3, MARCH 2012

687

Multisensor Data Fusion for Physical Activity Assessment Shaopeng Liu, Student Member, IEEE, Robert X. Gao∗ , Fellow, IEEE, Dinesh John, John W. Staudenmayer, and Patty S. Freedson

Abstract—This paper presents a sensor fusion method for assessing physical activity (PA) of human subjects, based on support vector machines (SVMs). Specifically, acceleration and ventilation measured by a wearable multisensor device on 50 test subjects performing 13 types of activities of varying intensities are analyzed, from which activity type and energy expenditure are derived. The results show that the method correctly recognized the 13 activity types 88.1% of the time, which is 12.3% higher than using a hip accelerometer alone. Also, the method predicted energy expenditure with a root mean square error of 0.42 METs, 22.2% lower than using a hip accelerometer alone. Furthermore, the fusion method was effective in reducing the subject-to-subject variability (standard deviation of recognition accuracies across subjects) in activity recognition, especially when data from the ventilation sensor were added to the fusion model. These results demonstrate that the multisensor fusion technique presented is more effective in identifying activity type and energy expenditure than the traditional accelerometer-alone-based methods. Index Terms—Physical activity assessment, sensor fusion, support vector machines, wearable sensing system.

I. INTRODUCTION HYSICAL activity (PA) is defined as bodily movement generated by skeletal muscles [1]. Engaging in physical activities on a regular basis by means of walking, jogging, or sport activities, is effective for maintaining health and preventing cardiovascular disease, diabetes, and obesity. Accurate monitoring of PA should provide information on the type, intensity, and duration of the activities that the person has been engaged in, thus is of significant interest to the research community [1]. The goal of PA assessment is to recognize the type, duration, and intensity of a broad range of activities and quantify the energy expenditure (PAEE) during physical activities. Dif-

P

Manuscript received May 5, 2011; revised August 21, 2011 and October 25, 2011; accepted Novbember 15, 2011. Date of publication December 5, 2011; date of current version February 17, 2012. This work was supported by the NIH under Grant UO1 CA130783. Asterisk indicates corresponding author. S. Liu is with the Electromechanical Systems Laboratory, Department of Mechanical Engineering, University of Connecticut, Storrs, CT 06269 USA (e-mail: [email protected]). ∗ R. X. Gao is with the Electromechanical Systems Laboratory, Department of Mechanical Engineering, University of Connecticut, Storrs, CT 06269 USA (e-mail: [email protected]). D. John and P. S. Freedson are with the Physical Activity and Health Laboratory, Department of Kinesiology, University of Massachusetts Amherst, MA 01003 USA (e-mail: [email protected]; [email protected]). J. W. Staudenmayer is with the Department of Mathematics and Statistics, University of Massachusetts Amherst, MA 01003 USA (e-mail: jstauden@ math.umass.edu). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TBME.2011.2178070

ferent lab-based methods have been used for PAEE assessment including direct [2] or indirect [3] calorimeters to quantify the intensity of activities. While accurate in assessing the energy expenditure associated with the PA, such methods are generally only performed in lab settings, require sophisticated lab equipment, and do not provide information about the specific type of PA. In the field, single-accelerometer-based PA assessment has become the device of choice to estimate PAEE and activity type, due to its low subject burden and noninvasive nature. Studies have shown good results using this method in monitoring PA types and PAEE [4]–[6]. However, accelerometers by nature cannot distinguish different types of activities that produce similar acceleration profiles but have different energy expenditure [7]. For example, walking at a certain speed may result in acceleration outputs similar to that of walking at the same speed while carrying a load, although the energy expenditure is different. To address the drawbacks of this method, researchers have investigated alternative techniques, e.g., by placing multiple accelerometers at different locations on the body [8]–[12] or combining accelerometers with other types of sensors, such as physiological sensors or GPS [13]–[15], to improve prediction accuracy. Also, advanced computational techniques, such as machine learning and sensor fusion, have been investigated in association with either single accelerometer or multiple sensors [8], [9], [13], [14], [16]–[18]. For example, k-nearest neighbor (kNN) and na¨ıve Bayes (NB) techniques have been investigated for classifying PA types, using features extracted from multiple accelerometers [8] or multiple types of sensors [17], [18]. Good classification results have been demonstrated by a customized decision-tree model, combined with an artificial neural network (ANN) [14], using data from two accelerometers and one GPS sensor [13]. These techniques, while demonstrating promising results, are subject to limitations in that a priori knowledge and “intuitive modeling” of different activities are needed to build a classification model, e.g., a customized decision trees for activity classification. Besides, the output is either PA type [8], [13], [15] or estimated PAEE [9]–[11], but not both simultaneously, which is important because either alone is not sufficient to assess the PA. As an example, two people may have run at various speeds for different durations, but produced the same energy expenditure. The method that only provides PA types may classify the two people as having engaged in two different types of activities, without accurately estimating the PAEE. On the other hand, the method that only estimates PAEE may correctly predict the energy expenditure, without classifying the two activities. As a result, no accurate activity assessment is

0018-9294/$26.00 © 2011 IEEE

688

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 3, MARCH 2012

possible using either the PA type or PAEE. Such limitations motivate research into PA assessment using methods that may be applied to free-living conditions. In recent years, the technique of support vector machines (SVMs) has been increasingly investigated for medical and biomedical applications, due to its classification and estimation capabilities [19]. Examples of successful application of SVM include fall detection [15], gesture classification [20], electroencephalogram (EEG) artifact removal [21], epileptic seizure prediction [22], and heart rate turbulence denoising [23]. Such prior applications make SVM an attractive candidate for PA assessment, where features extracted from the recorded multichannel signals could be redundant due to the internal redundancy of the data. Because SVM discriminates patterns and classes by constructing separating boundaries in a high-dimensional feature space, it condenses information from the training dataset and is thus able to generate a representation of the specific classes with a small amount of support vectors (SVs). As a result, both the number and locations and of the SVs are determined automatically during the training process for the SVM model selection. In addition, as a nonparametric regression and pattern recognition technique, SVM is capable of both classifying patterns and estimating specific values, which makes it well suited for purpose of PA assessment, which requires both activity type classification and energy expenditure estimation. Based on such motivation, an SVM-based multisensor fusion technique has been developed to analyze data acquired from a subject-worn integrated measurement system (IMS) [24] to predict both the PA type and PAEE. The performance of the developed method is experimentally evaluated by human subjects mimicking freeliving activities in a laboratory setting. II. METHODS A. SVM Framework The objective of investigating SVM is to fuse data (e.g., body motion and ventilation) measured by different types of sensors to more accurately assess the types, intensities, and energy expenditure of the physical activities a test subject has engaged in, than using a single sensor. Such a data fusion problem can be addressed by the SVM algorithm, which, in essence, formulates a decision boundary to separate one activity (or class of data) from another. When multiple activities need to be separated, a “one-against-one” approach can be taken for separating each set of two activities. For instance, to classify d > 2 activities, an SVM model can be built for each pair of activities from the training data to form a total of d(d − 1)/2 models. Using a new data point, each model will be tested, and a vote on which type of activity this data point should belong to will be cast. After all the models have been tested, the activity that has received the most votes will be identified as the activity that the new data point belongs to. To formulate an SVM pairwise-model, assume a dataset {xi } consisting of data measured by multiple sensors, where xi ∈ n (n is the dimension of the input vectors) and i = 1, . . . , N (N is the total number of data points). Within the duration of this dataset, the subject is assumed to have engaged in two types of

Fig. 1.

Separation of data into 2-D spaces.

activities {yi }, labeled as −1 and 1 (yi ∈ {−1, 1}), respectively. Each dataset {xi } can be associated with one of the two activity types yi . To distinguish the activity type which the dataset {xi } is associated with, a function f (x) is assumed to exist that draws a separation decision boundary of the two activities. Data points above the boundary (when f (x) yields a value ≥ 0) belong to the activity labeled as 1, whereas data points below the boundary (when f (x) produces a value < 0) are labeled as −1. Such a function f (x) can be expressed as  f (x) ≥ 0 ⇒ yi = 1 (1) f (x) < 0 ⇒ yi = −1. A drawback of such a separation boundary function built in the original feature space is that it is often a complex, nonlinear, and implicit function which is computationally demanding for determining activity types for each new data point added [25]. To overcome this limitation, the activities are assumed to be separable in an enlarged feature space with higher dimensionality than the original feature space [26] where a linear and explicit decision boundary—a hyperplane—can be formulated (as illustrated in Fig. 1). To achieve this, the SVM algorithm first transforms the data {xi } from the original lower-dimensional space to a higher-dimensional space [26] via a transformation function φ. A hyperplane f  (x ) = wT x + b = 0, where x = φ(x), is then built in the higher-dimensional space to separate the two activities. In this formulation, w and b are the weighing factors, and x is the transformed high-dimensional data. Similar to (1), the hyperplane function for separating activities is expressed as    f (xi ) = wT x + b ≥ 0 ⇒ yi = 1 (2) f  (xi ) = wT x + b < 0 ⇒ yi = −1. The hyperplane is built such that it maximizes the distance (or margin) to the closest training data point of either activity [27]. The process can be expressed as an optimization problem as expressed in the following [26]   D, subject to yi wT xi + b ≥ D, ∀i (3) max n w ∈ ,b∈

where D is the distance of the closest data point to the hyperplane and can be set as 1/ w after normalization. For practical applications when two activities may have overlapping (i.e., misclassified) data points in the feature space, slack variables ξ = {ξi }, where i = 1, . . . , N , ξi ≥ 0, are introduced, allowing certain points xi on the wrong side of the margin by an amount of Dξi . Points on the correct side of the margin are expressed

LIU et al.: MULTISENSOR DATA FUSION FOR PHYSICAL ACTIVITY ASSESSMENT

689

TABLE I PHYSICAL ACTIVITIES

Fig. 2.

Illustration of the SVM-based multisensor fusion algorithm.

with ξi = 0. Equation (3) can be rewritten as   N  1 w 2 + C ξi min w ,ξ ∈n ,b∈ 2 i=1  T  subject to ξi ≥ 0, yi w φ(xi ) + b ≥ 1 − ξi , ∀i

(4)

where C is the cost parameter for those sample points misclassified by the optimal separating plane. The hyperplane decision function f (x) is then determined as the following sign function (sgn(t) = 1 for t ≥ 0, and sgn(t) = −1 for t < 0):  N  T yi αi φ(xi ) φ(x) + b . (5) f (x) = sgn i=1

From (5), it is seen that the decision function can be determined without specifying the explicit form of the transformation φ, but only the kernel function K(xi , x) = φ(xi )T φ(x) that computes inner products. Replacing the inner product in (5), the decision function for distinguishing the two activities can be rewritten as  N  yi αi K(xi , x) + b . (6) f (x) = sgn i=1

For the present study, the Gaussian kernel [26] was selected due to its reported effectiveness in activity recognition [15]. B. Data Collection As illustrated in Fig. 2, the SVM-based multisensor data fusion system extracts features (e.g., the mean value, percentiles, and dominant frequency) from PA-relevant data measured on human subjects. These include breathing (or ventilation), and body motion [24]. These features are then fused by the SVM algorithm to identify the types of physical activity that the subject has engaged in, and the energy expenditure. The sensing system for subject measurement consists of two triaxial accelerometers (MMA7260QT, Freescale) placed at the hip and wrist, and one ventilation sensor (1325 Piezo Crystal Sensor, Ambu Sleepmate) secured to the abdomen (AB) at the level of umbilicus of the test subjects. The accelerometers measure the trunk and arm motions. The hip accelerometer was clipped to the AB ventilation sensor belt in line with the anterior axillary line of the dominant hip. The wrist accelerometer was attached to the wrist using a Velcro strap. The x-axis of the hip accelerometer pointed down along subject’s torso and its z-axis pointed out of the torso. For wrist accelerometer, the x-axis pointed up along the lateral side of subject’s dominant arm and the z-axis pointed into the lateral side of the same arm. The ventilation sensor measures the expansion and contraction associated with breathing rate and volume represent-

ing the physiological response to bodily movement. Data from the three sensors are sampled at 30 Hz and preprocessed by a microcontroller on board a data logger, worn at the subject’s waist and subsequently stored into a microflash memory. The subjects performed 13 types of activities of varying intensities (Table I) that are commonly seen in daily lives, and involve motions from different parts of the body, e.g., upper-limb-dominant activities such as computer work or filing papers, lower-limbdominant activities such as cycling and treadmill running, and whole body recreation activities such as basketball and tennis. Specifically for basketball activity, participants were instructed to start by shooting from one of the poly-spot targets which were distributed around a basketball net in a gymnasium, and then continue to dribble between and shoot from any of the targets. If they made the basket, they picked up the target and dribbled the ball to half-court, placed the target at half-court, and dribbled back to another target. For tennis, participants were given a standard racquet and a package of tennis balls. A court was outlined on the gymnasium floor, and participants were instructed to continue to rally against the wall as if playing with a partner. Further details of the basketball and tennis protocols are provided in [28]. In order to accurately assess the energy expenditure, the 13 activities were further separated into four categories, based on the intensity and similarity among the activities [29]: 1) sedentary activity, 2) household and other activity, 3) moderate locomotion, and 4) vigorous activity. The four categories were used for training the estimation model predicting the energy expenditure. For purpose of experimental organization, the activities were also classified into two routines, which include different combinations of the 13 activities. Selection of routine was balanced across subjects. C. Feature Extraction Instead of directly using raw sensor data that contain redundant information as input for the SVM data fusion algorithm, time- and frequency-domain features were first extracted to train the fusion model and determine model parameters. In this study, 63 features were extracted from the hip and wrist accelerometers plus the respiratory signals from the ventilation sensor for every 30-s data segment. The time-domain features (50 in total) included the mean value, standard deviation, 10th, 25th, 50th

690

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 3, MARCH 2012

Fig. 3. Distributions of typical features used in the PA assessment. (a) Standard deviation of hip accelerometer x-axis output. (b) Standard deviation of wrist accelerometer x-axis output. (c) Correlation between hip and wrist vector magnitude acceleration. (d) Breathing frequency from AB ventilation sensor.

(median), 75th, 90th percentiles, and correlation between the vector magnitudes of the hip and wrist accelerometer readings. The vector magnitude of an accelerometer signal is given by  (7) ar = a2x + a2y + a2z where ax , ay , and az are the outputs corresponding to the three axes of the accelerometers, respectively. Mean and standard deviation of the PA signals are calculated, providing a general description of the activity intensity levels [8], [13]. The middle three percentiles (25th, 50th, and 75th) characterize signal distributions, and the 10th and 90th percentiles represent an estimate of the low and high values in each signal. The correlation between the vector magnitudes provides a measure for the coordination or variation between the upper limb and body during an activity. The frequency-domain features (13 in total) were obtained from spectral analysis, including the dominant frequency of the respiratory signal that is the breathing frequency, spectral energy [8] and entropy [8], [13] of both the hip and wrist accelerometers. It was observed in experiments in our laboratory that measured respiratory signals are contaminated by highfrequency artifacts caused by muscles or tissues in the abdominal region. For reliable feature extraction, these artifacts were first removed [30]. The features were computed within the intervals of 3 to 6 min of each 7-min activity period. This ensures that the subject has achieved steady-state energy expenditure. Linear scaling was then applied on the extracted features in the range of [0, 1], to

avoid that features of greater numeric values would overwhelm those in the smaller numeric ranges. D. Feature Selection Removing irrelevant or redundant features and selecting “good” features are important for establishing an accurate SVM fusion model that minimizes classification/estimation errors and yields a general concept [31]. In this study, a two-step feature selection scheme is devised for selecting optimal features for the PA assessment. Selection of the 63 features was first based on a statistical analysis of the distribution of these features. Specifically, boxplots were plotted for each feature in each activity, which illustrate the distribution change among activities. A “good” feature is expected to have a less distribution overlap between the activities, which is considered to be better for the differentiation of activities [14]. For example, Fig. 3 shows the distribution boxplots of four typical features of the entire subject group, including standard deviations of the x-axis outputs of the hip and wrist accelerometers, correlation between the vector magnitudes of the hip and wrist accelerometers, and the breathing frequency from the AB ventilation sensor. The distributions of the standard deviation of the hip accelerometer x-axis output reveal significant differences among activities, e.g., for sedentary activities, CW and FP, this feature distributes below 0.2 G, while for vigorous activity such as T4-5 or T6-0, it distributes from 0.2 G to 1.5 G. The overlap between these two activity categories is 0%, indicating that the standard deviation is a “good” candidate feature. The 63 features were evaluated

LIU et al.: MULTISENSOR DATA FUSION FOR PHYSICAL ACTIVITY ASSESSMENT

by removing redundant features using the minimal-redundancymaximal-relevance (mRMR) heuristic [32]. The mRMR method measures the relevance and redundancy of the feature candidates with the target class based on mutual information, and selects a “promising” feature subset that has maximal relevance and minimal redundancy. A subset of 33 features with the best mRMR scores (difference between the relevance and redundancy) was selected for the PA assessment, including 6 features from the hip accelerometer, 25 features from the wrist accelerometer and 2 features from the AB ventilation sensor: 1) Hip accelerometer: 25th percentile, standard deviation and spectral entropy of x-axis, spectral energy of y-axis, 90th percentile and standard deviation of z-axis; 2) Wrist accelerometer: 25th, 50th, 75th, 90th percentiles, mean, standard deviation, spectral energy and entropy of x-axis, all time- and frequency-domain features of y-axis, all time-domain features and spectral energy of z-axis; 3) AB ventilation sensor: 90th percentile, standard deviation. E. PA Assessment A two-step procedure was used to predict the types of physical activity. First, a training dataset that consists of all the selected features obtained from all 50 subjects but one was constructed for building the SVM model and selecting the cost parameter C and Gaussian kernel parameter γ. The two parameters were selected through five-fold cross validation to prevent overfitting. The parameters that yielded the highest recognition rate were chosen during the process. Upon completion of training, the SVM model was applied to the feature set of the subject that was left out in the training process, to predict the activity type reflected in the 30-s data segments. Such a two-step procedure constitutes a “leave-one-subject-out” cross validation, and was executed on each subject data. To predict energy expenditure associated with each activity, the regression version of the SVM, support vector regression (SVR) [33], was implemented. Considering the broad range of intensity of the physical activities monitored, a separate SVR model was built for each of the four activity groups as listed in Table I. A prior study [29] has shown success in improving the METs estimation accuracy by grouping activities of similar intensity level and estimating METs separately with a different estimation model. Data from any subject who had performed activities in a particular group were used in the training and testing process of the respective SVR model. Construction of the SVR models followed the “leave-one-subject-out” crossvalidation procedure. A two-tier estimation scheme was devised to predict the physical activity energy expenditure. Specifically, the activity type of each 30-s data segment was first identified by the SVM model and grouped into one of the four activity categories. Subsequently, the energy expenditure during the 30-s period was estimated by the specific SVR model for that activity group. The results of activity type recognition using the SVM method were compared with those obtained by other commonly used techniques: kNN [17] and NB [8] classifiers, which are available in the MATLAB bioinformatics and statistics toolboxes.

691

The kNN parameter—the number of nearest neighbors—was determined through a five-fold cross validation during the training process, and the one that yielded the best recognition rate was selected. Euclidean distance was chosen as the distance metric for both the training and testing processes. For the NB classifier, normal distribution of the data was assumed. Recognition of activity type from both classifiers also followed the “leaveone-subject-out” cross-validation procedure. Furthermore, the results were compared among seven different fusion models listed below to investigate the effect of number of sensors and their specific locations on PA assessment 1) M1 : hip accelerometer only; 2) M2 : wrist accelerometer only; 3) M3 : AB ventilation sensor only; 4) M4 : hip, wrist accelerometers; 5) M5 : hip accelerometer, AB ventilation sensor; 6) M6 : wrist accelerometer, AB ventilation sensor; 7) M7 : hip, wrist accelerometers, AB ventilation sensor. III. EXPERIMENTAL STUDY Test subjects were recruited from responders to an advertisement about this study. Subjects who had musculoskeletal issues that would prevent them from performing the activities listed in Table I were excluded, e.g., if their blood pressures exceeded 140 mm/90 mm, had a positive stress test (evaluated by a physician for men more than 40 and women more than 45 years of age), or were taking medications that are known to influence metabolism and/or heart rate. A total of fifty test subjects (19 male and 31 female) were recruited. The group has the following characteristics (mean standard deviation): age = 32.6 ± 9.9 years, weight = 67.7 ± 12.3 kg, height = 171.2 ± 8.6 cm, and body mass index = 23.2 ±4.6 kg/m2 . Each participant read and signed an informed consent document approved by the institutional human subjects’ review board. Each subject was asked to perform one group of activities (as illustrated in Table I), and the specific activities were completed in a random order. Among the 50 subjects, 27 performed Routine 1 and the rest 23 performed Routine 2. If a subject did not feel comfortable to complete high-intensity activities, such as treadmill at 6.0 mph, basketball or tennis playing, he/she was not required to complete the activity. Each activity lasted for 7 min, followed by a 2-min rest period. Prior to the start of each test, subjects were asked to lie down on a bed to rest for 10 min, in order to achieve a baseline metabolic rate. All the tests were performed during the day, and the subjects were asked to eat 4 h before the test, after which no food or drink was allowed to be taken, except for water. During the tests, the actual PA types performed by the test subjects were noted, and the PAEEs were measured with a portable indirect calorimetry respiratory gas exchange system (Oxycon Mobile, Cardinal Health), which serves as a criterion energy expenditure measure. The respiratory gas exchange system, secured to the subject using an adjustable vest, provides physiological measurements such as breathing rate, ventilation volume, and the metabolic equivalent of each activity (MET). The MET

692

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 3, MARCH 2012

TABLE II ACTIVITY TYPE RECOGNITION FOR DIFFERENCE MODELS

is the ratio of activity energy expenditure to resting energy expenditure and expresses the absolute intensity of a given activity in multiples of resting metabolic rate. The measured data were then wirelessly transmitted to a laptop computer. The clocks for the integrated measurement system and the respiratory gas exchange system were synchronized and the time was noted at the beginning of each activity. The respiratory gas exchange system was calibrated prior to data collection with oxygen and carbon dioxide gases of known concentration and a certified 3-L syringe. IV. RESULTS A. Recognition of Activity Type Table II shows a comparison of the activity type recognition accuracy for the seven models. Within each model, the recognition is also compared among the three classifiers: SVM, kNN, and NB. The recognition accuracy is expressed as the mean and standard deviation for each model, computed (n = 50). It can be seen that, for both the SVM and kNN methods, the recognition performance is enhanced when more sensor data are included in the models. For example, the average accuracy obtained by SVM has increased from 75.8%, 77.6%, and 35.6% (achieved by the single-sensor models M1 , M2 , M3 , respectively) to 85.8%, 78.3%, and 80.1% for the dual-sensor models (M4 , M5 , M6 ), and to 88.1% for the multiple-sensor model (M7 ), respectively. The results from the NB classifier show different trends among the seven models but less recognition accuracies than the other two techniques. This indicates that the choice of an appropriate classifier has an effect on the performance of the assessment. The results from SVM were consistently better than those from using the kNN and NB. This has shown that the SVM algorithm has better performance on fusing data from multiple sensors than the other two techniques, which may be due to the more tunable model parameters of SVM than that of the other two methods. The standard deviation of the recognition accuracies reveals the subject-to-subject variability. The variability in SVM decreased from 19.3% when using a single hip accelerometer to 14.1% and 16.9% when using two sensors (either the hip accelerometer with wrist accelerometer, or hip accelerometer with ventilation sensor), and to 10.1% when using all three sensors. This indicates that with the ventilation sensor and wrist accelerometer data included in the fusion process, the subjectto-subject variability can be effectively reduced, making the

TABLE III ACTIVITY TYPE RECOGNITION FOR DIFFERENCE FEATURE SETS

Fig. 4.

PA recognition accuracy for each activity (model M 7 ).

fusion algorithm more generalizable than using data from only two accelerometers. The results obtained by kNN have shown similar trends. The individual contribution of each sensor/location was examined. The recognition accuracy from the single-sensor models have shown an approximate ratio of 4 : 4 : 2 (75.8% : 77.6% : 35.6% = 0.40 : 0.41 : 0.19 for SVM, and 70.7% : 76.2% : 31.1% = 0.40 : 0.43 : 0.17 for kNN) for both the SVM and kNN classifiers, respectively. This indicates that the hip and wrist accelerometer each contributes about 40% to the recognition of activity types, while the AB ventilation sensor contributes about 20%. Furthermore, individual contributions of the time- and frequency-domain feature sets were compared by substituting the frequency- and time-domain features with zero values in model M7 . Table III shows the activity type recognition accuracies for time- and frequency-domain feature sets by using the SVM classifier. It is noted that time-domain features has more contribution (84.7% : 66.5% = 0.56 : 0.44) than frequency-domain features for activity recognition. Similar findings have also been reported in previous studies [34]. Noteworthy is that the fusion of both time- and frequency-domain features increases the recognition accuracy while reduces the subject-to-subject variability. The activity-to-activity variability of the PA recognition accuracy of model M7 by SVM is shown in Fig. 4. Twelve of the thirteen activities were correctly recognized for more than 80% of the time. Furthermore, when the activities are classified into the four general activity categories (refer to Table I), the SVM fusion model with the two accelerometers and ventilation sensor reveals an overall average accuracy of 89.3% (88.9% for sedentary activity, 91.2% for household activity/other, 90.3% for moderate locomotion and 85.5% for vigorous activity, respectively). To further assess the SVM-based multisensor fusion method, confusion matrices are presented to evaluate the activity type

LIU et al.: MULTISENSOR DATA FUSION FOR PHYSICAL ACTIVITY ASSESSMENT

693

TABLE IV MET EESTIMATION SUMMARIES FOR DIFFERENT MODELS

Fig. 5. Confusion matrices of activity type recognition results of four models. The change in the color represents the recognition accuracy. (a) Confusion matrix of M1 . (b) Confusion matrix of M4 . (c) Confusion matrix of M5 . (d) Confusion matrix of M7 .

recognition results obtained from 4 (M1 , M4 , M5 , and M7 ) out of the seven fusion models. As shown in Fig. 5, the confusion matrix illustrates the percentage of one activity (row) classified into another activity (column). Hot colors (toward red) represent higher correct classification rates, while cold colors (toward blue) represent lower correct classification rates. These confusion matrices show the recognition and misclassification rates of each activity in each fusion model, therefore, complementing the estimation performance comparison (Table II). For example, recognition of activity MB and C2 reveal low correct classification rates (62% and 75%, respectively) when only data from the hip accelerometer were included in the SVM model. When the AB ventilation sensor was added, recognition rates of MB and C2 increased to 71% and 85%, respectively. When data from all the sensors were fused, 93% of MB and 89% of C2 activities were correctly recognized. Noteworthy is the rates of misclassifying MB, which is considered as a walking activity while carrying a load, as other walking activities such as T3 have decreased. For example, 32% of MB were misclassified as T3 when only the hip accelerometer was included in the model. When the AB ventilation sensor was added to the fusion model, the misclassification rate dropped to 23% (when MB was misclassified as T3). Only 5% of MB was misclassified as T3, when all the sensors were fused by the SVM algorithm. Thus, when the ventilation sensor and wrist accelerometer are fused by the SVM algorithm, improved recognition performance of each activity is attained, justifying the value of the SVM-based fusion method. B. Estimation of METs The MET characterizes the energy expenditure associated with activities, and was predicted in the present study by using the SVR fusion model. The estimated MET values were then compared with the values measured by the respiratory gas

Fig. 6. Comparison of the measured and predicted METs for different types of activities by model M 7 .

exchange system along with the resting metabolic rate measurement. The MET estimation results were compared among the seven fusion models, as shown in Table IV. The estimation performance was assessed by four statistical measures: bias, standard error (SE), root mean square error (RMSE) and mean absolute error in percentage (MAE). These measures were computed from the average estimation of each activity. The result shows that the SVR fusion model M7 with two accelerometers and AB ventilation sensor predicted the PAEE (with a bias of −0.01 MET, SE of 0.13 MET, RMSE of 0.42, and MAE of 6.3%) better than the other fusion models. Fig. 6 shows a comparison between the measured and predicted MET values by model M7 . The measured and predicted average METs of each activity were plotted as the abscissa and ordinate, and the width of each box shows the subject-to-subject variability (standard deviation) in both the measured and predicted METs. It is noted that the fitting equation is close to the line of identity (with a slope of 0.97), and there is a high coefficient of determination (R2 = 0.971). This indicates a good agreement between the predicted METs by the SVR fusion model with criterion METs. The RMSE of the sedentary activity, household and other activity, moderate locomotion, and vigorous activity are 0.09 METs, 1.75 METs, 0.36 METs, and 0.39 METs, respectively. The relatively weak estimation of the household

694

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 3, MARCH 2012

Fig. 7. Bland–Altman analysis of MET estimation for all activities by model M 7 . Dotted lines represent limits of agreement and solid line represents the mean bias.

and other activity group was primarily due to the estimation of activity VA and C2, which have shown relative large estimation errors seen in Fig. 6. This indicates that the presented method needs further enhancement to improve the PA assessment performance. Fig. 7 shows the Bland–Altman plot of the MET estimation for all activity types by model M7 . It is seen that 95% of the differences between the predicted and measured METs lie between −3.26 and 3.30 METs. V. DISCUSSION In this paper, an SVM-based multisensor data fusion method was devised and implemented for assessing physical activities from data measured by three body-worn sensors. Experimental results have demonstrated that the system was able to recognize 13 types of activities of varying intensities (sedentary to vigorous) and estimate the corresponding energy expenditure with good accuracy. Previous investigations have shown success in employing various types of pattern recognition techniques to identify activity types and estimate energy expenditure. For example, applying C4.5 decision tree to five accelerometers placed at the four limbs and right hip, Bao et al. [8] achieved an 84.3% overall accuracy of recognizing 20 activities. Using a combined model of custom decision tree and neural networks, Ermes et al. [13] reported a recognition rate of 89% for identifying nine activities from two accelerometers worn at hip and wrist and a GPS which provided speed data. Rothney et al. [9] predicted the METs of daily activities including deskwork, self-paced walking, jogging, biking, etc. with a mean squared error of 0.25 (RMSE: 0.50 METs) by applying an artificial neural network model to signals from a biaxial accelerometer worn at hip. Compared with the results reported in the previous studies, the presented algorithm is appealing, because it estimates both the PA types and PAEE, which other methods did not. Furthermore, it has recognized 13 different activities with an average success rate of 88.1%, which is 12.3% higher than the single accelerometer-based method that typically only uses a hip accelerometer. Also, it predicts energy expenditure with a RMSE of 0.42 METs, which is 43% less than using the hip accelerometer alone.

When future activities out of the 13 activities described in this study are analyzed, the activity type sensor fusion model will first assign the new activity with a “close” activity type based on the corresponding sensor measurements. The assigned activity type will be at a similar intensity level as the new activity, and will be allocated to one of the activity groups. The METs will be then estimated by the SVR model of that activity group. In this way, the presented method can be scaled to new activities that have not been included in the current study for estimating PAEE. While the presented method has shown better recognition accuracy than kNN and NB, it is computationally more demanding in the training process, primarily due to the search for optimal model parameters. This limits its applications where estimation models need to be updated in real-time. Another limitation is that the method has been tested on 50 subjects only, simulating 13 types of activities in a laboratory setting under controlled conditions. Expanded studies involving a larger population with more types of activities under free-living conditions are needed to further evaluate its robustness. Activities performed in a free-living setting are not performed in known time intervals as was the case in the current lab-based study. In future work, additional analytic methods will be used to identify when specific activities that occur in shorter more frequent and random intervals begin and end to evaluate the algorithm performance in free-living settings. In this study, accelerometer features were calculated from each axis instead of the combined vector magnitude. The advantage of using features of each axis is that it provides insight into the dynamics of activities in each direction along the axes, thus improving the discrimination of activities that have a similar vector length. For example, treadmill activity at 6.0 mph, 0% grade and basketball have shown a similar vector length to that of the accelerometer. However, basketball is inherently associated with more acceleration along the y-axis (pointed out of the lateral side of the subject) due to the nature of basketball where there is a large amount of lateral movement, as compared to treadmill activity where most movement is in the vertical plane. The disadvantage of including features of each axis separately is that it increases the size of the feature set, and consequently, increases the computational load. This may become a limiting factor if real-time activity mode detection is required. Furthermore, the relative high number of features may result in redundancy of the feature set. As an example, the spectral energy has shown patterns similar to the corresponding time-domain feature standard deviation, and thus may affect the generalizability of the results. The feature selection algorithm was able to eliminate redundant features and retain more relevant features. For example, for features extracted from the hip accelerometer, the algorithm selected the standard deviation but not the spectral energy from the x-axis. Conversely, from the y-axis, the spectral energy but not the standard deviation was chosen, whereas from the z-axis, the standard deviation was chosen again. These results confirm that proper feature selection is an important prerequisite toward accurate and reliable PA assessment. Future study will address this issue and enhance the generalizability of the results. In the present study, only one classifier was used for PA assessment. It is noted from previous study [13] that combining multiple classifiers (e.g., decision tree and neural networks) to

LIU et al.: MULTISENSOR DATA FUSION FOR PHYSICAL ACTIVITY ASSESSMENT

build a hybrid model may further improve the performance of estimation. Currently, the method merges features from multiple sensors together as one single feature set for the classifiers. An ensemble learning approach that employs separate classifiers on the feature set of each sensor and combines individual estimations may further improve the accuracy of physical activity assessment. This will be investigated in future studies. VI. CONCLUSION An SVM-based multisensor fusion method for physical activity assessment has been developed, which has demonstrated successful PA assessment through laboratory experiments. Two advantages of the algorithm have been verified: 1) improving the estimation accuracy of types and corresponding energy expenditure of physical activities, and 2) reducing subject-to-subject variability in activity type recognition by approximately 48%, when multisensor signals were included in the fusion models. REFERENCES [1] C. Caspersen, K. Powell, and G. Christensen, “Physical activity, exercise, and physical fitness: Definitions and distinctions for health-related research,” Public Health Rep., vol. 100, pp. 126–131, 1985. [2] J. Webster, G. Welsh, P. Pacy, and J. Garrow, “Description of a human direct calorimeter, with a note on the energy cost of clerical work,” Brit. J. Nutr., vol. 55, pp. 1–6, 1986. [3] V. Diaz, P. Benito, A. Peinado, M. Alvarez, C. martin, V. Salvo, F. Pigozzi, N. Maffulli, and F. Calderon, “Validation of a new portable metabolic system during an incremental running test,” J. Sport. Sci. Med., vol. 7, pp. 532–536, 2008. [4] D. Hendelman, K. Miller, C. Baggett, E. Debold, and P. Freedson, “Validity of accelerometry for the assessment of moderate intensity physical activity in the field,” Med. Sci. Sports Exerc., vol. 32, pp. 442–449, 2000. [5] C. Bouten, K. Koekkoek, M. Verduin, R. Kodde, and J. Janssen, “A triaxial accelerometer and portable data processing unit for the assessment of daily physical activity,” IEEE Trans. Biomed. Eng., vol. 44, no. 3, pp. 136–147, Mar. 1997. [6] B. Najafi, K. Aminian, A. Paraschiv-Ionescu, F. Loew, C. B¨ula, and P. Robert, “Ambulatory system for human motion analysis using a kinematic sensor: Monitoring of daily physical activity in the elderly,” IEEE Trans. Biomed. Eng., vol. 50, no. 6, pp. 711–723, Jun. 2003. [7] D. Pober, J. Staudenmayer, C. Raphael, and P. Freedson, “Development of novel techniques to classify physical activity mode using accelerometers,” Med. Sci. Sports Exerc., vol. 38, pp. 1626–1634, 2006. [8] L. Bao and S. Intille, “Activity recognition from user-annotated acceleration data,” Lecture Notes Comput. Sci., vol. 3001, pp. 1–17, 2004. [9] M. Rothney, M. Neumann, A. B´eziat, and K. Chen, “An artificial neural network model of energy expenditure using nonintegrated acceleration signals,” J. Appl. Physiol., vol. 103, pp. 1419–1427, Jul. 2007. [10] K. Chen, S. Acra, K. Majchrzak, C. Donahue, L. Baker, L. Clemens, M. Sun, and M. Buchowski, “Predicting energy expenditure of physical activity using hip- and wrist-worn accelerometers,” Diabetes Technol. The., vol. 5, pp. 1023–1033, 2003. [11] K. Zhang, F. Pi-Sunyer, and C. Boozer, “Improving energy expenditure estimation for physical activity,” Med. Sci. Sports Exerc., vol. 36, pp. 883– 889, 2004. [12] J. Lester, C. Hartung, L. Pina, R. Libby, G. Borriello, and G. Duncan, “Validated caloric expenditure estimation using a single body-worn sensor,” in Proc. 11th Int. Conf. Ubiquitous Comput., New York, 2009, pp. 225–234. [13] M. Ermes, J. Parkka, J. Mantyjarvi, and I. Korhonen, “Detection of daily activities and sports with wearable sensors in controlled and uncontrolled conditions,” IEEE Trans. Inf. Technol. Biomed., vol. 12, no. 1, pp. 20–26, Jan. 2008. [14] J. Parkka, M. Ermes, P. Korpipaa, J. Mantyjarvi, J. Peltola, and I. Korhonen, “Activity classification using realistic data from wearable sensors,” IEEE Trans. Inf. Technol. Biomed., vol. 10, no. 1, pp. 119–128, Jan. 2006. [15] E. Tapia, S. Intille, W. Haskell, K. Larson, J. Wright, A. King, and R. Friedman, “Real-time recognition of physical activities and their intensities using wireless accelerometers and a heart rate monitor,” in Proc. 2007

695

[16] [17] [18] [19] [20] [21] [22]

[23]

[24] [25] [26] [27] [28] [29]

[30]

[31] [32]

[33] [34]

11th IEEE Int. Symp. Wearable Comput., Washington DC, Oct. 11–13, 2007, pp. 37–40. T. Zhang, J. Wang, L. Xu, and P. Liu, “Fall detection by wearable sensor and one-class SVM algorithm,” Intell. Comput. Signal Process. Pattern Recog., vol. 345, pp. 858–863, 2006. U. Maurer, A. Rowe, A. Smailagic, and D. Siewiorek, “Location and activity recognition using eWatch: A wearable sensor platform,” Ambient Intell. Everyday Life, vol. 3864, pp. 86–102, 2006. P. Pirttikangas, K. Fujinami, and T. Nakajima, “Feature selection and activity recognition from wearable sensors,” Ubiquitous Comput. Syst., Lecture Notes Comput. Sci., vol. 4239, pp. 516–527, 2006. N. Cristianini and J. Shawe-Taylor, Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge, UK: Cambridge University Press, 2000. G. Naik and D. Kumar Jayadeva,, “Twin SVM for gesture classification using the surface electromyogram,” IEEE Trans. Inf. Technol. Biomed., vol. 14, no. 2, pp. 301–308, Mar. 2010. S. Shao, K. Shen, C. Ong, E. Wilder-Smith, and X. Li, “Automatic EEG artifact removal: A weighted support vector machine approach with error correction,” IEEE Trans. Biomed. Eng., vol. 56, no. 2, pp. 336–344, 2009. L. Chisci, A. Mavino, G. Perferi, M. Sciandrone, C. Anile, G. Colicchio, and F. Fuggetta, “Real-time epileptic seizure prediction using AR models and support vector machines,” IEEE Trans. Biomed. Eng., vol. 57, no. 5, pp. 1124–1132, May 2010. J. Rojo-Alvarez, O. Barquero-Perez, I. Mora-Jimenez, E. Everss, A. Rodriguez-Gonzalez, and A. Garcia-Alberola, “Heart rate turbulence denoising using support vector machines,” IEEE Trans. Biomed. Eng., vol. 56, no. 2, pp. 310–319, Feb. 2009. S. Liu, R. Gao, and P. Freedson, “Design of a wearable multi-sensor system for physical activity assessment,” in Proc. IEEE/ASME Int. Conf. Adv. Intell. Mechatron., Jul. 2010, pp. 254–259. V. Vapnik, Statistical Learning Theory. New York: Wiley, 1998. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York: Springer, 2009. V. Vapnik, The Nature of Statistical Learning Theory. New York: Springer, 1996. S. Kozey, K. Lyden, C. Howe, J. Staudenmayer, and P. Freedson, “Accelerometer output and MET values of common physical activities,” Med. Sci. Sports Exerc., vol. 42, pp. 1776–1784, 2010. J. Staudenmayer, D. Pober, S. Crouter, D. Bassett, and P. Freedson, “An artificial neural network to estimate physical activity energy expenditure and identify physical activity type from an accelerometer,” J. Appl. Physiol., vol. 107, pp. 1300–1307, 2009. S. Liu, Q. He, R. Gao, and P. Freedson, “Empirical mode decomposition applied to tissue artifact removal from respiratory signal,” in Proc. 30th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., Vancouver, British Columbia, Canada, Aug. 20–24, 2008, pp. 3624–3627. M. Dash and H. Liu, “Feature selection for classification,” Intell. Data Anal., vol. 1, pp. 131–156, 1997. H. Peng, F. Long, and C. Ding, “Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and minredundancy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 8, pp. 1226–1238, Aug. 2005. A. Smola and B. Schølkopf, “A Tutorial on support vector regression,” Stat. Comput., vol. 14, pp. 199–222, 2004. S. Preece, J. Goulermas, L. Kenney, and D. Howard, “A comparison of feature extraction methods for the classification of dynamic activities from accelerometer data,” IEEE Trans. Biomed. Eng., vol. 56, no. 3, pp. 871– 879, 2009.

Shaopeng Liu (S’08) received the B.S. and M.S. degrees in mechanical engineering from Tsinghua University, Beijing, China, in 2004 and 2007, respectively. He is working toward the Ph.D. degree in the Department of Mechanical Engineering of the University of Connecticut, Storrs, CT. His research interests include mechatronics system modeling and characterization, energy-efficient wireless sensor design and networking, multisensor data fusion, pattern recognition, human health monitoring, and physical activity assessment. Mr. Liu was a second place winner of the Philips Young Investigator Award at the 33rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’11), in 2011, and was awarded the Graduate Teaching Fellowship in the Department of Mechanical Engineering, in 2011. He is a student member of the ASME.

696

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 59, NO. 3, MARCH 2012

Robert X. Gao (M’91–SM’00–F’08) received the M.S. and Ph.D. degrees in mechanical engineering (subject area is measurement and control) from the Technical University, Berlin, Germany, in 1985 and 1991, respectively. He was a Professor with the Department of Mechanical and Industrial Engineering, University of Massachusetts Amherst, from 1995 to 2008, before joining the Department of Mechanical Engineering at the University of Connecticut, Storrs, CT, in Fall 2008, as the Pratt and Whitney Endowed Chair Professor. His research interests include the areas of physics-based sensing methodologies, energy-efficient design of sensors and sensor networks, RF and acousticbased wireless communications, and intelligent computation for health monitoring, diagnosis, and prognosis. Dr. Gao is an Associate Editor of the IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, the ASME Journal of Manufacturing Science and Engineering, and the IFAC Journal of Mechatronics. He received the National Science Foundation CAREER Award in 1996, and is the Faculty Advisor and co-recipient of several best student paper awards. He received the Barbara H. and Joseph I. Goldstein Outstanding Junior Engineering Faculty Award and the Outstanding Senior Engineering Faculty Award from the University of Massachusetts Amherst, in 1999 and 2007, and the Research Excellence Award from the Department of Mechanical Engineering at the University of Connecticut in 2011. He is a Fellow of the ASME, a Distinguished Lecturer of the IEEE Electron Devices Society, and an elected member of the Connecticut Academy of Science and Engineering (CASE).

Dinesh John received the Ph.D. degree in exercise physiology from the University of Tennessee, Knoxville, TN, in 2009. He is currently a Postdoctoral Research Associate with the Department of Kinesiology at the University of Massachusetts, Amherst, MA. His research interests include the objective measurement of physical activity and investigating the efficacy of novel lifestyle behavior interventions in improving health. Dr. John was awarded the Andy Kozar Graduate Research Scholarship Award and the Edward A. Capen Award for research achievements from the University of Tennessee, in 2009. He is a member of the editorial board for the Frontiers in Exercise Physiology. He is a member of the American College of Sports Medicine.

John W. Staudenmayer received the M.S. and Ph.D. degrees in operations research from the School of Operations Research at Cornell University, in 1999 and 2000, respectively. He completed postdoctoral training in biostatistics at Harvard School of Public Health in 2000–2001. He joined the University of Massachusetts faculty, in 2001, where he is currently an Associate Professor with the Department of Mathematics and Statistics. His applied research interest focuses on developing statistical methods to assess physical activity and sedentary behavior using wearable monitors. His statistical research is in the areas measurement error modeling, nonparametric regression, and Bayesian statistics. Dr. Staudenmayer is an Associate Editor for Biometrics and the Electronic Journal of Statistics. He is a former Associate Editor for the Journal of the American Statistical Association.

Patty S. Freedson received the M.S. and Ph.D. degrees in kinesiology from the University of Michigan, in 1976 and 1980, respectively. She completed postdoctoral training at the Institute of Environmental Stress, University of California, Santa Barbara, CA, in 1980–1981. She joined the University of Massachusetts faculty, in 1981, where she is currently a Professor and Chair in the Kinesiology Department in the School of Public Health and Health Sciences. Her research interests primarily focus on using wearable monitors to assess physical activity and sedentary behavior where she has developed numerous lab-based methods and approaches for calibrating and validating motion and physiological sensors for use in free-living settings. She has published more than 95 papers and has lectured in the United States and abroad. Dr. Freedson received the American Alliance for Health, Physical Education, Recreation, and Dance President’s Award, in 1996. In 2004, she was selected as one of the University of Massachusetts Distinguished Faculty Lecturers, and in 2007, she received a University of Massachusetts Outstanding Accomplishments in Research and Creative Activity Award. She is a former President of the New England Chapter of the American College of Sports Medicine, the Research Consortium and was a Vice-President of American College of Sports Medicine. She is the current President of the National Academy of Kinesiology. She is a fellow of the Research Consortium, American College of Sports Medicine, and the National Academy of Kinesiology.