Do you trust the data you get from your activity tracker? Are the steps accurate? What about the calories burned? How does having this data affect your life? If you have any of these questions then you are not alone. Come along with me on a quest to learn more about activity tracking devices to better determine if we can trust the information they provide.
In this blog installment we are going to follow the scientific method (sort of!) as we look to answer one seemingly simple question:
Is the caloric burn data provided by modern day activity monitor/tracker devices accurate enough on a day-by-day basis to utilize for purposes of influence on our caloric intake (diet)?
Now that we have the question, let’s do some research! If knowing your caloric burn is something that interests you, keep reading!
STEP ACCURACY – WHY IT MAY NOT MATTER
Wearable technology is on the rise. While maybe not all the rage of Google Glass, smart watches and activity trackers are still a hot item right now. Health professionals have collected data using wearable medical devices such as pedometers and heart rate monitors for years, but only recently have these devices gained mainstream traction. Suddenly people are buying them for personal health and recreational use. When it comes to step counting accuracy, articles like THIS ONE from the PT Journal have shown the Fitbit is spot-on. Then again, you can find reviews like THIS ONE, which while not exactly scientific, claims the opposite results. For the Fitbit official statement on accuracy, READ HERE. And if you don’t believe them, maybe THIS STUDY will help you trust the Fitbit’s accuracy a little bit more. But perhaps the accuracy doesn’t matter?
People should understand that, like the weight scale you have at home, you should care more about establishing a baseline first. Then you may use that data for comparison as you strive to improve, expecting only consistency from the device. Using these products should add awareness to your activity levels, hopefully increasing them over time. With that you can strive to become healthy, more fit. But what is your goal is to lose weight?
Knowing that most device manufacturers admit to the step count not always being 100% accurate, they are still quick to point out a % in relationship to accuracy, and remind you that the steps are indicative of at least your activity level. This gives merit to Nike’s fuel/points system, which eliminated the confusion with regards to steps altogether. Sure it is nice to know how many miles you walked one day to the next, for bragging rights, but from a health & fitness standpoint the most important measurement is your level of activity. How active are your normally? Have you increased your activity levels? Did you burn as many calories as you wanted? Are you healthier today than yesterday?
But just because you were able to increase your steps, or “activity” using some sort of measurement device, how do you determine how that relates to your physical well-being, and caloric burn? If you know anyone who works in the field of Physical Therapy or other fitness based occupation they will tell you that heart rate monitors are a vital part of determining health. (See THIS ARTICLE as an example of the HRM in education and health). Your doctor checks your pulse and blood pressure every time you visit him/her because it is a vital indicator of your well being.
Companies like Polar Electro have come up with so many ways to use your heart rate to determine your health over time that I’ll just show you THIS LINK that gives you an overview of their offerings! And the folks at DigiFit have their own OFFERINGS as well. When it comes down to, it the best health assessments require a heart rate monitor.
WHY ALL THE RAMBLING ABOUT HEART RATE MONITORS?
Because they are supposedly more accurate than pedometers for computing caloric burn. And that is the question we aim to answer with regards to our step counting device(s).
When I purchased my Fitbit Force for activity data I was also aware of the fact that it would show calories burned. One of the key selling points for me was the integration the Fitbit offered with MyFitnessPal. Since these “fancy pedometers” are using algorithms to estimate caloric burn– and my goal is to determine their accuracy– what better way to test that with than heart rate monitors?
Data collected from a heart rate monitor is useful for more than just instant caloric burn. It can also be evaluated over time and then used to determine health changes for that specific person. Polar and DigiFit know this, and suggest taking assessments before, during and after their cardio exercise programs. You can even use their various trainer programs to achieve fitness goals. But for our purposes here, we just want to use the heart monitors for calories burned comparisons.
For purpose of caloric data you can find a plethora of research studies, some for example cited in THIS ARTICLE on LiveStrong. That article reads: “the American College of Sports Medicine and the ‘British Journal of Medicine’ have put Polar’s heart rate monitors to the test over the years. In these studies, the measurement of calories expended has a non-significant margin of error, meaning the difference between the control and the Polar monitor is not enough to skew your results.” That is to say that recording your caloric burn using a Polar heart rate monitor is known to be one of the most accurate methods available.
Be aware that there are times when using an HRM isn’t the best option, such as STRENGTH TRAINING or other non-cardio-based activities. And if you do buy a heart rate monitor to help burn some fat, READ THIS to understand how fat burning and heart rates tie together. In my time researching this option I came to a conclusion that heart monitors are a viable tool for caloric burn measurements. They aren’t 100% accurate, but are touted as being far better than any other option available to the average consumer.
So can the more popular wrist-based devices (such as Fitbit, Jawbone, etc) prove as accurate and useful as you expect?
EQUIPMENT FOR TESTING
In an ideal environment we would gather data collected over a very long period of time, comparing more devices than just my Fitbit Force. And we might even use BOTH a heart rate monitor AND a face mask to measures oxygen intake. If you don’t know why the face mask matters, learn about VO2 and RMR by reading HERE. And just for additional fun reading, THIS article has some interesting data and fits here better than anywhere else. However I do not have access to the face mask equipment, and since this blog is a hobby for me I also lack the time & financial resources to make that long-term level of research happen. So I began looking for other options.
That lead me to learn about devices that I can source easily as a consumer, that are cost effective for my testing purpose. (Remember, I’m paying for myself, since nobody is sending me this stuff for free!) During my research for caloric burn and heart rate monitors, two devices seemed to get more attention than the others.
The first device is the BodyMedia LINK which is meant to be worn all day and measures your heart rate constantly. They went with an armband design that you affix to your upper arm. Their device comes with a free 3-month trial, but after that you are charged a monthly service fee for use of their software. Unlike the Fitbit which works for free (though you can pay for some extra Premium services), the only way to use the BodyMedia is to pay the fees. Bottom line– after 90-days, you must pay for their service or the device becomes useless.
BodyMedia wants you to wear their device 23-hours a day (pretty much all day, except when you shower or swim). That is more or less the same expectations as the Fitbit. That makes their device the ideal test candidate, as it does the same thing — but adding the some extra sensors for heart rate, skin temperature, and so forth.
The other device that seems to get mentioned as much as the BodyMedia is the Polar H7 Bluetooth Smart Heart Rate Sensor. When it comes to heart monitors as a wearable device, Polar’s product selection remains one of best Also, I liked their explanation of heart rates, as shown here in their Sport Zones Chart. And after reading enough reviews to make me dizzy it became obvious that hundreds of folks swear by their H7 chest strap as the best way to measure cardio workouts. Polar provides many options for monitoring the real-time hear rate data. Most people opt for a watch, or the iPhone app, but I decided to take their new Polar Loop Activity Tracker for a spin. (Check out my review of the Loop as compared to my Fitbit in THIS POST).
My local store did not have the BodyMedia FIT in stock, which ultimately meant it would not be included for this comparison. My impatience got the best of me, so the Polar products got purchased, and the BodyMedia will have to wait to get compared at a later date. Although frankly I’m not sure the product interests me personally… so unless they send me a free one, I may not test it. 😉
For purpose of control/comparison, I did purchase another in-stock and readily available HRM device: the Scosche Rhythm. This device attaches to your forearm in contrast to the Polar H7 which is a chest-worn device. This will be a nice way to compare heart measurements collected from two body locations, and then also to see how each company uses this data with their own “software/algorithms” to calculate burned calories.
For the the Scosche Rhythm I will use their iPhone app (LINK) to monitor the data in real-time and then review after the workout. As for the Polar HRM, I will be using their new Polar Loop Activity Tracker which stores the “session”data in the wristband and then syncs up to the new Polar Flow cloud using their new app (LINK). It was interesting to see how the Polar Loop worked when coupled with their H7 heart rate monitor. It has the potential to be a nice combination of cardio-accuracy plus daily step counting (again, check out THIS POST for more details).
Sidebar: If the only time you want caloric burn data is during cardiovascular exercise, then run out and buy a Polar FT7 Heart Rate Monitor Watch which comes with a chest HRM. That is all you need! This test is obviously taking things one step further.
Since some of my cardio workouts do not involve steps, and to provide more comparative data here, I will compare these heart monitors against website calculators. That only makes sense given that the Fitbit still requires you to manually log those activities anyway. Since the Fitbit dashboard provides “estimated” caloric burn for manually entered activities, we will see how that stacks up against the heart rate devices. And we’ll throw in MyFitnessPal’s data too!
TEST METHODS & PREDICTIONS
In final then, we will be testing using this equipment:
- FOR STEP-BASED CARDIO ACTIVITIES
- Fitbit Force – Calculates steps, calories burned, and distance
- Polar H7 HRM – Records heart rate and estimates calories burned
- Scosche Rhythm – Records heart rate and estimates calories burned
- Treadmill – Records distance and estimates calories burned
- MyFitnessPal – Estimates caloric burn based on user input
- Argus by Azumio – Calculates steps & distance, and estimates calories burned
- FOR NON-STEP BASED CARDIO ACTIVITIES
- Polar H7 HRM – Records heart rate and estimates calories burned
- Scosche Rhythm – Records heart rate and estimates calories burned
- Bike – Records distance and estimates calories burned
- MyFitnessPal – Estimates caloric burn based on user input
- Fitbit – Estimates caloric burn based on user input
For the purpose of recording step-based data I will be using my Nautilus Pro Series T718 treadmill. For non-step-based cardio data the activity of choice will be Spinning on a stationary Schwinn exercise bike. Note that this bike is not a Schwinn Airdyne. Because the intensity changes with speed, the use of a heart rate monitor may prove useful for this piece of cardio equipment. The catch is that trying to enter this data into an online calculator is tough, not knowing what resistance/wattage option to use. For lack of any better options I’ll use what I’ve been using for the last year– “stationary bike, vigorous effort, 200 watts” — and let the calculators do their best.
The above data will then be compared to attempt to determine the accuracy. Even if the HRM data is recorded the same on both the Scosche and the Polar, it is possible that the two heart rate monitors may differ based on algorithms used by the respective companies
Based on the devices I’ve procured we are only going to be comparing data during short periods of cardiovascular activity. It is unknown how comparing these small cardio exercises ultimately relates to the rest of the day where the Fitbit calculates calories based on just walking/etc. However, as THIS POST shows, data from the Fitbit Force and Polar Loop are so strikingly similar at the end of the day that I’m left to trust the data. (This is where the BodyMedia might be better suited for comparison, but that is another study for another day).
My prediction is that, for step-based cardio, the Fitbit will be accurate (within 10% of the HRM data). However, I’m expecting that the caloric burn data from non-step-based cardio activities as recorded by the heart rate monitors and then compared to these online calculators will be off, by staggering amounts (20% or more?!). Of course, the only way to find out is to start conducting some tests. SO LET’S GO!
COLLECTING & COMPARING THE DATA
Since drafting the above introduction I’ve now managed to complete my full week of running two tests at the same time. First, the one you are reading about here, where I would perform cardio activities and log the data between the step counter(s) and the heart rate monitors, and compare those to the online calculators. Second, my shootout was occurring as documented in THIS ARTICLE between the two wristband devices.
Let’s look at the figures and see what we can learn….
Above is the chart logging my treadmill activity (click to enlarge). My average speed setting on the treadmill was 3.5 MPH so that is what I used for the MyFitnessPal calculator.
You may note that on December 6th I forgot to log the Argus steps/distance data. And for reference only, two of the three times I did record Argus the device was strapped to my arm, versus one where it just say in my pants pocket. Here is that data charted visually:
Although the Fitbit, Polar and Scosche items were typically very close to each other, the data on the final test day was off. You may also notice that my pulse was also considerably lower than the other days, as recorded by both the Flow and Rhythm. I’m guessing this had something to do with the results on that day.
For the first three tests the Scosche always ran the highest of the top three, which you will see again when we get to the bike testing. It is interesting to see just how far off the treadmill was even with me having input my gender, age and weight. MyFitnessPal is equally as off, but the Argus data is the worst of the bunch!
As one might expect, this data all has margins of error and should be taken lightly. That being said, the remarkable similarities between the Fitbit Force’s step-based data when compared here to the Polar Loop’s heart-rate based data is a good indicator of the Fitbit’s accuracy. Remember, the data shown here for the Loop is based on the H7 HRM, not based on step calculations.
RAW DATA: My spreadsheet was compiled using data available on the iPhone app and/or the website dashboard from Fitbit, Polar, Scosche, and so forth. I’ve placed some samples of the dashboard data I used later in the article for reference, but did not want to overburden you folks with all of the raw data. If anyone would like any screen shots of the raw data please feel free to email me.
POLAR LOOP – TEST #1 – IMPORTANT NOTE: On the first test there are ~9 minutes total (spread before/after the actual run) that the Polar Loop was recording heart rate and caloric burn. I had forgotten that as soon as you hook up the HRM it starts recording, but in subsequent tests I was able to be more accurate for starting/stopping the logs. Since you cannot “trim” the data when viewed on their website (or in the app), I’m forced to have to enter the entire sequence here. By my best estimation there are around 15 calories of extra data — meaning the “adjusted” figure would probably be 128 kcal, which would bring it even closer to the Fitbit and more likely accurate.
In review then on the treadmill, some interesting “numbers” to chew on:
(based on exclusion of the outlying data found in test #4)
- Fitbit’s caloric data is within 5% (average) when compared to the Polar HRM data
- Fitbit’s caloric data is within 16 kcal at worst when compared to the Polar HRM data
- At most the HRM devices showed a variation of 35 kcal
- Pulse data recorded was identical as recorded at both locations (arm and chest)
My prediction was that we would be within 10% accuracy. For two of the four recorded sessions, the Fitbit was within that level of accuracy compared to the Loop. Based on this I’m not going to argue that the HRM is probably more accurate, and I trust the Polar Loop/H7 data. But if 15% accuracy (or BETTER!) is okay with you, the Fitbit seems to be pretty darn good. Especially when you consider that at the end of the day the differences become even smaller (again see my othe review comparing the two devices, with HRM data logged).
Now onto the bike…
Above is the chart for the stationary bike testing (again, click to view larger). Because of the design of the Schwinn Airdyne it is impossible to record exact resistance. For our testing here I used “stationary bike, vigorous effort, 200 watts” when using online calculator sites. Note that on two occasions I forgot to collect the bike data, but we’re still able to compare the more important values which were logged.
With regards to the Fitbit, there are two values recorded because I did indeed wear my wristband during exercise. The “auto” values are what the armband saw, which is obviously inaccurate for cycling. I manually created an “activity” (using the stop watch feature of the Force), and then later went back and entered the data manually for that time (which is the “manual” data you see).
SCOSCHE RHYTHM – TEST #2 – IMPORTANT NOTE: Heart rate data kept dropping out on the Scosche during the second test, as I had poor arm placement. It was a good learning lesson to make sure, before starting to log a session, that the HRM was working. Hence I would NOT trust the data from that device on that run, but I still did show it here.
POLAR H7 HRM – TEST #4 – IMPORTANT NOTE: During the final run I logged the HRM data from Polar using their Beat iPhone app, rather than the Loop band. This was done for my own personal software testing, but I’ve been assured by them that the data is computed by the same algorithms and would be the same as expressed here in this testing.
Here are the charts:
Immediately you will notice that the calculators offered on the Fitbit and MyFitnessPal sites are nearly identical, which is to be expected. But what jumps out after that is how close (almost IDENTICAL) is the data coming off the Polar Flow. If indeed the Polar data is to be trusted, then the calculators those two sites use are amazingly precise.
Again this data makes me question the Scosche numbers. As if to say their math for caloric calculation wrong, or at least VERY different from Polar. Why is it so much higher? Especially given it recorded the same data (even the same avg/max pulse data) as the Polar. Odd!
As to be expected, the exercise equipment has extremely low values. In the case of the bike that makes sense, since it does not have user input for age/weight/gender. However the treadmill does ask that data but as shown prior was also rather low. In the future I may utilize the HRM features of my treadmill for further testing, but that was not included here.
NON-STEP BASED: Although we are comparing the Fitbit during the bike recording, this is a non-step based activity. As such, the Fitbit needs a calculator to help determine calories burned. But when you note the similarities of the data to that of the Polar Flow’s data it makes me wonder why I might bother wearing a HRM. If indeed the data I’m going to get from just entering in my workout manually is the same, hy do I really need an HRM?
Of course if you could have one device that did both that would be the best of both worlds. But as I showed in my REVIEW HERE, the Loop isn’t ready for prime-time just yet. Down the road, however, it will be a device that should compete well with the likes of the Fitbit. For now, however, it still needs some software polishing and a better battery life.
Same as I did in the prior section, here are some “numbers” to chew on:
- At worst the Polar Flow was off from the Fitbit calculator by 10% however that is in just one instance. In the other recorded instances, it was within 4%
- At worst the Scosche Rhythm was off from the Fitbit calculator by 36% and differed from the Polar Flow by as much as 40%
- If I had recorded my calories manually and not used the Polar HRM, the worst I would have done is counted my calories off by 13 kcal
And as promised, here are some dashboard views from the respective web sites. These are just a sampling of the data that I used to fill the charts above. On many of these sites you could drill in deeper if you wanted or hover to see actual heart BPM, etc.
Above: Fitbit Activity as logged by using the stopwatch feature on the wristband. In this view you could view calories, steps, floors and page more precisely if you wanted.
Above: Polar Session as logged by using a heart rate monitor in conjunction with their Polar Loop wristband. (you can also use Polar Beat and their HRM to log same data). You cannot trim these sessions, but you could get HRM data by sliding the mouse over the red graph line.
Above: Scosche Rhythm data collected using their iOS app on my iPhone device. Although not as clean as the Polar site, the Scosche data was nice and easy to understand. In the case of both HRM sites, the inclusion of additional sensors or using the GPS features would have provided more data (distance, speed, etc).
Over the past week I’ve learned that…
- My Fitbit Force may have step variations from other devices, but this doesn’t ultimately make a huge difference in caloric burn at the end of the day – This is spoken when compared to the Loop in my other blog post comparison/battle
- The new Polar Loop Activity Tracker provided very similar results as the Fitbit for daily caloric burn, but it also adds the HRM connectivity feature that provides additional data (to see the daily caloric data comparison chart click HERE)
- For purpose of my testing I’ve learned that the data I have been estimating using the Fitbit calculator is nearly identical to the data I logged using the Polar H7 HRM
So can we trust the caloric burn data from the Fitbit? I’d give that a resounding YES!
Keep in mind that calorie counting can be a FLAWED science. I’ve read elsewhere that people tend to be +/- 20% accuracy when recording their caloric intake, be it because of portion measurement inaccuracies or other reasons. With that kind of variation, having caloric burn data is only one part of the accuracy equation. Do your best to eat healthy foods and if you are going to record them, take the time to do it right.
Given all of this very interesting data, my final thoughts would be as follows:
- More than I had expected, the Fitbit seems quote accurate. Most importantly the calories recorded for exercise are spot-on. Based on these findings I’m left to hope/assume it translates into accuracy for the rest of the day– the “non-workout” regular steps that I take. For me this satiates me concerns.. the Fitbit works!
- Using a heart rate monitor was fun, but probably overkill for most people. These online calculators have a percentage of error, but proved to be accurate enough that using them should not ruin your diet if you are using it for caloric burn and intake comparison. However when in doubt, using an HRM is always the best option.
- Assuming you have a more intense workout regime, then yes, I can see where using a heart rate monitor is a good idea. However I’d always urge you to compare that data to online calculators and more than one device. Not just for fun but for accuracy — which is a good thing to have.
Although I’ve given up (for now) on the Polar Loop band, I am still playing around with the Polar Beat app for using the H7 HRM on occasion. I’m also considering using the DigiFit app for the Scosche. For the moment I’m having trouble justifying the practical use of an HRM in my workout routine but I’d love to test some of the more advanced features and testing they can offer. More data is fun… and as you now must realize, I’m a data geek!
UPDATE (12-MAR-2014) – As you may be aware, Fitbit has issued a recall on the Fitbit Force (LINK). I’d like to point out that I personally have never once experienced any skin irritation issues. I have a half dozen friends with Force devices who also have had ZERO issue. Never the less you should visit that page if you have any issues. And note, the Fitbit Force remains my #1 pick even if it is (possibly) unavailable as you read this post.