Continuing on progress without eye diameter feedback for now, this week I focused on creating a metric feedback script that would combine what I have working so far and export data to an excel script. This would include the facial coding system and the heartbeat sensor working together to export a single excel file from Unity.
Also, I decided to spend some time researching other research projects that have also worked with biofeedback technologies to learn about their findings. I want to make sure that I am immersed in that community so I can successfully build upon whats been done before me. The first step in doing this is prioritizing all the papers from CHI PLAY 2018 that seemed most relevant to my project. ( https://chiplay.acm.org/2018/proceedings/ ).
CHI PLAY is the international and interdisciplinary conference (by ACM SIGCHI) for researchers and professionals across all areas of play, games and human-computer interaction (HCI).
The papers that were most inspirational to me so far in thinking about this project was "Augumenting Co-Located Social Play With Biofeedback: An Interactional Approach" and "Effects of Manipulating Physiological Feedback in Immersive Virtual Environments".
Biofeedback Excel Report
Exporting to an excel document from Unity was not as easy as I was hoping. Mainly because I wanted to export an excel document that would carry dynamic graphs and multiple sheets as well. I found that DLLs that people used in the past were outdated, and things were breaking. I created my own library targeting the correct framework for Unity, and then imported it into Unity. Things would almost work using Micosoft.Interlop.Excel in my library, but Unity didn't like the InterlopServices that it was using. I did end up finding Neuzilla's library NPOI that worked just fine with Unity except when it came to creating graphs in Excel. It kept giving me an error that said "No implementation" when trying to use the CreateChart() method. So I downloaded DOTPEEK from Jetbrains so I could decompile the NPOI DLL to see what was going on under the hood. (https://www.jetbrains.com/decompiler/ )
Once I found the method inside of the IDrawing Interface, turns out, its exactly like I thought. It literally doesn't have implementation for drawing charts. All it has is a throw that says "No implementation" LOL someone didn't finish the job perhaps? It started to feel like I was running out of options with Unity because it felt like nothing was working. Until I thought of a back-door way to make this work the way I want. I would already have a pre-built excel document that has a table in it, then just populate the data dynamically then save it as a copy.
I asked my faculty adviser for this project, Dr. Tom Carbone, to be my first test subject in capturing a recovery heart rate. To do this, subjects must find their target heart rate for their age. Target heart rates are based on 60%-80% of estimated maximum heart rates (220 minus age). Then immediately after exercising, we record 2 minutes of heart rate data. Lastly, to get the recovery heart rate we subtract the 2-minute mark heart rate from the heart rate right after exercising. To accomplish this, he ran around the school with the MIO Heartbeat sensor on to get his heart rate up, and then I captured the data through my metric script right when he got back. Results showed that his "biological age is moderately younger" than his calendar age according to https://www.azumio.com/blog/fitness/what-does-heart-say-about-exercise-recovery.
I also tested this against a heart beat app on a Apple watch to see that the heart rates were accurate with each other. I still would like to optimize the speed of the algorithm for it still feels a little slow.