As an associate data analyst and full-stack developer at Gallaudet University, I provided consulting services to professors and software engineers and mentored researchers. I conducted statistical and cost-benefit analyses and developed AI models using Python and R. I also developed several apps in JavaScript and TypeScript, incorporating full-stack technologies. One of my notable projects wasFingerspelling Battleship.
App Demo
Objective Gallaudet University aimed to expand their existing sign language dataset by collecting additional data on modern sign languages and fingerspelling. Despite already having one of the largest sign language datasets, they sought to ensure that the data remains current and reflective of the evolving nature of sign language.
Challenge
The number of individuals participating in surveys across research institutes has decreased over time, slowing the data collection process due to a lack of incentive.
Developing game apps to speed up data collection risks introducing gamesmanship actions that differ from real-world actions, potentially skewing the data.
Limited staffing resources make it challenging to ensure the quality of sign language data prior to training, validating, and testing AI models.
My Approach
Technology Selection: Implemented AWS S3 + Express.js + Node.js + Socket.io + React.js tech stack for the app, with GitHub for version control and CI/CD, Docker for containerization, and AWS Elastic Beanstalk for deployment at a cost of $40 monthly.
Quality Assessment: Combined class functions with video segmentation to verify hand poses and movements corresponding to fingerspelling letters. Applied the Randomized Controlled Trials with Wizard-of-Oz experiment to evaluate the model.
Data Integration: Tracked users using UUIDs from Okta accounts with consent, automatically uploading fingerspelling videos and metadata (device type, geolocation, session times) to AWS S3, integrated with Gallaudet's existing dataset for AI model training.
Results
Real-Time Fingerspelling Recognition: The app can capture and analyze fingerspelling in real-time, aiding in the collection of high-quality sign language data.
Automated Video Uploads with Metadata: User videos, along with session metadata like device type, geolocation, and timestamps, are automatically stored in AWS S3, streamlining data collection.
Enhanced User Engagement Through Multiplayer Gameplay: The online multiplayer mode is designed to handle player interactions efficiently, including features like an idle timer, wait time management, and automatic disconnection if a server is busy, helping maintain a smooth, engaging user experience.
Note: The link to this app is currently unavailable as development is still ongoing.