My research projects are all done on my own time, generally with assistance from academia, and are often not more than just crazy ideas I wish to explore further.
Realtime voice AI generating UI on the fly
An experiment in realtime, multimodal AI: a voice-driven assistant that streams spoken input and renders adaptive UI components in response — interfaces composed on the fly to match what the user is asking for, instead of being pre-designed.
Try it at persona.rgbknights.com.
TFT
A sandbox for exploring realtime video filters and effects in the browser — testing pipelines, shaders, and processing techniques against live camera input.
Try it at tft.rgbknights.com.
What is Screeps you may ask?
"It's an open-source sandbox MMO RTS game for programmers, wherein the core mechanic is programming your units' AI. You control your colony by writing JavaScript which operates 24/7 in the single persistent real-time world."
Why Screeps? The most interesting thing about AI deployment around Screeps is the built-in GYMS. The real-time game loop also creates a restriction not often seen in other AI development where training and execution must be done in a timely manner — in very small time steps over a long period as the game operates in ticks with a fixed amount of CPU time.
Check out the source code for different versions of static systems I have created that the AI is attempting to learn from / play against.
NOTE: Some may be incomplete or missing files required to compile, as I was using a local NPM package at one point.
Future work: exploring moving from the private server repo (which is always months or years behind) to using the official repo.
References: openai self-play, 1810.08575, 1810.12894, 1810.12282, 1807.04742, GOAP.
The crazy idea to combine these two different technologies came about after a 6-hour VR session when I thought to myself:
I wish the game knew I was getting tired and would change the difficulty so I could have kept playing…
At that lightbulb moment I was drawn to Electroencephalogram (EEG) technologies, exploring options to measure the bio feedback of the player. The plans evolved into combining VR and EEG together so the VR environment receives bio and mental feedback about the player. VR was the best fit because you would need to wear a headset to capture the EEG readings anyway. This had the unexpected benefit of the EEG component offsetting the weight of the VR component, making the overall headset easier to wear for longer periods. After successfully creating a proof of concept with a mobile-device VR adaptor and a homemade EEG kit, it was time to upgrade. I am currently waiting to acquire an Emotiv Epoc+ and an Oculus Rift to continue the research, mainly because Emotiv's commercial headset and SDK can produce basically the same set of states my models can.
References: PMC3990628, PMC4957988, PubMed 27547552.
This research started as an exploration of predictions and recommendation engines and why they suck. It quickly evolved as I needed to create my own data sets. The one I felt would benefit most from a better recommendation engine was music, so I switched from YouTube to SoundCloud. Through the SoundCloud API I could access track titles, artists, and also things I didn't think would be interesting but turned out very useful — track art, waveform, and reposting chains.
The system is not so much an AI (although it does have some neural networks to transform data) as it is a set of logic checks against my training set, plus a black list. It will automatically repost tracks from my stream based on a series of checks, evaluating: title (recently reposted?), description & tags (against the black list), artist & publishers (weighted from my likes), reposting chain, track art (similarity score), track waveform (intensity + similarity), and an audio sample (Fourier series, speech-to-text, filtered against the black list).
Most processes run in parallel and are non-blocking thanks to a graph database of transient details. Final results feed an AI with two outputs: yes/no to repost and which playlist to add to. If "no" but component values are high, the track ends up in a review playlist for me to look at later.
See the results on SoundCloud.
NOTE: I do not run the AI all the time as it generates a lot of data and the SoundCloud API is throttling me, so the results show up in spurts.
References: IJEDR1404092, arXiv 1703.09109, VU Fernandez.
The idea behind Geo Adventures was conceived after being introduced to Pokémon GO. The combined mapping and AR functionality created a new generation of games. Early SDKs for AR had a lot of issues, but with Google's ARCore and Mapbox's Maps for Unity the framework finally existed.
POI: creating my own point-of-interest database, different from Niantic's. Currently researching mapping real-world POIs to fantasy & sci-fi themes.
NavMesh generation: creating navigation data so agents can walk paths in parks, malls, etc.
Indoor play: exploring options around combining outdoor play with indoor elements — needed for themes that include gathering / crafting or attack / defence.
WiFi: using the current wifi access point name/ID to signal that you are inside a fort or protected building, enabling attack/defend or raid themes on static locations.
References: semanticscholar, dbis ulm, doi:10.1016/j.procs.2016.08.017, ARCore.
Autonomous Neural Training and Sharing
My ANTS research started shortly after my SPIDERS research. Without any knowledge of AI and various sub-fields years ago, I stumbled into the beginnings of neural network design without understanding what a neural network was. After SPIDERS I wanted to expand the ability for the system to learn — and ANTS was born. Its greatest achievement was the self-assembly of a system that could analyze .NET assemblies for their type information and create an abstract syntax tree (AST) of the resulting flow of types from different functions, then predict and test chains of those function calls. I even moved all the code into Azure for some large-scale tests against the default .NET libraries, but in the end the cost of the project was too much to continue at scale, so the research was archived in 2013.
Self Propagation for Identification of Data Equality and Relationships in Subsystems
In the early years of the internet there was no Google. (I know — hard to believe…) In those days, your site list came from a dog's breakfast of different sources. It wasn't until the introduction of web crawlers that true search engines were created. While all this madness over who had the best web crawler was going on, I was interested in the idea of crawling code the same way crawlers were exploring the web. I created my own version of a crawler, SPIDERS. It helped me explore the C and Assembly languages. I also created a version that could explore the web the same way web crawlers do, and I still use it to this day to do deep dives into search engines.