RecycPal

Hackathon: Humble Beginnings

RecycPal started as a hackathon project for the DeltaHacks 8, and is slowly evolving from a personal project into a real commercial AIaaS (Artificial Intelligence as a Service).

The first rendition was a image classifier for recyclable materials that we made in 36 hours. I was primarily in charge training the AI and connecting it to our iOS application. Initially, my teammates and I wanted to use PyTorch, an open source machine learning framework developed by Facebook(Meta), however we quickly figured out that there was no way we were going to learn and figure out how to train, develop and deploy an app with it within the time frame. We did what great scientist like what all great engineers do, and we pivoted to a different framework. Specifically we pivoted into Apple's machine learning framework, CreateML. Seeing how seemingly easy it was to implement and the amount of documentation and examples we decided to run with it.

Training the image classifier was actually really easy, CreateML was pretty intuitive and the only thing we had to really worry about was how to format the folder structure and finding all the images for training. We were fine for the most part, until it came down for me to implement the model into the application.

Hackathon: Implementation Issues

What I thought were good examples and a plethora of documentation turned out to be outdated. All of the documents and tutorials I found were ultimately useless, and I had never really come across the issue of 'outdated documentation'. I spent 5 hours trying to initalize our model, reading through more documentation, scouring GitHub for examples, and even going on the Chinese search engine Baidu to see if I could find anything that could help. My teammate and I put our collective brains together to figure out the issue and after another our of effort we were able to initialize our model and even connect it to our camera view. Which is where the AI would update the label and tell the user what recyclable material is being seen.

We finished, uploaded on GitHub, and submitted. Spectacularly, we ended up in 3rd place!! However, that was only the first step for RecycPal. Our initial vision was to make RecycPal a object detection artificial intelligence, not an image classifier. Riding the waves of dopamine after our victory, I began the groundwork for our new AI.

The Journey: Choosing the Framework

Initially I tried using CreateML's object detection model, because training was easy and required zero code. I ended up training the first four versions of RecycPal on CreateML. This was done by using IBM's Cloud Annotation software to do all of the labelling. Now that annotating and training were done, I figured it was time to deploy a prototype. Now using the model in an iOS application proved to be insanely difficult because I had no updated documentation, and any YouTube video that was uploaded prior to iOS 15 was of no help anymore. The last nail in the coffin was when I wasn't able to easily control the confidence threshold, this was when I finally decided to look into PyTorch again.

Going in with no AI experience I knew PyTorch's learning curve was going to be challenging. Just like any normal developer, I looked to their documentation, and YouTube tutorials. The first problem to tackle was to learn how to properly annotate images, and of course it sounds simple at first. However, the amount of work and nuance that goes into vetting the dataset, pre-processing, and balancing classes was something that no tutorial prepared me for. This whole process of learning how to decently prepare an object detection dataset took a good month to consolidate.

The Journey: Training Montage

The only way I learned how to properly prepare the dataset, was to also train the model. Before the actual training, I had to annotate which were done in RoboFlow, because I was able to format my dataset to match the new YOLOv5 file structure and label map. I figured out through YouTube tutorials that I could train my models on Google Colab using their free GPU. That being said, RecycPal versions 5 through 8 were unrecyclable, pun intended. They were garbage and couldn't detect anything, because I preprocessed my dataset by changing hues, saturations, too much noise/grain, etc. This basically trained the AI to see things that were never even going to be used in a real world application, hence why it's performance was so inaccurate.

Discouraged yes, but with my heart ablaze I wasn't going to take these failures sitting down. I discarded my old dataset, and started anew. Thrusting RecycPal into a new era with versions 9 through 17 (Latest version at the time of writing).

The Journey: Graphics Card vs Google Colab

Versions 9-16 were actually done on my Linux laptop because the free Google Colab GPUs were just moving too slow, and I wasn't lucky enough to get a Tesla T4 on a weekday. Cloning the git, training, recording with wandb.ai (Weight and Biases), and running inferences. I was able to maximize on my gaming laptop's GPU (RTX 2060 Max Q), and with all these experiments I was able to fine tune the dataset even more. Finding out that to add a new item at RecycPal's current scale only requires about 40 images. These images, require different lighting, lighting temperatures, angles, backgrounds, depth, etc. and by using some rotation, brightness, and flip image processing I'm able to increase those initial 40 images to 120 images per class/item.

This system has now allowed RecycPal to begin blossoming and to accurately identify 16 different recyclable items, ranging from Crushed Zephyrhills Bottles to Arizona Cans. I took notice that as the datatset grew larger the training time slowly went from 45 minutes to 114 minutes (version 16). Monitoring my graphics card's temperature it began pushing up to 95C and I was starting to get concered about it's life span. This is when I bought Google Colab Pro. This would guarantee me a P100 or Tesla T4 every time I trained my models, which both clocked faster than my GPU, and also had 16GB of RAM (10GB more than mine).

The Journey: Where Are We Now? Where Are We Going?

RecycPal is currently at version 17, and focus has shifted from training the AI to deployment. Work on deploying a web application (Django), iOS application, and on IoT devices (ESP32) have already begun.

The web application is going to require an API to be able to process the user uploaded images and eventually a live video using OpenCV. I am currently looking into usingSyndicai, a low-code MLOps platform, as our API, however I am still running into some model initalizaztion errors. Using TorchServe or making an API are still in the realm of possibilities.

The iOS application options are still being explored. Using a YOLOv5 model straight into Swift isn't an option yet, but using the old YOLOv2 framwork in CreateML is still a strong possibility. The only thing holding the application back is the lack of updated documentation, or my lack of ability to find the updated documentation. Even if that doesn't work, we're still not dead in the water yet, because Turi Create, an open source Python library made by Apple, is also in the cards for deployment.

Deploying on IoT devices, refers to microcontrollers (MCUs) and microprocessors, such as Arduinos, ESP32/8266, STM32s, and Raspberry Pies. Integration into these devices will require MicroPython and TinyML, and the reason for deployment on these devices are to eventually help the global effort of cleaning up pollution in the oceans and assisting local waste management facilities in optimizing the separation of recyclables.

Hopefully, by the time I revisit this article I'll be able to document the journy of how I overcame these deployment issues. 👏🏽