Thursday, January 2, 2020

Misty and Azure

In June of 2018, I purchased Misty from Misty Robotics as a part of a Kickstarter. I'm backer #277 and got the t-shirt to prove it.

I got my Misty on December 4, 2019 and it was like Christmas. It was truly like being a kid again with a shiny reflective toy. I was so excited, I skipped everywhere I went. 


Unfortunately, being a grown up - I had to put her on the back burner with minimal play time until the holiday break.

During the holiday break, Misty and I spent all my free time together - trying things out and getting a feel for each other. I really like her - but I think the jury is still out on whether she likes me. :)

My dog has been getting up there in age. Butters turned 12 last September and lately he's been showing his age. He has more grey now, sleeps more, goes through coughing spells, and his stomach has become very sensitive.

Misty has several QUALCOMM processors and one of them is using Windows IoT. As a Azure Dev, naturally I wanted to experiment.

Isn't he so stinking cute!?
Since I'm spending more time worried about Butters, I would like Misty to look after my dog while I'm away for more than a few hours. Here are my ultimate first goals:
  1. Have Misty roam the house looking for butters.
  2. Upon recognition, take picture.
  3. Text to me.
  4. Sigh of relief that he's okay.
Here's what I've got so far. I went off the beaten path just a tad - but making my way around again.
  1. Misty recognizes me, my daughter and my grandson through her built in facial recognition.
  2. I purposely did not train her to know my husband to test stranger identification.
  3. Upon recognition, she sends a http request with a message (i.e., Hi Tammy) param to an Azure Function using a HTTP trigger.
  4. The Azure function calls to Text to Speech api service with the message where it's converted to speech.
  5. The response is stored in Azure storage container as a blob with content-type 'audio/x-wav'.
  6. Misty calls the storage api where the blob is stored.
  7. She downloads the wav file and plays it.
So really, there is very minimal processing going on in Misty - everything is done in the cloud.

What happens if she doesn't recognize you? Well .... she displays an angry face, growls and says "Stranger Danger!".

Just in case you're lost and would like to be more lost - here is a list of the SDK's/Extensions that I used.
  1. Azure Storage SDK
  2. Azure Tools Extension
  3. NodeJS
  4. Azure Function App - HTTP Trigger
  5. Text to Speech APIs
  6. Visual Studio Code/Azure Functions for VSC - great for debugging issues.
Once I got this working, I tried to train Misty to learn my dog's face using her pre-built facial recognition software. But alias, she can only recognize humans through her existing facial recognition software. 

So .... this will be one of the next problems to solve. I'll need to use Azure Cognitive Services computer vision to take a picture of him and send for recognition and analysis.

I also just started playing with the her built in navigation capabilities, so she can travel around my apartment in the search of Butters.


Here's a video of her recognizing me (cause I'm sitting in front of her), responding visually and saying my name. Kind of lame in itself, unless you understand what's behind it.

But ... baby steps. Here's some code sources to get you started.

I started with this great tutorial written by Chris Meyer. Chris goes on to do some computer vision stuff which is next for me. He also does a great job in the actual walk through of setting the Azure Function up.

And here's mine so far. Here you'll find all the files you need for Misty and Azure. Most of what I changed was using the BlobService to store the speech files instead of converting to base64 string and streaming back.

I'm really thinking maybe Misty needs a tutu or something. I feel a fashion design coming in her near future.


No comments:

Post a Comment