Viewing a single comment thread. View all comments

RedditLovingSun t1_jddyo6g wrote

I can see a future where apple and android start including apis and tools/interface for LLM models to navigate and use features of the phone, smart home appliance makers can do the same, along with certain web apps and platforms (as long as your user is authenticated). If that kind of thing takes off so businesses can say they are "GPT friendly" (same way they say "works with Alexa") or something we could see actual Jarvis level tech soon.

Imagine being able to talk to google assistant and it's actually intelligent and can operate your phone, computer, home, execute code, analyze data, and pull info from the web and your google account.

Obviously there are a lot of safety and alignment concerns that need to be thought out better first but I can't see us not doing something like that in the coming years, it would suck tho if companies got anti-competitive with it (like if google phone and home ml interfaces are kept only available to google assistant model)

83

nightofgrim t1_jdehy1h wrote

I crafted a prompt to get ChatGPT to act as a home automation assistant. I told it what devices we have in the house and their states. I told it how to end any statement with one or more specially formatted commands to manipulate the accessories in the house.

It was just a fun POC, but it immediately became clear how much better this could be over Alexa or Siri.

I was able to ask it to do several things at once. Or be vague about what I wanted. It got it.

46

iamspro t1_jdeq8jw wrote

Awesome I did the same, plus a step to send those commands to the home assistant API. Then with Shortcuts I added a way to send the arbitrary sentence from Siri to this server. Still a bit awkward though because you have to say something like "hey siri tell gpt to turn off the kitchen light"

9

nightofgrim t1_jdevw3a wrote

I didn’t hook up voice because of that awkward part. If I could get my hands on a raspberry pi I might make my own listening device.

6

RedditLovingSun t1_jdemr0b wrote

That's awesome I've been thinking of trying something similar with a raspberry pi with various inputs and outputs but am having trouble thinking of practical functions it could provide. Question, how did you hook the model to the smart home devices, did program your own apis that chatgpt could use?

3

nightofgrim t1_jdewhmx wrote

I'm at work so I don't have the prompt handy, but I instructed chat GPT to output commands in the following format:

[deviceName:state]

So chatGPT might reply with:

> I turned on your bedroom light [bedroom light:on] and turned up the temperature [thermostat:72]

All you have to do is parse the messages for [:] and trigger the thing.

EDIT:

I told it to place all commands at the end, but it insists on inlining them. Easy enough to deal with.

7

---AI--- t1_jdey54g wrote

GPT is really good at outputting json. Just tell it you want the output in json, and give an example.

So far in my testing, it's got a success rate of 100%, although I'm sure it may fail occasionally.

9

nightofgrim t1_jdf00h9 wrote

If it fails, reply that it screwed up and needs to fix it. I bet that would work.

5

iJfbQd t1_jdf9cqi wrote

I've just been parsing the json output using a json5 parser (ie in Python, import json5 as json). In my experience, this catches all of the occasional json output syntax errors (like putting a comma after the terminal element).

2

Smallpaul t1_jdejh9x wrote

>I crafted a promoted to get ChatGPT

?

1

nightofgrim t1_jdew090 wrote

Prompt. Thanks. Damn auto correct needs ChatGPT level intelligence.

5

frequenttimetraveler t1_jdemv5n wrote

Google will more likely come up with its own version of this. It's already in every android phone and the iphone search box. It's a natural fit

Despite being there first, microsoft will have a hard time when google gatekeeps everything

8

signed7 t1_jdfcyxr wrote

Models need to get a lot smaller (without sacrificing too much capability) and/or phone TPUs need to get a lot better first

2

Wacov t1_jdfh05b wrote

Don't typical home assistants already do voice recognition in the cloud? It's just the attention phrase ("ok Google" etc) they recognize locally

6

RedditLovingSun t1_jdfds8b wrote

I'm optimistic, between the hardware and algorithmic advances being made

1

bernaferrari t1_jdhy0qz wrote

Good news is, deep learning APIs are decoupled from android, so Google can just update via play store (as long as the device gpu supports it).

2

drunk-en-monk-ey t1_jde2an2 wrote

It’s not so straight forward

−7

RedditLovingSun t1_jde2yvh wrote

I'm not disagreeing with you but out of curiosity can you elaborate on any factors I may have overlooked?

20

wywywywy t1_jde6ltj wrote

Yes but a lot of not-so-straight-forward things happened in the last few weeks already!

5

ghostfaceschiller t1_jdekaf3 wrote

People really need to update their priors on what kind of things are straightforwardly possible or not. Like if you majorly updated your expectations last week, you are way behind and need to update them again.

4

ZenDragon t1_jde4uj8 wrote

Agreed, but it's not like they have to implement everything all at once. Such integration would already be useful as soon as a small selection of the most basic features are working.

3