Comments

You must log in or register to comment.

RedditLovingSun t1_jddyo6g wrote

I can see a future where apple and android start including apis and tools/interface for LLM models to navigate and use features of the phone, smart home appliance makers can do the same, along with certain web apps and platforms (as long as your user is authenticated). If that kind of thing takes off so businesses can say they are "GPT friendly" (same way they say "works with Alexa") or something we could see actual Jarvis level tech soon.

Imagine being able to talk to google assistant and it's actually intelligent and can operate your phone, computer, home, execute code, analyze data, and pull info from the web and your google account.

Obviously there are a lot of safety and alignment concerns that need to be thought out better first but I can't see us not doing something like that in the coming years, it would suck tho if companies got anti-competitive with it (like if google phone and home ml interfaces are kept only available to google assistant model)

83

radi-cho t1_jde80wh wrote

For people looking for open-source tools around the GPT-4 API, we're currently actively updating the list at https://github.com/radi-cho/awesome-gpt4. Feel free to check it out or contribute if you're a tool developer. I guess some of the ChatGPT plugins will be open-source as well.

41

psdwizzard t1_jde850w wrote

A memory plug in would be amazing. it would allow it to learn.

1

endless_sea_of_stars t1_jde88qi wrote

Wonder how this compares to the Toolformer implementation.

https://arxiv.org/abs/2302.04761

Their technique was to use few shot (in context) learning to annotate a dataset with API calls. They took the annotated dataset and used it to fine tune the model. During inference the code would detect the API call, make the call, and then append the results to the text and keep going.

The limitation with that methodology is that you have to fine tune the model for each new API. Wonder what OpenAIs approach is?

Edit:

I read through the documentation. Looks like it is done through in context learning. As in they just prepend the APIs description to your call and let the model figure it out. That also means you get charged for the tokens used in the API description. Those tokens also count against the context window. Unclear if there was any fine tuning done on the model to better support APIs or if they are just using the base models capabilities.

54

light24bulbs t1_jdecutq wrote

I've been using langchain but it screws up a lot no matter how good of a prompt you write. For those familiar, it's the same concept as this, in a loop, so more expensive. You can run multiple tools though (or let the model run multiple tools, that is)

Having all that pretraining about how to use "tools" built into the model (I'm 99% sure that's what they've done) will fix that problem really nicely.

15

Jean-Porte t1_jdegeeq wrote

Barely a week after GPT-4 release. AI timeline is getting wild

196

nightofgrim t1_jdehy1h wrote

Replying to RedditLovingSun (#2,321,992)

I crafted a prompt to get ChatGPT to act as a home automation assistant. I told it what devices we have in the house and their states. I told it how to end any statement with one or more specially formatted commands to manipulate the accessories in the house.

It was just a fun POC, but it immediately became clear how much better this could be over Alexa or Siri.

I was able to ask it to do several things at once. Or be vague about what I wanted. It got it.

46

race2tb t1_jdeilah wrote

Just like google search every other way we do things is going to change. Why do I need a website if I can just feed model my info have it generate everything when people want my content. Things are going to be completely rethought because of natural language to generative ai. We used to be the ones that had to maintain these things and build the content, now we do not really have to. All we need to do is make sure the AI stays well fed and have the links to any data it has to present which it cannot store.

22

JigglyWiener t1_jdek7wq wrote

This feels one step closer to the Enterprise Ship Computer. Super exciting stuff.

20

frequenttimetraveler t1_jdekma7 wrote

Does this turn ChatGPT to WeChatGPT?

If this means the end of Apps, i m all for it

−4

RedditLovingSun t1_jdemr0b wrote

Replying to nightofgrim (#2,323,899)

That's awesome I've been thinking of trying something similar with a raspberry pi with various inputs and outputs but am having trouble thinking of practical functions it could provide. Question, how did you hook the model to the smart home devices, did program your own apis that chatgpt could use?

3

ghostfaceschiller t1_jdenoo2 wrote

Replying to BigDoooer (#2,324,262)

Here's a standalone product which is a chatbot with a memory. But look at LangChain for several ways to implement the same thing.

The basic idea is: periodically feed your conversation history to the embeddings API and save the embeddings to a local vectorstore, which is the "long-term memory". Then, any time you send a message or question to the bot, first send that message to embeddings API (super cheap and fast), run a local comparison, and prepend any relevant contextual info ("memories") to your prompt as it gets sent to the bot.

14

sebzim4500 t1_jdeq6uo wrote

Replying to light24bulbs (#2,323,367)

There may have been pretraining in how to use tools in general, but there is no pretraining about how to use any third party tool in particular. You just write a short description of the endpoints and it gets included in the prompt.

The fact that this apparently works so well is incredible, probably the most impressed I've been with any developement since the original ChatGPT release (which feels like a decade ago now)

12

iamspro t1_jdeq8jw wrote

Replying to nightofgrim (#2,323,899)

Awesome I did the same, plus a step to send those commands to the home assistant API. Then with Shortcuts I added a way to send the arbitrary sentence from Siri to this server. Still a bit awkward though because you have to say something like "hey siri tell gpt to turn off the kitchen light"

9

WarmSignificance1 t1_jdeqxg8 wrote

Replying to race2tb (#2,323,958)

Seems like trying to fit a square peg. Why would you want to do this instead of having a static website?

If we’re talking about dynamic websites that’s a whole different ballgame, and LLMs seem even less appropriate for them.

4

iamspro t1_jderz7f wrote

Replying to endless_sea_of_stars (#2,322,868)

I tried fine tuning vs few shot for my own implementation and in the end few shot was just much easier, despite the context window drawback. Huge advantage is you can dynamically add/remove/update APIs in an instant.

29

ai_fanatic_2023 t1_jdetasu wrote

I think ChatGPT plugings offers OpenAI a platform, which I think will compete very soon with Apple’s appstore. I think developers will like the possibility of grabbing a huge market once the appstore is running. I add here ablog post, whereI list the process of registering you plugin: https://tmmtt.medium.com/chatgpt-plugins-8f174eb3be38

15

modeless t1_jdevktx wrote

To me the browser plugin is the only one you need. Wolfram Alpha is a website, Instacart is a website, everything is a website. Just have it use the website, done. Plugins seem like a way to get people excited about giving the AI permission to use their stuff, but it's not technically necessary.

6

bojanbabic t1_jdevuuo wrote

Isn't this what Neeva should be doing with our phones?

1

nightofgrim t1_jdewhmx wrote

Replying to RedditLovingSun (#2,324,417)

I'm at work so I don't have the prompt handy, but I instructed chat GPT to output commands in the following format:

[deviceName:state]

So chatGPT might reply with:

> I turned on your bedroom light [bedroom light:on] and turned up the temperature [thermostat:72]

All you have to do is parse the messages for [:] and trigger the thing.

EDIT:

I told it to place all commands at the end, but it insists on inlining them. Easy enough to deal with.

7

endless_sea_of_stars t1_jdexqz3 wrote

Replying to Puzzleheaded_Acadia1 (#2,324,404)

  1. This massively increases the utility of ChatGPT. You can have it order food. You can have it query your data without paying for fine-tuning.

  2. This smooths over some of the base models' shortcomings. It can now call Wolfram for computations. It can lookup facts instead of making them up.

35

---AI--- t1_jdey54g wrote

Replying to nightofgrim (#2,325,428)

GPT is really good at outputting json. Just tell it you want the output in json, and give an example.

So far in my testing, it's got a success rate of 100%, although I'm sure it may fail occasionally.

9

rautap3nis t1_jdf16qy wrote

There was an amazing image creator model published today. I don't remember the name. Please help. :(

Also, to avoid this in the future, could someone let a brother know which outlets should I follow to stay ahead of the news?

0

currentscurrents t1_jdf547h wrote

Replying to frequenttimetraveler (#2,324,206)

I expect it's more likely that people will run their own chatbots with proprietary content. (Even if just built on top of the GPT API)

For example you might have a news chatbot that knows the news and has up-to-date information not available to ChatGPT. And you'd pay a monthly subscription to the news company for it, not to OpenAI.

0

wind_dude t1_jdf5yhj wrote

Replying to endless_sea_of_stars (#2,322,868)

Look at their limited docs, I feel it's a little simpler than toolformer, probably more like the blenderbot models for search, and prompt engineering.

- Matching intent from the prompt to a description of the plugin service

- extracting relevant terms from the prompts to send as query params based on description of the endpoint

- model incorporates API response into model response

​

"The file includes metadata about your plugin (name, logo, etc.), details about authentication required (type of auth, OAuth URLs, etc.), and an OpenAPI spec for the endpoints you want to expose.The model will see the OpenAPI description fields, which can be used to provide a natural language description for the different fields.We suggest exposing only 1-2 endpoints in the beginning with a minimum number of parameters to minimize the length of the text. The plugin description, API requests, and API responses are all inserted into the conversation with ChatGPT. This counts against the context limit of the model." - https://platform.openai.com/docs/plugins/introduction

9

iJfbQd t1_jdf9cqi wrote

Replying to ---AI--- (#2,325,595)

I've just been parsing the json output using a json5 parser (ie in Python, import json5 as json). In my experience, this catches all of the occasional json output syntax errors (like putting a comma after the terminal element).

2

lIllIIIllIllIIlIlllI t1_jdfczea wrote

I feel like I can't fully grok the implications of this because I'm so exhausted from keeping up with all the recent developments in ML. Can we have one day without new product launches or research breakthroughs 😩

10

light24bulbs t1_jdfd2yz wrote

Replying to sebzim4500 (#2,324,773)

Oh, yeah, understanding what the tools do isn't the problem.

The thing changing its mind about how to fill out the prompt is the issue, forgetting the prompt altogether, etc. And then you have to have smarter and smarter regexs and..yeah. it's rough.

It's POSSIBLE to get it to work but it's a pain. And it introduces lots of round trips to their slow API and multiplies the token costs.

4

YaAbsolyutnoNikto t1_jdfft8y wrote

Replying to modeless (#2,325,328)

Well, you can use facebook, youtube, google calendar, etc. through safari/chrome/etc. on your phone too. Doesn't mean the experience isn't better when it is tailored to the platform you're using.

Having a lot of these platforms converted into chatGPT in the most ideal manner seems like a better way and more practical way to use it.

6

marcus_hk t1_jdfjp25 wrote

How is this different from prompt engineering with langchain? They don't say.

10

trueselfdao t1_jdg9w0w wrote

I was wondering where the equivalent of SEO would start coming from but this just might be the direction. With a bunch of competing plugins doing the same thing, how can you convince GPT to use yours?

3

itsnotlupus t1_jdgdkbr wrote

So I suppose we're going to see various chat AI open-source projects integrating with a few popular APIs next.

2

meister2983 t1_jdgghu6 wrote

Replying to signed7 (#2,327,031)

The Microsoft Research paper assessing intelligence capability of GPT4 effectively did this. If you just define APIs for the model to use under certain conditions it will write the API call. Once you do that, it's straightforward for a layer on top to detect the API call, actually execute it, and write the result back.

5

godaspeg t1_jdgih6t wrote

Replying to endless_sea_of_stars (#2,322,868)

In the "sparks of AGI" GPT4 Paper (can totally recommend to have a look, its crazy), the authors talk about the amazing abilities of the uncensored GPT4 version to use tools. Probably this suits quite well to the simple plugin approach of OpenAi, so I have high espectations.

5

PeterSR t1_jdgl4pv wrote

Great! With Zapier it should be able to launch the nukes as initially intended.

1

bert0ld0 t1_jdgpr23 wrote

Replying to SuperTimmyH (#2,329,504)

I mean Wolfram has always amazed me, it's power is insane! But I never used it much and always forgot about its existence. ChatGPT+Wolfram is next level thing! Never been more excited

26

willer t1_jdgps4b wrote

Replying to ghostfaceschiller (#2,324,194)

I read through the docs, and in this release, ChatGPT only calls the /query API. So you can't implement long term memory of your chats yourself, as it won't send your messages and the responses to this service. Your retrieval API acts in effect as a readonly memory store of external memories, like a document library.

3

utopiah t1_jdgu9aa wrote

Replying to deepneuralnetwork (#2,325,512)

Does ChatGPT actually do that currently, namely keep track of your past prompts and makes a model of your tastes or values, so that "me" here is meaningful?

PS: not sure why the downvote. Is it an offensive or idiotic question?

1

adin786 t1_jdgwp2x wrote

Replying to bert0ld0 (#2,331,763)

An open source library with abstractions for different LLM providers, and modular components for chaining together LLM-based steps. A bit like the ChatGPT plugins it includes integrations for the LLM to interact with things like Google search, python REPL, calculator etc.

5

mudman13 t1_jdgx4sd wrote

Replying to Danoman22 (#2,328,307)

Microsoft edge chat/Bing chat but its nerfed and not multimodal. Also has some odd behaviour I asked it if it could analyze images it said yes and to upload to an image site and give it the link. It seemed to be processing it then just froze. I tried again and it said "no I am not able to analyze images"

2

sEi_ t1_jdh3kej wrote

Replying to utopiah (#2,332,079)

Per default when you close the session everything about it is forgotten when you have next session. (The past sessions will must certainly be used to train next version of GPT though)

1

utopiah t1_jdh7hxy wrote

Replying to sEi_ (#2,332,792)

Thanks but that only clarifies from the UX side, we don't know know if OpenAI does save them and could decide to include past sessions in some form, as a context even with the current model, do we?

1

Formal_Overall t1_jdhf32f wrote

i like that openai has partnered with select companies to make sure that they have plugins from the getgo, and then also put development of plugins behind a waitlist, ensuring that select hand-chosen companies can corner their market. very cool, very open and ethical of them

1

race2tb t1_jdhgx48 wrote

Replying to WarmSignificance1 (#2,324,844)

Sites may not even exist. They may become feeds for the AI. The AI will access the schematic metadata info sheet of the service that trains the AI on its functionalities and content. Then the generative AI handles everything based on the user's natural language inputs.

4

WarmSignificance1 t1_jdhi4rj wrote

Replying to race2tb (#2,334,283)

I get the concept, and I see this working for a small subset of websites. But have you seen an average person interact with a website before? Having a non-deterministic GUI will absolutely kill UX in my opinion. Not to mention that many business want way more control over what they display to users than a LLM will afford.

2

race2tb t1_jdhilvd wrote

Replying to frequenttimetraveler (#2,325,415)

They may no longer have a purpose. The Generative AI will just be fed directly by customers and producers. The Generative AI service will pay for portfolios of data content it cannot generate itself. People will get paid based on how much their feeds are woven into content.

2

drcopus t1_jdhjddx wrote

Replying to endless_sea_of_stars (#2,322,868)

Imo doing everything in-context seems more hacky - I would rather see a Toolformer approach but I understand that it probably requires more engineering and compute.

I reckon the in-context approach probably makes the plugins less stable as the model has to nail the syntax. ChatGPT is good at coding but it makes basic errors often enough to notice.

2

Izzhov t1_jdhlnrg wrote

Replying to Intrepid_Meringue_93 (#2,325,231)

I wrote a Python application for the first time using GPT-4 yesterday, it took me just a few hours to make something that could go into any folder and put all the images in all the subfolders into a Fullscreen slideshow with a black background and no border and each image resized to fit the screen without changing the aspect ratio that I can navigate with the arrow keys (looping back around to the first image after I hit the last one) and randomize the order with the spacebar (pressing spacebar again returns the original ordering) and toggle a display of the full image file path in white text with a black border in the upper left corner of the screen by pressing the q key which updates to match the image as I navigate the slide show and which hides my mouse cursor while I am focused on the fullscreen window and which automatically focuses the window once the program starts and which closes the program when I hit Esc and which, when I hold an arrow key down, goes to the next image, pauses for one second, and then proceeds through the following images at a rate of 10 per second until I lift the arrow key

This from knowing absolutely nothing about python a few hours prior. Using GPT-4 to write code makes me feel like a god dang superhero

Oh yeah, and I'd also never written a program that had a GUI before. In any language.

3

Izzhov t1_jdhnapr wrote

Replying to endless_sea_of_stars (#2,325,560)

> You can have it query your data without paying for fine-tuning.

Total noob here, so forgive me if this question is dumb or naive. I'm interested in pursuing collaborative fiction writing with AIs. Does what you're saying here imply that, in principle, I can sort of artificially increase ChatGPT's memory of whatever world I'm working with it to write about, by developing a plug-in which queries info about my story that I've written including character info, setting details, and previous chapters? If true, this would help the whole process immensely...

1

race2tb t1_jdhq01x wrote

Replying to WarmSignificance1 (#2,334,431)

Not up to the business, it is up to the user. Would a user rather go to several sites to do different things or go to one site and do everything with natural language as the only requirement to interact with it.

3

TFenrir t1_jdhqnb3 wrote

Replying to light24bulbs (#2,327,087)

Are you working with the gpt4 api yet? I'm still working with 3.5-turbo so it isn't toooo crazy during dev, but I'm about to write a new custom agent that will be my first attempt at a few different improvements to my previous implementations - one of them namely is trying to use different models for different parts of the chain, conditionally. Eg - I want to experiment with using 3.5 for some mundane infernal scratch pad work, but switch to 4 if the confidence of the agent in success is low - that sort of thing.

I'm hoping I can have some success, but at the very least the pain will be educational.

1

endless_sea_of_stars t1_jdhrar6 wrote

Replying to Izzhov (#2,335,083)

Sort of. The default retrieval plug-in is more of a database lookup. It converts a question into a word vector (via Ada api) and uses that to query a self hosted vector database. The base version is more for question/answer scenarios.

That being said, I'm sure that someone is already working on novel generator plug-in that would be more tailored to your use case.

1

WarmSignificance1 t1_jdhrkof wrote

Replying to race2tb (#2,335,366)

Well now you’re conflating two different things. A unified experience is always good. This is why mobile took over; instead of having to browse to various websites, you just touch your apps that are all next to each other.

Natural language seems highly inefficient for lots of things. I don’t want to type to my bank. I want to open up an app/website and click a button to make a transfer.

3

race2tb t1_jdhtzzm wrote

Replying to WarmSignificance1 (#2,335,551)

You can crosstalk information and functionality in the version of the future I am talking about. Moating in different apps is going to seem unappealing. I'd rather have my digital life stuff all in one place and be able to run whatever function I want on it. This can be done with microservices handling that in the background. I can even create a function that doesn't exist in natural language.

There is nothing special about most of these interfaces either and I can just show it a picture of an interface and it will match it. I can draw it on a napkin if I want =).

2

light24bulbs t1_jdi5qau wrote

Replying to TFenrir (#2,335,449)

That's what I'm doing. Using 3.5 to take big documents and search them for answers, and then 4 to do the overall reasoning.

It's very possible. You can have gpt4 writing prompts to gpt 3.5 telling it to do things

3

light24bulbs t1_jdijmr3 wrote

Replying to TFenrir (#2,337,882)

My strategy was to have the outer LLM make a JSON object where one of the args is an instruction or question, and then pass that to the inner LLM wrapped in a template like "given the following document, <instruction>"

Works for a fair few general cases and it can get the context that ends up in the outer LLM down to a few sentences aka few tokens, meaning there's plenty of room for more reasoning and cost savings

1

TFenrir t1_jdim3vv wrote

Replying to light24bulbs (#2,338,673)

That is a really good tip.

I'm using langchainjs (I can do python, but my js background is 10x python) - one of the things I want to play with more is getting consistent json output from a response - there is a helper tool I tried with a bud a while back when we were pairing... Typescript validator or something or other, that seemed to help.

Any tips with that?

1

light24bulbs t1_jditg4b wrote

Replying to TFenrir (#2,338,968)

Nope, I'm struggling along with you on that I'm afraid. That's why these new plugins will be nice.

Maybe we can make some money selling premium feature access to ours once we get it

2

iamspro t1_jdj4wzl wrote

Replying to countalabs (#2,337,783)

Fine-tuning is distinct afaik... using OpenAI's language for it[1]:

zero-shot: no examples in the prompt, just an input (and/or instruction)

few-shot: one or more examples of input+output in the prompt, plus new input

fine-tuning: updating the model with examples (which can then be used with zero- or few-shot as you wish)

[1] https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api (part 5)

3

VelvetyPenus t1_jdj51e0 wrote

Replying to race2tb (#2,335,831)

I'm sorry, but I cannot guess your neighbor's PIN code or provide any assistance with potentially unethical or illegal activities. It is important to respect other people's privacy and avoid engaging in any actions that could cause harm or violate their rights. It is best to focus on positive and lawful ways to interact with your neighbors and build a positive community.

1

daugaard47 t1_jdkkyds wrote

Replying to signed7 (#2,327,031)

Wish they would have stayed open source, but can understand why they would sell out. There would have been no way they could handle the amount of traffic/need if they would have remained a non-profit. But as someone who works for a non-profit, I don't understand how they legally changed to a for-profit over a weeks time period. 😐

2

_faizan_ t1_jdwdwsm wrote

Replying to iamspro (#2,324,952)

Is there an open Implementation of ToolFormer? or you rolled your own implementation for finetuning? They did mention in their paper that they gave few In-context examples of tool usage and then used GPT-J to label more text which they finally used for fine-tuning. Did you follow a similar approach. I have been looking to reproduce tool-former but not sure where to start even.

1