Viewing a single comment thread. View all comments

currentscurrents t1_jcziz0q wrote

I'm gonna end up buying a bunch of 24GB 3090s at this rate.

93

Straight-Comb-6956 t1_jd08cq1 wrote

LLaMa/Alpaca work just fine on CPU with llama.cpp/alpaca.cpp. Not very snappy(1-15 tokens/s depending on model size), but fast enough for me.

39

lurkinginboston t1_jd0zr7c wrote

I will assume you are much more knowledgeable than I am in this space.. have few basic questions that have been bothering me since all the craze started around GPT and LLM recently.

I managed to get Alpaca working on my end using the above link and get very good result. LLaMa biggest takeaway was it is able to reproduce quality comparable to GPT and much lower compute size. If this is the case, why is the ouput much shorter on LLaMa than what I get on OpenGPT? I would imagine the OpenGPT reponse is much longer because ... it is just bigger? What is the limiting factor to not for us to get longer generated response comparable to GPT?

ggml-alpaca-7b-q4.bin is only 4 gigabyt - I guess this what it means by 4bit and 7 billion parameter. Not sure if rumor or fact, GPT3 model is 128B, does it mean if we get trained model of GPT, and manage to run 128B locally, will it give us the same results? Will it be possible to retrofit GPT model within Alpaca.cpp with minor enhancement to get output JUST like OpenGPT? I have read to fit 128B, it requires muliple Nvidia A100.

Last question, inference means that it gets output from a trained model. Meta/OpenAI/Stability.ai have the resources to train a model. If my understanding is correct, Alpaca.cpp or https://github.com/ggerganov/llama.cpp are a sort of 'front-end' for these model. They allow us to provide an input to get an output by inference with the model. The question I am trying to ask is, what is so great about llama.cpp? Is it because it's in C? I know there is Rust version of it out, but it uses llama.cpp behind the scene. Is there any advantage of an inference to be written in Go or Python?

10

KerfuffleV2 t1_jd1kfyp wrote

Note: Not the same person.

> I would imagine the OpenGPT reponse is much longer because ... it is just bigger?

llama.cpp recently added a commandline flag to disable the end of message marker from getting generated, so that's one way you can try to force responses to be longer. (It doesn't always work, because the LLM can start generating irrelevant content.)

The length of the response isn't directly related to the size of the model, but just having less information available/relevant could mean it has less to talk about in a response.

> GPT3 model is 128B, does it mean if we get trained model of GPT, and manage to run 128B locally, will it give us the same results?

If you have the same model and you give it the same prompt, you should get the same result. Keep in mind if you're using some other service like ChatGPT you aren't directly controlling the full prompt. I don't know about OpenGPT, but from what I know ChatGPT has a lot of special sauce not just in the training but other stuff like having another LLM write summaries for it so it keeps track of context better, etc.

> Last question, inference means that it gets output from a trained model.

Inference is running a model that's already been trained, as far as I know.

> If my understanding is correct, Alpaca.cpp or https://github.com/ggerganov/llama.cpp are a sort of 'front-end' for these model.

The model is a bunch of data that was generated by training. Something like llama.cpp is what actually uses that data: keeping track of the state, parsing user input into tokens that can be fed to the model, performing the math calculations that are necessary to evaluate its state, etc.

"Gets its output from", "front end" sound like kind of weird ways to describe what's going on. Just as an example, modern video formats and compression for video/audio is pretty complicated. Would you say that a video player "gets its output" from the video file or is a front-end for a video file?

> The question I am trying to ask is, what is so great about llama.cpp?

I mean, it's free software that works pretty well and puts evaluating these models in reach of basically everyone. That's great. It's also quite fast for something running purely on CPU. What's not great about that?

> I know there is Rust version of it out, but it uses llama.cpp behind the scene.

I don't think this is correct. It is true that the Rust version is (or started out) as a port of the C++ version but it's not using it behind the scenes. However, there's a math library called GGML that both programs use, it does the heavy lifting of doing the calculations for the data in the models.

> Is there any advantage of an inference to be written in Go or Python?

Same advantage as writing anything in Go, which is... Just about nothing in my opinion. See: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride

Seriously though, this is a very, very general question and can be asked about basically any project and any set of programming languages. There are strengths and weaknesses. Rust's strength is high performance, ability to do low level stuff like C, and it has a lot of features aimed at writing very reliable software that handles stuff like edge cases. This comes at the expense of having to deal with all those details. On the other hand, a language like Python is very high level. You can just throw something together and ignore a lot of details and it still can work (unless it runs into an unhandled case). It's generally a lot slower than languages like Rust, C, C++ and even Go.

However, for running LLMs, most of the processing is math calculations and that will mean calling into external libraries/modules that will be written in high performance languages like C, Rust, etc. Assuming a Python program is taking advantage of that kind of resource, I wouldn't expect it to be noticeably slow.

So, like a lot of the time, it comes down to personal preference of what the developer wants to use. The person who wrote the Rust version probably like Rust. The person who wrote the C++ version likes C++, etc.

13

keeplosingmypws t1_jd5xygm wrote

I have the 16B parameter version of Alpaca.cpp (and a copy of the training data as well as the weights) installed locally on a machine with an Nvidia 3070 GPU. I know I can launch my terminal using the Discrete Graphics Card option, but I also believe this version was built for CPU use and I’m guessing that I’m not getting the most out of my graphics card

What’s the move here?

1

KerfuffleV2 t1_jd7sb4u wrote

llama.cpp and alpaca.cpp (and also related projects like llama-rs) only use the CPU. So not only are you not getting the most out of your GPU, it's not getting used at all.

I have an old GPU with only 6GB so running larger models on GPU isn't practical for me. I haven't really looked at that aspect of it much. You could start here: https://rentry.org/llama-tard-v2

Keep in mind you will need to be pretty decent with technical stuff to be able to get it working based on those instructions even though they are detailed.

1

keeplosingmypws t1_jd9wpwm wrote

Thanks for leading me in the right direction! I’ll letcha know if I get it working

1

Unlucky_Excitement_2 t1_jdavhcr wrote

Bro what are you talking about LOL. Its context length he's discussing. There are multiple ways[all of which I'm expertimenting with] ->

  1. flash attention
  2. strided context window
  3. finetuning on a dataset with longer sequences
0

KerfuffleV2 t1_jdbrkc1 wrote

Uh, did you reply to the wrong person or something? Your post doesn't have anything to do with either mine or the parent.

3

gliptic t1_jd2bsc7 wrote

In fact, GPT3 is 175B. But GPT3 is old now and doesn't make effective use of those parameters.

1

uspmm2 t1_jd1jh1b wrote

are you talking about the 30b one?

1

Straight-Comb-6956 t1_jd1srkd wrote

Haven't tried the 30B model. 65B takes 900ms/token on my machine.

3

msgs t1_jd46yf9 wrote

do you have a link to a torrent/download for the 30B or 65B weights that works with Alpaca.cpp? reddit DMs are fine if don't want to post it publicly.

1

SpiritualCyberpunk t1_jd1m06i wrote

Idk why, but after the first answer to a question addressed to it, mine spewed out random nonsense. Literally unrelated things.

1

gybemeister t1_jczucbf wrote

Any reason, beside price, to buy 3090s instead of 4090s?

13

currentscurrents t1_jczuqo8 wrote

Just price. They have the same amount of VRAM. The 4090 is faster of course.

26

satireplusplus t1_jczz8e6 wrote

VRAM is the limiting factor to run these things though, not tensor cores

13

currentscurrents t1_jd007aa wrote

Right. And even once you have enough VRAM, memory bandwidth limits the speed more than tensor core bandwidth.

They could pack more tensor cores in there if they wanted to, they just wouldn't be able to fill them with data fast enough.

17

pointer_to_null t1_jd0bv74 wrote

This is definitely true. Theoretically you can page stuff in/out of VRAM to run larger models, but you won't be getting much benefit over CPU compute with all that thrashing.

5

Enturbulated t1_jd1x9uu wrote

You are absolutely correct. text-gen-webui offers "streaming" via paging models in and out of VRAM. Using this your CPU no longer gets bogged down with running the model, but you don't see much improvement in generation speed as the GPU is churning with loading and unloading model data from main RAM all the time. It can still be an improvement worth some effort, but it's far less drastic of an improvement than when the entire model fits in VRAM.

2

shafall t1_jd2380o wrote

To give some more specifics, most of the time its not the CPU that copies the data on modern systems, it is the PCI DMA chip (that may be on the same die though). CPU just sends address ranges to DMA Info

1

wojtek15 t1_jd0p206 wrote

Hey, recently I was thinking if Apple Silicon Macs may be best thing for AI in the future. Most powerful Mac Studio has 128Gb of Uniform RAM which can be used by CPU, GPU or Neural Engine. If only memory size is considered, even A100, let alone any consumer oriented model, can't match. With this amount of memory you could run GPT3 Davinci size model in 4bit mode.

2

pier4r t1_jd0pf1x wrote

> 128Gb of Uniform RAM which can be used by CPU, GPU or Neural Engine.

But it doesn't have the same bandwidth as the VRAM on the GPU card iirc.

Otherwise every integrated GPGPU would be better due to available ram.

The neural engine on M1 and M2 is usable IIRC only with apple libraries, that may not be used by notable models yet.

10

currentscurrents t1_jd10ab5 wrote

Llamma.cpp uses the neural engine, so does StableDiffusion. And the speed is not that far off from VRAM, actually.

>Memory bandwidth is increased to 800GB/s, more than 10x the latest PC desktop chip, and M1 Ultra can be configured with 128GB of unified memory.

By comparison, the Nvidia 4090 is clocking in at ~1000GB/s

Apple is clearly positioning their devices for AI.

14

pier4r t1_jd39md4 wrote

> Llamma.cpp uses the neural engine

I am trying to find confirmation for this but I didn't. I saw some ports, but weren't from the LLaMa team. Do you have any source?

1

remghoost7 t1_jd1k0l6 wrote

>...Uniform RAM which can be used by CPU, GPU or Neural Engine.

Interesting....

That's why I've seen so many M1 implementations of machine learning models. It really does seem like the M1 chips were made with AI in mind....

2

SWESWESWEh t1_jd2s9ml wrote

Unfortunately, most code out there is using calls to cuda explicitly rather then checking the GPU type you have and using that. You can fix this yourself, (I use an m1 macbook pro for ML and it is quite powerful) but you need to know what you're doing and it's just more work. You might also run into situations where things are not fully implemented in Metal Performance Shaders (the mac equivalent to cuda), but Apple does put a lot of resources into making this better

2

LetMeGuessYourAlts t1_jd02jkq wrote

Used availability is better on the 3090 as well. I got one for $740 on eBay. Little dust on the heatsinks but at half price it was a steal.

5

I_will_delete_myself t1_jczvx4j wrote

That or just use the cloud until Nvidia releases a 48gb gpu (which will happen sooner than one would think. Games are getting limited by VRAM)

12

Educational-Net303 t1_jd03se1 wrote

What game is limited by vram? I haven't heard of any game running over 24gb unless it's Skyrim with a bunch of 8k mods

18

currentscurrents t1_jd0f76v wrote

I mean of course not, nobody would make such a game right now because there are no >24GB cards to run it on.

18

frownyface t1_jd6q1qi wrote

There was an insane age of PC gaming where hardware was moving so fast that game developers were releasing games with max-settings that didn't run on any current hardware to try to future proof themselves from having a game suddenly feeling obsolete shortly after launch.

2

I_will_delete_myself t1_jd04mia wrote

people are demanding more and more interactivity in their video games (look at the trend of open worlds). It’s only gonna get bigger.

−1

Educational-Net303 t1_jd051kh wrote

Cyberpunk on max with psycho takes ~16gb max. It's gonna be a few years before we actually see games demanding more than 24.

13

I_will_delete_myself t1_jd05atn wrote

Now try that on 2-4 monitors. You would be surprised how premium gamers like their hardware. It’s like checking out sports cars but for nerds like me.

−1

Educational-Net303 t1_jd05hmc wrote

Are we still taking consumer grade hardware or specialized GPU made for a niche crowd?

6

42gether t1_jd0juau wrote

Niche supercar gamers start up the industry which then will lead into realistic VR which will then lead into consumer high quality stuff?

1

Educational-Net303 t1_jd0k6p6 wrote

Which takes years

4

42gether t1_jd2rfb6 wrote

Okay, thank you for your input.

And?

Newsflash everything we did started because some cunt felt like growing lungs and wanting oxygen from the air.

It all takes time, what are you trying to argue?

1

Educational-Net303 t1_jd2rsax wrote

My whole point is that it will take years before we get to 48GB vram consumer GPUs. You just proved my point again without even reading it.

3

2muchnet42day t1_jczj8da wrote

>I'm gonna end up buying a bunch of 24GB 3090s at this rate.

Better hurry up...

4

currentscurrents t1_jczkbue wrote

Honestly, they already cost more than I can afford to spend on a side project.

I'm just gonna have to wait and hope that AMD gets their act together on AI support.

9

UnusualClimberBear t1_jczl0bn wrote

Better light a candle rather than buy an AMD GC for anything close to cutting edge.

16

currentscurrents t1_jczods2 wrote

I'm hoping that non-Vonn-Neumann chips will scale up in the next few years. There's some you can buy today but they're small:

>NDP200 is designed natively run deep neural networks (DNN) on a variety of architectures, such as CNN, RNN, and fully connected networks, and it performs vision processing with highly accurate inference at under 1mW.

>Up to 896k neural parameters in 8bit mode, 1.6M parameters in 4bit mode, and 7M+ In 1bit mode

An arduino idles at about 10mw, for comparison.

The idea is that if you're not shuffling the entire network weights across the memory bus every inference cycle, you save ludicrous amounts of time and energy. Someday, we'll use this kind of tech to run LLMs on our phones.

14

VodkaHaze t1_jd11vhm wrote

There's also the tenstorrent chips coming out to public which are vastly more efficient than nvidia stuff

2

currentscurrents t1_jd1c52o wrote

Doesn't look like they sell in individual quantities right now but I welcome any competition in the space!

1

mycall t1_jd0yi8i wrote

> if you're not shuffling the entire network weights across the memory bus every inference cycle

Isn't this common though?

1

2muchnet42day t1_jczooi6 wrote

Yeah, I wouldn't buy AMD either. It's a shame that NVIDIA is basically a monopoly in a AI, but it is what it is.

9

whyvitamins t1_jd0j5zq wrote

> hope that AMD gets their act together on AI support

walking around picking up coins from the ground to buy a 3090 should be faster honestly

6

rolexpo t1_jd0fvle wrote

You'll have better luck waiting for Intel

4

AnOnlineHandle t1_jd1k2un wrote

They haven't been sold in Australia for months, only second hand.

1

whyvitamins t1_jd1mddg wrote

realistically, what's the cheapest one can get a used functioning 3090 rn? like 700 usd minimum?

2

C0demunkee t1_jd8svm2 wrote

Tesla P40 24gb VRAM, $150 only 1 or 2 gen behind the 3090

1

tOSUfever t1_jdfj8k9 wrote

where are you finding 24gb p40's for $150?

1

C0demunkee t1_jdg030x wrote

eeeeeeebay

Maybe $200 on a bad day, but still far better than anything newer

1

C0demunkee t1_jd8l0tg wrote

maybe consider Tesla P40s

24gb, lots of CUDA cores, $150 each

1

westingtyler t1_jdgtfb2 wrote

tell me about it. i never cared about video cards before this ai stuff, but now I want an awesome video card.

1