[D] The best way to train an LLM on company data

Submitted by jaxolingo t3_125qztx in MachineLearning

Hey guys, I want to train any LLM on my company’s data we have stored in Azure and Snowflake
It’s all in tabular form, and I was wondering how can I train an LLM on the data, and be able to ask it questions about it. No computations required from the model, but at least be able to tell answer questions such as: What was Apple’s return compared to it’s sector last month ( we have financial data)

- is it possible to train an LLM to understand tabluar data

- is it possible to train it on Snowflake/Azure

Any help or links would be appreciated!

261

Comments

You must log in or register to comment.

TheDeviousPanda t1_je5ddm3 wrote

It’s going to be a lot easier to just take something like GPT-4 and feed in your data directly and ask questions.

3

master-leaf t1_je5dtrm wrote

There was a paper I read a few months ago (I think it was called tapas). In this paper they show how to ingest tabular data to a transformer model.

71

master-leaf t1_je5hhu6 wrote

Replying to jaxolingo (#2,450,250)

I would check the paper, but I think they fine tune a pre trained local LM. They also created their own encodings to account for the structure of tabular data, such as the column headers, entity rows etc.

I will note though, from what I remember the table sizes were pretty small.

12

13ass13ass t1_je5nc8b wrote

You could look at the natural language -> sql query tools that are all the rage right now. I’d recommend checking out langchains sqlchainagent since it’s open source.

136

SkinnyJoshPeck t1_je5ue3b wrote

I'm not 100% sure what your infrastructure or background is, but generally you can just transform data to whatever data format works best for the model.

So, you would build a pipeline that goes

 Snowflake -> Some ETL process -> Transformed Data Storage -> Model Training -> Model Saving -> Model Loading for API to ask questions

where that Some ETL process is a process that transforms your data to whatever the model needs, and your model trains from that.

For example, on AWS you might have something like

Redshift/RDS/Whatever -> SageMaker -> Output Model to S3 -> API for your model or something idk

or if it's all going to be on-prem and you won't have Cloud tech, you'd do something like

Snowflake/Azure/Any Data Source -> Airflow for running training -> Model Upload to Some Folder -> API in a docker container in Kubernetes or something for users to hit

or they can just download the model locally and use some script to ask it questions, I'm not 100% sure it all depends on the model/language/etc that you use.

This is a fairly complicated task; if your company is getting serious about this, y'all should hire someone who is an ML engineer to do this task. :)

32

patniemeyer t1_je5wc4u wrote

This may not be what you want, but I was not aware until recently that OpenAI offers an API to fine tune GPT-3/4 on your own data: https://platform.openai.com/docs/guides/fine-tuning

They charge your for training and for usage of your custom model, so it may or may not be economical for your use case.

11

abnormal_human t1_je60s31 wrote

Yes, it's totally possible to train an LLM to understand tabular data. It's a very general purpose architecture. With enough resources it is well suited to a wide range of problems, and yes, Azure/Snowflake can do everything you need (at some price, assuming you know what to do with them).

You need to make a decision about whether you want to bake the info into the LLM, or whether you want to teach the LLM to find the answers and then format them for humans.

This will depend on your use case, budget, team size, competencies, data set size, and time to market requirements. Baking the info into the LLM is a lot harder than doing the other thing, like potentially 100x-1000x harder and more expensive, and without people with experience doing it, you will waste a lot of time/energy getting there.

3

EverythingGoodWas t1_je612lg wrote

You aren’t going to train an LLM on company data. You could fine tune an existing one with company data, but creating an LLM from scratch is an absolutely massive compute task. If you are trying to make a closed domain question answering system, that uses your company’s data, you basically need to create a full pipeline from parsing, searching, and finally pushing the context and question to a language model.

15

evergreensphere t1_je613n6 wrote

Replying to patniemeyer (#2,453,745)

Fine tuning is only available for the old GPT3. It is not available for GPT3.5 or GPT4.

Also, most people I've talked to found that fine tuning did not work as well as using things like vectorized search, or vectorized search combined with graph index.

20

LetGoAndBeReal t1_je65ffo wrote

The comments here so far have addressed three possible approaches to this. Two of those approaches - ie training your own model and fine-tuning an existing model - are not currently viable. Training your model would require a ridiculous amount of human and compute power and not result in something where data could be easily added. Fine-tuning a model does not result in the model absorbing new data - it only conditions the output patterns from the model using data/knowledge the model gained during initial training.

The only viable approach is to use retrieval augmented generation, where data relating to user questions are retrieved from outside the model and fed to model as part of the prompt. Tools like LangChain can help you build a RAG solution on your own. There are also many services coming out that provide this sort of capability, such as humata.ai.

86

memberjan6 t1_je65xrg wrote

Go wstch YouTube for pinecone.ai and milvus. Also, go watch office365 copilot video.

4

zeoNoeN t1_je6f3b5 wrote

I had a lot of success with implementing huggingface models in the last week, so that could be a starting point

2

Rawvik t1_je6f9w8 wrote

I am also currently looking to do something like this for my own company data. Please let me know if you find something useful.

2

detached-admin t1_je6gbz7 wrote

Don't do that if you're dealing with other people's money in any way. If it's yours, try langchain.

2

visarga t1_je6k74d wrote

Replying to Im_Unlucky (#2,458,004)

Often it seems that the model can't properly synthesise information from a bunch of snippets, it lacks the context of those snippets, so it will combine the information incorrectly or hallucinate an explanation.

Retrieval + loading data in the context is far from solved.

16

visarga t1_je6kqvw wrote

Replying to EverythingGoodWas (#2,454,668)

I'd rather fine-tune the LLM on company documentations than feeding it through retrieval. Does anyone have experience with fine-tuning GPT-3 on a new text? Can it answer questions or freely use information from this text?

11

kromem t1_je6uv46 wrote

Replying to visarga (#2,458,474)

"Moar layers" doesn't only need to apply to the NN.

CoT prompting works by breaking analysis down into smaller steps that each generate their own additional context.

Doing something similar with DB analysis is absolutely possible, such as preemptively summarizing schema and using that summary as part of the retrieval to contextualize the specific fragments.

Additionally, having static analysis examples on hand for related tables that's fed in to go from zero shot to few shot would go a long way at reducing some of the issues you highlight.

6

MadDanWithABox t1_je6vh9x wrote

Just a caveat. Any training (fine-tuning really) that you do on an LLM is NOT guaranteed to be able to give you correct answers. The answers to your questions will seem plausible, and could be correct, but you'll need some system to verify if it's something you want to take action on.

5

Im_Unlucky t1_je6vjqd wrote

Replying to visarga (#2,458,474)

This is true. Good prompt engineering has definitely gone a long way for me in improving rag performance though. I don’t think shoving snippets into the prompt is the long term solution, but at this point in time it’s the best option in my experience.

8

Cantareus t1_je6vn79 wrote

Replying to Im_Unlucky (#2,458,004)

>shove data into vector db
>
>embed incoming query
>
>return topk matches from vector db

Can these steps be done through the openai API?

You only reply using JSON. Write 5 queries that will return useful data to aid you in answering the users questions "What was Apple’s return compared to it’s sector last month" return in JSON array with no explanations. Each query must run independently. Use the return format [{"Reason":"Why the query is useful","query":"The sql query"}] The schema is:

(Imagine a valid schema is here)

GPT-4 Response:

[

{

"Reason": "Retrieve Apple's stock return last month",

"query": "SELECT stock_return FROM stocks WHERE company = 'Apple' AND date >= DATEADD(month, -1, GETDATE()) AND date < GETDATE();"

},

{

"Reason": "Retrieve sector for Apple",

"query": "SELECT sector FROM companies WHERE company_name = 'Apple';"

},

{

"Reason": "Retrieve companies in Apple's sector",

"query": "SELECT company_name FROM companies WHERE sector = (SELECT sector FROM companies WHERE company_name = 'Apple');"

},

{

"Reason": "Calculate average return for Apple's sector last month",

"query": "SELECT AVG(stock_return) as sector_return FROM stocks WHERE company IN (SELECT company_name FROM companies WHERE sector = (SELECT sector FROM companies WHERE company_name = 'Apple')) AND date >= DATEADD(month, -1, GETDATE()) AND date < GETDATE();"

},

{

"Reason": "Compare Apple's return to the average return of its sector last month",

"query": "SELECT (SELECT stock_return FROM stocks WHERE company = 'Apple' AND date >= DATEADD(month, -1, GETDATE()) AND date < GETDATE()) - (SELECT AVG(stock_return) as sector_return FROM stocks WHERE company IN (SELECT company_name FROM companies WHERE sector = (SELECT sector FROM companies WHERE company_name = 'Apple')) AND date >= DATEADD(month, -1, GETDATE()) AND date < GETDATE()) as comparison;"

}

]

5

EarthquakeBass t1_je6wa0g wrote

I think Azure might actually have support for private OpenAI stuff

2

machineko t1_je70llx wrote

Replying to LetGoAndBeReal (#2,455,558)

Why would you say that fine-tuning is not viable? There are many production use cases of fine-tuning a model using in-house proprietary data.
If fact, if you have the resources you can do both fine-tuning of an existing model (whether is just supervised or unsupervised) and also use that for retrieval augmented generation.

10

LetGoAndBeReal t1_je71r0g wrote

Replying to machineko (#2,462,055)

Fine-tuning can be great for getting better output from the model based on the knowledge that model already contains. I only meant fine-tuning is not viable for getting new data/knowledge into a model. Fine-tuning does not accomplish knowledge absorption.

3

athos45678 t1_je7ercw wrote

Train a Llama LoRa model. The 30 b model isn’t too expensive to tune (40 bucks ish), and is ridiculously capable.

You just need to format the data in a long text doc with each prompt separated by two line breaks. I found it worked best in the alpaca style, where you have a single line break after the prompt, like “write a function that sorts this table in python def sort():” followed by the written out code, and then the double line break to signal the start of the next input.

Then use the simple-llama trainer app to make it all easy.

3

t_minus_1 t1_je7gsc7 wrote

Please look at sketch and langchain pandas/SQL plugins. I have seen excellent results with both of these approaches. Both of these approaches will require you to send metadata to openAI.

If you would like to do it yourself may be start with GPT-J / LORA and use the same instruction approach/fine tuning like databricks did .

9

light24bulbs t1_je7ilvq wrote

Replying to LetGoAndBeReal (#2,455,558)

I disagree that it's not viable to train a model. The problem is that the best public model (llama) is non-commercial.

That said, it's extremely possible to train things into it. There's a ton of new scripts floating around online. The Lora training is especially good.

The trouble with vectors is they are so limited. They're fine if you need to look up one distinct thing (and the vector gets the match right) but they're utterly useless if you'd like the model to learn about something in general.

9

light24bulbs t1_je7mr9p wrote

Replying to LetGoAndBeReal (#2,466,723)

False, they all do. The process of fine-tuning is identical to the initial pre-training, though perhaps with different settings. They're mostly setup to take q&a data for getting llama to take instructions better, but actually that's just text wrapped in some context and passed in straight up.

I was very confused by this as well but no, you can train new stuff.

−1

gstatton t1_je7mwmv wrote

Replying to jaxolingo (#2,464,593)

I’ve done something similar to this, but I created a view in snowflake, dumped the ‘describe table <table-name>’ to CSV, used that CSV descriptor in the head of the prompt, and asked the LLM to generate a SQL query from the user’s question using the provided table schema…I’ll take the model’s return and pass it directly to snowflake, return it as a data frame, and then send that back to the LLM to get a text summary of the query results

You can also use LangChain to set up the chain to do this automatically..

3

LetGoAndBeReal t1_je7re7y wrote

Replying to light24bulbs (#2,467,481)

Of course the fine-tuning data itself can have knowledge not in the model - that doesn’t prove anything.

What you need to show is that knowledge presumably added during fine-tuning was then retrieved from the model after fine-tuning.

0

DonutListen2Me t1_je7rr2p wrote

You don't want to input tabular data to a LLM. You should use a model specifically for tabular data, like XGBoost.

What you want to do is embed your company's documents and use a vector search on the embeddings to feed as context to an LLM.

6

light24bulbs t1_je7sn5e wrote

Replying to LetGoAndBeReal (#2,467,826)

The fact that the fine tuning can improve instruction following is EXACTLY that. There's no distinction between predicting the next word, following instructions, or deep knowledge. They are all the same thing as far as an LLM.

3

bubudumbdumb t1_je7yrmq wrote

In the last few days someone posted on hacker news about a system allowing the integration of a gpt with a postgress database

2

athos45678 t1_je82thk wrote

Replying to netham91 (#2,466,232)

So as far as set up goes, you just need to: “”” Git clone https://github.com/lxe/simple-llama-finetuner Cd simple-llama-finetuner Pip install -r requirements.txt Python app.py ## if you’re on a remote machine (Paperspace is my go to) then you may need to edit the last line of this script to set ‘share=True’ in the launch args “””

Then you should get a link for the gradio web app. Copy and paste the code samples in the format described before in the input text box. It will look something like this:

“”” Write a code snippet that sorts a function Def sort(arr):

  Return arr.sorted()

Some other code snippet input

Some answer

Etc. “””

Edit: I’m drinking with friends sorry i can’t format better. Single line break between prompt and observed correct response, double line break between prompt instances.

3

Tostino t1_je847jg wrote

Replying to kromem (#2,460,802)

Literally just worked through this today manually as a proof of concept, using the LLM to augment the DB schema with comments describing any relevant info or corner cases. I'm essentially just manually feeding it as context to my prompts when I need to know something related to that set of tables, but it seems pretty powerful. Automating this is going to be nuts.

5

LetGoAndBeReal t1_je8akb1 wrote

Replying to light24bulbs (#2,471,164)

I would agree with that last statement. You think you understand this, but you don’t seem to understand what does and doesn’t happen during fine-tuning or to realize that the problem of adding knowledge to LLMs is a notoriously difficult problem that ongoing research is trying to solve.

Try looking at some of the research: https://openreview.net/forum?id=vfsRB5MImo9

Or read what OpenAI says fine-tuning accomplishes: https://platform.openai.com/docs/guides/fine-tuning

Or, better yet, try actually getting a LLM to learn new facts by fine-tuning it. Then you will understand.

4

huyouare t1_je8fby1 wrote

Replying to Appropriate_Ant_4629 (#2,471,841)

I was wondering how this relates to retrieval or SQL queries but it sounds like you’re suggesting that OP finetunes on their dataset regularly. Might be good to try in combination with retrieval, but how would you represent the tabular data as training examples?

1

VelvetyPenus t1_je8gktg wrote

First person to use AI to embezzle majority of company profits arrested. Convict used Reddit to ask how to turn company data into AI dataset.

3

elbiot t1_je8i0i2 wrote

Replying to LetGoAndBeReal (#2,472,128)

The second link says fine tuning is a substitute for lengthy prompts, including putting more into it than can fit in the longest prompt. Prompts are a way to give the model new information. What is your definition of knowledge that isn't something you can put into a prompt?

2

LetGoAndBeReal t1_je8j7hw wrote

Replying to elbiot (#2,473,803)

The key word in that OpenAI link is “examples”. It says “more examples” and not “more knowledge”, because it’s referring to few shot training, which is about conditioning rather than providing new data.

In other words, if you want to get the model to classify sentiment of user comments as positive or negative, you can provide several examples in the prompt of both positive and negative comments. Fine-tuning allows you to provide many more such examples to the model than can fit in a prompt.

The key point is that through fine-tuning these examples can condition the model to classify sentiment but do not cause new facts to be absorbed by the model. You cannot get new facts to be readily absorbed through fine-tuning, which is why the OP should not look to fine-tuning to endow the model with the external dataset they want to use for question answering.

1

VinceD6 t1_je8kaza wrote

I am currently trying to do the same thing. Take a look at LlamaIndex, build a POC yesterday and it seemed to work really good.

1

LetGoAndBeReal t1_je8m6y9 wrote

Replying to light24bulbs (#2,472,746)

Include new factual statements in your training data like “Joe Biden’s cat is named Fluffy.” Ask the model the name of Joe Biden’s cat before and after training and let us know the answers you get back. See if you get reliable answers across a set of data/questions.

2

disbeam t1_je920cv wrote

What some people have done is to use Azure Cognitive Search as a pre-cursor to the LLM.

You use Cognitive Search to extract information from your organisation's own documentation and ask the LLM to only provide the correct answer from the details found in the search, otherwise responding with saying it doesn't know. It then answers complete with references. Having seen it in action with one of our customer's, I've been quite impressed.

2

LetGoAndBeReal t1_je9a3hb wrote

Replying to elbiot (#2,475,011)

Of course, that’s what allows RAG to work in the first place. I didn’t say you couldn’t provide new knowledge through the prompt. I only said you cannot provide new knowledge through the fine-tuning data. These are two completely separate things. This distinction is the reason RAG works for this use case and fine-tuning does not.

1

LetGoAndBeReal t1_je9c66v wrote

Replying to WokeAssBaller (#2,469,393)

Instead of insisting that fine-tuning reliably adds new knowledge to an LLM, why not instead show some evidence of this claim. Per my links above, this is a notoriously challenging problem in ML.

Apart from these resources, let's think critically for a second. If the approach were viable at this point, then there would be tons of commercial solutions using fine-tuning instead of RAG for incorporating external knowledge in an LLM application. Can you find even one?

2

Tiny_Arugula_5648 t1_je9oyfo wrote

Not sure why no one is calling this out but there is no indication a LLM is going to be useful here.. you have tabular data, unless it's unstructured text held in there it's not goung to be useful, pick the right model for the job..

2

elbiot t1_je9s53t wrote

Replying to LetGoAndBeReal (#2,479,113)

Your claim that prompting can achieve what fine tuning can't contradicts the documentation for openai that you posted that said fine tuning can do whatever prompting can without the length limit

1

sandys1 t1_je9thds wrote

Hi. We are actually building a framework for this. P.S. 100% open-source so there is no paid component to it right now.

Would you be willing to talk about what you want ? We may be able to help get you what you want.

3

Goldenier t1_je9uruu wrote

Replying to LetGoAndBeReal (#2,462,290)

This is false, and actually most of the time the opposite is the problem: the model learns too much of the new data it's finetuned on (overfitting on it), but forgets the "knowledge" in the original model. The simplest and most popularly used example right now is when you use the dreambooth, Lora or other finetuning methods to finetune parts of the big image diffusion models and if you overtrain it will place the newly trained face or object in almost all of it's output, so it easily learns new data but also easily forgets old one. ( One mitigation for this is to use preservation loss to make sure it also keeps the old knowledge. ) And there is no reason why the same methods wouldn't work on LLMs too, for example there is already Lora for LLMs too.

2

LetGoAndBeReal t1_je9zfyb wrote

Replying to Goldenier (#2,484,187)

>And there is no reason why the same methods wouldn't work on LLMs too, for example there is already Lora for LLMs too.

It's really not helpful to make strong assertions like this without referring to specific, verifiable sources. Fine-tuning very typically is done in a way where certain layers/parameters of the model are frozen. This is done to avoid the sort of loss we are discussing. The LoRA paper itself states that LoRA "freezes the pre-trained model weights".

0

WokeAssBaller t1_jea0o2f wrote

Replying to LetGoAndBeReal (#2,472,128)

Again you are using an incredibly limited definition of fine tuning based on what the open ai api allows, which once again tells me you don’t know ML.

Fine tuning is ANY additional training on a foundational model, this can be MLM training on the model base or selectively training the subsequent layers.

OF COURSE this can add knowledge as you are doing the same training that got it knowledge in the first place. Glad to see you jumped on the chatgpt band wagon last week, build a transformer from scratch and come talk to me

2

LetGoAndBeReal t1_jea1id9 wrote

Replying to elbiot (#2,483,444)

I believe you are referring to this statement from the link: "Ability to train on more examples than can fit in a prompt." Correct?

If so, as I explained, the key word here is "examples." And if you understand why, you will see that there is no contradiction. I will try to clarify why.

There are two methods that we are discussing for extending the capability of an LLM:

  1. Prompt engineering
  2. Fine-tuning

There are also different types of capability that might be extended. We are discussing the following two:

  1. Adding new knowledge/facts to the model
  2. Improving downstream processing tasks, such as classification, sentiment analysis, etc.

Both of these capabilities are readily done through prompt engineering. Adding new knowledge with prompt engineering involves including that knowledge as context in the prompt. Improving tasks such as classification is done by include examples of the processing you want done in the prompt.

What the article says is that for the case where you want to provide examples in the prompt to make the model perform better, you can alternatively use fine-tuning. The article does not say "Ability to add more knowledge than can fit in a prompt." Examples = downstream processing tasks. Examples != new knowledge.

1

lgastako t1_jeayn8v wrote

Replying to WokeAssBaller (#2,491,385)

I know training a model from scratch will work, but the context of the conversation is fine tuning an existing model and I'm saying I would love to see examples of the claims people are making actually working, because I have only been able to find and create examples of it not working very well at all.

1

icm76 t1_jebgew9 wrote

!remind me in 3 days

1

WokeAssBaller t1_jebpjog wrote

Replying to lgastako (#2,494,397)

fine tuning is just additional training, so if it works from scratch it works with fine tuning. And no it may not be as effective as other methods but the poster was claiming it was impossible

1

j_lyf t1_jebvmnp wrote

Where do embeddings come into all of this?

1