What is your ML model supposed to do?

> a model that can generate more accurate embeddings in general

What do you mean by this? Accuracy is an evaluation metric that doesn't really mean anything "in general" but only with respect to a specific prediction being made. I think this is a slightly ill-posed question

More accurate, as in it can more accurately capture the semantic meaning of jargon that is typical in financial documents.

It’s a retrieve/rerank model

In my experience NLP models are released as public science when trained on datasets scraped from the web.

Things like "models that solve this problem in finance" or "datasets of annotated football matches" or "medical records of millions of people" are not likely to follow the publication patterns of open science.

If you have a model like the one you asked for you likely have a way to profit from it and you are unlikely to publish it.

finBERT maybe?

There are teams of people working on and refining this in the mortgage and quantitative finance industries, it is highly unlikely that they would be willing to open source it for competitors to compare notes and take market share.

Right. But to be able to assess this you need to define a task and evaluate your model’s performance to perform that task. Embedding accuracy cannot be discussed completely in the ether. Even the most general comments you will read about a model beating another, will refer to that new model performing better for specific tasks on benchmark datasets.

It would be a lot easier to help you if you explained what you are trying to accomplish that requires “higher accuracy”

[deleted]

First I would say there exist versions of FinBERT which aren't just tuned for sentiment analysis. There are two groups who developed models they called FinBERT https://arxiv.org/abs/1908.10063 and https://arxiv.org/abs/2006.08097. The first paper's model can be fond here and is tuned for seniment analysis but the second model, found here, was pre-trained using masked language modelling on general financial text. So that one can be fine-tuned for other tasks.

Since you're interested in text embeddings, you may also be interested in this paper https://arxiv.org/pdf/2111.00526.pdf. The focus of that paper is sentiment analysis, but the general idea of using a sentence-BERT model to get better textual embeddings (as opposed to using vanilla BERT) should hold more generally.

Unrelated question: what dataset are you using for the task?