I am really enjoying the course by https://dlsyscourse.org. You build a DL library called Needle from scratch throughout the lectures. They also offer notebooks and tests to pass, so it's more than just videos.

It's also by CMU, but the videos are different than the one referenced by /u/verboseEqualsTrue, though I plan on watching that one too.

They cover transformers and other network types, but I haven't gotten that far in the videos yet. I'm still implementing an automatic differentiator.

I have found that Karpathy videos are really good too, though I'm not following them too closely yet.

[deleted]

It might be. But then tensorflow/pytorch are out there, too, have documentation and have CPU-only modes.

I say this as someone who coded a number of numeric algorithms (BFGS, Nelder-Mead, etc) from scratch just to understand them, while knowing all along I wasn't competing with real OR libs. On a few occasions, my own implementations were handy for proof-of-concept work when adding 3rd party libs was a hassle.

but could it be useful for a portfolio?

If you are still learning, you can continue. But with the knowledge that your library likely won't be competitive with what's already out there.

What are you doing in a deep learning sub?

You should do something else. There are a lot of small C++ nn libraries. To make this one competitive with a real deep learning framework you would need to implement everything with gpu which would be painful. Also, python libraries also have the huge benefit of great data science libraries which make it much more convenient to preprocess data for training networks in python vs cpp.

Additionally there are ways to deploy python models to cpp so there’s not much benefit in training with a cpp library.

Hi, thanks a lot for this very detailed answer. I will have a look at your suggestions. Yes we are looking into the mathematical equations too. We fear that it will be quite un-precise but in other hand I understand very well the issue with the small dataset currently available. We may end up exploring different solutions.

After you have played with scikitlearn, spend some time studying its endpoint (it's like an industry standard), then go on and play with tensoflow/torch

Oh, I hadn't realized that you meant that the embedding would contain information about other features, I get it now. I was referencing an RNN since I thought it was the only option due to the variable input size. Thanks

Yeah I'm gonna try both options (with BERT and the bigger models) but since I'm working with a big dataset I'm not sure I'll be able to use the larger models due to the token and request limits. Thanks for the help

The embedding contains all information like sentiment or tf-idf. You just need to train a model to predict trait from post embedding then average over all posts by a person. I didn’t suggest using RNN. Are you sure you were replying my comment?

Generally, yes. If you use a model with a long context length (BigBird or OpenAI’s ada02), you’ll likely be fine unless the articles you’re embedding are greater than the token limit. If your using BERT or another, smaller model, you have to chunk/average; that can produce fixed sized vectors but you gotta put the work in haha

[deleted]

That is very likely. I wonder how this works for multimodality. Weights would probably have to hold more.

But even then, with the other features (sentiment analysis, tf-idf) how would I feed a vector containing a varying number of tokens and other types of features? I can't see how you would this using an RNN. That is for each post

Oh, so those models are able to produce fixed size embeddings of texts? I wasn't aware of that

Personally I think the limits with those models is just the amount of information that each weight can hold is limited.

I would embed the whole post (BigBird or OpenAI embeddings have really long context lengths), and just feed that vector into an RNN. As long as the post is between one and 9000 tokens, the embedding shape will remain the same

Tensorflow is used to make neural networks. Start with simpler machine learning algorithms. Look into the Scikit-learn library. Scikit-learn is used for simpler and more commonly used machine learning algorithms. Look into kaggle tutorials for better understanding. Hope this helps!

How will I build projects if I don't know how to use tensorflow?

You wouldn't want to go into designing machine learning systems and life cycle right now. Just build projects. Don't get stuck into a cycle of MOOCs

Thanks!

What are people's thoughts about dive into deep learning book?

I was also thinking about this to gain a basic experience of how projects are made and structured.