That is certainly an option that I was considering, but then I would have to make my own job planner / multirunner, (which I actually already have done for my current project, but this whole refactoring was to try and move away from my own custom functions and try to use some more standardized methods)

> https://github.com/fidelity/spock

This looks quite promising, and I like the Post hooks you linked below, but I do not see any way of running a series of experiment in a non-combinatoric way? There is Optuna api (though I can't tell whether early pruning is supported in this?), but I don't see any way of grouping parameters for a set of experiments.

Another solution is to use a memory efficient neural network: https://arxiv.org/pdf/1905.10484.pdf With this type of neural network you can easily fit those size images into your neural network. However the problem with them is that they are very difficult to make (you manually have to code up the backpropagation). So depending on your math proficiency and ambitions this might just be too much.

It really depends what you are intending to use this for. There are many sides to machine learning, but you don't have to know all of them. To name a few very different concepts:

MLOps (Corsera has an excellent series on this) Reinforcement learning GANs Graph neural networks

I would say that once you have an idea about what most of these topics involve it is time to actively dive into some of them by actually trying to code up solutions in them, or downloading well known github projects and trying to run them yourself.