While testing models and playing with hyperparams can be fun, the real problem is that you are trying to apply deep learning to 100 images.

Get more images.

Have you tried platforms like Upwork? I had a lot of success on it 3 years ago

Looks cool, thanks for pointing it out

For every image in your database, you could use features from the penultimate layer in a CNN, index them.

Then to search over images, simply calculate the distance between the query image features and the database features.

This can be expensive computationally and memory wise if you have a lot of images. Some solutions could be to cluster your database embeddings, use sparse matrices, use approximate KNN, add some explore-exploit heuristics (take the images with the lowest distance compared to the first 37% images in the database, this cuts down search time by up to 63%, but might not be great). There is possibly more out there in SoTA, but I am not up to date there.