RicketyCricket t1_je50mbi wrote on March 29, 2023 at 1:44 PM

_Arsenie_Boca_ t1_je5d04j wrote on March 29, 2023 at 3:09 PM

Looks interesting, a bit more lightweight than hydra. But also misses a lot of cool features like composing multiple yaml configs

RicketyCricket t1_je5j2n9 wrote on March 29, 2023 at 3:48 PM

Most of the cool stuff is buried in the docs under advanced features :-)

https://fidelity.github.io/spock/advanced_features/Composition

(full transparency I'm the author/maintainer/core-developer. I know the docs need a re-org to surface more of the useful features)

RicketyCricket t1_je5jchr wrote on March 29, 2023 at 3:50 PM

This being my favorite hidden one:

https://fidelity.github.io/spock/advanced_features/Evolve#maintaining-cli-and-python-api-configuration-parity

RicketyCricket t1_je5kgy4 wrote on March 29, 2023 at 3:57 PM

second favorite:

https://fidelity.github.io/spock/advanced_features/Post-Hooks

Basically lets you do any validation necessary on your configs. Spock provides some basics (greater than, within bounds, etc) but it's totally up to the user via any simple asserts or validation functions a user wants to write.

_Arsenie_Boca_ t1_je6ayl7 wrote on March 29, 2023 at 6:45 PM

Thanks, looks like your library isn't far behind hydra in terms of functionality. Will definitely look into it more closely the next time I set up a project.

What would you say are the pros and cons between hydra and spock?

RicketyCricket t1_je9lp7z wrote on March 30, 2023 at 12:39 PM

Mainly that Spock is much lighter weight and really focuses on just configuration management and stateful ness. Hydra has all these crazy bells and whistles (Ray integration etc) that could be useful for certain things but kinda starts meandering from the original purpose of configuration management imo. Hydra is great and if it works for you then use it. We built Spock internally when I was at Fidelity because Hydra didn’t exist… just so happens that FB/Meta was doing the same thing at the same time so both libraries end up covering a very similar usage space

_Arsenie_Boca_ t1_je9n0ea wrote on March 30, 2023 at 12:50 PM

Thanks, I basically use only the config part of hydra and am regularly annoyed that its so clunky, so spock might be a good alternative. Gonna check it out :)

alyflex OP t1_je5y5gh wrote on March 29, 2023 at 5:24 PM

> https://github.com/fidelity/spock

This looks quite promising, and I like the Post hooks you linked below, but I do not see any way of running a series of experiment in a non-combinatoric way? There is Optuna api (though I can't tell whether early pruning is supported in this?), but I don't see any way of grouping parameters for a set of experiments.