I'd like to have one too 😀.

I believe a common approach is to use a linear activation function for regression problems unless target variable has certain semantics that suggest the use of other non-linearities (sigmoid, tanh etc.). Also consider rescaling your targets instead of trying to match the desired output with activation functions.

From you description (I might be wrong though), it seems like the 0 output is a special case. In this case you might want to use a binary classifier to classify input samples into two classes first. For class 0 the output is 0. For class 1 you use another model (regressor) that outputs a prediction.

This should be considered in the first place. For instance, gradient boosting trees that are mostly implemented in C/C++ and have GPU compute backends - XGBoost, CatBoost and LightGBM. Given daily updates, you'll have enough time not only to train a model, but also optimize its hyperparameters. In my experience, XGBoost + RayTune work just fine.

Back in 2012 we were experimenting with an engineering-based approach to extract relations and events from texts. Examples of events are company announcements, merger and acquisitions, management position changes, customer complaints about products etc. Our NLP pipelined included two major steps - named entity recognizers and rule-based engine over graph of annotations. The former step extracts various types of entities - names of companies and people, geographical locations, temporal expressions and dictionary-based extractor that extracts anchor verbs (e.g., acquire, purchase, announce, step down). The latter step uses a rule-based engine that tries to match tokens and named entities into high-level concepts using regular expression-type syntax, e.g., 'annotate[COMPANY_ANNOUNCEMENT] if match[COMPANY ANNOUNCEMENT_VERB]'. Then, if I recall correctly, we switched to use rules over dependency structure of sentences (something like subject - verb - object) - with slightly lower precision this resulted in much better recall. But this was 10 years ago, and a lot has changed since then.