yazriel0 OP t1_j68nwcd wrote on January 28, 2023 at 3:15 PM

Reply to [N] OpenAI has 1000s of contractors to fine-tune codex by yazriel0

So, this is +++ for codex quality.

But a --- for future prospects of GPT5-ish, AGI and our new overlords ?

yazriel0 t1_j237x73 wrote on December 29, 2022 at 9:10 AM

Reply to comment by evanthebouncy in [D] DeepMind has at least half a dozen prototypes for abstract/symbolic reasoning. What are their approaches? by valdanylchuk

> we cannot understate the value of (1) , reasoning in existing symbolic system

ofc. and (1) may be a good way to bootstrap (2) ..

why arent we seeing more (un)supervised learning on code? perhaps with handcrafted auxiliary tasks.

when will this loop exit? how much memory will this function allocate? etc, etc. this seems to be a huge underutilized dataset.

am i missing something? (yes, its a lot of compute)

yazriel0 t1_iwvipkh wrote on November 18, 2022 at 5:46 PM

Reply to [R] RWKV-4 7B release: an attention-free RNN language model matching GPT-J performance (14B training in progress) by bo_peng

Great stuff, and much needed!! Transformer are expensive.

Is the RNN mode suitable for update-able-neural-net NNEU used in tree-search games? This is where the next tree node evaluation re-uses the previous node.

yazriel0 t1_isbv4hp wrote on October 14, 2022 at 7:44 PM

Reply to [R] Mind's Eye: Grounded Language Model Reasoning through Simulation - Google Research 2022 by Singularian2501

Why arent we doing for code domain? Generate programs, try to run them, auto correct the model?

This can probably be iterated with far more samples than a physical simulator