ezelikman
ezelikman t1_j6kkch1 wrote
Reply to comment by ihateshadylandlords in Parsel: A (De-)compositional Framework for Algorithmic Reasoning with Language Models - Stanford University Eric Zelikman et al - Beats prior code generation sota by over 75%! by Singularian2501
Here's another slightly longer TL;DR:
Humans solve hard problems by them down into parts and solving them part by part. We normally ask language models to solve algorithmic problems in one go (or if they revise their solutions, we expect them to revise everything). This has been known to be a problem for a while. It turns out, maybe unsurprisingly, that by asking language models to break problems down and then implementing subparts independently, we get way better results.
We do this by writing a programming language (basically, English with indentation plus a small amount of syntax for tests and references). We design an LLM-powered compiler around it to generate programs efficiently. We show it works on solving competitive coding problems, robotic task planning, and math theorem proving. We also show that it's decently robust - able to implement a bare-bones lisp compiler in a few dozen lines.
ezelikman t1_j6kib8v wrote
Reply to comment by starstruckmon in Parsel: A (De-)compositional Framework for Algorithmic Reasoning with Language Models - Stanford University Eric Zelikman et al - Beats prior code generation sota by over 75%! by Singularian2501
>having a compiler LM that can reliably convert Parcel ( or something simmilar ) to actual code would be a massive win. Imagine coding in natural language psudocode
We made this available here!: https://github.com/ezelikman/parsel
And there's a notebook here: https://colab.research.google.com/github/ezelikman/parsel/blob/main/parsel.ipynb
Hopefully, there'll be a nicer IDE integration at some point in the nearish future!
ezelikman t1_j6lx0vm wrote
Reply to comment by farmingvillein in [R] Parsel: A (De-)compositional Framework for Algorithmic Reasoning with Language Models - Stanford University Eric Zelikman et al - Beats prior code generation sota by over 75%! by Singularian2501
Hi, author here!
There are a few ways to interpret this question.
The first is, "why generate a bunch of composable small functions - why not generate complete Python/Lean/etc. implementations directly from the high-level sketch?" If you generate 10 complete implementations, you have 10 programs. If you generate 10 implementations of four subfunctions, you have 10,000 programs. By decomposing problems combinatorially, you call the language model less. You can see the benefits in Fig. 6 and our direct compilation ablation. There's also the context window: a hundred 500-token functions from Parsel is a 50,000-token program. You won't get that with Codex alone.
Another interpretation is, "why do you need to expose intermediate language when you can use a more abstract intermediate representation." You suggest "leveraging the value of LLMs--through a more natural language interface." That's the goal. Parsel is intentionally basically indented natural language w/ unit tests. There's minimal extra syntax for efficiency and generality - ideally, people who've never used Python can understand and write Parsel. The "expert" details here aren't syntax: most people are unfamiliar with the nuances of writing natural language that automatically compiles to code, like the value of comprehensive unit tests.
Another is, "why design a new language instead of writing this as, e.g., a Python library?" My response is we did this too. Internally, Parsel is in Python, and a "Function" class already exists - you can find it on GitHub. Still, you need a process to generate implementations and select one satisfying the constraints, which we call the compiler.
Hope this answers your question!