Viewing a single comment thread. View all comments

theunixman t1_j6jff5n wrote

Reply to comment by farmingvillein in [R] Parsel: A (De-)compositional Framework for Algorithmic Reasoning with Language Models - Stanford University Eric Zelikman et al - Beats prior code generation sota by over 75%! by Singularian2501

We have to learn APIs all the time, and basically they're all DSLs that just don't admit they are so they're even harder.

3

farmingvillein t1_j6jgv48 wrote

And this isn't a good thing, it is a necessary thing--we do it because someone bundled some logic together and you need to interact with it.

None of this addresses whether or why something like Parsel is necessary as an intermediate step. The authors do very little to justify the necessity of an intermediate representation; there is no meaningful analysis of why it apparently performs better, nor an ablation analysis to try to close the gaps.

The key benefits--like enforced test cases--could, hypothetically, very easily be enforced in something like Python, or many other languages.

And given the massive volumes of training data we have for these other languages, there are a lot of good reasons to think that we should be able to see equal or better behavior than with a wholly manufactured pseudocode (effectively) language.

The paper would have been much more convincing and interesting if, e.g., they started with something like python and progressively added the restrictions that apparently helped Parsel provide higher quality results.

0

abcdchop t1_j6m17n8 wrote

wait bro the key benefit is the the hierarchical description -- the "language" is just a format for explaining the hierarchical description of the problem in natural language, I think that the improvements your suggesting pretty much describe the paper itself

6

farmingvillein t1_j6n4hqy wrote

> wait bro the key benefit is the the hierarchical description

agreed

> I think that the improvements your suggesting pretty much describe the paper itself

Allow users to work in actual unstructured language, or an extant programming language, and I'd agree.

1

theunixman t1_j6jhf69 wrote

Right, turning it into an actual DSL would be much better, and then you'd have better semantics for the library. But honestly I'm bored talking about aesthetics already, peace.

1