Disentangling Computation from Reasoning for Numerical Reasoning Tasks

View a PDF of the paper Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks by Wenhu Chen and 3 other authors.
view pdf

abstract:Recently, significant progress has been made in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (COT) is the state-of-the-art method so far for these tasks. COT uses language models to perform both reasoning and computation in a multi-step ‘thinking’ process. To separate computation from reasoning, we propose ‘Programs of Thought’ (POT), which uses language models (mainly codecs) to express the reasoning process as a program. The calculations are performed on an external computer, which executes the generated program to obtain the answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) for both few-shot and zero-shot setups. Under both few-shot and zero-shot settings, PoT can show an average performance gain of about 12\% over CoT across all evaluated datasets. By combining PoT with self-consistent decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets. All our data and code is released on Github at this https URL