AI Case Study

Researchers at UCL developed a generic algorithm called GENERE which "correctly identifies the correct data distribution, and hence leads to better generalization performances than what the model learns without the generator. To illustrate the benefit of using generative regularization, we considered a class of real world problems for which obtaining data is costly: learning to answer math exam problems. In this illustrative experiment, a simple text-to-equation translation problem is created, where inputs are sentences describing an equation such as 'compute one plus three minus two', and outputs are symbolic equations, such as 'X = 1 + 3 - 2'. The only publicly available word problem datasets we are aware of contain between 400 and 600 problems which is not enough to properly train sufficiently rich models that capture the link between the words and the quantities involved in the problem."

"We proposed to allow data generators to be 'weakly' specified, leaving the undetermined coefficients to be learned from data. We derived an efficient algorithm called GENERE, that jointly estimates the parameters of the model and the undetermined sampling coefficients, removing the need for costly cross-validation. While this procedure could be viewed as a generic way of building informative priors, it does not rely on a complex integration procedure such as Bayesian optimization, but corresponds to a simple modification of the standard stochastic optimization algorithms, where the sampling alternates between the use of real and generated data."