Harnessing GPTs for the next significant advancement in probabilistic programming
By Marc on
It was invigorating to tune into the latest PyMC Labs discussion led by Thomas Wiecki, PhD and Daniel Lee on GPTs in probabilistic programming, available here. Though it could have delved into a dry lecture, it emerged as a treasure trove of insights. They presented thought-provoking open questions that encourage us to explore new directions and consider the practical implications.
At ValueGrid, we've recently begun brainstorming on this exact topic, especially in the context of a new project with one of our customers. We'd like to reflect on a few of the captivating open questions from the talk.
Towards the end of the discussion (@ 56:20), Thomas poses an intriguing question: Can GPTs be used to build internal representations that allow reasoning about the world in a probabilistic way, and answer questions requiring causal Bayesian modeling? In our humble opinion, this concept holds the potential for the next disruptive breakthrough in probabilistic programming.
In our own tests, GPT-4 is currently great at generating boilerplate code for PyMC (notably older versions, predating v4/5) and Stan models. This in itself is commendable. Yet, the significant transformation will stem from its ability to craft actual internal representations—effectively defining the model, assigning priors to unknown parameters, outlining the likelihood and so on...
Daniel rightly points out two current limitations: 1) hallucination, and 2) GPTs have not been trained for the specialized, often exotic and complex models he's focused on. While he has a point in these niche areas, we believe the broader impact of GPTs will be most evident in more traditional industry sectors. In arenas such as engineering, corporate finance, operations and supply chain, there's significant potential for GPTs to excel in crafting these internal representations (as defined by Thomas).
Thomas mentions (@ 50:25) that "the benefit in Bayesian modeling is only really realized to its fullest potential when we are building the type of models that we're usually building where we have an understanding of what each parameter means but, once that understanding goes away, and we are in some 65 billion-dimensional space and we have no idea what those parameters actually relate to, what does even uncertainty mean in that in that space?". He is talking in the context of quantifying different types of uncertainties for GPTs' outputs, which, indeed, sounds quite extreme and may remain intractable for a while. However, we think that, in addition to the capability of building internal representations, as previously mentioned, GPTs can really shine by tackling the realm of mildly higher dimensional spaces, where today, most modeling teams do not venture, because of the loss of understanding of the different parameters, as well expressed by Thomas. Daniel reiterates the challenges posed by hallucinations. Still, as a community, it's up to us to harness GPTs effectively for such tasks.
Think about the intrinsic strengths of GPTs. They are the perfect tools to generate or formulate these probabilistic models based on the questions or scenarios it's presented with; to bring-in domain knowledge from virtually any (LLM-based) subject matter expert, nicely bringing your log loss down; to gain insights into your data, proposing a likelihood; to judiciously assign priors for all unknown parameters; to maintain explainability/transparency at scale, by breaking it down and explain/justify in plain English all the choices it makes, allowing modelers to follow and validate. This would pave the way for Bayesian modeling to make even greater strides in the industry.
In conclusion, we believe that harnessing GPTs to construct internal representations holds the key to the next significant advancement in probabilistic programming. We're thrilled to have the opportunity to work on practical applications for the industry.
At ValueGrid, we're genuinely interested in hearing your perspectives on this. Have you gathered any experiences or insights on the topic? Please share them in the comments. Let's continue the conversation and learn from each other.
Follow and participate in this discussion here