[{"content":"Imagine the large scale deployment of AI agents that make decisions on behalf of individuals or firms. For example, in online advertising auctions, an advertiser gives their high-level goal and constraints to a personal bidding agent. The responsibility of the bidding agent is to convert the advertiser\u0026rsquo;s goal and constraints into per-auction bids without human review. How do we ensure: (1) the advertiser\u0026rsquo;s goal is maximized, and (2) the outcome (who wins the auction) is efficient?\nThis example related to automated bidding in the online-ads industry is part of a more general problem I think we will continue to see more of in the future. AI agents are increasingly being used to negotiate and make decisions in the real world on behalf of the individuals and firms that deploy them. To ensure that the outcomes amongst a set of self-interested agents is efficient, we should adopt methods from the field of mechanism design 1.\nThe problems When developing such a system, there are fundamentally two core problems that need to be solved: the agent\u0026rsquo;s problem and the designer\u0026rsquo;s problem. The agent is the AI agent whom is responsible for making decions on behalf of the individual or firm whereas the designer is the platform or entity responsible for developing the mechanism that governs how scarce resources are allocated amongst strategically interacting self-interested agents.\nAgent problem Individuals or firms have an objective they would like to maximize subject to some constraints. Agents are often either: utility or value maximizing 2. Constraints often encode the private preferences of the individual or firm. Constraints can be budgets, return-on-spend, or cost-based, among many others. For example, an advertiser may have a budget they cannot exceed. Or a supply chain vendor may have a cost associated with transporting a product to a warehouse. The AI agent solves this optimization problem and sends the result to the designer. For example, the bidding agent computes the optimal bid amount to send to the auctioneer.\nDesigner problem Given the output of the agent\u0026rsquo;s problem, what mechanism does the designer use such that the outcome is efficient? Determining whether an outcome is efficient or not depends on the objective of the designer. Do they want to maximize revenue? Fairness? For example, a utility provider may be required to curtail power generated by residiential photovoltaics (PVs). Should the utility: minimize total curtailment? Equal curtailment? Proportional curtailment? Different curtailment rules result in different outcomes.\nOne of the hardest parts for the designer is that the inputs to the mechanism ultimately come from the private preferences of the individual or firm. Thus, the mechanism must incentivize them to truthfully reveal their private preferences to their agents as well as the incentive to participate in the mechanism.\nI define the field of mechanism design loosely here and it is meant to encapsulate concepts from social-choice theory and contract-design, among others.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAgents can also be both (hybrid).\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://gstechschulte.github.io/posts/2026-07-31-why-mechanism-design/","summary":"\u003cp\u003eImagine the large scale deployment of AI agents that make decisions on behalf of individuals or firms. For example, in online advertising auctions, an advertiser gives their high-level goal and constraints to a personal bidding agent. The responsibility of the bidding agent is to convert the advertiser\u0026rsquo;s goal and constraints into per-auction bids without human review. How do we ensure: (1) the advertiser\u0026rsquo;s goal is maximized, and (2) the outcome (who wins the auction) is efficient?\u003c/p\u003e","title":"Coordinating Agents With Mechanism Design"},{"content":"This post is an ongoing investigation of my alternative view of AI and the role of algorithms for multi-agent systems in markets.\nGame theory is concerned with predicting the behavior of agents participating in an strategic interaction. We can also ask the inverse. Given a desired behavior of the agents, what strategic interaction would give rise to this behavior? For example, there are two firms $A$ and $B$. Each firm can choose to cooperate $C$ or defect $D$ which results in the following payoff matrix\nFirm 2: C Firm 2: D Firm 1: C (3, 3) (0, 5) Firm 1: D (5, 0) (1, 1) The methods of game theory looks at this interaction and predicts what each agent will do. In this interaction, defecting $D$ is the strictly dominant strategy for both firms. Correspondingly, the Nash Equilibrium (NE) is $(D, D)$ with payoffs $(1, 1)$, but the socially efficient outcome is $(C, C)$ with payoffs $(3, 3)$. We can resolve this inefficency using mechanism design.\nIn this example, we would like to introduce a mechanism such that when the firms play the modified game, they find it in their interest to produce the socially efficient outcome. The mechanism is not changing the agents\u0026rsquo; preferences, rather, the payoff is changed. Imagine a regulation is introduced such that a tax $\\tau$ is imposed on firms that engage in $D$. The firm pays $\\tau$ to the regulator regardless of what the other firms do. The payoff matrix is now\nFirm 2: C Firm 2: D Firm 1: C (3, 3) (0, 5 - $\\tau$) Firm 1: D (5 - $\\tau$, 0) (1 - $\\tau$, 1 - $\\tau$) For $D$ to longer be the dominant strategy, $C$ needs to be at least as good as $D$ in both actions. In other words, the value of $\\tau \\geq 2$ which results in\nFirm 2: C Firm 2: D Firm 1: C (3, 3) (0, 3) Firm 1: D (3, 0) (-1, -1) $D$ is now weakly dominated by $C$ and $(C, C)$ becomes the NE which is the socially efficient outcome we desired. Notice that the designer of the mechanism does not force the firms to cooperate. The firms remain rational and self-interested.\n","permalink":"https://gstechschulte.github.io/posts/2026-05-08-thinking-about-mechanism-design/","summary":"\u003cp\u003eThis post is an ongoing investigation of my \u003ca href=\"https://gstechschulte.github.io/posts/2025-09-02-future-of-ai/\"\u003ealternative view of AI\u003c/a\u003e and the role of algorithms for multi-agent systems in markets.\u003c/p\u003e\n\u003cp\u003eGame theory is concerned with predicting the behavior of agents participating in an strategic interaction. We can also ask the inverse. Given a desired behavior of the agents, what strategic interaction would give rise to this behavior? For example, there are two firms $A$ and $B$. Each firm can choose to cooperate $C$ or defect $D$ which results in the following payoff matrix\u003c/p\u003e","title":"Mechanism Design - A Primer"},{"content":"Being able to build something, or at least having some knowledge about how something is built is sort of beautiful. It\u0026rsquo;s a kind of hidden superpower. Whenever I see a new apartment building or street being built, it always crosses my mind how the people in this line of work are paid less, on average, than those in offices. It appears that the work of those building the houses and streets is more valuable than those of the management class, so why are they paid less?\nTo answer this question, you need to stop thinking \u0026ldquo;in totality\u0026rdquo;, and rather \u0026ldquo;on the margin\u0026rdquo;. The total importance of a good or service does not determine its price. Its scarcity at the margin does. Keyne\u0026rsquo;s describes this in the Diamond-Water Paradox: The total value of water is much greater than that of diamonds but since water is abundant, the marginal value of water is much less than that of diamonds, and thus the reason for the low price.\nWe would say that the marginal revenue of product (MRP) of the construction worker is low relative to the finance worker. The construction worker\u0026rsquo;s marginal output is limited by their physical capability (e.g., how many bricks can be laid). Yes, machines may increase output, albeit, we are still bottlenecked by physics. On the other hand, the finance worker is much more leveraged. The trader may have access to a large trading account which can result in significant profits for the company relating to a much higher MRP. Moreover, the construction field is considered to have an elastic supply curve, i.e., the amount of available workers is much higher than that of finance because barriers to entry are lower and skills can be learned in a shorter amount of time.\nSo yeah, think on the margin.\n","permalink":"https://gstechschulte.github.io/posts/2026-04-12-labor-margin/","summary":"\u003cp\u003eBeing able to build something, or at least having some knowledge about how something is built is sort of beautiful. It\u0026rsquo;s a kind of hidden superpower. Whenever I see a new apartment building or street being built, it always crosses my mind how the people in this line of work are paid less, on average, than those in offices. It appears that the work of those building the houses and streets is more valuable than those of the management class, so why are they paid less?\u003c/p\u003e","title":"The Construction Versus Finance Worker"},{"content":"I recently read the paper Carbon-Aware Computing for Data Centers with Probabilistic Performance Guarantees which demonstrated the benefits and power of a Distributionally Robust Optimization method and I wanted to dive deeper into the method. This blog post is a result of the deep dive.\nIntroduction In Stochastic Optimization (SO), we are interested in solutions that account for different possible outcomes of uncertain data. Accounting for this uncertainty requires us to define a probability distribution over the uncertain quantities—a forecast error, a demand signal, a compute load—and then optimize an objective defined with respect to that distribution. The choice of how to use the distribution depends on the application: one might minimize expected cost, guard against a high-percentile loss using Conditional Value-at-Risk (CVaR), or enforce a chance constraint that a capacity limit is violated with probability no greater than some threshold. What all of these approaches share is the assumption that we know, or can accurately estimate, the distribution itself. Distributionally Robust Optimization is the framework we reach for when that assumption breaks down.\nIn Distributionally Robust Optimization (DRO), we are uncertain not just about the outcome of a random variable (RV), but about the data-generating process that produced the observations we have collected. Suppose there is a RV in our objective function or constraints that is distributed according to some true but unknown distribution $\\mathbb{P}$. We have collected $N$ historical observations of this quantity, and from them we can construct an empirical distribution $\\hat{\\mathbb{P}}$ that places equal probability weight $1/N$ on each observed value:\n$$ \\begin{equation} \\hat{\\mathbb{P}} = \\frac{1}{N} \\sum_{i=1}^{N} \\delta_{s^i} \\end{equation} $$\nwhere $\\delta_{s^i}$ is a Dirac mass centered at the $i$-th historical observation $s^i$.\nGiven these samples, a natural approach is Sample Average Approximation (SAA): simply treat $\\hat{\\mathbb{P}}$ as if it were the true distribution $\\mathbb{P}$ and optimize directly against your historical values. SAA is easy to implement and works well when $N$ is large and the future resembles the past. The problem is that $\\hat{\\mathbb{P}}$ places zero probability on any outcome not in your historical data. If tomorrow\u0026rsquo;s data looks nothing like you have ever seen before such as a public holiday, a viral product launch or a seasonal shif. The SAA solution provides no protection whatsoever. The empirical distribution is a point estimate in the space of distributions, and like any point estimate, it carries sampling error that grows more severe as $N$ decreases.\nDRO\u0026rsquo;s central idea is to hedge against this uncertainty by optimizing against the worst-case distribution in a neighborhood around $\\hat{\\mathbb{P}}$. This neighborhood is called the ambiguity set, and it quantifies the collection of distributions that are plausibly consistent with the data you have observed. Formally, we define the ambiguity set as a Wasserstein ball:\n$$ \\begin{equation} \\mathcal{B}^{\\varepsilon} := { \\mathbb{Q} \\in \\mathcal{P}_1(\\mathcal{S}) \\mid d_W(\\hat{\\mathbb{P}}, \\mathbb{Q}) \\leq \\varepsilon } \\end{equation} $$\nwhere $\\mathcal{P}_1(\\mathcal{S})$ is the space of probability distributions supported on the feasible set $\\mathcal{S}$, and $d_W(\\hat{\\mathbb{P}}, \\mathbb{Q})$ is the Wasserstein-1 distance between $\\hat{\\mathbb{P}}$ and $\\mathbb{Q}$. The Wasserstein distance (aka earth mover\u0026rsquo;s distance) measures the minimum cost required to transport the probability mass of one distribution into the shape of another, where cost accounts for both how much mass is moved and how far it travels. A distribution that has the same shape as $\\hat{\\mathbb{P}}$ but shifted slightly upward is close in Wasserstein distance, because all of the mass only needs to travel a short distance. A distribution with probability mass concentrated in a completely different region is far away, because the mass would need to travel a long distance to reach it.\nThe radius $\\varepsilon$ is the central tuning parameter:\nA small $\\varepsilon$ means the adversary can only choose distributions that look very similar to $\\hat{\\mathbb{P}}$. The worst-case distribution won\u0026rsquo;t deviate much from your historical data, and the DRO solution will be close to the SAA solution. A large $\\varepsilon$ means the adversary can choose distributions that look quite different from $\\hat{\\mathbb{P}}$, including distributions with mass in regions your data has never visited. The solution will be more conservative but more robust to distributional shift. The DRO problem then takes the form of a minimax optimization: choose the decision $x$ that minimizes cost under the worst-case distribution in $\\mathcal{B}^\\varepsilon$:\n$$ \\begin{equation} \\min_{x \\in \\mathcal{X}} \\sup_{\\mathbb{Q} \\in \\mathcal{B}^\\varepsilon} \\mathbb{E}^{s \\sim \\mathbb{Q}}[f(x, s)] \\end{equation} $$\nThe $\\sup$ over $\\mathbb{Q}$ represents an adversary that, after you commit to a decision $x$, picks the distribution within the ambiguity set that makes your decision look as bad as possible.\nImplementation Conceptually, the ambiguity set makes sense but it is not very intuitive to me. For starters, equation (1) defines an infinite-dimensional set, i.e., there are uncountably many probability distributions within any Wasserstein ball. How does one solve the worst-case optimization over such a set? Moreover, how can we develop an intuition for what the ball actually contains? The goal in the proceeding sections is to develop a geometric intuition for what the ambiguity set contains. Then, in a subsequent post, we will address how we actually solve the optimization problem analytically.\nNote: All code for this blog post can be viewed or downloaded via the embedded Marimo notebook below.\nWasserstein distance For our empirical dataset, we will sample data from a Normal distribution with $\\mu=5$ and $\\sigma=1$. In 1D, the Wasserstein distance between two equal-weight empirical distributions on $N$ points each reduces to the average absolute difference between their sorted quantiles,\n$$ \\begin{equation} W_1(\\hat{\\mathbb{P}}, \\mathbb{Q}) = \\frac{1}{N} \\sum_{i=1}^{N} \\left| x_{(i)} - y_{(i)} \\right| \\end{equation} $$\nwhere $x_{(i)}$ and $y_{(i)}$ are the $i$-th order sorted values of $\\hat{\\mathbb{P}}$ and $\\mathbb{Q}$ respectively. Geometrically, this is the area between the two CDFs. This formula reveals something important. Suppose we parameterize a candidate distribution $\\mathbb{Q}$ by a perturbation vector $\\delta \\in \\mathbb{R}^N$, where $\\delta_i = y_{(i)} - x_{(i)}$ is the shift applied to the $i$-th sorted quantile of $\\hat{\\mathbb{P}}$. Then the Wasserstein ball constraint $W_1(\\hat{\\mathbb{P}}, \\mathbb{Q}) \\leq \\varepsilon$ becomes exactly:\n$$ \\begin{equation} \\frac{1}{N} \\sum_{i=1}^{N} |\\delta_i| \\leq \\varepsilon \\end{equation} $$\nThis is an L1 ball constraint on $\\delta$ in $\\mathbb{R}^N$. Placing a Wasserstein ball of radius $\\varepsilon$ around $\\hat{\\mathbb{P}}$ in the infinite-dimensional space of distributions is equivalent, in 1D, to constraining a finite perturbation vector to lie in an L1 ball of radius $\\varepsilon$.\nSampling from the Wasserstein ball To sample a distribution $\\mathbb{Q}$ from $\\mathcal{B}^\\varepsilon$, we need to generate a perturbation vector $\\delta$ that satisfies the L1 constraint above. We will implement the following three steps to achieve this:\nChoose a random direction. Draw $\\delta_i \\sim \\mathcal{N}(0, 1)$ for each $i = 1, \\ldots, N$. This gives a random direction in $\\mathbb{R}^N$.\nProject onto the L1 sphere. Normalize $\\delta$ so that its L1 radius equals exactly $\\varepsilon$:\n$$\\delta \\leftarrow \\delta \\cdot \\frac{\\varepsilon}{\\frac{1}{N}\\sum_i |\\delta_i|}$$\nPull into the interior. Multiply by $u \\sim \\text{Uniform}(0, 1)$ to land somewhere between the origin and the sphere surface: $$\\delta_{\\text{final}} = u \\cdot \\delta$$\nThe resulting perturbation satisfies $\\frac{1}{N}\\sum_i |\\delta_{\\text{final},i}| \\leq \\varepsilon$ by construction. Adding $\\delta_{\\text{final}}$ to the sorted quantiles of $\\hat{\\mathbb{P}}$ and re-sorting gives the support of a valid sample distribution $\\mathbb{Q} \\in \\mathcal{B}^\\varepsilon$.\nPlay with the slider below to change the value of $\\epsilon$ and observe how $\\mathbb{Q}$ changes.\n","permalink":"https://gstechschulte.github.io/posts/2026-03-06-dro-part-one/","summary":"\u003cp\u003eI recently read the paper \u003ca href=\"https://ieeexplore.ieee.org/document/11250739\"\u003eCarbon-Aware Computing for Data Centers with Probabilistic Performance Guarantees \u003c/a\u003e which demonstrated the benefits and power of a Distributionally Robust Optimization method and I wanted to dive deeper into the method. This blog post is a result of the deep dive.\u003c/p\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eIn Stochastic Optimization (SO), we are interested in solutions that account for different possible outcomes of uncertain data. Accounting for this uncertainty requires us to define a probability distribution over the uncertain quantities—a forecast error, a demand signal, a compute load—and then optimize an objective defined with respect to that distribution. The choice of how to use the distribution depends on the application: one might minimize expected cost, guard against a high-percentile loss using Conditional Value-at-Risk (CVaR), or enforce a chance constraint that a capacity limit is violated with probability no greater than some threshold. What all of these approaches share is the assumption that we know, or can accurately estimate, the distribution itself. Distributionally Robust Optimization is the framework we reach for when that assumption breaks down.\u003c/p\u003e","title":"Distributionally Robust Optimization - Part 1"},{"content":"Many applications of optimization and control are performed in a deterministic setting. That is, the quantities of the problem such as the state variables, control variables, and or parameters of a model are treated as fixed known values. This assumption may be reasonable in applications such as robotics, but in others such as resource allocation, we may need to incorporate the uncertainty of various quantities in order to compute the expectation of the objective.\nFor example, when performing battery arbitrage, the goal is to develop a system that uses a battery to buy energy from the grid when prices are low and sell back to the grid when prices are high. Electricity prices can fluctuate greatly due to supply and demand. Thus, our system needs to take into account the uncertainty of the electricity prices. Optimizing and taking uncertainty into account means that we seek an optimal solution that accounts for different outcomes of the data.\nFor example, in the bottom subplot at 12:00, we face different price scenarios for the remainder of the day. Each price scenario can result in a differerent optimal decision (second to top plot) which impacts the reward (top plot) and consequently the battery state of charge (SoC).\nStochastic optimization essentially boils down to putting expectations in front of things and or computing probabilities and then solving the optimization problem. Most basic stochastic optimization problems, like the one above, can be solved via strategies that either: compute expectations directly, or by methods that approximate them.\nIn this post, we will develop a model predictive control (MPC) framework utilizing sampled approximations to solve the battery arbitrage problem described above. To simplify the problem setup, to generate the samples, a set of price paths will be sampled from the historical data and we will act like these samples are the price forecasts from the posterior predictive distribution of a Bayesian model.\nThe code to reproduce the results of this post can be found in the following GitHub repository.\nModel predictive control MPC is a flexible control framework that uses a model of a system to predict its future behavior and optimize control actions over a time horizon. At each time step, an MPC controller solves an optimization problem to find the best sequence of control actions that minimizes a cost function while satisfying system constraints. In MPC, only the first control action from the control trajectory is implemented, and the process is repeated at the next time step allowing the controller to adapt to new information.\nProblem formulation The objective is to develop a MPC controller that maximizes the expected profit by choosing how much energy to buy from or sell to the grid $e$ given the electricity price $p$ at each timestep $t$ over some horizon $T$.\n$$ \\begin{equation} \\max_{u} \\mathbb{E} \\sum_{t=0}^T -p_t e_t \\end{equation} $$\nThe decision variables (controls) are subject to the following constraints\n$$ \\begin{align} e_t \u0026amp;\\leq \\frac{1}{\\eta} (R^{\\max} - R_t) \\\\ e_t \u0026amp;\\geq -\\frac{1}{\\eta} R_t \\end{align} $$\nwhere $R$ is the battery SoC at each time step $t$. So far, the system is comprised of the following state vector $\\mathbf{x}$ and control vector $\\mathbf{u}$\n$$ \\begin{align} \\mathbf{x} = (R_t, p_t) \\\\ \\mathbf{u} = (e_t) \\end{align} $$\nThe variables in the state vector evolve according to the following transition functions\n$$ \\begin{align} R_{t+1} = R_t + \\eta x_t \\\\ p_{t+1} = f(p_t) \\end{align} $$\nwhere $f$ is a Bayesian model that forecasts the electricity price over the horizon $T$.\nSample average approximation Our objective function (1) involves an expectation over uncertain price forecasts. Rather than computing the expectation analytically, we can use sample average approximation (SAA) by drawing $N$ samples from the posterior predictive distribution obtained from a Bayesian model. Let $c_n$ represent the $n$\u0026lsquo;th sample from the posterior predictive distribution. The SAA of equation (1) becomes\n$$ \\begin{align} \\min_{x, x_{n=1}^N} \\frac{1}{N} \\sum_{n=1}^N c_n^T x_n \\end{align} $$\nwhere $x_n \\in \\mathbb{R}^H$ is the control trajectory for price sample $n$ and $c_n \\in \\mathbb{R}^H$ is the price forecast trajectory for price sample $n$. Since the SAA problem is embeded into the MPC framework, this problem is solved at each timestep\n$$ \\begin{align} \\min_{x, x_{n=1}^N} \\frac{1}{N} \\sum_{n=1}^N \\sum_{h=0}^{H-1} c_h^n x_h^n \\end{align} $$\nwhere $x_h^n$ is the $n$\u0026lsquo;th control trajectory at the $h$\u0026lsquo;th hour and $c_h^n$ is the $n$\u0026lsquo;th price forecast at the $h$\u0026lsquo;th hour. In practice (and shown in the implementation below), we will stack the control vector into $\\bar{x}$ to obtain a single large LP. Then, when you take the dot product between $c$ and $x$, you effectively compute the expected profit.\n$$ \\begin{align} \\min_{x} \\bar{c}^T \\bar{x} \\\\ \\text{s.t.} \\space A_{ub} \\bar{x} \\leq b_{ub} \\\\ A_{eq} \\bar{x} = b_{eq} \\\\ l \\leq \\bar{x} \\leq u \\end{align} $$\nConstraints Constraints are one of those things that are easy to state in words, but can be tricky when it comes to implementing them in code. The constraints of our problem quickly become complex as we are optimizing over scenarios (samples) and time (the control horizon). Here, we introduce and use triangular and block diagonal matrices to encode the constraints of our system.\nIn our MPC problem we need to ensure the following at each timestep\nThe battery SoC cannot exceed its minimum or maximum capacity. The amount of power bought from the grid cannot cause the battery SoC to go above its maximum capacity. The amount of power sold to the grid cannot cause the battery SoC to go below its minimum capacity. For each scenario (sample), the first decision (control) must be equal to each other. Recall the standard LP formulation\n$$ \\begin{align} \\min_{x} c^Tx \\\\ \\text{s.t.} \\space A_{ub} x \\leq b_{ub} \\\\ A_{eq} x = b_{eq} \\\\ l \\leq x \\leq u \\end{align} $$\nwhere $x$ is a control vector, $c$ is a coefficient vector, $A_{ub}$ encodes inequality constraints, $A_{eq}$ encodes equality constraints, and $\\ell$ and $u$ encode lower and upper bounds (box constraints) on controls.\nConstraint (1) is the easiest. These are the box constraints $\\alpha$ of $R_t$ and are traditionally passed to the SciPy linprog function as a list of tuples\nbounds = [(-alpha, alpha) for _ in range(n_vars)] Constraints (2) and (3) are more complex. We need to translate the decision variable constraints from above to matrix-form $A_{ub} x \\leq b_{ub}$. First, lets imagine we only have one sample. Reformulate the constraint equations above to\n$$ \\begin{align} e_t \u0026amp;\\leq \\frac{1}{\\eta} (R^{\\max} - R_t) \u0026amp;\\rightarrow \\eta e_t \\leq R^{max} - R_t \\\\ e_t \u0026amp;\\geq -\\frac{1}{\\eta} R_t \u0026amp;\\rightarrow -\\eta e_t \\leq R_t \\end{align} $$\nNotice how the left hand side terms $\\eta$ can be expressed as a matrix of coefficients and the right hand side terms can be expressed in terms of a column vector.\nAt each time step, we solve the LP for the next 24 hours subject to these \u0026ldquo;upper\u0026rdquo; and \u0026ldquo;lower\u0026rdquo; constraints on the decision variables. The optimizer evaluates the constraints via the dot product A_ub @ x \u0026lt;= b_ub. To enforce the respective constraints on each decision variable $u_0, u_1, \\ldots, u_N$ at each timestep we use a lower triangular matrix for each upper and lower constraint of shape (24, 24).\nupper_constraint = np.tri(H, H) * eta lower_constraint = np.tri(H, H) * (-eta) which would give us the following upper constraint in matrix-form\n$$ \\begin{bmatrix} \\eta \u0026amp; 0 \u0026amp; 0 \u0026amp; \\ldots \u0026amp; 0 \\\\ \\eta \u0026amp; \\eta \u0026amp; 0 \u0026amp; \\ldots \u0026amp; 0 \\\\ \\eta \u0026amp; \\eta \u0026amp; \\eta \u0026amp; \\ldots \u0026amp; 0 \\\\ \\vdots \u0026amp; \\vdots \u0026amp; \\vdots \u0026amp; \\ddots \u0026amp; \\vdots \\\\ \\eta \u0026amp; \\eta \u0026amp; \\eta \u0026amp; \\ldots \u0026amp; \\eta \\\\ \\end{bmatrix} \\begin{bmatrix} x_0 \\\\ x_1 \\\\ x_2 \\\\ \\vdots \\\\ x_{H-1} \\\\ \\end{bmatrix} \\leq \\begin{bmatrix} R^{max} - R_t \\\\ R^{max} - R_t \\\\ R^{max} - R_t \\\\ \\vdots \\\\ R^{max} - R_t \\\\ \\end{bmatrix} $$\nand the lower constraint\n$$ \\begin{bmatrix} -\\eta \u0026amp; 0 \u0026amp; 0 \u0026amp; \\ldots \u0026amp; 0 \\\\ -\\eta \u0026amp; -\\eta \u0026amp; 0 \u0026amp; \\ldots \u0026amp; 0 \\\\ -\\eta \u0026amp; -\\eta \u0026amp; -\\eta \u0026amp; \\ldots \u0026amp; 0 \\\\ \\vdots \u0026amp; \\vdots \u0026amp; \\vdots \u0026amp; \\ddots \u0026amp; \\vdots \\\\ -\\eta \u0026amp; -\\eta \u0026amp; -\\eta \u0026amp; \\ldots \u0026amp; -\\eta \\\\ \\end{bmatrix} \\begin{bmatrix} x_0 \\\\ x_1 \\\\ x_2 \\\\ \\vdots \\\\ x_{H-1} \\\\ \\end{bmatrix} \\leq \\begin{bmatrix} R_t \\\\ R_t \\\\ R_t \\\\ \\vdots \\\\ R_t \\\\ \\end{bmatrix} $$\nIf we only had a single sample we could just vertically stack these two matrices and we would have the final $A_{ub}$ matrix. However, since each sample needs to satsify these upper and lower constraints, we can use a block diagonal matrix to encode this behavior. A block diagonal matrix is a square matrix where the diagonal entries are themselves square matrices (blocks) and all other off-diagonal entries are zero matrices. For example, if there were four posterior samples the upper constraint block diagonal matrix is\n$$ \\begin{bmatrix} A_{ub_1} \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\\\ 0 \u0026amp; A_{ub_2} \u0026amp; 0 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; A_{ub_3} \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; A_{ub_4} \\end{bmatrix} $$\nwhere there are four diagonal matrices; one for each sample. Each diagonal matrix is an upper_constraint matrix. The structure for the lower constraint block diagonal matrix it the same. We use scipy.linalg.block_diag to build the block matrices. Lastly, these two block matrices are vertically stacked to obtain the final $A_{ub}$ matrix.\n# Repeat upper/lower constraints for each sample upper_blocks = [upper_constraint.copy() for _ in range(n_samples)] lower_blocks = [lower_constraint.copy() for _ in range(n_samples)] # Each diagonal block contains the constraints for this sample A_ub_upper = scipy.linalg.block_diag(*upper_blocks) A_ub_lower = scipy.linalg.block_diag(*lower_blocks) A_ub = np.vstack([A_ub_upper, A_ub_lower]) The $b_{ub}$ column vector is where the inequality constraint evaluations happen. We just need to concatenate the evaluation for the upper and lower constraints.\nb_ub = np.concatenate([ np.full(n_samples * H, R_max - R_current), # Upper constraint eval np.full(n_samples * H, R_current) # Lower constraint eval ]) Lastly, constraint (4) encodes non-anticipativity constraints which is a fancy name for saying that \u0026ldquo;the first control of each sample must be equal to each other\u0026rdquo;. This constraint is primarily required to ensure that the optimal solution is implementable in practice. Without this constraint, the optimal solution may contain different values for the first control sequence in each posterior predictive sample sparking the question—\u0026ldquo;which control should I implement?\u0026rdquo;.\nNon-anticipativity constraints can be encoded using equality constraints. Below, we encode arithmetic that the difference between the first control value for each sample must be zero, i.e., the control values are the same.\nfor i in range(n_na_constraints): # Constraint: u^i_0 - u^{i+1}_0 = 0 # Position of u^{i+1}_0 in the stacked vector: (i+1) * H + 0 A_eq_na[i, i * H] = 1.0 # Coefficient for u^i_0 A_eq_na[i, (i + 1) * H] = -1.0 # Coefficient for u^{i+1}_0 b_eq_na = np.zeros(n_na_constraints) Implementation The full sample average approximation implementation is the following\ndef solve_saa(R_current, p_forecast_samples): n_samples, H, dim = p_forecast_samples.shape n_vars = n_samples * H # Coefficient vector is the forecasted prices # Now in terms of the expected profit c = (1 / n_samples) * p_forecast_samples.flatten() upper_constraint = np.tri(H, H) * eta lower_constraint = np.tri(H, H) * (-eta) upper_blocks = [upper_constraint.copy() for _ in range(n_samples)] lower_blocks = [lower_constraint.copy() for _ in range(n_samples)] # Each diagonal block contains the constraints for this sample A_ub_upper = scipy.linalg.block_diag(*upper_blocks) A_ub_lower = scipy.linalg.block_diag(*lower_blocks) A_ub = np.vstack([A_ub_upper, A_ub_lower]) b_ub = np.concatenate([ np.full(n_samples * H, R_max - R_current), # Upper constraint eval np.full(n_samples * H, R_current) # Lower constraint eval ]) bounds = [(-alpha, alpha) for _ in range(n_vars)] # Build non-anticipativity constraints n_na_constraints = n_samples - 1 A_eq_na = np.zeros((n_na_constraints, n_vars)) for i in range(n_na_constraints): # Constraint: u^i_0 - u^{i+1}_0 = 0 # Position of u^i_0 in the stacked vector: i * H + 0 # Position of u^{i+1}_0 in the stacked vector: (i+1) * H + 0 A_eq_na[i, i * H] = 1.0 # Coefficient for u^i_0 A_eq_na[i, (i + 1) * H] = -1.0 # Coefficient for u^{i+1}_0 b_eq_na = np.zeros(n_na_constraints) res = linprog( c=c, A_ub=A_ub, b_ub=b_ub, A_eq=A_eq_na, b_eq=b_eq_na, bounds=bounds, method=\u0026#34;highs\u0026#34; ) return res Running the MPC controller over the control horizon, we can visualize the trajectories over time.\n# System parameters T = 48 H = 24 n_states = 2 n_controls = 1 R_max = 10 alpha = 5 eta = 0.9 # Trajectories x_traj = np.zeros((H, n_states)) u_traj = np.zeros((H, n_controls)) obj_traj = np.zeros((H,)) # Initial conditions R_current = 5.0 actual_p_idx = 0 # Realized price scenario p_current = p_sampled[actual_p_idx, 0].item() total_obj = 0.0 future_xs = range(H) for k in range(H): current_xs = range(k) # Get price forecast p_forecast = p_sampled[:, k: H + k] # Solve the sample average approx. problem out = solve_saa(R_current, p_forecast) profit = -out.fun u_opt = out.x # Apply first control sequence uk_opt = u_opt[0] # Compute contribution obj = objective(uk_opt, p_current) # State transition R_current = R_current + (eta * uk_opt) p_current = p_sampled[actual_p_idx, k + 1].item() # Compute cumulative reward total_obj += obj # Track trajectories x_traj[k] = np.array([R_current, p_current]) u_traj[k] = np.array([uk_opt]) obj_traj[k] = total_obj # Propogate uncertainty u_opt_all = u_opt.reshape(4, 24, 1) obj_all = objective(u_opt_all, p_forecast) meanobj = obj_all.mean(axis=0).flatten() stdobj = obj_all.std(axis=0).flatten() r_current_all = R_current + (eta * u_opt_all) # Plotting code not shown here... It is interesting to observe the uncertainty of the reward (profit) given the samples for the decision and state variables. For example, we can observe that when the controls are all the same, e.g., from timestep 15:00 to 16:00, the reward uncertainty is very small. Whereas in the time period from 16:00 to 20:00, prices and control decisions are more uncertain, and thus the uncertainty in the reward is much higher. The realized trajectory (indicated by the solid black line following the dashed vertical line) shows that the reward roughly followed the expected reward and that the solver chooses controls that maximize the expected reward.\n","permalink":"https://gstechschulte.github.io/posts/2025-11-18-stochastic-mpc/","summary":"\u003cp\u003eMany applications of optimization and control are performed in a deterministic setting. That is, the quantities of the problem such as the state variables, control variables, and or parameters of a model are treated as fixed known values. This assumption may be reasonable in applications such as robotics, but in others such as resource allocation, we may need to incorporate the uncertainty of various quantities in order to compute the \u003cstrong\u003eexpectation\u003c/strong\u003e of the objective.\u003c/p\u003e","title":"Stochastic Model Predictive Control"},{"content":"Uber allocates money across different regions and programs to incentivize riders and drivers to use Uber products. This incentive structure ultimately influences the market. This leads to the natural question of \u0026ldquo;how much to allocate to each city and which program\u0026rdquo; to maximize business objectives? Uber has a finite amount of money that must be allocated accordingly. Given a total budget of say, $1,000,000, how should it be divied up amongst the cities and programs?\nUber has developed an automated system for answering such a question. In this post, I will be reproducing the paper Practical Marketplace Optimization at Uber Using Causally-Informed Machine Learning in JAX on a simulated marketplace dataset. The main objective of this post is to: (1) develop a neural network that learns the association between budget allocations and marketplace outcomes, and (2) implement the Alternating Direction Method of Multipliers (ADMM) algorithm in the context of resource allocation problems using the neural network.\nThe code contained in this post can be found in this repo.\nData simulation In order to better understand how the neural network is embedded into ADMM and how ADMM works, a simple marketplace dataset with three cities and one program for each city is simulated. To maintain some of the real-life characteristics of the data, non-linearity and interaction effects are introduced. This results in the following graphs\nDepending on the total budget allocation constraint, a reasonable budget allocation should be approximately $$2500, $3500, and $5000 for city 0, 1, and 2, respectively.\nImplementation There are four main methodological components in Uber\u0026rsquo;s automated budget allocation system\nUser Causal Effect Estimator. A deep learning (DL) model that uses historical data to predict marketplace outcomes such as gross bookings for each region. Smoothing Layer. Used to generate a differentiable low-dimensional response surface from the DL model outputs in order to more efficiently perform optimization. Optimizer. Computes the optimal budget allocation for each region and lever subject to the marketplace model dynamics and business operation constraints. Business Value Evaluation. An evaluation framework to measure the quality of the optimal budget allocations. In this post, we will discuss components 1-3 and implement the DL model and ADMM algorithm.\nUser Causal Effect Estimator For the user causal effect estimator, I implement a simple neural network with an embedding layer for the \u0026ldquo;sparse\u0026rdquo; categorical features (cities) and a linear layer for the \u0026ldquo;dense\u0026rdquo; numerical features (budgets). The sparse and dense feature embeddings are then concatenated and fed through a series of ResNet blocks. However, for simplicity, I use MLPs.\nclass MarketModel(nnx.Module): def __init__( self, num_cities: int, city_embedding_dim: int, budget_embedding_dim: int, *, rngs: nnx.Rngs, ): input_dim = city_embedding_dim + budget_embedding_dim # Sparse embedding layer (for categorical features) self.city_embedding = nnx.Embed( num_embeddings=num_cities, features=city_embedding_dim, rngs=rngs ) # Dense feature extractor (for numerical features) self.budget_embedding = nnx.Linear(1, budget_embedding_dim, rngs=rngs) self.linear1 = nnx.Linear(input_dim, 128, rngs=rngs) self.linear2 = nnx.Linear(128, 256, rngs=rngs) self.linear3 = nnx.Linear(256, 128, rngs=rngs) self.linear4 = nnx.Linear(128, 16, rngs=rngs) self.out = nnx.Linear(16, 1, rngs=rngs) def __call__(self, city_idx: Array, budget: Array) -\u0026gt; Array: budget_embedding = nnx.relu(self.budget_embedding(budget)) city_embedding = self.city_embedding(city_idx) # Sparse and dense feature embeddings are concatenated combined_input = jnp.concatenate([city_embedding, budget_embedding], axis=-1) x = nnx.relu(self.linear1(combined_input)) x = nnx.relu(self.linear2(x)) x = nnx.relu(self.linear3(x)) x = nnx.relu(self.linear4(x)) x = self.out(x) return x The input to the MarketModel is an array of city indices and budgets. Training the model and plotting the model\u0026rsquo;s predictions results in the following graph below. This simple neural network architecture is able to learn the association between per-city budget allocations and business outcomes (gross bookings) reasonably well.\nThis model will then be embedded into the optimizer where we will differentiate and evaluate the response surface of the model. Repeatedly evaluating the response surface is like asking \u0026ldquo;what happens if we increase driver incentives by 100 dollars in Chicago while decreasing rider promotions by 50 dollars in Detriot\u0026rdquo;? Then, when we taken the gradient of the outcome (bookings) with respect to the input (budgets), we get a sense of which direction maximizes our objective.\nSmoothing model The deep learning (DL) model produces a mapping from budgets to marketplace outcomes (supply, demand, gross bookings, etc.), which can be fed into an optimizer. However, the optimization routine require numerous passes over the response surface, necessitating many evaluations of the DL model, which is computationally costly. The response surface can be thought of a non-convex landscape where each point represents a different combination of budget amounts across various levers and cities where the height at each point shows the predicted business outcome.\nThe goal with the smoothing model is to generate a low-dimensional \u0026ldquo;smooth\u0026rdquo; differential surface for derivative-based optimization algorithms. Uber achieves this using Adaptive Sparse Grids (ASG) to strategically sample a grid-based representation of the DL surface. B-spline functions are then used as a smoothing model to approximate the response surface. B-splines are choosen because of their flexibility and analytical derivative forms. This B-spline model is then embeded into the ADMM algorithm to provide low-cost evaluations of the response surface.\nAs I simulate a dataset where I can visually see the optimal budget allocations for each city, I will not implement the smoothing model in this post. Instead, I will embed the DL model directly into the ADMM algorithm. Although, once more levers are added to each city or more features are added to the DL model, it would be interesting to analyze the impacts the smoothing model has on the ADMM results.\nOptimizer ADMM is used to solve the non-linear, non-convex resource allocation problem. The objective is to break up a total budget into smaller budgets for each city and program. This allows us to separate the single-city cross-lever problem and the cross-city problem and solve both problems iteratively. The objective function to solve is\n$$ \\begin{aligned} \\min_{\\mathbf{b}, \\mathbf{z}} \\quad \u0026amp; \\sum_{c \\in \\text{cities}} f_c(\\mathbf{b}_c) + g(\\mathbf{z}) \\\\ \\text{s.t.} \\quad \u0026amp; \\mathbf{b}_c - \\mathbf{z}_c = \\mathbf{0}, \\quad \\forall c \\in \\text{cities} \\end{aligned} $$\nand the ADMM algorithm consists of three steps:\n$\\mathbf{b}$ update. This step optimizes each city independently given the current consensus $\\mathbf{z}$ and duals $\\mathbf{y}$ effectively balancing the primary objective (maximize the outcome for this city), consensus penalty (stay close to the consensus target allocation), and the business penalty (don\u0026rsquo;t stray too far from the reference budget allocation). $$ \\begin{aligned} \\mathbf{b}_c^{k+1} := \\underset{\\mathbf{b}_c}{\\operatorname{argmin}} \\left( f_c(\\mathbf{b}_c) + \\frac{\\rho}{2} \\left\\lVert \\mathbf{b}_c - \\mathbf{z}^k + \\mathbf{y}_c^k \\right\\rVert_2^2 \\right) \\\\ \\end{aligned} $$\n$\\mathbf{z}$ update. Coordinates across all cities to satisfy the global (total) budget constraint and computes the consensus allocation closest to the city preferences $\\mathbf{b}_c^{k+1} + \\mathbf{y}_c^k$. (Note: The summation term sums over all cities. Hugo is being a pain and won\u0026rsquo;t compile a summation with multiple subscripts) $$ \\begin{aligned} \\mathbf{z}_c^{k+1} := \\underset{\\mathbf{z}}{\\operatorname{argmin}} \\left( g(\\mathbf{z}) + \\frac{\\rho}{2} \\sum_c \\left\\lVert \\mathbf{b}_c^{k+1} - \\mathbf{z}_c + \\mathbf{y}_c^k \\right\\rVert_2^2 \\right) \\\\ \\end{aligned} $$\n$\\mathbf{y}$ update. Accumulates constraint violations between city budget and the consensus allocation. $$ \\begin{aligned} \\mathbf{y}_c^{k+1} := \\mathbf{y}_c^{k} + \\mathbf{b}_c^{k+1} - \\mathbf{z}_c^{k+1} \\end{aligned} $$\nwhere $\\rho$ is a penalty for total budget constraint violations. This parameter controls the tradeoff between how many gross bookings we are willing to sacrifice to reduce the total budget constraint violation. Small $\\rho$ typically results in loose constraints and takes the ADMM algorithm longer to converge, and vice-versa. Uber also introduces another penalty term in the form of Hellinger distance that forces the algorithm to prefer allocations similar to a reference budget allocation. Instead of the Hellinger distance, I implement a simple quadratic penalty term scaled by $\\phi$.\nIn primal-dual optimization methods, the dual variables, also known as Lagrange multipliers, provide valuable insights into the marginal effect of constraints on the objective function. These variables represent the shadow price of a given constraint, such as a city\u0026rsquo;s budget, quantifying how much the total objective would improve if that budget were increased by one unit.\nOne of the reasons for the popularity of ADMM is that the $\\mathbf{b}$ update step can be implemented in parallel. Although this step uses variables from the previous iteration, within this step, all the per-city problems can be optimized in parallel. JAX primitives lends itself well to a high-performance implementation of ADMM. We can easily perform the per-city update in parallel with vmap, use lax.scan for loops with state, and jit for computation heavy functions.\nWith that being said, the implementation is fairly straightforward. The ADMM algorithm is implemented via ADMMAlgorithm and is stateless. It takes state and a step function and then operates on this state. For each step, the algorithm outputs a new state. We can then run the algorithm for $n$ steps and accumulate the results. Moreover, the step function is JIT compiled so the XLA compiler traces and optimizes all the function therein.\ndef z_update(budgets: Array, duals: Array, total_budget: float) -\u0026gt; Array: n_cities = len(budgets) u_k = budgets + duals adjustment = (total_budget - jnp.sum(u_k)) / n_cities consensus = adjustment + u_k return consensus def y_update(budgets: Array, consensus: Array, duals: Array) -\u0026gt; Array: return duals + budgets - consensus def city_objective(model_fn: Callable) -\u0026gt; Callable: def objective( budget: float, city: int, consensus: float, dual: float, reference: float, rho: float, phi: float, ): outcome = model_fn(jnp.array([city]), jnp.array([[budget]])).squeeze() admm_penalty = rho / 2 * (budget - consensus + dual) ** 2 business_penalty = phi * (budget - reference) ** 2 return -outcome + admm_penalty + business_penalty return objective def b_update( objective_fn: Callable, min_budget: float, max_budget: float, lr: float, n_steps: int, ) -\u0026gt; Callable: optimizer = optax.adam(learning_rate=lr) def update( city: int, budget: float, consensus: float, dual: float, reference: float, rho: float, phi: float, ): def obj_fn(x): return objective_fn(x, city, consensus, dual, reference, rho, phi) def scan_fn(carry, _): budget_val, opt_state_val = carry # Compute gradient w.r.t this budget loss_val, grad_val = jax.value_and_grad(obj_fn)(budget_val) # Update and apply updates, new_opt_state = optimizer.update( grad_val, opt_state_val, budget_val ) new_budget_unconstrained = optax.apply_updates(budget_val, updates) # Projection projected_budget = optax.projections.projection_box( new_budget_unconstrained, min_budget, max_budget ) return (projected_budget, new_opt_state), loss_val current_budget = budget opt_state = optimizer.init(budget) (final_budget, _), losses = jax.lax.scan( scan_fn, (current_budget, opt_state), None, length=n_steps ) return final_budget return update def build_admm(model_fn: Callable, config: ADMMConfig) -\u0026gt; ADMMAlgorithm: city_obj = city_objective(model_fn) b_update_ = b_update( city_obj, config.min_budget, config.max_budget, config.lr, config.n_optim_steps ) b_update_vmap = jax.vmap( b_update_, in_axes=(0, 0, 0, 0, 0, None, None), # Vectorize over cities ) def init_fn(initial_budgets: Array) -\u0026gt; ADMMState: return ADMMState( budgets=initial_budgets, consensus=initial_budgets, duals=jnp.zeros(config.n_cities), ) def step_fn(state: ADMMState) -\u0026gt; Tuple[ADMMState, ADMMInfo]: # Vectorized per-city budget updates city_indices = jnp.arange(config.n_cities) new_budgets = b_update_vmap( city_indices, state.budgets, state.consensus, state.duals, config.reference_budget, config.rho, config.phi, ) new_consensus = z_update(new_budgets, state.duals, config.total_budget) new_duals = y_update(new_budgets, new_consensus, state.duals) # Updated state new_state = ADMMState( budgets=new_budgets, consensus=new_consensus, duals=new_duals ) new_objective = jnp.sum(jax.vmap(model_fn)(city_indices, new_budgets[:, None])) primal_residual = jnp.linalg.norm(new_budgets - new_consensus) dual_residual = jnp.abs(jnp.sum(new_budgets) - config.total_budget) # Updated info new_info = ADMMInfo( objective=new_objective, primal_residual=primal_residual, dual_residual=dual_residual, converged=primal_residual \u0026lt; config.tol, ) return new_state, new_info return ADMMAlgorithm(init_fn, step_fn) Results The ADMM algorithm is initialized as follows\n# Budget constraints scaled by the maximum allowed budget total_budget = 15000.0 / max_budget min_budget_per_city = 3000.0 / max_budget max_budget_per_city = 8000.0 / max_budget tolerance = 50 / max_budget # Initial and reference budgets scaled by the maximum allowed budget init_budgets = jnp.array([1000.0, 1000.0, 1000.0]) / max_budget ref_budgets = jnp.array([5000.0, 5000.0, 5000.0]) / max_budget config = ADMMConfig( n_cities=n_cities, total_budget=total_budget, reference_budget=ref_budgets, rho=0.25, phi=0.001, lr=0.01, tol=tolerance, min_budget=min_budget_per_city, max_budget=max_budget_per_city, n_optim_steps=50, ) admm = build_admm(model, config) and the algorithm is ran for $n$ steps (or until convergence).\n@partial(jax.jit, static_argnames=[\u0026#34;algorithm\u0026#34;, \u0026#34;n_steps\u0026#34;]) def run(algorithm, initial_state, n_steps): def scan_fn(state, _): new_state, info = algorithm.step(state) return new_state, (info, new_state.budgets) final_state, (trace, budget_trace) = jax.lax.scan( scan_fn, initial_state, None, length=n_steps ) return final_state, trace, budget_trace state = admm.init(init_budgets) # Warmup JIT compilation _ = run(admm, state, 10) jax.block_until_ready(_) start = time.time() final_state, trace, budget_trace = run(admm, state, 10) jax.block_until_ready((final_state, trace, budget_trace)) duration = time.time() - start Running ADMM optimization ============================== Completed in : 0.0248 sec. Final budgets : [3964.27 4028.075 8000.0005] Final objective : 186.1875 Final primal residual : 0.038195 Final dual residual : 0.066156 Budget sum : 15992.35 (target: 15000.0) Running the algorithm for 10 iterations results in 186 gross bookings with allocations of [3964, 4028, 8000] for city 0, 1 and 2, and go over budget by almost $1,000. Visualizing the budget allocations over the ADMM iterations shows that the algorithm initially (iterations 1-3) explores different regions of the response surface for each city and then quickly identifies a near-optimal budget allocation for each city that maximizes the total gross bookings.\nWhen performing non-convex optimization with methods such as ADMM, we do not have theoretical guarantees that the optimization algorithm has converged to an optimal solution. However, we can analyze various variables and or metrics to assess convergence and the quality of the solution. For ADMM, we will analyze the dual and primal residuals as well as the objective function (gross bookings) over the ADMM iterations.\nAt a high-level, the dual residual measures the consensus allocation stability, i.e., \u0026ldquo;is the algorithm converging on a consensus allocation?\u0026rdquo;, whereas the primal residual measures the original constraint violation, i.e., \u0026ldquo;how much are we violating the total budget constraint?\u0026rdquo;. The calculation for the primal and dual residuals are computed as follows\n# Magnitude of consensus constraint violation primal_residual = jnp.abs(jnp.sum(new_budgets) - config.total_budget) # Magnitude of consensus variable variation dual_residual = config.rho * jnp.linalg.norm(new_consensus - state.consensus) and plot the results\nThe amount of gross bookings is expected to increase and then flat line. In the paper, the authors state that they look for \u0026ldquo;small\u0026rdquo; changes in the objective value from the previous ADMM iteration.\nThe dual variables for each city\u0026rsquo;s budget are expected to converge to a single common value. This convergence signifies an optimal allocation where the marginal return per dollar is equalized across all cities. For instance, if one city\u0026rsquo;s dual variable were higher than another\u0026rsquo;s it would indicate a higher marginal return for one city over another signaling that more budget should be allocated there.\nThe primal residuals, measuring how much we are violating the total budget constraint, should be \u0026ldquo;small\u0026rdquo; and exhibit stability over the ADMM iterations.\nConclusion With the simplified dataset and neural network, it appears that the ADMM implementation is computing the optimal budget allocations per-city. The implementation herein is vastly simplified compared to the system outlined by the authors. It would be interesting to add another lever dimension to each city to see the results from the ADMM algorithm.\nMoreover, Uber uses Adaptive Sparse Grids (ASG) to learn a low-dimensional representation of the response surface generated by the DL model. A B-spline model is then used to generate a differentiable \u0026ldquo;smooth\u0026rdquo; response surface which is then embedded into the ADMM routine instead of the neural network directly. As more dimensions are added, it would also be interesting to analyze the impact of such methods on the optimization result.\n","permalink":"https://gstechschulte.github.io/posts/2025-09-15-marketplace-optimization/","summary":"\u003cp\u003eUber allocates money across different regions and programs to incentivize riders and drivers to use Uber products. This incentive structure ultimately influences the market. This leads to the natural question of \u0026ldquo;how much to allocate to each city and which program\u0026rdquo; to maximize business objectives? Uber has a finite amount of money that must be allocated accordingly. Given a total budget of say, $1,000,000, how should it be divied up amongst the cities and programs?\u003c/p\u003e","title":"Reproducing Uber's Marketplace Optimization"},{"content":"Nowadays, AI is primarily associated with LLMs, where the goal is to develop autonomous agents with human-level intelligence (whatever that means). I have always found the goal of creating something with \u0026ldquo;intelligence\u0026rdquo; boring for some reason. Maybe it is because the objective is too diffuse—create something at least as smart as humans and hope it figures out \u0026lt;insert your problem here\u0026gt;. Regardless, these LLM-based agents are useful tools and should be used, but these systems are a narrow view of intelligence.\nIntelligence is often overlooked when considering it as a property of the collective and not the individual. For example, a market intelligently allocates resources without a central entity due to its unique ability to process and coordinate dispersed knowledge that no single agent could ever possess. Hayek presents an an example of this in The Use of Knowledge in Society.\nConsider a new use for tin, or a major source of tin is disrupted, making supply scarce relative to demand causing the price of tin to rise. Producers, seeing the higher price, are incentivized to increase output, and entrepreneurs are motivated to find substitutes for tin. Consumers on the other hand do not necessarily need to know why the price increased. Rather, they adjust their behavior to use tin more sparingly or to seek substitutes. Without any central order being issued, thousands of people adjust their behavior in a way that conserves the scarce resource, guiding it toward its most highly-valued uses.\nThe methods and systems that can control and fairly distribute finite resources across multiple-agents is intelligence. When you begin to think of this collective or market-view of intelligence you begin to think of the systems that\nLearn from data to provide better services. Agents can express or reveal their preferences. Algorithms provide the glue between the data and the market. This distributed view of intelligence is better suited for solving large-scale problems with many iteracting agents, for example, in power grids, transportation, and commerce systems. In my view\n$$\\text{AI} = \\text{Data} + \\text{Algorithms} + \\text{Markets}$$\nWhen you combine methods from the fields of computer science, statistics, and economics in a new way, you get systems that focus on the allocation of finite resources by understanding, optimizing, and controlling large-scale multi-agent systems. Examples of such fields are Mean Field Game Theory, Mean Field Control, and Statistical Contract Theory.\n","permalink":"https://gstechschulte.github.io/posts/2025-09-02-future-of-ai/","summary":"\u003cp\u003eNowadays, AI is primarily associated with LLMs, where the goal is to develop autonomous agents with human-level intelligence (whatever that means). I have always found the goal of creating something with \u0026ldquo;intelligence\u0026rdquo; boring for some reason. Maybe it is because the objective is too diffuse—create something at least as smart as humans and hope it figures out \u003ccode\u003e\u0026lt;insert your problem here\u0026gt;\u003c/code\u003e. Regardless, these LLM-based agents are useful tools and should be used, but these systems are a narrow view of intelligence.\u003c/p\u003e","title":"Alternative View of AI"},{"content":"Operator execution In OLAP systems, sequential scans are the primary method for query execution. The goal is two-fold: (1) minimize the amount of data fetched from the disk or a remote object store, and (2) maximize the use of hardware resources for efficient query execution.\nAndy’s (unscientific) top three execution optimization techniques:\nData parallelization (vectorization). Breaking down a query into smaller tasks and running them in parallel on different cores, threads, or nodes. Task parallelization (multi-threading). Breaking down a query into smaller independent tasks and executing them concurrently. This allows the DBMS to take full advantage of hardware capabilities and or multiple machines to improve query execution time. Code specialization (pre-compiled / JIT). Code generation for specific queries, e.g. JIT or pre-compiled parameters. which fall into three primary ways for speeding up queries:\nReduce instruction count. Use fewer instructions to do the same amount of work. Reduce cycles per instruction. Execute more CPU instructions in fewer cycles, i.e. data that we need is in L1/L2/L3 cache to maximize data locaclity. Parallelize execution. Use multiple threads to compute each query in parallel. Query execution The DBMS converts a SQL statement into a query plan. This plan is a DAG of operators where each operator instance is an invocation of an operator on a unique segment of data. A task is a sequence of one or more operator instances. A task set is the collection of executable tasks for a logical pipeline.\nSELECT A.id, B.value FROM A JOIN B USING (id) WHERE A.value \u0026lt; 99 AND B.value \u0026gt; 100; TODO: Add query plan diagram here..\nThe distinction between operators and operator instances is made because we can have multiple operators run in parallel, e.g. if table A is a billion rows, we can divide up the scan operator into 10 instances where each instance scans different files or row-groups in an object store.\nProcessing models A DBMS processing model defines how the system executes a query plan and moves data from one operator to the next. It specifies how things like the direction in which the query plan is evaluated and what kind of data is passed between the operators along the way. There are different processing models with various tradeoffs for different workloads, e.g OLTP and OLAP. The two different plan processing directions are:\nTop-to-Bottom. Starts with the root node and \u0026ldquo;pulls data up\u0026rdquo; from the children. This approach always passes tuples with function calls. Bottom-to-Top. Starts with leaf nodes and \u0026ldquo;pushes data up\u0026rdquo; to its parents. The three main processing models to consider are Iterator, Materialization, and Vectorization where each model is comprised of two types of execution paths:\nControl flow. How the DBMS invokes an operator. Data flow. How an operator sends its results. The output of an operator can be either whole tuples as in the N-ary storage model (row-oriented storage) or subsets of columns as in the decomposition storage model (column-oriented storage).\nIterator model Also known as the Volcano or Pipeline model, is a model where each query plan operator implements a Next function. On each invocation, the operator returns either a single tuple or an end of file (EOF) marker if there are no more tuples. The operator implements a loop that calls next on its children to retrieve their tuples and then process them. Each tuple is then processed up the plan as far as possible before the next tuple is retrieved.\nQuery plan operators in an iterator model are highly composible because each operator can be implemented indepedent from its parent or child so long as it implements the Next function.\nThe iterator model also allows for pipelining where the DBMS can process a tuple through as many operators as possible before having to retrieve the next tuple. The series of tasks performed for a given tuple in the query plan is called a pipeline. However, some operators may be blocked until their children emit all of their tuples, e.g. with joins, subqueries, and order bys.\nTODO: insert diagram\nThe downside with the iterator model is that we are basically calling Next for every single tuple. If there are a billion tuples, then there will be one billion Next calls.\nMaterialization model The materialization model is a specialization of the iterator model where instead of having a Next function that returns a single output, each operator processes all of its input and then emits its output all at once. The operator \u0026ldquo;materializes\u0026rdquo; its output as a single result. To avoid scanning too much input, the DBMS can push down hints, e.g. limits. The output can be either a whole tuple as in row-oriented storage or a subset of columns as in columnar storage.\nTODO: insert diagram\nEvery query plan operator implements the Output function. The operator proceesses all of the tuples from its children at once. The return result of this function is all of the tuples that operator will ever emit.\nThis approach is better for OLTP workloads because queries typically only access a small number of tuples at a time. Thus, there are fewer function calls to retrieve tuples. The materialization model is not suited for OLAP queries with large intermediate results because the DBMS may have to spill those results to disk between operators.\nVectorization model The vectorization model is a hybrid approach of the iterator and materialization model. Like the iterator model where each operator implements a Next function, but each operator emits a batch of tuples instead of a single tuple. The operators internal loop processes multiples tuples at a time. The size of the batch can very based on hardware or query properties. Each batch will contain one or more colums with each having their own null bitmaps.\nTODO: insert diagram\nThe vectorized model is considered ideal for OLAP queries because it greatly reduces the number of invocations per operator, removes tuple-navigation overhead, and allows operators to use vectorized (SIMD) instructions to process batches of tuples.\nPlan processing direction In the previous sections, the DBMS starts executing a query by invoking a Next at the root of the query plan and pulls data up from leaf operators. This pull based is how most DBMSs implement their execution engine. However, there is also push based.\nTop-to-bottom (pull) Starts with the root and pulls data up from its children. Tuples are always passed between operators using function calls (unless it\u0026rsquo;s a pipeline breaker).\nTODO: insert diagram\nBottom-to-top (push) Start with leaf nodes and push data to the parents. With push-based, you can fuse operators together within a for loop to minimize intermediate result staging.\nTODO: insert diagram\nFilter representation With the iterator model, if a tuple does not satisfy a filter, then the DBMS just invokes Next again on the child operator to get another tuple. In the vectorized model, however, a vector/batch may contain some tuples that do not satisfy filters. Therefore, we need some logical representation to identify tuples that are valid and need to be processed to include in the materialized results. There are two primary approaches to do this: selection vectors, and bitmaps.\nSelection filters Selection filters are used to store the indices or identifiers of tuples that are valid and should be considered for further processing. These filters are typically a dense sorted list of tuple identifiers that indicate which tuples in a batch are valid.\nTODO: insert diagram\nBitmaps A bitmap is created to indicate whether a tuple is valid and can be used as an input mask for further processing. The bitmap is positionally-asligned that indicates whether a tuple is valid at an offset. Some SIMD instructions natively use these bitmaps as input masks.\nTODO: insert diagram\n","permalink":"https://gstechschulte.github.io/posts/db-systems-query-processing/","summary":"\u003ch2 id=\"operator-execution\"\u003eOperator execution\u003c/h2\u003e\n\u003cp\u003eIn OLAP systems, sequential scans are the primary method for query execution. The goal is two-fold: (1) minimize the amount of data fetched from the disk or a remote object store, and (2) maximize the use of hardware resources for efficient query execution.\u003c/p\u003e\n\u003cp\u003eAndy’s (unscientific) top three execution optimization techniques:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eData parallelization (vectorization)\u003c/strong\u003e. Breaking down a query into smaller tasks and running them in parallel on different cores, threads, or nodes.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTask parallelization (multi-threading)\u003c/strong\u003e. Breaking down a query into smaller independent tasks and executing them concurrently. This allows the DBMS to take full advantage of hardware capabilities and or multiple machines to improve query execution time.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCode specialization (pre-compiled / JIT)\u003c/strong\u003e. Code generation for specific queries, e.g. JIT or pre-compiled parameters.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003ewhich fall into three primary ways for speeding up queries:\u003c/p\u003e","title":"[WIP] Database Systems - Query Execution and Processing"},{"content":"Introduction As the business landscape embraces data-driven approaches for analysis and decision-making, there is a rapid surge in the volume of data requiring storage and processing. This surge has led to the growing popularity of OLAP database systems.\nAn OLAP system workload is characterized by complex queries that require scanning over large portions of the database. In OLAP workloads, the database system is often analyzing and deriving new data from existing data collected on the OLTP side. In contrast, OLTP workloads are characterized by fast, relatively simple and repetitive queries that operate on a single entity at a time (usually involving an update or insert).\nThis blog aims to provide an overview of the popular data storage representations and encoding within OLTP and OLAP database systems. First, OLTP data storage is discussed followed by OLAP systems.\nStorage models The difference in access patterns between OLTP and OLAP means that each system can optimize for their respective data access patterns, e.g. OLAP systems can optimize for techniques like sequential scans where the system can scan through large chunks of data at a time. Due to the different access patterns, OLTP and OLAP systems have different storage models. A DBMS\u0026rsquo;s storage model specifies how it physically organizes tuples on disk and in memory. There are three primary storage models: (1) N-ary (row-oriented), (2) decomposition (columnar), and (3) hybrid. Each of these storage models is discussed below.\nStorage manager Assuming the DB is not in-memory, a DBMS stores a DB as files on disk. The DBMS storage manager is responsible for managing the DB\u0026rsquo;s files, e.g. keeping track of what has been read and written to pages as well as how much free space is in these pages. It represents these files as a collection of pages\nRow-oriented Assuming the DB is not in-memory, a DBMS stores a DB as files on disk. The DBMS storage manager is responsible for managing the DB\u0026rsquo;s files, e.g. keeping track of what has been read and written to pages as well as how much free space is in these pages. It represents these files as a collection of pages and in row-oriented storage, the DBMS stores (almost) all the attributes for a single tuple (row) contiguously in a single page in a file. This is ideal for OLTP workloads because transactions usually access individual entities and are insert-heavy.\nAt a high level, the DBMS is managing a bunch of files. Within these files there are pages to break up the data into different chunks. Then, within this page, theere are tuples, i.e. the data (rows/records) of the tables.\nPages The DBMS organizes the DB across one or more files in fixed-size blocks called pages. Pages contain different kinds of data such as tuples, indexes, and log records. Most systems will not mix these types within a page. Additionally, some systems require that pages are self-contained, i.e. all the information needed to read each page is on the page itself.\nEach page is given a unique identifier, and most DBMSs have an indirection layer that maps a page id to a file path and offset. The upper levels of the system will ask for a specific page number. Then, the storage manager will have to turn that page number into a file and an offset to find the page.\nTODO: insert diagram here\nPage storage architecture When accessing data from the DB, there needs to be a way to find the location of the page where the data resides. Heap file organization is a method to achieve this. A heap file is an unordered collection of pages where tuples are stored in random order.\nIt is easy to locate the desired page if there is only one file. Where there are multiple files, there needs to be metadata to keep track of what pages exist in multiple files and which ones have free space. The DBMS can locate a page on disk given a page id by using a linked list of pages or a page directory.\nLinked list. Header page holds pointers to a list of free pages and a list of data pages. However, if the DBMS is looking fo a specific page, it has to do a sequential scan on the data page list until it finds the page it is looking for. Page directory. The DBMS maintains special pages that tracks the location of data pages in the database files. TODO: insert diagram here\nOther methods for managing files on disk include:\nTree file organization Sequential/sorted organization (ISAM) Hash organization Page layout Every page contains a header of metadata about the pages\u0026rsquo;s:\nPage size Checksum DBMS version Transaction visibility TODO: insert diagram here\nThere are three main approaches to organizing data within a page: (1) slotted (tuple-oriented), (2) log-structured, and (3) index-organized.\nSlotted pages Slotted pages is the most common approach for row-oriented DBMSs for laying out data within pages. Slotted pages map slots to offsets. A header keeps track of the number of slots used with an offset of the starting location of the last used slot, and a slot array, which keeps track of the location of the start of each tuple. To add a tuple, the slot array will grow from the beginning to the end, and the data of the tuples will grow from end to beginning. The page is considered full when the slot array and the tuple data meet.\nTODO: insert diagram here\nTuple layout The DBMS assigns each logical tuple a unique identifier that represents its physical location in the DB. A tuple is essentially a sequence of bytes and it is the DBMSs job to interpret those bytes into attribute types and values. Typically, the layout of a tuple consists of: a header, the data, unique identifier, and optionally denormalized tuple data.\nTODO: insert diagram here\nTuple header. Each tuple is prefixed with a header that contains metadata about it such as visibility information for the DBMS\u0026rsquo;s concurrency control protocol, and a bit map for NULL values. Tuple data. The actual data for attributes. Attributes are typically stored in the order that you specify them when the table is created. Most DBMSs do not allow a tuple to exceed the size of a page. Unique identifer. Each tuple in the DB is assigned a unique identifier. Most commonly this is page_id + (offset or slot. Denormalized tuple data. A DBMS can physically denormalize (e.g. pre-join) related tuples and store them together in the same page. This makes reads faster since the DBMS only has to load one page rather than two separate pages. However, it can make updates more expensive since the DBMS needs more space for each tuple. TODO: insert diagram here\nSummary A DB is organized as files on disk. These files are composed of pages. There are multiple ways to organize data within these pages (e.g. slotted). The data we care about is stored as tuples within the pages.\nColumn-oriented OLAP workloads typically require scanning over large portions of a table using a few columns (relative to the total number of columns) to analyze data. File sizes in OLAP workloads are relatively large (usually 100MB+) and are primarily read-only. Thus, the DBMS should store a single attribute for all tuples contiguously in a block of data in memory for column-oriented storage. Although the file sizes are \u0026ldquo;large\u0026rdquo;, the DBMS may still organize data into groups with the file.\nTODO: insert diagram here\nMoreover, for column-oriented storage, all variable length data needs to be converted to fixed length so that simple arithmetic can be used to jump to an offset to find a tuple. This can be done by using dictionary compression. The DBMS stores the dictionary in the header of the page and stores the actual data in the body of the page. The DBMS can then use the dictionary to reconstruct the data. To identify the tuples (data in page), there are two primary options:\nFixed length offsets. Each value is the same length for an attribute. The DBMS can reach locations of other attributes of the same tuple by inferring from the length of the value and the current offset. TODO: insert diagram here\nEmbedded tuple ids. Each value is stored with its tuple ID in a column. TODO: insert diagram here\nFurthermore, OLAP queries rarely select a single column, i.e. the projection and predicates often involve different columns. For example:\nSELECT product_id, AVG(price) FROM sales WHERE time \u0026gt; \u0026#39;2024-01-01\u0026#39; GROUP BY product_id; Thus, a columnar scheme that still stores attributes separately but keeps the data for each tuple physically close to each other is desired.\nHybrid (PAX) Partition across attributes (PAX) is a hybrid storage model that horizontally partitions data into row groups, and then vertically partitions their attributes into column chunks—All within a DB page. This is what Parquet and Orc use. The goal with PAX is to get the benefit of faster processing on columnar storage while retaining the spatial locality benefits of row storage.\nTODO: insert diagram here\nIn most PAX model implementations such as Apache Parquet and Orc, the global metadata is the footer of the file. This is because most distributed file systems and OLAP workloads are very append-friendly and may not support in-place updates efficiently.\nSummary TODO: add content\nFormat design decisions Modern row-oriented and columnar systems need to make certain design decisions when designing and engineering file formats. Here, the major design decisions behind file formats for OLAP workloads are discussed:\nFile metadata Format layout Type system Encoding schemes Block compression Filters Nested data File metadata Files are self-contained to increase portability, i.e. they contain all the relevant information to interpret their contents without external data dependencies. Each file maintains global metadata (usually in the footer) abouts it contents such as: table schema, row group offsets, and tuple/zone counts.\nThis is opposite of, for example, Postgres. In Postgres, you have a bunch of files that keep track of the catalog (schema, tables, types, etc.). Then, you have pages for the actual data. In order for you to understand what is in the data pages, you need to read the catalog first.\nFormat layout The most common file formats like Parquet and Orc use the PAX storage model that splits data into row groups that contain one or more column chunks. However, the size of row groups varies per implementation and makes compute/memory trade offs.\nParquet. Number of tuples (e.g. 1 million) Orc. Physical storage size (e.g. 250MB) Arrow. Number of tuples (e.g. 1020 * 1024) TODO: insert PAX or Parquet diagram here\nType system The type system defines the data types that the file format supports. A DB system typically has both physical and logical types.\nPhysical type. Low-level byte representation, e.g. IEEE-754, that focuses on the actual storage representation.\nLogical type. Auxiliary types are higher-level representation that focus on the semantic meaning, e.g. DATE, INT64, VARCHAR, that are then mapped to physical types.\nEncoding schemes Encoding schemes specify how the file format stores the bytes for contiguous data. There are several encoding schemes (given below), and one can apply multiple encoding schemes on top of each other to further improve compression).\nDictionary encoding. The most common. It replaces frequent values with smaller fixed-length codes and then maintains a mapping (dictionary) from the codes to the original values. Codes could either be positions (using a hash table) or byte offsets into a dictionary. Additionally, values in the dictionary can be sorted and compressed further.\nTODO: insert dictionary encoding diagram here\nRun-length encoding (RLE). Bitpacking. Delta encoding. Frame-of-reference (FOR). Compression Compression compresses data using a general-purpose algorithm (e.g. LZO, LZ4, Snappy, Zstd) and saves storage space, but can introduce computational overhead (compress versus decompress) and data opacity for the execution engine. Data opacity means if you run data through Snappy or Zstd, the DB system does not know what those bytes mean and you cannot go and jump to arbitrary offsets to find the data you are looking for. You need to decompress the whole block to interpret the data.\nCompression made more sense in the 2000s and 2010s because the main bottleneck was disk and network, so we were willing to pay CPU costs. But now, the CPU is actually one of the slower components and we have cheap object stores.\nFilters First, the difference between a filter and an index. An index tells you were data is, whereas a filter tells you if something exists. There are several types of filters in DB systems to boost search performance.\nZone maps. Maintain min/max values per column at the file and row group level. Parquet and Orc store zone maps in the header of each row group.\nBloom filters. A probabilistic data structure (can get false positives but never false negatives) that tracks the existence of values for each column in a row group.\nNested data Real-world data sets often contain semi-structured objects, e.g. JSON and Protobufs. In order to store semi-structured data as regular columns, most modern formats add additional fields that make querying the data easier and faster. There are two main approaches to storing semi-structured data as columns: (1) record shredding, and (2) length + presence encoding.\nRecord shredding. When storing semi-structured data as a \u0026ldquo;blob\u0026rdquo; in a column, every single time you need to parse the blob, you need to run JSON functions to extract the structure from it. Instead, split it up so that every level in the path is treated as a separate column. Now we can rip through a column for a given field in the JSON. There is always a schema! It does not make sense to have applications inserting random documents into a table!\nTo achieve this, two additional fields are stored:\nRepetition level. At what repeated field in the field\u0026rsquo;s path the value has repeated, i.e. for each path store it as a separate column and record how many steps deep we are into a given document for that hierarchy. Essentially, we are storing paths as separate columns with additional metadata about the paths. Definition level. Specifies how many columns in the path of the field that could be undefined are actually present. TODO: insert shreding diagram here\nLength + presence encoding.\nSummary ","permalink":"https://gstechschulte.github.io/posts/db-systems-file-formats/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eAs the business landscape embraces data-driven approaches for analysis and decision-making, there is a rapid surge in the volume of data requiring storage and processing. This surge has led to the growing popularity of OLAP database systems.\u003c/p\u003e\n\u003cp\u003eAn OLAP system workload is characterized by complex queries that require scanning over large portions of the database. In OLAP workloads, the database system is often analyzing and deriving new data from existing data collected on the OLTP side. In contrast, OLTP workloads are characterized by fast, relatively simple and repetitive queries that operate on a single entity at a time (usually involving an update or insert).\u003c/p\u003e","title":"[WIP] Database Systems - Storage"},{"content":"When interfacing with libraries written in C/C++ from Rust, it may require writing native callbacks to provide functionality or logic to the library. A C Callback is a function pointer that is passed as an argument to another function, allowing that function to \u0026ldquo;call back\u0026rdquo; and execute the passed function at runtime.\nWhen interfacing with Python from Rust, there may be scenarios where the Rust code also needs to be able to call a Python function. Rust\u0026rsquo;s foreign function interface (FFI) and pyo3 crate in fact lets you do this. However, calling Python from Rust involves invoking the Python interpreter, which can reduce performance. If one of the goals for using Rust is to improve the performance of your application or library, this overhead might be undesirable. To avoid invoking the Python interpreter, you can use Numba. Numba allows you to create a C callback, pass this function pointer to Rust, and perform the callback without incurring the overhead associated with Python.\nThis post will briefly explain how to create C callbacks using Numba, and how to pass and call them from within Rust.\nCreating C Callbacks with Numba To create a C callback of a Python function using Numba, the cfunc function is used. Alternatively, one could use the @cfunc decorator. Regardless of the technique, passing a signature of the Python function is required as it determines the visible signature of the C callback. The C function object exposes the address of the compiled C callback as the address attribute, so that you can pass it to a foreign library. The object is also callable from Python.\nimport ctypes import numpy as np from numba import cfunc, carray, types from callback import initialize # Define the C signature c_sig = types.void(types.CPointer(types.double), types.CPointer(types.double), types.intc, types.intc) @cfunc(c_sig) def my_callback(in_, out, m, n): in_array = carray(in_, (m, n)) out_array = carray(out, (m, n)) for i in range(m): for j in range(n): out_array[i, j] = 2 * in_array[i, j] # Prepare input data m, n = 3, 2 input_array = np.array([1., 2., 3., 4., 5., 6.]) output_array = np.zeros(m * n, dtype=np.float64) # Get pointers to the data input_ptr = input_array.ctypes.data_as(ctypes.POINTER(ctypes.c_double)) output_ptr = output_array.ctypes.data_as(ctypes.POINTER(ctypes.c_double)) # Call the function using ctypes my_callback.ctypes(input_ptr, output_ptr, ctypes.c_int(m), ctypes.c_int(n)) print(f\u0026#34;Input array : {input_array}\u0026#34;) print(f\u0026#34;Output array: {output_array}\u0026#34;) Input array : [1. 2. 3. 4. 5. 6.] Output array: [2. 4. 6. 8. 10. 12.] Even though the code is being executed within Python, my_callback does not invoke the Python interpreter each time the function is called—making the code much faster.\n\u0026ldquo;Call back\u0026rdquo; from Rust What if part of our library is written in Rust and needs to be able to call this Python function? Performance is critical so Numba is used to create a C callback from the original Python function. This function pointer will then be passed to Rust where the \u0026ldquo;callback\u0026rdquo; is performed, i.e., my_callback is called from within Rust without ever invoking the Python interpreter.\nThe boundary between Rust and the C callback can be crossed using Rust\u0026rsquo;s FFI. FFI lets Rust code call functions written in other programming languages (typically C/C++), and is ultimately all about accessing bytes that originate somewhere outside the Rust code. For that, Rust provides two primary building blocks:\nSymbols. Names assigned to particular addresses in a given segment of your binary that allow you to share memory between the external origin and your Rust code. Calling convention. Provides a common understanding of how to call functions stored in such shared memory. Rust\u0026rsquo;s extern keyword is used to link with external functions and variables defined outside of the Rust environment libraries. This is achieved by declaring external blocks where these functions and variables are specified. In our example, we need to define an external block with the function signature (calling convention) of my_callback. The call to this function is then wrapped in an unsafe block due to the potential risks associated with calling code that originates outside of Rust.\nBelow, the function signature of my_callback is defined in Rust as the type alias Callback. The alias represents a function pointer type that can be used to call C functions from Rust. The ffi and os::raw modules provide type definitions required for C-compatible data types.\nuse std::{ ffi::{c_double, c_void}, os::raw::c_int, }; // Declare calling convention of `my_callback` using C types from the `ffi` module type Callback = unsafe extern \u0026#34;C\u0026#34; fn(in_: *const c_double, out: *mut c_double, m: c_int, n: c_int); Notice how the function signature of Callback matches that of my_callback. Now, we need a way of passing the pointer of my_callback to Rust. To enable this interface, we will use pyo3 to create a Python extension module. Details of how to use pyo3 will not be given here. Rather, we will focus on the declaration of my_python_fn. Here, the fn_ptr parameter is cast to the Callback type using std::mem::transmute. This casting is an unsafe operation that converts fn_ptr from a usize type to the function pointerCallback type enabling the callback.\nNow, we can perform the callback by passing the appropriate data to my_python_fn. The callback also happens in an unsafe block as it involves dereferencing raw pointers. Lastly, the output vector is returned as a Python object.\n#[pyfunction] fn compute_from_rust(fn_ptr: usize) -\u0026gt; PyResult\u0026lt;(PyObject)\u0026gt; { // Cast `fn_ptr` from usize to the `Callback` type (aka the function pointer) let my_python_fn: Callback = unsafe { std::mem::transmute(fn_ptr as *const c_void) }; // Create data to be passed to `my_python_fn` let m: c_int = 3; let n: c_int = 2; let in_ = vec![1., 2., 3., 4., 5., 6.]; let mut out = vec![0.0; (m * n) as usize]; // Perform callback within an unsafe block unsafe { (my_python_fn)(in_.as_ptr(), out.as_mut_ptr(), m, n); } // Return the data to Python Python::with_gil(|py| Ok(PyList::new_bound(py, \u0026amp;out).into())) } #[pymodule] fn callback(m: \u0026amp;Bound\u0026lt;\u0026#39;_, PyModule\u0026gt;) -\u0026gt; PyResult\u0026lt;()\u0026gt; { m.add_function(wrap_pyfunction!(compute_from_rust, m)?)?; Ok(()) } We can now compile the Rust code, and import and call the extension module in Python.\nimport ctypes import numpy as np from numba import cfunc, carray, types from callback import compute_from_rust c_sig = types.void(types.CPointer(types.double), types.CPointer(types.double), types.intc, types.intc) @cfunc(c_sig) def my_callback(in_, out, m, n): in_array = carray(in_, (m, n)) out_array = carray(out, (m, n)) for i in range(m): for j in range(n): out_array[i, j] = 2 * in_array[i, j] # Call `my_callback` within Rust using FFI result = compute_from_rust(my_callback.address) print(f\u0026#34;Rust output: {result}\u0026#34;) Rust output: [2.0, 4.0, 6.0, 8.0, 10.0, 12.0] Voilà, the result is the same as above. The code in this blog post can be found here.\n","permalink":"https://gstechschulte.github.io/posts/2024-11-30-c-callbacks/","summary":"\u003cp\u003eWhen interfacing with libraries written in C/C++ from Rust, it may require writing native callbacks to provide functionality or logic to the library. A C Callback is a function pointer that is passed as an argument to another function, allowing that function to \u0026ldquo;call back\u0026rdquo; and execute the passed function at runtime.\u003c/p\u003e\n\u003cp\u003eWhen interfacing with Python from Rust, there may be scenarios where the Rust code also needs to be able to call a Python function. Rust\u0026rsquo;s foreign function interface (FFI) and \u003ccode\u003epyo3\u003c/code\u003e crate in fact lets you do this. However, calling Python from Rust involves invoking the Python interpreter, which can reduce performance. If one of the goals for using Rust is to improve the performance of your application or library, this overhead might be undesirable. To avoid invoking the Python interpreter, you can use Numba. Numba allows you to create a C callback, pass this function pointer to Rust, and perform the callback without incurring the overhead associated with Python.\u003c/p\u003e","title":"Creating C Callbacks with Numba and Calling Them From Rust"},{"content":"A blog series consisting of my notes on the Carnegie Mellon University (CMU) Introduction and Advanced Database Systems Lectures by Andy Pavlo and Jignesh Patel. The primary goal of this series is to: (1) consolidate my notes, and (2) act as a reference guide for my future self. Perhaps some readers may extract some value, but I would highly recommend watching the lectures for yourself.\nThe series will cover:\nDatabase storage Indexes Join algorithms Query execution and processing Query optimization Query scheduling and coordination Concurrency control OLAP database management system components The series will primarily focus on the components of OLAP database management systems (DBMS). A recent trend of the last decade is the breakout of OLAP DBMS components into standalone services and libraries for:\nSystem catalogs Intermediate representations Query optimizers File format Execution engines Given these components, the general architecture of an OLAP system is the following:\nWhere the components perform the following functions (generally speaking):\nFront end. Takes in the user query and parses it into an intermediate representation using a language parser. Planner. Takes in the intermediate representation from the front-end uses the binder, rewriter, and optimizer to generate a query plan. Scheduler. Takes in the query plan, organizes worker nodes, and schedules the execution by breaking the plan up into fragments. Execution engine. Takes in plan fragments and executes them. I/O service. Takes in block requests from the execution engine and returns the data after retrieving it from the object store. Catalog. Keeps track of data locations and metadata of the data for the DBMS and communicates with all components of the DBMS except the front-end. ","permalink":"https://gstechschulte.github.io/posts/db-systems-blog-series-overview/","summary":"\u003cp\u003eA blog series consisting of my notes on the Carnegie Mellon University (CMU) Introduction and Advanced Database Systems Lectures by Andy Pavlo and Jignesh Patel. The primary goal of this series is to: (1) consolidate my notes, and (2) act as a reference guide for my future self. Perhaps some readers may extract some value, but I would highly recommend watching the lectures for yourself.\u003c/p\u003e\n\u003cp\u003eThe series will cover:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDatabase storage\u003c/li\u003e\n\u003cli\u003eIndexes\u003c/li\u003e\n\u003cli\u003eJoin algorithms\u003c/li\u003e\n\u003cli\u003eQuery execution and processing\u003c/li\u003e\n\u003cli\u003eQuery optimization\u003c/li\u003e\n\u003cli\u003eQuery scheduling and coordination\u003c/li\u003e\n\u003cli\u003eConcurrency control\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"olap-database-management-system-components\"\u003eOLAP database management system components\u003c/h2\u003e\n\u003cp\u003eThe series will primarily focus on the components of OLAP database management systems (DBMS). A recent trend of the last decade is the breakout of OLAP DBMS components into standalone services and libraries for:\u003c/p\u003e","title":"Database Systems - Series Overview"},{"content":" Hierarchical regression, also known as multilevel modeling, is a powerful modeling technique that allows one to analyze data with a nested structure. This approach is particularly useful when dealing with data that has natural groupings, such as students within schools, patients within hospitals, or in the example below, product configurations within manufacturing processes. One of the key advantages of hierarchical regression lies in its ability to handle missing data in groups, i.e., when one group may not share the same covariates as another group or some groups may contain missong observations.\nSimulated Example: Manufacturing Process Analysis To illustrate the ability of hierarchical regression in handling missing data, let\u0026rsquo;s consider a simulated example from a manufacturing context. We\u0026rsquo;ll analyze how different machine process parameters impact production speed across two product groups. In this simulation, we generate data for two product groups with different dependencies on machine process parameters:\nProduct group 1 depends on feed speed and pull acceleration. Product group 2 depends on feed speed, pull acceleration, and cutting wait time. The cutting wait time for product group 1 is set to NaN, simulating a scenario where this parameter is not applicable or not measured for this group.\nimport matplotlib.pyplot as plt import numpy as np import numpyro import pandas as pd import seaborn as sns import jax.numpy as jnp import numpyro.distributions as dist from jax import random from numpyro.infer import Predictive n_samples = 500 # Generate machine process parameters for both groups feed_speed = np.random.uniform(1, 10, n_samples) pull_acceleration = np.random.uniform(0.5, 5, n_samples) cutting_wait_time = np.random.uniform(0.1, 2, n_samples) # Create product groups product_group = np.array([\u0026#39;Product_Group_1\u0026#39;] * (n_samples // 2) + [\u0026#39;Product_Group_2\u0026#39;] * (n_samples // 2)) # Calculate production speed production_speed = np.zeros(n_samples) # Product Group 1: depends on feed speed and pull acceleration (Cutting Wait Time is NaN) mask_pg1 = product_group == \u0026#39;Product_Group_1\u0026#39; # Increased coefficients for Product Group 1 to ensure higher production speed production_speed[mask_pg1] = (feed_speed[mask_pg1] * 2.5 + pull_acceleration[mask_pg1] * 3 + np.random.normal(0, 0.5, sum(mask_pg1))) # Product Group 2: depends on all three parameters mask_pg2 = product_group == \u0026#39;Product_Group_2\u0026#39; production_speed[mask_pg2] = (feed_speed[mask_pg2] * 1 + pull_acceleration[mask_pg2] * 0.5 + cutting_wait_time[mask_pg2] * 3 + np.random.normal(0, 0.5, sum(mask_pg2))) # Create DataFrame with NaN for Cutting Wait Time in Product Group 1 simulated_dataset = pd.DataFrame({ \u0026#39;Product_Group\u0026#39;: product_group, \u0026#39;Feed_Speed\u0026#39;: feed_speed, \u0026#39;Pull_Acceleration\u0026#39;: pull_acceleration, \u0026#39;Cutting_Wait_Time\u0026#39;: [np.nan if x == \u0026#39;Product_Group_1\u0026#39; else cutting_wait_time[i] for i,x in enumerate(product_group)], \u0026#39;Production_Speed\u0026#39;: production_speed }) plt.figure(figsize=(7, 3)) sns.histplot( data=simulated_dataset, x=\u0026#34;Production_Speed\u0026#34;, hue=\u0026#34;Product_Group\u0026#34;, binwidth=1 ) plt.xlabel(\u0026#34;Production Speed\u0026#34;) plt.title(\u0026#34;Simulated Data Distribution\u0026#34;); # sns.pairplot(simulated_dataset.iloc[:, 0:], hue=\u0026#34;Product_Group\u0026#34;, corner=True); simulated_dataset[\u0026#34;Product_Group\u0026#34;] = (simulated_dataset[\u0026#34;Product_Group\u0026#34;] .map({ \u0026#34;Product_Group_1\u0026#34;: 0, \u0026#34;Product_Group_2\u0026#34;: 1 }) ) Parameter Masking for Missing Data To handle the missing data in our hierarchical model, I will use a technique called parameter masking. This approach allows us to effectively \u0026ldquo;turn off\u0026rdquo; certain parameters for specific groups where they are not applicable when data X is passed to the model. Key aspects of this approach:\nWe use an indicators array to specify which parameters are relevant for each product group. The input data X is masked to replace NaNs with zeros, and then multiplied by the indicators. The weights w are also masked using the indicators, ensuring that irrelevant parameters don\u0026rsquo;t contribute to the predictions. parameter_mask = jnp.array([ [1, 1, 0], [1, 1, 1] ]) config_idx = jnp.array(simulated_dataset[\u0026#34;Product_Group\u0026#34;].values) X = jnp.array(simulated_dataset.iloc[:, 1:4].values) y = jnp.array(simulated_dataset[\u0026#34;Production_Speed\u0026#34;].values) def model(indicators, config_idx, X, y=None): n_configs, n_params = indicators.shape # Create a masked version of X where NaNs are treated as zeros based on the indicators X_masked = jnp.where(jnp.isnan(X), 0.0, X) * indicators[config_idx] # Group-specific effects with numpyro.plate(\u0026#34;config_i\u0026#34;, n_configs): alpha = numpyro.sample(\u0026#34;alpha\u0026#34;, dist.Normal(0., 5.)) with numpyro.plate(\u0026#34;param_i\u0026#34;, n_params): w = numpyro.sample(\u0026#34;w\u0026#34;, dist.Normal(0., 5.)) # Compute the weighted sum of features using indicators to zero-out unused parameters w_masked = jnp.multiply(w.T, indicators) eq = alpha[config_idx] + jnp.sum(jnp.multiply(X_masked, w_masked[config_idx]), axis=-1) mu = numpyro.deterministic(\u0026#34;mu\u0026#34;, eq) scale = numpyro.sample(\u0026#34;scale\u0026#34;, dist.HalfNormal(5.)) with numpyro.plate(\u0026#34;data\u0026#34;, X.shape[0]): numpyro.sample(\u0026#34;obs\u0026#34;, dist.Normal(mu, scale), obs=y) numpyro.render_model( model, model_args=( parameter_mask, config_idx, X, y ), render_params=True ) rng_key = random.PRNGKey(seed=42) rng_key, rng_subkey = random.split(key=rng_key) kernel = numpyro.infer.NUTS(model) mcmc = numpyro.infer.MCMC( kernel, num_warmup=200, num_samples=300, num_chains=4, chain_method=\u0026#34;vectorized\u0026#34; ) mcmc.run( rng_subkey, parameter_mask, config_idx, X, y ) sample: 100%|██████████| 500/500 [00:03\u0026lt;00:00, 148.27it/s] Model Inference and Results After running MCMC inference on our model, we can examine the parameter estimates and make predictions.\n# Parameter estimates for w[2, 0] are actually 0.0 once data is passed # through the program mcmc.print_summary() mean std median 5.0% 95.0% n_eff r_hat alpha[0] -0.03 0.11 -0.04 -0.20 0.15 598.37 1.00 alpha[1] 0.05 0.12 0.05 -0.18 0.22 797.79 1.01 scale 0.52 0.02 0.52 0.50 0.55 1058.94 1.00 w[0,0] 2.51 0.01 2.51 2.49 2.53 837.47 1.00 w[0,1] 1.00 0.01 1.00 0.98 1.02 1350.56 1.00 w[1,0] 2.98 0.02 2.98 2.94 3.02 740.98 1.00 w[1,1] 0.49 0.03 0.49 0.45 0.53 1002.80 1.00 w[2,0] 0.11 5.30 -0.03 -7.42 9.98 893.63 1.00 w[2,1] 3.05 0.07 3.05 2.93 3.15 891.06 1.01 Number of divergences: 0 This summary shows us the estimated parameters, including group-specific intercepts (alpha) and weights for each parameter. Importantly, we\u0026rsquo;ll see that the weight for the third parameter (cutting wait time) in Product group 1 is effectively zero as expected. We can now use the model to make predictions.\nrng_key, rng_subkey = random.split(key=rng_key) samples = mcmc.get_samples() predictive = Predictive(model, posterior_samples=samples) pps = predictive( rng_subkey, parameter_mask, config_idx, X, None ) simulated_dataset[\u0026#34;preds\u0026#34;] = pps[\u0026#34;obs\u0026#34;].mean(axis=0) plt.figure(figsize=(7, 3)) sns.histplot(data=simulated_dataset, x=\u0026#34;Production_Speed\u0026#34;, binwidth=1, label=\u0026#34;obs\u0026#34;) sns.histplot(data=simulated_dataset, x=\u0026#34;preds\u0026#34;, binwidth=1, label=\u0026#34;pps\u0026#34;) plt.xlabel(\u0026#34;Production Speed\u0026#34;) plt.title(\u0026#34;Posterior Predictive Distribution\u0026#34;); This plot compares the observed production speeds with the model\u0026rsquo;s predictions, allowing us to assess how well our model captures the underlying patterns in the data.\nconfig_idx_new = jnp.array([0, 1], dtype=jnp.int32) X_test = jnp.array([ [3.0, 2.0, 0.0], [3.0, 2.0, 2.0] ]) rng_key, rng_subkey = random.split(key=rng_key) pps_new = predictive( rng_subkey, parameter_mask, config_idx_new, X_test, None ) pps_new[\u0026#34;obs\u0026#34;].mean(axis=0) Array([13.455311, 10.143251], dtype=float32) config_idx_new = jnp.array([0, 1], dtype=jnp.int32) X_test = jnp.array([ [6.0, 2.0, 2.0], [6.0, 2.0, 2.0] ]) rng_key, rng_subkey = random.split(key=rng_key) pps_new = predictive( rng_subkey, parameter_mask, config_idx_new, X_test, None ) pps_new[\u0026#34;obs\u0026#34;].mean(axis=0) Array([20.98238 , 13.135274], dtype=float32) Conclusion Hierarchical regression, combined with parameter masking, provides a powerful framework for analyzing grouped data with missing values. This approach allows us to:\nAccount for group-specific variations in the relationships between predictors and outcomes. Handle missing data without requiring imputation or discarding incomplete cases. Make predictions for new data, even when some predictors are not applicable to certain groups. ","permalink":"https://gstechschulte.github.io/posts/hierarchical-regression-missing-data/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003cp\u003eHierarchical regression, also known as multilevel modeling, is a powerful modeling technique that allows one to analyze data with a nested structure. This approach is particularly useful when dealing with data that has natural groupings, such as students within schools, patients within hospitals, or in the example below, product configurations within manufacturing processes. One of the key advantages of hierarchical regression lies in its ability to handle missing data in groups, i.e., when one group may not share the same covariates as another group or some groups may contain missong observations.\u003c/p\u003e","title":"Hierarchical Regression With Missing Data"},{"content":"Introduction In some scenarios, one needs to enrich an event stream with data from another source that holds \u0026ldquo;state\u0026rdquo;. This state provides additional context to the event stream.\nFor example, in manufacturing, a machine may use a set of machine process parameters (pressure, speed, force, etc.) when producing an item. The process parameters represent the \u0026ldquo;state\u0026rdquo; of the machine at production time $t$. However, the software services that publishes messages on what is being produced and the machine process parameters currently used are separate. Furthermore, to avoid the duplication of data, the service that publishes process parameters only publishes a message when there is a change in state, e.g when an operator changes one of process parameters.\nData simulation Lets simulate some data with TimescaleDB.\nCREATE TABLE production ( time timestamptz NOT NULL, product_id INT NOT NULL ); INSERT INTO production SELECT *, 1 as product_id FROM generate_series(\u0026#39;2024-01-01 05:00:00\u0026#39;, \u0026#39;2024-01-01 05:05:00\u0026#39;, INTERVAL \u0026#39;1m\u0026#39;) AS time UNION ALL SELECT *, 2 as product_id FROM generate_series(\u0026#39;2024-01-01 05:10:00\u0026#39;, \u0026#39;2024-01-01 05:13:00\u0026#39;, INTERVAL \u0026#39;1m\u0026#39;) AS time SELECT * FROM production; time product_id 2024-01-01 05:00:00+00 1 2024-01-01 05:01:00+00 1 2024-01-01 05:02:00+00 1 2024-01-01 05:03:00+00 1 2024-01-01 05:04:00+00 1 2024-01-01 05:05:00+00 1 2024-01-01 05:10:00+00 2 2024-01-01 05:11:00+00 2 2024-01-01 05:12:00+00 2 2024-01-01 05:13:00+00 2 CREATE TABLE machine ( time timestamptz NOT NULL, speed NUMERIC NOT NULL ); INSERT INTO machine (time, speed) VALUES (\u0026#39;2024-01-01 02:00:00\u0026#39;::timestamptz, 40.0), (\u0026#39;2024-01-01 05:07:00\u0026#39;::timestamptz, 60.0); SELECT * FROM machine; time speed 2024-01-01 02:00:00+00 40.0 2024-01-01 05:07:00+00 60.0 Postgres stateful join We would like to enrich the production data with the process parameters from machine. Thus, we need to join the most recent process parameter with a production event where a production event most occur greater than or equal to the change in machine state.\nThis enrichment can be achieved with a stateful join using PostgreSQL\u0026rsquo;s LATERAL JOIN expression. The LATERAL keyword allows a subquery or derived table to reference columns from tables listed before it in the FROM clause. A LATERAL join is like a for loop: for each row returned by the tables listed before LATERAL in the FROM clause, PostgreSQL will evaluate the LATERAL subquery using the current row\u0026rsquo;s values. The resulting rows from the LATERAL subquery are joined to the current row, typically using a JOIN condition of ON TRUE since the real join conditions are inside the LATERAL subquery. This process is then repeated for each row or set of rows from the tables preceding LATERAL.\nSELECT * FROM production prod LEFT JOIN LATERAL ( SELECT time as change_time, speed FROM machine WHERE time \u0026lt;= prod.time ORDER BY time DESC LIMIT 1 ) ON TRUE; time product_id change_time speed 2024-01-01 05:00:00.000000 +00:00 1 2024-01-01 02:00:00.000000 +00:00 40 2024-01-01 05:01:00.000000 +00:00 1 2024-01-01 02:00:00.000000 +00:00 40 2024-01-01 05:02:00.000000 +00:00 1 2024-01-01 02:00:00.000000 +00:00 40 2024-01-01 05:03:00.000000 +00:00 1 2024-01-01 02:00:00.000000 +00:00 40 2024-01-01 05:04:00.000000 +00:00 1 2024-01-01 02:00:00.000000 +00:00 40 2024-01-01 05:05:00.000000 +00:00 1 2024-01-01 02:00:00.000000 +00:00 40 2024-01-01 05:10:00.000000 +00:00 2 2024-01-01 05:07:00.000000 +00:00 60 2024-01-01 05:11:00.000000 +00:00 2 2024-01-01 05:07:00.000000 +00:00 60 2024-01-01 05:12:00.000000 +00:00 2 2024-01-01 05:07:00.000000 +00:00 60 2024-01-01 05:13:00.000000 +00:00 2 2024-01-01 05:07:00.000000 +00:00 60 In our hypothetical manufacturing example, the machine process parameters changed when product 2 began producing. Before this enrichment process, it wouldn\u0026rsquo;t have been known why the time to produce product 2 was faster. However, the LATERAL JOIN allows us to see that the speed increased from 40 to 60.\n","permalink":"https://gstechschulte.github.io/posts/sql-stateful-joins/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eIn some scenarios, one needs to enrich an event stream with data from another source that holds \u0026ldquo;state\u0026rdquo;. This state provides additional context to the event stream.\u003c/p\u003e\n\u003cp\u003eFor example, in manufacturing, a machine may use a set of machine process parameters (pressure, speed, force, etc.) when producing an item. The process parameters represent the \u0026ldquo;state\u0026rdquo; of the machine at production time $t$. However, the software services that publishes messages on what is being produced and the machine process parameters currently used are separate. Furthermore, to avoid the duplication of data, the service that publishes process parameters only publishes a message when there is a change in state, e.g when an operator changes one of process parameters.\u003c/p\u003e","title":"Stateful Joins in SQL"},{"content":" Alternative sampling backends This blog post is a copy of the alternative samplers documentation I wrote for Bambi. The original post can be found here.\nIn Bambi, the sampler used is automatically selected given the type of variables used in the model. For inference, Bambi supports both MCMC and variational inference. By default, Bambi uses PyMC\u0026rsquo;s implementation of the adaptive Hamiltonian Monte Carlo (HMC) algorithm for sampling. Also known as the No-U-Turn Sampler (NUTS). This sampler is a good choice for many models. However, it is not the only sampling method, nor is PyMC the only library implementing NUTS.\nTo this extent, Bambi supports multiple backends for MCMC sampling such as NumPyro and Blackjax. This notebook will cover how to use such alternatives in Bambi.\nNote: Bambi utilizes bayeux to access a variety of sampling backends. Thus, you will need to install the optional dependencies in the Bambi pyproject.toml file to use these backends.\nimport arviz as az import bambi as bmb import bayeux as bx import numpy as np import pandas as pd WARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions. bayeux Bambi leverages bayeux to access different sampling backends. In short, bayeux lets you write a probabilistic model in JAX and immediately have access to state-of-the-art inference methods.\nSince the underlying Bambi model is a PyMC model, this PyMC model can be \u0026ldquo;given\u0026rdquo; to bayeux. Then, we can choose from a variety of MCMC methods to perform inference.\nTo demonstrate the available backends, we will fist simulate data and build a model.\nnum_samples = 100 num_features = 1 noise_std = 1.0 random_seed = 42 np.random.seed(random_seed) coefficients = np.random.randn(num_features) X = np.random.randn(num_samples, num_features) error = np.random.normal(scale=noise_std, size=num_samples) y = X @ coefficients + error data = pd.DataFrame({\u0026#34;y\u0026#34;: y, \u0026#34;x\u0026#34;: X.flatten()}) model = bmb.Model(\u0026#34;y ~ x\u0026#34;, data) model.build() We can call bmb.inference_methods.names that returns a nested dictionary of the backends and list of inference methods.\nmethods = bmb.inference_methods.names methods {'pymc': {'mcmc': ['mcmc'], 'vi': ['vi']}, 'bayeux': {'mcmc': ['tfp_hmc', 'tfp_nuts', 'tfp_snaper_hmc', 'blackjax_hmc', 'blackjax_chees_hmc', 'blackjax_meads_hmc', 'blackjax_nuts', 'blackjax_hmc_pathfinder', 'blackjax_nuts_pathfinder', 'flowmc_rqspline_hmc', 'flowmc_rqspline_mala', 'flowmc_realnvp_hmc', 'flowmc_realnvp_mala', 'numpyro_hmc', 'numpyro_nuts']}} With the PyMC backend, we have access to their implementation of the NUTS sampler and mean-field variational inference.\nmethods[\u0026#34;pymc\u0026#34;] {'mcmc': ['mcmc'], 'vi': ['vi']} bayeux lets us have access to Tensorflow probability, Blackjax, FlowMC, and NumPyro backends.\nmethods[\u0026#34;bayeux\u0026#34;] {'mcmc': ['tfp_hmc', 'tfp_nuts', 'tfp_snaper_hmc', 'blackjax_hmc', 'blackjax_chees_hmc', 'blackjax_meads_hmc', 'blackjax_nuts', 'blackjax_hmc_pathfinder', 'blackjax_nuts_pathfinder', 'flowmc_rqspline_hmc', 'flowmc_rqspline_mala', 'flowmc_realnvp_hmc', 'flowmc_realnvp_mala', 'numpyro_hmc', 'numpyro_nuts']} The values of the MCMC and VI keys in the dictionary are the names of the argument you would pass to inference_method in model.fit. This is shown in the section below.\nSpecifying an inference_method By default, Bambi uses the PyMC NUTS implementation. To use a different backend, pass the name of the bayeux MCMC method to the inference_method parameter of the fit method.\nBlackjax blackjax_nuts_idata = model.fit(inference_method=\u0026#34;blackjax_nuts\u0026#34;) Different backends have different naming conventions for the parameters specific to that MCMC method. Thus, to specify backend-specific parameters, pass your own kwargs to the fit method.\nThe following can be performend to identify the kwargs specific to each method.\nbmb.inference_methods.get_kwargs(\u0026#34;blackjax_nuts\u0026#34;) {\u0026lt;function blackjax.adaptation.window_adaptation.window_adaptation(algorithm: Union[blackjax.mcmc.hmc.hmc, blackjax.mcmc.nuts.nuts], logdensity_fn: Callable, is_mass_matrix_diagonal: bool = True, initial_step_size: float = 1.0, target_acceptance_rate: float = 0.8, progress_bar: bool = False, **extra_parameters) -\u0026gt; blackjax.base.AdaptationAlgorithm\u0026gt;: {'logdensity_fn': \u0026lt;function bayeux._src.shared.constrain.\u0026lt;locals\u0026gt;.wrap_log_density.\u0026lt;locals\u0026gt;.wrapped(args)\u0026gt;, 'is_mass_matrix_diagonal': True, 'initial_step_size': 1.0, 'target_acceptance_rate': 0.8, 'progress_bar': False, 'algorithm': blackjax.mcmc.nuts.nuts}, 'adapt.run': {'num_steps': 500}, blackjax.mcmc.nuts.nuts: {'max_num_doublings': 10, 'divergence_threshold': 1000, 'integrator': \u0026lt;function blackjax.mcmc.integrators.generate_euclidean_integrator.\u0026lt;locals\u0026gt;.euclidean_integrator(logdensity_fn: Callable, kinetic_energy_fn: blackjax.mcmc.metrics.KineticEnergy) -\u0026gt; Callable[[blackjax.mcmc.integrators.IntegratorState, float], blackjax.mcmc.integrators.IntegratorState]\u0026gt;, 'logdensity_fn': \u0026lt;function bayeux._src.shared.constrain.\u0026lt;locals\u0026gt;.wrap_log_density.\u0026lt;locals\u0026gt;.wrapped(args)\u0026gt;, 'step_size': 0.5}, 'extra_parameters': {'chain_method': 'vectorized', 'num_chains': 8, 'num_draws': 500, 'num_adapt_draws': 500, 'return_pytree': False}} Now, we can identify the kwargs we would like to change and pass to the fit method.\nkwargs = { \u0026#34;adapt.run\u0026#34;: {\u0026#34;num_steps\u0026#34;: 500}, \u0026#34;num_chains\u0026#34;: 4, \u0026#34;num_draws\u0026#34;: 250, \u0026#34;num_adapt_draws\u0026#34;: 250 } blackjax_nuts_idata = model.fit(inference_method=\u0026#34;blackjax_nuts\u0026#34;, **kwargs) blackjax_nuts_idata Tensorflow probability tfp_nuts_idata = model.fit(inference_method=\u0026#34;tfp_nuts\u0026#34;) tfp_nuts_idata NumPyro numpyro_nuts_idata = model.fit(inference_method=\u0026#34;numpyro_nuts\u0026#34;) numpyro_nuts_idata flowMC flowmc_idata = model.fit(inference_method=\u0026#34;flowmc_realnvp_hmc\u0026#34;) flowmc_idata Sampler comparisons With ArviZ, we can compare the inference result summaries of the samplers. Note: We can\u0026rsquo;t use az.compare as not each inference data object returns the pointwise log-probabilities. Thus, an error would be raised.\naz.summary(blackjax_nuts_idata) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } az.summary(tfp_nuts_idata) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } az.summary(numpyro_nuts_idata) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } az.summary(flowmc_idata) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Summary Thanks to bayeux, we can use three different sampling backends and 10+ alternative MCMC methods in Bambi. Using these methods is as simple as passing the inference name to the inference_method of the fit method.\n%load_ext watermark %watermark -n -u -v -iv -w Last updated: Sat Apr 13 2024 Python implementation: CPython Python version : 3.12.2 IPython version : 8.20.0 bambi : 0.13.1.dev25+g1e7f677e.d20240413 pandas: 2.2.1 numpy : 1.26.4 bayeux: 0.1.10 arviz : 0.18.0 Watermark: 2.4.3 ","permalink":"https://gstechschulte.github.io/posts/bambi-alternative-samplers/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003ch1 id=\"alternative-sampling-backends\"\u003eAlternative sampling backends\u003c/h1\u003e\n\u003cp\u003eThis blog post is a copy of the alternative samplers documentation I wrote for \u003ca href=\"https://bambinos.github.io/bambi/\"\u003eBambi\u003c/a\u003e. The original post can be found \u003ca href=\"https://bambinos.github.io/bambi/notebooks/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eIn Bambi, the sampler used is automatically selected given the type of variables used in the model. For inference, Bambi supports both MCMC and variational inference. By default, Bambi uses PyMC\u0026rsquo;s implementation of the adaptive Hamiltonian Monte Carlo (HMC) algorithm for sampling. Also known as the No-U-Turn Sampler (NUTS). This sampler is a good choice for many models. However, it is not the only sampling method, nor is PyMC the only library implementing NUTS.\u003c/p\u003e","title":"Alternative Samplers to NUTS in Bambi"},{"content":" Interpret Advanced Usage The interpret module is inspired by the R package marginaleffects and ports the core functionality of {marginaleffects} to Bambi. To close the gap of non-supported functionality in Bambi, interpret now provides a set of helper functions to aid the user in more advanced and complex analysis not covered within the comparisons, predictions, and slopes functions.\nThese helper functions are data_grid and select_draws. The data_grid can be used to create a pairwise grid of data points for the user to pass to model.predict. Subsequently, select_draws is used to select the draws from the posterior (or posterior predictive) group of the InferenceData object returned by the predict method that correspond to the data points that \u0026ldquo;produced\u0026rdquo; that draw.\nWith access to the appropriately indexed draws, and data used to generate those draws, it enables for more complex analysis such as cross-comparisons and the choice of which model parameter to compute a quantity of interest for. Additionally, the user has more control over the data passed to model.predict. Below, it will be demonstrated how to use these helper functions. First, to reproduce the results from the standard interpret API, and then to compute cross-comparisons.\nimport warnings import arviz as az import numpy as np import pandas as pd import bambi as bmb from bambi.interpret.helpers import data_grid, select_draws warnings.simplefilter(action=\u0026#39;ignore\u0026#39;, category=FutureWarning) Zero Inflated Poisson We will adopt the zero inflated Poisson (ZIP) model from the comparisons documentation to demonstrate the helper functions introduced above.\nThe ZIP model will be used to predict how many fish are caught by visitors at a state park using survey data. Many visitors catch zero fish, either because they did not fish at all, or because they were unlucky. We would like to explicitly model this bimodal behavior (zero versus non-zero) using a Zero Inflated Poisson model, and to compare how different inputs of interest $w$ and other covariate values $c$ are associated with the number of fish caught. The dataset contains data on 250 groups that went to a state park to fish. Each group was questioned about how many fish they caught (count), how many children were in the group (child), how many people were in the group (persons), if they used a live bait and whether or not they brought a camper to the park (camper).\nfish_data = pd.read_stata(\u0026#34;http://www.stata-press.com/data/r11/fish.dta\u0026#34;) cols = [\u0026#34;count\u0026#34;, \u0026#34;livebait\u0026#34;, \u0026#34;camper\u0026#34;, \u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;] fish_data = fish_data[cols] fish_data[\u0026#34;child\u0026#34;] = fish_data[\u0026#34;child\u0026#34;].astype(np.int8) fish_data[\u0026#34;persons\u0026#34;] = fish_data[\u0026#34;persons\u0026#34;].astype(np.int8) fish_data[\u0026#34;livebait\u0026#34;] = pd.Categorical(fish_data[\u0026#34;livebait\u0026#34;]) fish_data[\u0026#34;camper\u0026#34;] = pd.Categorical(fish_data[\u0026#34;camper\u0026#34;]) fish_model = bmb.Model( \u0026#34;count ~ livebait + camper + persons + child\u0026#34;, fish_data, family=\u0026#39;zero_inflated_poisson\u0026#39; ) fish_idata = fish_model.fit(random_seed=1234) Create a grid of data data_grid allows you to create a pairwise grid, also known as a cross-join or cartesian product, of data using the covariates passed to the conditional and the optional variable parameter. Covariates not passed to conditional, but are terms in the Bambi model, are set to typical values (e.g., mean or mode). If you are coming from R, this function is partially inspired from the data_grid function in {modelr}.\nThere are two ways to create a pairwise grid of data:\nuser-provided values are passed as a dictionary to conditional where the keys are the names of the covariates and the values are the values to use in the grid. a list of covariates where the elements are the names of the covariates to use in the grid. As only the names of the covariates were passed, default values are computed to construct the grid. Any unspecified covariates, i.e., covariates not passed to conditional but are terms in the Bambi model, are set to their \u0026ldquo;typical\u0026rdquo; values such as mean or mode depending on the data type of the covariate.\nUser-provided values To construct a pairwise grid of data for specific covariate values, pass a dictionary to conditional. The values of the dictionary can be of type int, float, list, or np.ndarray.\nconditional = { \u0026#34;camper\u0026#34;: np.array([0, 1]), \u0026#34;persons\u0026#34;: np.arange(1, 5, 1), \u0026#34;child\u0026#34;: np.array([1, 2, 3]), } user_passed_grid = data_grid(fish_model, conditional) user_passed_grid.query(\u0026#34;camper == 0\u0026#34;) Default computed for unspecified variable: livebait .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Subsetting by camper = 0, it can be seen that a combination of all possible pairs of values from the dictionary (including the unspecified variable livebait) results in a dataframe containing every possible combination of values from the original sets. livebait has been set to 1 as this is the mode of the unspecified categorical variable.\nDefault values Alternatively, a list of covariates can be passed to conditional where the elements are the names of the covariates to use in the grid. By doing this, you are telling interpret to compute default values for these covariates. The psuedocode below outlines the logic and functions used to compute these default values:\nif is_numeric_dtype(x) or is_float_dtype(x): values = np.linspace(np.min(x), np.max(x), 50) elif is_integer_dtype(x): values = np.quantile(x, np.linspace(0, 1, 5)) elif is_categorical_dtype(x) or is_string_dtype(x) or is_object_dtype(x): values = np.unique(x) conditional = [\u0026#34;camper\u0026#34;, \u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;] default_grid = data_grid(fish_model, conditional) default_grid.shape, user_passed_grid.shape Default computed for conditional variable: camper, persons, child Default computed for unspecified variable: livebait ((32, 4), (24, 4)) Notice how the resulting length is different between the user passed and default grid. This is due to the fact that values for child range from 0 to 3 for the default grid.\ndefault_grid.query(\u0026#34;camper == 0\u0026#34;) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Compute comparisons To use data_grid to help generate data in computing comparisons or slopes, additional data is passed to the optional variable parameter. The name variable is an abstraction for the comparisons parameter contrast and slopes parameter wrt. If you have used any of the interpret functions, these parameter names should be familiar and the use of data_grid should be analogous to comparisons, predictions, and slopes.\nvariable can also be passed user-provided data (as a dictionary), or a string indicating the name of the covariate of interest. If the latter, a default value will be computed. Additionally, if an argument is passed for variable, then the effect_type needs to be passed. This is because for comparisons and slopes an epsilon value eps needs to be determined to compute the centered and finite difference, respectively. You can also pass a value for eps as a kwarg.\nconditional = { \u0026#34;camper\u0026#34;: np.array([0, 1]), \u0026#34;persons\u0026#34;: np.arange(1, 5, 1), \u0026#34;child\u0026#34;: np.array([1, 2, 3, 4]) } variable = \u0026#34;livebait\u0026#34; grid = data_grid(fish_model, conditional, variable, effect_type=\u0026#34;comparisons\u0026#34;) Default computed for contrast variable: livebait idata_grid = fish_model.predict(fish_idata, data=grid, inplace=False) Select draws conditional on data The second helper function to aid in more advanced analysis is select_draws. This is a function that selects the posterior or posterior predictive draws from the ArviZ InferenceData object returned by model.predict given a conditional dictionary. The conditional dictionary represents the values that correspond to that draw.\nFor example, if we wanted to select posterior draws where livebait = [0, 1], then all we need to do is pass a dictionary where the key is the name of the covariate and the value is the value that we want to condition on (or select). The resulting InferenceData object will contain the draws that correspond to the data points where livebait = [0, 1]. Additionally, you must pass the InferenceData object returned by model.predict, the data used to generate the predictions, and the name of the data variable data_var you would like to select from the InferenceData posterior group. If you specified to return the posterior predictive samples by passing model.predict(..., kind=\u0026quot;pps\u0026quot;), you can use this group instead of the posterior group by passing group=\u0026quot;posterior_predictive\u0026quot;.\nBelow, it is demonstrated how to compute comparisons for count_mean for the contrast livebait = [0, 1] using the posterior draws.\ndraw_1 = select_draws(idata_grid, grid, {\u0026#34;livebait\u0026#34;: 0}, \u0026#34;count_mean\u0026#34;) draw_1 = select_draws(idata_grid, grid, {\u0026#34;livebait\u0026#34;: 0}, \u0026#34;count_mean\u0026#34;) draw_2 = select_draws(idata_grid, grid, {\u0026#34;livebait\u0026#34;: 1}, \u0026#34;count_mean\u0026#34;) comparison_mean = (draw_2 - draw_1).mean((\u0026#34;chain\u0026#34;, \u0026#34;draw\u0026#34;)) comparison_hdi = az.hdi(draw_2 - draw_1) comparison_df = pd.DataFrame( { \u0026#34;mean\u0026#34;: comparison_mean.values, \u0026#34;hdi_low\u0026#34;: comparison_hdi.sel(hdi=\u0026#34;lower\u0026#34;)[\u0026#34;count_mean\u0026#34;].values, \u0026#34;hdi_high\u0026#34;: comparison_hdi.sel(hdi=\u0026#34;higher\u0026#34;)[\u0026#34;count_mean\u0026#34;].values, } ) comparison_df.head(10) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } We can compare this comparison with bmb.interpret.comparisons.\nsummary_df = bmb.interpret.comparisons( fish_model, fish_idata, contrast={\u0026#34;livebait\u0026#34;: [0, 1]}, conditional=conditional ) summary_df.head(10) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Albeit the other information in the summary_df, the columns estimate, lower_3.0%, upper_97.0% are identical.\nCross comparisons Computing a cross-comparison is useful for when we want to compare contrasts when two (or more) predictors change at the same time. Cross-comparisons are currently not supported in the comparisons function, but we can use select_draws to compute them. For example, imagine we are interested in computing the cross-comparison between the two rows below.\nsummary_df.iloc[:2] .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The cross-comparison amounts to first computing the comparison for row 0, given below, and can be verified by looking at the estimate in summary_df.\ncond_10 = { \u0026#34;camper\u0026#34;: 0, \u0026#34;persons\u0026#34;: 1, \u0026#34;child\u0026#34;: 1, \u0026#34;livebait\u0026#34;: 0 } cond_11 = { \u0026#34;camper\u0026#34;: 0, \u0026#34;persons\u0026#34;: 1, \u0026#34;child\u0026#34;: 1, \u0026#34;livebait\u0026#34;: 1 } draws_10 = select_draws(idata_grid, grid, cond_10, \u0026#34;count_mean\u0026#34;) draws_11 = select_draws(idata_grid, grid, cond_11, \u0026#34;count_mean\u0026#34;) (draws_11 - draws_10).mean((\u0026#34;chain\u0026#34;, \u0026#34;draw\u0026#34;)).item() 0.2143627093182434 Next, we need to compute the comparison for row 1.\ncond_20 = { \u0026#34;camper\u0026#34;: 0, \u0026#34;persons\u0026#34;: 1, \u0026#34;child\u0026#34;: 2, \u0026#34;livebait\u0026#34;: 0 } cond_21 = { \u0026#34;camper\u0026#34;: 0, \u0026#34;persons\u0026#34;: 1, \u0026#34;child\u0026#34;: 2, \u0026#34;livebait\u0026#34;: 1 } draws_20 = select_draws(idata_grid, grid, cond_20, \u0026#34;count_mean\u0026#34;) draws_21 = select_draws(idata_grid, grid, cond_21, \u0026#34;count_mean\u0026#34;) (draws_21 - draws_20).mean((\u0026#34;chain\u0026#34;, \u0026#34;draw\u0026#34;)).item() 0.053678256991883604 Next, we compute the \u0026ldquo;first level\u0026rdquo; comparisons (diff_1 and diff_2). Subsequently, we compute the difference between these two differences to obtain the cross-comparison.\ndiff_1 = (draws_11 - draws_10) diff_2 = (draws_21 - draws_20) cross_comparison = (diff_2 - diff_1).mean((\u0026#34;chain\u0026#34;, \u0026#34;draw\u0026#34;)).item() cross_comparison -0.16068445232635978 To verify this is correct, we can check by performing the cross-comparison directly on the summary_df.\nsummary_df.iloc[1][\u0026#34;estimate\u0026#34;] - summary_df.iloc[0][\u0026#34;estimate\u0026#34;] -0.16068445232635978 Summary In this notebook, the interpret helper functions data_grid and select_draws were introduced and it was demonstrated how they can be used to compute pairwise grids of data and cross-comparisons. With these functions, it is left to the user to generate their grids of data and quantities of interest allowing for more flexibility and control over the type of data passed to model.predict and the quantities of interest computed.\n%load_ext watermark %watermark -n -u -v -iv -w Last updated: Tue Dec 05 2023 Python implementation: CPython Python version : 3.11.0 IPython version : 8.13.2 numpy : 1.24.2 pandas: 2.1.0 bambi : 0.13.0.dev0 arviz : 0.16.1 Watermark: 2.3.1 ","permalink":"https://gstechschulte.github.io/posts/bambi-advanced-marginal-effects/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003ch1 id=\"interpret-advanced-usage\"\u003eInterpret Advanced Usage\u003c/h1\u003e\n\u003cp\u003eThe \u003ccode\u003einterpret\u003c/code\u003e module is inspired by the R package \u003ca href=\"https://marginaleffects.com\"\u003emarginaleffects\u003c/a\u003e and ports the core functionality of {marginaleffects} to Bambi. To close the gap of non-supported functionality in Bambi, \u003ccode\u003einterpret\u003c/code\u003e now provides a set of helper functions to aid the user in more advanced and complex analysis not covered within the \u003ccode\u003ecomparisons\u003c/code\u003e, \u003ccode\u003epredictions\u003c/code\u003e, and \u003ccode\u003eslopes\u003c/code\u003e functions.\u003c/p\u003e\n\u003cp\u003eThese helper functions are \u003ccode\u003edata_grid\u003c/code\u003e and \u003ccode\u003eselect_draws\u003c/code\u003e. The \u003ccode\u003edata_grid\u003c/code\u003e can be used to create a pairwise grid of data points for the user to pass to \u003ccode\u003emodel.predict\u003c/code\u003e. Subsequently, \u003ccode\u003eselect_draws\u003c/code\u003e is used to select the draws from the posterior (or posterior predictive) group of the InferenceData object returned by the predict method that correspond to the data points that \u0026ldquo;produced\u0026rdquo; that draw.\u003c/p\u003e","title":"Advanced Interpret Usage in Bambi"},{"content":" #| code-fold: true import matplotlib.pyplot as plt import torch import numpy as np from botorch.acquisition import qLogExpectedImprovement from botorch.fit import fit_gpytorch_model from botorch.models import SingleTaskGP from botorch.optim import optimize_acqf from gpytorch.mlls import ExactMarginalLogLikelihood from torch.distributions import Normal plt.style.use(\u0026#34;https://raw.githubusercontent.com/GStechschulte/filterjax/main/docs/styles.mplstyle\u0026#34;) Outcome constraints In optimization, it is often the goal that we need to optimize an objective function while satisfying some constraints. For example, we may want to minimize the scrap rate by finding the optimal process parameters of an manufacturing machine. However, we know the scrap rate cannot be below 0. In another setting, we may want to maximize the throughput of a machine, but we know that the throughput cannot exceed the maximum belt speed of the machine. Thus, we need to find regions in the search space that both yield high objective values and satisfy these constraints. In this blog, we will focus on inequality outcome constraints. That is, the domain of the objective function is\n$$\\text{lower} \\le f(x) \\le \\text{upper}$$\nwhere $\\text{lower}$ and $\\text{upper}$ are the lower and upper bounds of the objective function. You need not both bounds, but rather one or the other. The set of points $x\u0026rsquo;$ that satisfy the constraint are called feasible points and the set of points that do not satisfy the constraint are called infeasible points. Often, in tutorials and or books teaching Bayesian optimization, it is assumed we know a cost function a priori that restricts the outcome space in some way, and then an additional model is used to model the constraint. However, in practice, we often only know a lower and or upper bound according to technical specifications. These bounds do not require an additional model.\nIn this blog, it will be shown how to use BoTorch to optimize a one-dimensional function with an outcome constraint without using an additional model for the cost (constraint) function. The remainder of the post assumes the reader is already familiar with Bayesian optimization.\nProbability of feasibility In BoTorch it is common to use a Gaussian Process (GP) to model the objective function. The output of the GP is a Gaussian distribution over the predicted values for a given set of input points. It provides not just a single point estimate but a probabilistic prediction that accounts for uncertainty in the predictions. Thus, for each point in the search space, we have a corresponding Gaussian distribution representing the belief of the objective value at that point.\nIntuitively, if we have defined an outcome constraint, we can compute the probability that $f(x)$ is feasible by taking the cumulative distribution function (CDF) of the predictive distribution and computing the area between the lower bound and the upper bound. For example, imagine a GP has made a prediction given an input $x$. This predictive distribution of the outcome $y$ is shown below. The prediction is normally distributed around $0.0$ with plausible predictions ranging from $-3$ to $3$. Additionally, there is an outcome constraint of the form\n$$0 \u0026lt; f(x)$$\nThe probability that the prediction is feasible (above $0$) is computed using the CDF of the predictive distribution. In this example, the probability of feasibility is $0.5$. As will be shown below, this probability can then multiplied by the policy score to get the constrained policy score.\n#| code-fold: true xs = torch.linspace(-3, 3, 100) normal = Normal(torch.tensor([0.0]), torch.tensor([1.0])) xs_eval = torch.exp(normal.log_prob(xs)) / torch.exp(normal.log_prob(xs)).sum() cdf = normal.cdf(xs) above = torch.where(cdf \u0026gt; 0.5) prob_feasibility = normal.cdf(torch.zeros(1)).item() plt.figure(figsize=(7, 3)) plt.plot(xs, xs_eval, label=\u0026#34;Predictive distribution\u0026#34;) plt.fill_between(xs, xs_eval, where=xs \u0026gt; 0, alpha=0.5, label=\u0026#34;Feasible region\u0026#34;) plt.xlabel(\u0026#39;Predicted outcome $y$\u0026#39;) plt.ylabel(\u0026#39;Probability\u0026#39;) plt.title(f\u0026#34;Area under the shaded curve: {prob_feasibility:.2f}\u0026#34;) plt.legend() plt.show() Constrained Policy With the probability of feasibility computed, we can scale the policy, e.g. expected improvement (EI), score of each unseen point in the search space by the probability the point is feasible.\nIf the data point is likely to satisfy the constraints, then its EI score will be multiplied by a large number (a high probability of feasibility), thus keeping the EI score high. If the data point is unlikely to satisfy the constraints, then its EI score will be multiplied by a small number (a small probability of feasibility), thus keeping the EI score small. To implement inequality outcome constraints, acquisition functions that utilize Monte-Carlo (MC) sampling are used as this allows us to directly pass a list of constraint callables. These are any acquisition functions that inherit from SampleReducingMCAcqquisitionFunction.\nImplementation To implement inequality outcome constraints, only a list of constraint callables which map a Tensor of posterior samples of dimension sample_shape x batch-shape x q x m-dim to a sample_shape x batch-shape x q-dim Tensor. The associated constraints are considered satisfied if the output is less than zero. In the example below, we aim to minimize the Forrester function subject to the following constraint that $$f(x) \u0026lt; 0$$\nNote: Since we are minimizing, the objective function is inverted, and thus the inequality is also inverted.\ndef objective_fn(x): return -((x + 1) ** 2) * torch.sin(2 * x + 2) / 5 + 1 + x / 3 #| code-fold: true lb, ub = -5, 5 bounds = torch.tensor([[lb], [ub]], dtype=torch.float) xs = torch.linspace(lb, ub, 100).unsqueeze(1) ys = -objective_fn(xs) n = 5 train_x = bounds[0] + (bounds[1] - bounds[0]) * torch.rand(n, 1, dtype=torch.double) train_y = -objective_fn(train_x) plt.figure(figsize=(7, 3)) plt.plot(xs, ys, label=\u0026#34;Objective\u0026#34;) plt.scatter(train_x, train_y, color=\u0026#34;black\u0026#34;, label=\u0026#34;Observations\u0026#34;) plt.axhline(y=0, color=\u0026#34;k\u0026#34;, linestyle=\u0026#34;--\u0026#34;, label=\u0026#34;Upper Bound\u0026#34;) plt.legend(); The Bayesian optimization loop below uses the qLogExpectedImprovement policy. To impose the desired inequality outcome constraint $f(x) \u0026lt; 0$, a list of callables [lambda Z: Z.squeeze(-1) - upper] is passed to constraints. This callable subtracts the posterior samples $Z$ by upper which is $0.0$. If the result of this is less than zero, then the constraint is satisfied.\nNote that $Z$ here would be passing in all outcomes if a multi-task GP had been defined, so you want to index into $Z$ appropriately and make separate callables for each outcome, e.g. constraints=[lambda Z: Z[..., constraint_outcome_idx]]. However, in this example, there is only one outcome, so we can just use Z.squeeze(-1) to select the correct (and only) outcome dimension.\nn_iterations = 20 upper = 0. for iteration in range(n_iterations): print(f\u0026#34;iter: {iteration}\u0026#34;) model = SingleTaskGP(train_x, train_y, ) mll = ExactMarginalLogLikelihood(model.likelihood, model) fit_gpytorch_model(mll) logEI = qLogExpectedImprovement( model, best_f=train_y[train_y \u0026lt; 0].max(), constraints=[ lambda Z: Z.squeeze(-1) - upper # if there was also a lower bound # lambda Z: lower - Z.squeeze(-1), ] ) new_x, _ = optimize_acqf( acq_function=logEI, bounds=bounds, q=1, num_restarts=5, raw_samples=20, ) new_y = -objective_fn(new_x) train_x = torch.cat([train_x, new_x]) train_y = torch.cat([train_y, new_y]) We can then evaluate the policy on unseen data and plot the proposed samples (queries).\nwith torch.no_grad(): acquisition_score = logEI(xs.unsqueeze(1)) fig, ax = plt.subplots( 2, 1, figsize=(8, 4), sharex=True, gridspec_kw={\u0026#34;height_ratios\u0026#34;: [2, 1]} ) ax[0].scatter(train_x[:5], -train_y[:5], color=\u0026#34;black\u0026#34;, label=\u0026#34;observed samples\u0026#34;) ax[0].scatter(train_x[5:], -train_y[5:], color=\u0026#34;red\u0026#34;, label=\u0026#34;proposal samples\u0026#34;) ax[0].plot(xs, -ys, label=\u0026#34;objective function\u0026#34;) ax[0].legend() ax[1].plot(xs, acquisition_score, label=\u0026#34;acquisition score\u0026#34;) ax[1].legend(); The objective function has been flipped back to its original form to visually evaluate the optimization loop. Notice how the majority of the proposed points that minimize the objective function are near $0$. Points that are below $0$ happen due to the fact that we are using a probabilistic surrogate model to compute the probability of feasibility. The predictions of this model are not perfect, and thus, it is possible that the optimized policy score informs the next query to be a point below $0.0$. Nonetheless, the minimizing points are found near $-4.2$, $-2.5$, and $-1.5$.\n","permalink":"https://gstechschulte.github.io/posts/constrained-bayesian-optimization/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e#| code-fold: true\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e matplotlib.pyplot \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e plt\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e torch\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e numpy \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e np\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e botorch.acquisition \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e qLogExpectedImprovement\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e botorch.fit \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e fit_gpytorch_model\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e botorch.models \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e SingleTaskGP\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e botorch.optim \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e optimize_acqf\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e gpytorch.mlls \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e ExactMarginalLogLikelihood\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e torch.distributions \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e Normal\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eplt\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003estyle\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003euse(\u003cspan style=\"color:#e6db74\"\u003e\u0026#34;https://raw.githubusercontent.com/GStechschulte/filterjax/main/docs/styles.mplstyle\u0026#34;\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003ch1 id=\"outcome-constraints\"\u003eOutcome constraints\u003c/h1\u003e\n\u003cp\u003eIn optimization, it is often the goal that we need to optimize an objective function while satisfying some constraints. For example, we may want to minimize the scrap rate by finding the optimal process parameters of an manufacturing machine. However, we know the scrap rate cannot be below 0. In another setting, we may want to maximize the throughput of a machine, but we know that the throughput cannot exceed the maximum belt speed of the machine. Thus, we need to find regions in the search space that both yield high objective values and satisfy these constraints. In this blog, we will focus on inequality \u003cem\u003eoutcome constraints\u003c/em\u003e. That is, the domain of the objective function is\u003c/p\u003e","title":"Outcome Constraints in Bayesian Optimization"},{"content":" Survival Models Survival models, also known as time-to-event models, are specialized statistical methods designed to analyze the time until the occurrence of an event of interest. In this notebook, a review of survival analysis (using non-parametric and parametric methods) and censored data is provided, followed by a survival model implementation in Bambi.\nThis blog post is a copy of the survival models documentation I wrote for Bambi. The original post can be found here.\nSurvival and censoring times Sometimes the right way to model discrete, countable events is to model not the counts themselves but rather the time between events. This gives us information regarding the rate of an event. Survival models are models for countable things, but the outcomes we want to predict are durations. Durations are continuous deviations from some point of reference (so they are all positive values).\nThe tricky part with survival models is not the probability distribution assigned to the durations, but dealing with censoring. Censoring occurs when the event of interest does not occur in the window of observation. In a simple scenario, this can happen because the observation period ends before the event occurred. Censored individuals (or units) can not just be dropped from the sample. As an example, we use Richard McElreath\u0026rsquo;s cat adoption example from chapter 11.4 of Statistical Rethinking: Imagine a cohort of 100 cats who start waiting for adoption at the same time. After one month, half of them have been adopted. Now what is the rate of adoption? You can’t compute it using only the cats who have been adopted. You need to also account for the cats who haven’t yet been adopted. The cats who haven’t been adopted yet, but eventually will be adopted, clearly have longer waiting times than the cats who have already been adopted. So the average rate among those who are already adopted is biased upwards—it is confounded by conditioning on adoption.\nIncluding censored observations requires a new type of model. The key idea is that the same distribution assumption for the outcome tells us both the probability of any observed duration that end in the event as well as the probability that we would wait the observed duration without seeing the event. For each unit, we assume there is a true survival time $T$ as well as a true censoring time $C$. The survival time represents the time at which the event of interest occurs. The censoring time is the time at which censoring occurs. We observe either: the survival, or the censoring time:\n$$Y = \\text{min}(T, C)$$\nIf the event occurs, then we observe the survival time, else we observe the censoring time. In order to analyze survival data, we first need to understand the two types of censoring: left and right censoring, and how to estimate the survival function.\nLeft and right censoring There are two main \u0026ldquo;types\u0026rdquo; of censoring: right and left. Right censoring occurs when $T \\ge Y$, i.e. the true event time $T$ is at least as large as the observed time $Y$. This is a consequence of $Y = \\text{min}(T, C)$. Right censoring derives its name from the notion that time is typically read and displayed from left to right. Left sensoring occurs when the true event time $T$ is less than or equal to the observed time $Y$. An example of left censoring could be in a study of pregnancy duration. Suppose that patients are surveyed 250 days (8.2 months) after conception. Some patients may have already had their babies. For these patients, pregnancy duration is less than 250 days.\nEstimating the survival function Survival analysis is concerned with estimating the survival function $S(t)$\n$$S(t) = Pr(T \u0026gt; t) = 1 - F(t)$$\nwhich is a decreasing function that quantifies the probability of surviving past time $t$. Alternatively, $S(t)$ can be expressed as one minus the cumulative distribution function (CDF) $F$ of the event time $T$—referred to as the complementary cumulative distribution function (CCDF). The focus on the survival function is important because for censored observations, we only know that the time-to-event exceeds the observed time $Y$.\nHere, continuing with the cat adoption example, we consider the task of estimating the survival function for cat adoptions. To estimate $S(30) = Pr(T \u0026gt; 30)$, the probability that a cat is not adopted after 30 days, it is tempting to compute the proportion of cats who were adopted before 30 days and subtract this from 1. However, this would be incorrect because it ignores the cats who were not adopted before 30 days but who will be adopted later—these cats clearly have longer adoption rates. Thus, if we continued with the naive approach, the average rate of adoption would be biased upwards—it is confounded by conditioning on adoption.\nHowever, it is possible to overcome this challenge by using the Kaplan-Meier estimator. The Kaplan-Meier estimator is a non-parametric estimator of the survival function that accounts for censoring. Let $d_1 \u0026lt; d_2 \u0026lt; . . . \u0026lt; d_K$ denote the $K$ unique adoption times among the non-censored cats, and $q_k$ denote the number of cats adopted at time $d_k$. For $k = 1,\u0026hellip;,K$, let $r_k$ denote the number of cats not adopted at time $d_k$. By the law of total probability\n$$Pr(T \u0026gt; d_k) = Pr(T \u0026gt; d_k | T \u0026gt; d_{k-1}) Pr(T \u0026gt; d_{k-1}) + Pr(T \u0026gt; d_k | T \\leq d_{k-1}) Pr(T \\leq d_{k-1})$$\nThe fact that $d_{k-1} \u0026lt; d_k$ implies that $Pr(T \u0026gt; d_k | T \\leq d_{k-1}) = 0$ (as it is impossible for a cat to be adopted past time $d_k$ if the cat was adopted before time $d_{k-1}$). Thus, if we simplify the above equation and plug into the survival function, we obtain\n$$S(d_k) = Pr(T \u0026gt; d_k | T \u0026gt; d_{k-1})S(d_{k-1})$$\nNow we must estimate the terms on the right-hand side. It is common to use the following estimator\n$$\\hat{Pr}(T \u0026gt; d_j | T \u0026gt; d_{j-1}) = \\frac{r_j - q_j}{r_j}$$\nwhich leads us to the Kaplan-Meier estimator of the survival function\n$$\\hat{S}(d_k) = \\prod_{j=1}^k \\frac{r_j - q_j}{r_j}$$\nwhere $\\hat{S}(d_k)$ represents the estimated survival probability up to time $d_k$. The product is taken over all time points up to $k$, where an event occurred. The variables $r_j$ and $q_j$ denote the number of subjects at risk and the number of events at time $d_j$, respectively. The term $\\frac{r_j - q_j}{r_j}$ is the conditional probability of surviving the $j$-th time point given that an individual has survived just before $d_j$. Specifically, $r_j - q_j$ are the number of individuals who survived just before $d_j$ and $r_j$ is the number of individuals who survived just after $d_j$, and $r_j$ are those who were at risk $d_j$.\nCat adoption survival function Below we use the KaplanMeierFitter class of the lifelines package to compute and visualize the survival curve for cat adoptions from an animal shelter in Austin, Texas beginning October 1st, 2013 until May 30th, 2018 (the last day the shelter rescued a cat). The dataset comes from the City of Austin Open Data Portal and contains columns such as animal name, date of birth, species, and many more. However, for the purpose of this notebook we are interested in the following columns:\ndays_to_event - number of days until the cat was adopted (date_in - date_out) out_event - the reason for the cat leaving this particular shelter, e.g. adopted or transfered. color - the color of the cat, e.g. white, blue, brown tabby, black. #| code-fold: true import arviz as az import matplotlib.pyplot as plt import numpy as np import pandas as pd import scipy from lifelines import KaplanMeierFitter import bambi as bmb url = \u0026#34;https://raw.githubusercontent.com/rmcelreath/rethinking/master/data/AustinCats.csv\u0026#34; cats_df = pd.read_csv(url, sep=\u0026#34;;\u0026#34;) plt.figure(figsize=(7, 3)) plt.hist(cats_df[\u0026#34;days_to_event\u0026#34;], bins=250, label=\u0026#34;Uncensored data\u0026#34;) plt.xlim(0, 186) # limit to 6 months for visibility plt.title(\u0026#34;Days Until Adoption\u0026#34;) plt.ylabel(\u0026#34;Count\u0026#34;) plt.xlabel(\u0026#34;Days\u0026#34;) plt.legend(); The distribution of days until adoption exhibits a long tail with most cats (if we observe the adopt event) being adopted within the first month of inception. Note that the plot has been truncated to six months for better visibility. Below, we estimate the survival function using the KaplanMeierFitter class from the lifelines package.\nkm = KaplanMeierFitter() km_adoptions = km.fit( cats_df[\u0026#34;days_to_event\u0026#34;], cats_df[\u0026#34;out_event\u0026#34;].apply(lambda x: 1 if x == \u0026#34;Adoption\u0026#34; else 0) ) fig, ax = plt.subplots(figsize=(7, 3)) km_adoptions.plot(label=\u0026#34;Kaplan-Meier Estimator\u0026#34;, ax=ax) ax.set_ylabel(\u0026#34;Probability of Adoption\u0026#34;) ax.set_xlabel(\u0026#34;Days\u0026#34;) ax.set_xlim(0, 365) ax.grid(True) ax.set_title(\u0026#34;Cat Adoption Survival Curve\u0026#34;); The Kaplan-Meier estimator shows that by 100 days, the probability of a cat not being adopted is about $0.15$ percent. After 100 days, the probability of cat not being adopted decreases, albeit at a much slower rate. Thus, if a cat hasn\u0026rsquo;t been adopted by the 100th day, it is more likely the cat will continue to wait for adoption. In the next section, we discuss pm.Censored, a PyMC distrbution that allows us to model censored data.\nThe pm.Censored distribution The censored distribution from PyMC allows us to make use of a sequential construction, similar to the Kaplan-Meier estimator outlined above, to model censored data. To understand the pm.Censored distribution, lets reason how a distribution may be used to model censored data. For observed adoptions, the probability of observed waiting time can be distributed according to an exponential with some rate $\\lambda$ $$D_i \\sim \\text{Exponential}(\\lambda_i)$$ or $$f(D_i | \\lambda_i) = \\lambda_i \\text{exp}(-\\lambda_i D_i)$$ It’s the censored cats that are tricky. If something else happened before a cat could be adopted, or it simply hasn’t been adopted yet, then we need the probability of not being adopted, conditional on the observation time so far. One way to motivate this is to image a cohort of 100 cats, all joining the shelter on the same day.\nIf half have been adopted after 30 days, then the probability of waiting 30 days and still not being adopted is 0.5. If after 60 days, only 25 remain, then the probability of waiting 60 days and not yet being adopted is 0.25. Thus, any given rate of adoption implies a proportion of the cohort of 100 cats that will remain after any given number of days. This probability comes from the cumulative probability distribution. A cumulative distribution gives the proportion of cats adopted before or at a certain number of days. So $1 - \\text{CDF}$, which is the CCDF, gives the probability a cat is not adopted by the same number of days. Remember from the Estimating the survival function section, this is equivalent to the survival function. If the exponential distribution is used, the CDF is\n$$F(D_i | \\lambda_i) = 1 - \\text{exp}(-\\lambda_i D_i)$$\nwhere the complement is (here we use $S$ to denote the equivalence of the survival function and CCDF)\n$$S(D_i|\\lambda) = \\text{exp}(-\\lambda_i D_i)$$\nWhich is what we need in our model since it is the probability of waiting $D_i$ days without being adopted yet. The pm.Censored from PyMC offers a convenient way to model censored data and the probability density function (PDF) is defined as\n\\begin{cases} 0 \u0026amp; \\text{for } x \u0026lt; \\text{lower}, \\ \\text{CDF}(\\text{lower}, \\text{dist}) \u0026amp; \\text{for } x = \\text{lower}, \\ \\text{PDF}(x, \\text{dist}) \u0026amp; \\text{for } \\text{lower} \u0026lt; x \u0026lt; \\text{upper}, \\ 1 - \\text{CDF}(\\text{upper}, \\text{dist}) \u0026amp; \\text{for } x = \\text{upper}, \\ 0 \u0026amp; \\text{for } x \u0026gt; \\text{upper}. \\end{cases}\nwhere lower is left-censored and upper is right-censored. Our cat adoption dataset is right-censored. Therefore, lower can be None, and upper is the observed times when an event occurs. The pm.Censored uses the CCDF to answer the question we are interested in: what is the probability of not being adopted yet, given the observation time so far?\nImplementation in Bambi To understand how this is used, lets use Bambi to recover the parameters of the censored distribution with no predictors. Before the model is fit, days_to_event is scaled to represent months as the raw values contain very large values. This scaling ensures a smoother sampling process.\nAdditionally, modeling censored data in Bambi requires a new formula syntax censored(time, event) on the response term. censored indicates we want to model censored data and gets parsed where time and event are passed into a Bambi transformation function censored. This function takes two arguments: the first being the observed value $Y$ (in this example time), and the second being the type of censoring of the event. In Bambi, it is possible to have left, none, right, and interval censoring. event needs to be encoded as one of the censoring types. In our cat adoption example, we will encode the adoption event as right.\nLastly, the exponential distribution is used to model the cat adoption rate parameter. But why not enter censored as the likelihood like we normally do in Bambi? The pm.Censored is indeed eventually used as the likelihood. However, there also needs to be a distribution that models the rate parameter. In this example it is the exponential distribution. This distribution is then used as input into the pm.Censored distribution. For more information on how to use the pm.Censored distribution, see the following PyMC documentation: Bayesian regression models with truncated and censored data and Censored data models.\ncats = cats_df.copy() cats[\u0026#34;adopt\u0026#34;] = np.where(cats[\u0026#34;out_event\u0026#34;] == \u0026#34;Adoption\u0026#34;, \u0026#34;right\u0026#34;, \u0026#34;none\u0026#34;) cats[\u0026#34;color_id\u0026#34;] = np.where(cats[\u0026#34;color\u0026#34;] == \u0026#34;Black\u0026#34;, 1, 0) cats = cats[[\u0026#34;days_to_event\u0026#34;, \u0026#34;adopt\u0026#34;, \u0026#34;color_id\u0026#34;]] model_1 = bmb.Model( \u0026#34;censored(days_to_event / 31, adopt) ~ 1\u0026#34;, data=cats, family=\u0026#34;exponential\u0026#34;, link=\u0026#34;log\u0026#34; ) model_1.build() model_1.graph() idata_1 = model_1.fit( tune=500, draws=500, random_seed=42, chains=4, cores=10 ) Auto-assigning NUTS sampler... Initializing NUTS using jitter+adapt_diag... Multiprocess sampling (4 chains in 10 jobs) NUTS: [Intercept] Sampling 4 chains for 500 tune and 500 draw iterations (2_000 + 2_000 draws total) took 5 seconds. az.plot_trace(idata_1); summary = az.summary(idata_1) summary /Users/gabestechschulte/miniforge3/envs/bambinos/lib/python3.11/site-packages/xarray/core/concat.py:546: FutureWarning: unique with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version. common_dims = tuple(pd.unique([d for v in vars for d in v.dims])) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Interpreting the intercept (the cat adoption rate parameter) alone is of not much value. Therefore, lets use the survival function to compute the probability of not being adopted after a range of months, given the learned rate parameter $\\lambda$. We could dervive the survival function and pass the intercept parameter to it, but SciPy already implements it as scipy.stats.expon.sf, so we will just use this implementation.\nCat adoption plots The plot below shows the estimated survival function and CCDF for cat adoptions. First, we compute the $0.95$ credible interval (CI) and median value for the intercept. Then, since a log-link was used, the values are exponentiated.\nlambda_preds = np.quantile( idata_1[\u0026#34;posterior\u0026#34;][\u0026#34;Intercept\u0026#34;], [0.025, 0.5, 0.975] ) lambda_lower = 1 / np.exp(lambda_preds[0]) lambda_median = 1 / np.exp(lambda_preds[1]) lambda_upper = 1 / np.exp(lambda_preds[2]) t = np.linspace(0, max(cats[\u0026#34;days_to_event\u0026#34;] / 31), 100) S0 = scipy.stats.expon.sf cdf = scipy.stats.expon.cdf fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10, 3), sharey=True) ax[0].plot(t, S0(lambda_median * t)) ax[0].fill_between(t, S0(lambda_lower * t), S0(lambda_upper * t), alpha=0.25) ax[0].grid(True) ax[0].set_xlim(0, 12) ax[0].set_xlabel(\u0026#34;Months\u0026#34;) ax[0].set_ylabel(\u0026#34;Probability\u0026#34;) ax[0].set_title(\u0026#34;Probability Not Being Adopted by Time $d_k$\u0026#34;) ax[1].plot(t, cdf(lambda_median * t)) ax[1].fill_between(t, cdf(lambda_lower * t), cdf(lambda_upper * t), alpha=0.25) ax[1].grid(True) ax[1].vlines(1, 1, 1, linestyles=\u0026#34;dashed\u0026#34;) ax[1].set_xlabel(\u0026#34;Months\u0026#34;) ax[1].set_ylabel(\u0026#34;Probability\u0026#34;) ax[1].set_title(\u0026#34;Probability of Being Adopted by Time $d_k$\u0026#34;); Analyzing the CCDF (the left plot), the probability of a cat waiting one month without being adopted is about $0.60$, whereas the probability of a cat being adopted by the first month is about $0.40$. Analyzing the CDF (right plot), the majority of cats, about $0.97$, are adopted by the sixth month. Now that we have an intuition on how pm.Censored is used for modeling censored data, in the next section, we will discuss how to model censored data with predictors.\nRegression models with a survival response It is often the case that we would like to understand how various predictors are associated with the survival function. For example, we may want to know if the survival function for cats of different colors or species is different. As outlined above, we cannot simply run a regression on the observed times $Y$ given some predictors $X$. What we are actually interested in is predicting the survival time $T$ given the predictors $X$. To achieve this, we first need to understand the Hazard function and the Cox proportional hazards model.\nThe hazard function In survival analysis, it is often more convenient to express the survival function in terms of the hazards rate, which is closely related to the survival function $S(t)$, and is the instantaneous rate of an event occuring at time $t$ given that the event has not yet occured.\n\\begin{split}\\begin{align*} \\lambda(t) \u0026amp; = \\lim_{\\Delta t \\to 0} \\frac{P(t \u0026lt; T \u0026lt; t + \\Delta t\\ |\\ T \u0026gt; t) / \\Delta t}{Pr(T \u0026gt; t)} \\ \u0026amp; = \\lim_{{\\Delta t \\to 0}} \\frac{Pr(t \u0026lt; T \\leq t + \\Delta t) / \\Delta t}{Pr(T \u0026gt; t)} \\ \u0026amp; = \\frac{f(t)}{S(t)} \\end{align*}\\end{split}\nwhere\n$$f(t) = \\lim_{{\\Delta t \\to 0}} \\frac{Pr(t \u0026lt; T \\leq t + \\Delta t)}{\\Delta t}$$\nwhere $T$ is the (unobserved) survival time and $f(t)$ is the PDF associated with $T$. The relationship between the hazard function and the survival function can be described in terms of the likelihood $L$\n\\begin{equation} L_i = \\begin{cases} f(y_i) \u0026amp; \\text{if the } i\\text{th observation is not censored} \\ S(y_i) \u0026amp; \\text{if the } i\\text{th observation is censored} \\end{cases} \\end{equation}\nIf $Y = y_i$ and the $i\\text{th}$ observation is not censored, then the likelihood is the probability of the event in a tiny interval around time $y_i$. If the $i\\text{th}$ observation is censored, then the likelihood is the probability of surviving at least until time $y_i$. We have now seen two ways to model the survival times: (1) a non-parametric estimator such as Kaplan-Meier, and (2) a parametric model using the PDF $f(t)$ to estimate the hazard rate. However, what we would really like to do is to model the survival time as a function of the predictors. Thus, instead of working with the PDF $f(t)$, we work directly with the hazard function to model the survival time as a function of predictors.\nThe Cox proportional hazards model Above, we developed a model with no predictors to recover the parameters of the cat adoption rate, and used this as input into the pm.Censored distribution. Since we would now like to add predictor(s), we need to reformulate our modeling task into a risk regression model as it allows us to model the hazard rate as a function of our predictors. Specifically, the Cox proportional hazards model. With predictors $x_j$ and regression coefficients $\\beta$, the hazard rate is modeled as\n$$\\lambda(t|x_i) = \\lambda_0(t)\\text{exp}(\\sum_{j=1}^p x_{ij}\\beta_j)$$\nwhere $\\lambda_{0}t$ is the baseline hazard rate independent of the predictors. This baseline hazard rate is unspecified (or unidentified) and means that we allow the instantaneous probability of an event at time $t$, given that a subject has survived at least until time $t$, to take any form. Practically speaking, this means that the hazard function is very flexible and can model a wide range of relationships between the covariates and survival time. One can interpret the Cox proportional hazards model as a one-unit increase in $x_{ij}$ corresponds to an increase in $\\lambda(t, x_i)$ by a factor of $\\text{exp}(\\beta_j)$. In the next section, it is discussed how to develop a regression model with survival responses and predictors.\nImplementation in Bambi Adding predictors to model the hazard rate as a function of our predictors is trivial in Bambi. We simply continue to use the formula syntax. In the backend, the rate is modeled as a function of the specified predictors in the Bambi model. For example, if in the Bambi model, we specified censored(y, event) ~ 1 + x with an exponential likelihood, then the latent rate $\\lambda$ is modeled as an exponential distribution according to\n$$\\alpha \\sim \\mathcal{N}(0, 1)$$ $$\\beta \\sim \\mathcal{N}(0, 1)$$ $$\\mu = \\text{exp}(\\alpha + \\beta X)$$ $$\\lambda = 1 / \\mu$$ $$Y \\sim \\text{Exponential}(\\lambda)$$\nwhere $Y$ is then passed to the dist argument of the pm.Censored distribution.\nCat adoption rates by color However, thanks to Bambi\u0026rsquo;s formula syntax, we can just include the predictors of interest. Below, color_id is added to model the survival probability of black and other colored cats.\ncat_model = bmb.Model( \u0026#34;censored(days_to_event / 31, adopt) ~ 0 + color_id\u0026#34;, data=cats, center_predictors=False, priors={\u0026#34;color_id\u0026#34;: bmb.Prior(\u0026#34;Normal\u0026#34;, mu=0, sigma=1)}, categorical=[\u0026#34;color_id\u0026#34;], family=\u0026#34;exponential\u0026#34;, link=\u0026#34;log\u0026#34; ) cat_model.build() cat_model.graph() /Users/gabestechschulte/miniforge3/envs/bambinos/lib/python3.11/site-packages/formulae/terms/variable.py:87: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead elif is_string_dtype(x) or is_categorical_dtype(x): cat_model Formula: censored(days_to_event / 31, adopt) ~ 0 + color_id Family: exponential Link: mu = log Observations: 22356 Priors: target = mu Common-level effects color_id ~ Normal(mu: 0.0, sigma: 1.0) ------ * To see a plot of the priors call the .plot_priors() method. * To see a summary or plot of the posterior pass the object returned by .fit() to az.summary() or az.plot_trace() idata = cat_model.fit( tune=500, draws=500, random_seed=42, chains=4, cores=10 ) Auto-assigning NUTS sampler... Initializing NUTS using jitter+adapt_diag... Multiprocess sampling (4 chains in 10 jobs) NUTS: [color_id] Sampling 4 chains for 500 tune and 500 draw iterations (2_000 + 2_000 draws total) took 368 seconds. az.summary(idata) /Users/gabestechschulte/miniforge3/envs/bambinos/lib/python3.11/site-packages/xarray/core/concat.py:546: FutureWarning: unique with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version. common_dims = tuple(pd.unique([d for v in vars for d in v.dims])) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The summary output informs us that, on average, the rate parameter for other cats color_id[0] is lower than the rate for black cats color_id[1]. As performed above, lets plot the survival curves and CDFs for black and other colored cats to get a better understanding of the rate parameters.\nCat adoption plots In the inference data, we have posterior draws for color_id (with corresponding coordinates for other and black cats) where the values represent the sampled rates. However, it is also possible to obtain $Y$, in this example, months to event (as our data has been scaled) by calling model.predict() on the observed or new data. This will add a new data variable censored(y, event) to the posterior group of the inference data.\nnew_data = pd.DataFrame({\u0026#34;color_id\u0026#34;: [0, 1]}) cat_model.predict(idata, data=new_data, kind=\u0026#34;mean\u0026#34;) /Users/gabestechschulte/miniforge3/envs/bambinos/lib/python3.11/site-packages/xarray/core/concat.py:546: FutureWarning: unique with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version. common_dims = tuple(pd.unique([d for v in vars for d in v.dims])) other_cats = (idata[\u0026#34;posterior\u0026#34;][\u0026#34;censored(days_to_event / 31, adopt)_mean\u0026#34;] .sel({\u0026#34;censored(days_to_event / 31, adopt)_obs\u0026#34;: 0}) .values .flatten() ) other_cats_preds = np.quantile(other_cats, [0.025, 0.5, 0.975]) black_cats = (idata[\u0026#34;posterior\u0026#34;][\u0026#34;censored(days_to_event / 31, adopt)_mean\u0026#34;] .sel({\u0026#34;censored(days_to_event / 31, adopt)_obs\u0026#34;: 1}) .values. flatten() ) black_cats_preds = np.quantile(black_cats, [0.025, 0.5, 0.975]) lambdas = { \u0026#34;Other cats\u0026#34;: 1 / other_cats_preds, \u0026#34;Black cats\u0026#34;: 1 / black_cats_preds } fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10, 3), sharey=True) for key, value in lambdas.items(): lower, median, upper = value ax[0].plot(t, S0(median * t), label=f\u0026#34;{key}\u0026#34;) ax[0].fill_between(t, S0(lower * t), S0(upper * t), alpha=0.25) ax[0].grid(True) ax[0].set_xlim(0, 10) ax[0].legend() ax[0].set_title(\u0026#34;Probability Not Being Adopted by Time $d_k$\u0026#34;) for key, value in lambdas.items(): lower, median, upper = value ax[1].plot(t, cdf(median * t), label=f\u0026#34;{key}\u0026#34;) ax[1].fill_between(t, cdf(lower * t), cdf(upper * t), alpha=0.25) ax[1].grid(True) ax[1].set_xlim(0, 10) ax[1].legend() ax[1].set_title(\u0026#34;Probability of Being Adopted by Time $d_k$\u0026#34;); Analyzing the CCDF (left plot), we can see that black cats have a slightly higher probability of not being adopted throughout the whole range of $k$. Furthermore, analyzing the CDF (right plot), we can see it also takes a longer time for the majority of black cats to be adopted compared to other colored cats. Below, we plot the distribution of days until adoption for the two groups.\nplt.figure(figsize=(7, 3)) plt.hist( other_cats * 31, bins=50, density=True, label=\u0026#34;Other cats\u0026#34; ) plt.hist( black_cats * 31, bins=50, density=True, label=\u0026#34;Black cats\u0026#34; ) plt.legend() plt.xlabel(\u0026#34;Days\u0026#34;) plt.ylabel(\u0026#34;Density\u0026#34;) plt.title(\u0026#34;Distribution of Adoption Times\u0026#34;); Scaling adoption times back to days (multiplying by 31), we can see that black cats have longer and a wider range of time until adoptions (about 55 days) than cats that are not black (about 51 days).\nSummary In this notebook, we introduced censored data, left and right censoring, and why such data lends itself to specialized statistical methods and models. First, the non-parametric Kaplan-Meier estimator to estimate the survival curve of censored data was introduced. Subsequently, motivated by modeling the survival function as a function of predictors, the hazards rate and Cox proportional hazards model was introduced. Modeling censored data in Bambi requires defining the response as censored(y, event) where event is left or right censoring. To add predictors to the model, simply include them in the formula. Bambi leverages the pm.Censored distribution from PyMC as the likelihood for censored data.\n%load_ext watermark %watermark -n -u -v -iv -w The watermark extension is already loaded. To reload it, use: %reload_ext watermark Last updated: Mon Oct 23 2023 Python implementation: CPython Python version : 3.11.0 IPython version : 8.13.2 pymc : 5.8.1 arviz : 0.16.1 pandas : 2.1.0 bambi : 0.13.0.dev0 matplotlib: 3.7.1 numpy : 1.24.2 scipy : 1.11.2 Watermark: 2.3.1 ","permalink":"https://gstechschulte.github.io/posts/bambi-survival-models/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003ch1 id=\"survival-models\"\u003eSurvival Models\u003c/h1\u003e\n\u003cp\u003eSurvival models, also known as time-to-event models, are specialized statistical methods designed to analyze the time until the occurrence of an event of interest. In this notebook, a review of survival analysis (using non-parametric and parametric methods) and censored data is provided, followed by a survival model implementation in Bambi.\u003c/p\u003e\n\u003cp\u003eThis blog post is a copy of the survival models documentation I wrote for \u003ca href=\"https://bambinos.github.io/bambi/\"\u003eBambi\u003c/a\u003e. The original post can be found \u003ca href=\"https://bambinos.github.io/bambi/notebooks/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e","title":"Survival Models in Bambi"},{"content":" Predict New Groups In Bambi, it is possible to perform predictions on new, unseen, groups of data that were not in the observed data used to fit the model with the argument sample_new_groups in the model.predict() method. This is useful in the context of hierarchical modeling, where groups are assumed to be a sample from a larger group.\nThis blog post is a copy of the zero inflated models documentation I wrote for Bambi. The original post can be found here.\nBelow, it is first described how predictions at multiple levels and for unseen groups are possible with hierarchical models. Then, it is described how this is performed in Bambi. Lastly, a hierarchical model is developed to show how to use the sample_new_groups argument in the model.predict() method, and within the interpret sub-package. For users coming from brms in R, this is equivalent to the sample_new_levels argument.\nHierarchical models and predictions at multiple levels A feature of hierarchical models is that they are able to make predictions at multiple levels. For example, if we were to use the penguin dataset to fit a hierchical regression to estimate the body mass of each penguin species given a set of predictors, we could estimate the mass of all penguins and each individual species at the same time. Thus, in this example, there are predictions for two levels: (1) the population level, and (2) the species level.\nAdditionally, a hierarchical model can be used to make predictions for groups (levels) that were never seen before if a hyperprior is defined over the group-specific effect. With a hyperior defined on group-specific effects, the groups do not share one fixed parameter, but rather share a hyperprior distribution which describes the distribution for the parameter of the prior itself. Lets write a hierarchical model (without intercepts) with a hyperprior defined for group-specific effects in statistical notation so this concept becomes more clear:\n$$\\beta_{\\mu h} \\sim \\mathcal{N}(0, 10)$$ $$\\beta_{\\sigma h} \\sim \\mathcal{HN}(10)$$ $$\\beta_{m} \\sim \\mathcal{N}(\\beta_{\\mu h}, \\beta_{\\sigma h})$$ $$\\sigma_{h} \\sim \\mathcal{HN}(10)$$ $$\\sigma_{m} \\sim \\mathcal{HN}(\\sigma_{h})$$ $$Y \\sim \\mathcal{N}(\\beta_{m} * X_{m}, \\sigma_{m})$$\nThe parameters $\\beta_{\\mu h}, \\beta_{\\sigma h}$ of the group-specific effect prior $\\beta_{m}$ come from hyperprior distributions. Thus, if we would like to make predictions for a new, unseen, group, we can do so by first sampling from these hyperprior distributions to obtain the parameters for the new group, and then sample from the posterior or posterior predictive distribution to obtain the estimates for the new group. For a more in depth explanation of hierarchical models in Bambi, see either: the radon example, or the sleep study example.\nSampling new groups in Bambi If data with unseen groups are passed to the new_data argument of the model.predict() method, Bambi first needs to identify if that group exists, and if not, to evaluate the new group with the respective group-specific term. This evaluation updates the design matrix initially used to fit the model with the new group(s). This is achieved with the .evaluate_new_data method in the formulae package.\nOnce the design matrix has been updated, Bambi can perform predictions on the new, unseen, groups by specifying sample_new_groups=True in model.predict(). Each posterior sample for the new groups is drawn from the posterior draws of a randomly selected existing group. Since different groups may be selected at each draw, the end result represents the variation across existing groups.\nHierarchical regression To demonstrate the sample_new_groups argument, we will develop a hierarchical model on the OSIC Pulmonary Fibrosis Progression dataset. Pulmonary fibrosis is a disorder with no known cause and no known cure, created by scarring of the lungs. Using a hierarchical model, the objective is to predict a patient’s severity of decline in lung function. Lung function is assessed based on output from a spirometer, which measures the forced vital capacity (FVC), i.e. the volume of air exhaled by the patient.\n#| code-fold: true import arviz as az import matplotlib.pyplot as plt import numpy as np import pandas as pd import warnings import bambi as bmb warnings.simplefilter(action=\u0026#34;ignore\u0026#34;, category=FutureWarning) The OSIC pulmonary fibrosis progression dataset In the dataset, we were provided with a baseline chest computerized tomography (CT) scan and associated clinical information for a set of patients where the columns represent the following\npatient- a unique id for each patient weeks- the relative number of weeks pre/post the baseline CT (may be negative) fvc - the recorded lung capacity in millilitres (ml) percent- a computed field which approximates the patient\u0026rsquo;s FVC as a percent of the typical FVC for a person of similar characteristics sex - male or female smoking_status - ex-smoker, never smoked, currently smokes age - age of the patient A patient has an image acquired at time week = 0 and has numerous follow up visits over the course of approximately 1-2 years, at which time their FVC is measured. Below, we randomly sample three patients and plot their FVC measurements over time.\ndata = pd.read_csv( \u0026#34;https://gist.githubusercontent.com/ucals/\u0026#34; \u0026#34;2cf9d101992cb1b78c2cdd6e3bac6a4b/raw/\u0026#34; \u0026#34;43034c39052dcf97d4b894d2ec1bc3f90f3623d9/\u0026#34; \u0026#34;osic_pulmonary_fibrosis.csv\u0026#34; ) data.columns = data.columns.str.lower() data.columns = data.columns.str.replace(\u0026#34;smokingstatus\u0026#34;, \u0026#34;smoking_status\u0026#34;) data .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } def label_encoder(labels): \u0026#34;\u0026#34;\u0026#34; Encode patient IDs as integers. \u0026#34;\u0026#34;\u0026#34; unique_labels = np.unique(labels) label_to_index = {label: index for index, label in enumerate(unique_labels)} encoded_labels = labels.map(label_to_index) return encoded_labels predictors = [\u0026#34;patient\u0026#34;, \u0026#34;weeks\u0026#34;, \u0026#34;fvc\u0026#34;, \u0026#34;smoking_status\u0026#34;] data[\u0026#34;patient\u0026#34;] = label_encoder(data[\u0026#39;patient\u0026#39;]) data[\u0026#34;weeks\u0026#34;] = (data[\u0026#34;weeks\u0026#34;] - data[\u0026#34;weeks\u0026#34;].min()) / ( data[\u0026#34;weeks\u0026#34;].max() - data[\u0026#34;weeks\u0026#34;].min() ) data[\u0026#34;fvc\u0026#34;] = (data[\u0026#34;fvc\u0026#34;] - data[\u0026#34;fvc\u0026#34;].min()) / ( data[\u0026#34;fvc\u0026#34;].max() - data[\u0026#34;fvc\u0026#34;].min() ) data = data[predictors] patient_id = data.sample(n=3, random_state=42)[\u0026#34;patient\u0026#34;].values fig, ax = plt.subplots(1, 3, figsize=(12, 3), sharey=True) for i, p in enumerate(patient_id): patient_data = data[data[\u0026#34;patient\u0026#34;] == p] ax[i].scatter(patient_data[\u0026#34;weeks\u0026#34;], patient_data[\u0026#34;fvc\u0026#34;]) ax[i].set_xlabel(\u0026#34;weeks\u0026#34;) ax[i].set_ylabel(\u0026#34;fvc\u0026#34;) ax[i].set_title(f\u0026#34;patient {p}\u0026#34;) plt.tight_layout() The plots show variability in FVC measurements, unequal time intervals between follow up visits, and different number of visits per patient. This is a good scenario to use a hierarchical model, where we can model the FVC measurements for each patient as a function of time, and also model the variability in the FVC measurements across patients.\nPartial pooling model The hierarchical model we will develop is a partially pooled model using the predictors weeks, smoking_status, and patient to predict the response fvc. We will estimate the following model with common and group-effects:\ncommon-effects: weeks and smoking_status group-effects: the slope of weeks will vary by patient Additionally, the global intercept is not included. Since the global intercept is excluded, smoking_status uses cell means encoding (i.e. the coefficient represents the estimate for each smoking_status category of the entire group). This logic also applies for weeks. However, a group-effect is also specified for weeks, which means that the association between weeks and the fvc is allowed to vary by individual patients.\nBelow, the default prior for the group-effect sigma is changed from HalfNormal to a Gamma distribution. Additionally, the model graph shows the model has been reparameterized to be non-centered. This is the default when there are group-effects in Bambi.\npriors = { \u0026#34;weeks|patient\u0026#34;: bmb.Prior(\u0026#34;Normal\u0026#34;, mu=0, sigma=bmb.Prior(\u0026#34;Gamma\u0026#34;, alpha=3, beta=3)), } model = bmb.Model( \u0026#34;fvc ~ 0 + weeks + smoking_status + (0 + weeks | patient)\u0026#34;, data, priors=priors, categorical=[\u0026#34;patient\u0026#34;, \u0026#34;smoking_status\u0026#34;], ) model.build() model.graph() idata = model.fit( draws=1500, tune=1000, target_accept=0.95, chains=4, random_seed=42, cores=10, ) Model criticism Hierarchical models can induce difficult posterior geometries to sample from. Below, we quickly analyze the traces to ensure sampling went well.\naz.plot_trace(idata) plt.tight_layout(); Analyzing the marginal posteriors of weeks and weeks|patient, we see that the slope can be very different for some individuals. weeks indicates that as a population, the slope is negative. However, weeks|patients indicates some patients are negative, some are positive, and some are close to zero. Moreover, there are varying levels of uncertainty observed in the coefficients for the three different values of the smoking_status variable.\naz.summary(idata, var_names=[\u0026#34;weeks\u0026#34;, \u0026#34;smoking_status\u0026#34;, \u0026#34;fvc_sigma\u0026#34;, \u0026#34;weeks|patient_sigma\u0026#34;]) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The effective sample size (ESS) is much lower for the weeks and weeks|patient_sigma parameters. This can also be inferred visually by looking at the trace plots for these parameters above. There seems to be some autocorrelation in the samples for these parameters. However, for the sake of this example, we will not worry about this.\nPredict observed patients First, we will use the posterior distribution to plot the mean and 95% credible interval for the FVC measurements of the three randomly sampled patients above.\npreds = model.predict(idata, kind=\u0026#34;mean\u0026#34;, inplace=False) fvc_mean = az.extract(preds[\u0026#34;posterior\u0026#34;])[\u0026#34;fvc_mean\u0026#34;] # plot posterior predictions fig, ax = plt.subplots(1, 3, figsize=(12, 3), sharey=True) for i, p in enumerate(patient_id): idx = data.index[data[\u0026#34;patient\u0026#34;] == p].tolist() weeks = data.loc[idx, \u0026#34;weeks\u0026#34;].values fvc = data.loc[idx, \u0026#34;fvc\u0026#34;].values ax[i].scatter(weeks, fvc) az.plot_hdi(weeks, fvc_mean[idx].T, color=\u0026#34;C0\u0026#34;, ax=ax[i]) ax[i].plot(weeks, fvc_mean[idx].mean(axis=1), color=\u0026#34;C0\u0026#34;) ax[i].set_xlabel(\u0026#34;weeks\u0026#34;) ax[i].set_ylabel(\u0026#34;fvc\u0026#34;) ax[i].set_title(f\u0026#34;patient {p}\u0026#34;) plt.tight_layout() The plots show that the posterior estimates seem to fit the three patients well. Where there are more observations, the credible interval is smaller, and where there are fewer observations, the credible interval is larger. Next, we will predict new, unseen, patients.\nPredict new patients Imagine the cost of acquiring a CT scan increases dramatically, and we would like to interopolate the FVC measurement for a new patient with a given set of clinical information smoking_status and weeks. We achieve this by passing this data to the predict method and setting sample_new_groups=True. As outlined in the Sampling new groups in Bambi section, this new data is evaluated by formulae to update the design matrix, and then predictions are made for the new group by sampling from the posterior draws of a randomly selected existing group.\nBelow, we will simulate a new patient and predict their FVC measurements over time. First, we will copy clinical data from patient 39 and use it for patient 176 (the new, unseen, patient). Subsequently, we will construct another new patient, with different clinical data.\n# copy patient 39 data to the new patient 176 patient_39 = data[data[\u0026#34;patient\u0026#34;] == 39].reset_index(drop=True) new_data = patient_39.copy() new_data[\u0026#34;patient\u0026#34;] = 176 new_data = pd.concat([new_data, patient_39]).reset_index(drop=True)[predictors] new_data .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } preds = model.predict( idata, kind=\u0026#34;mean\u0026#34;, data=new_data, sample_new_groups=True, inplace=False ) # utility func for plotting def plot_new_patient(idata, data, patient_ids): fvc_mean = az.extract(idata[\u0026#34;posterior\u0026#34;])[\u0026#34;fvc_mean\u0026#34;] fig, ax = plt.subplots(1, 2, figsize=(10, 3), sharey=True) for i, p in enumerate(patient_ids): idx = data.index[data[\u0026#34;patient\u0026#34;] == p].tolist() weeks = data.loc[idx, \u0026#34;weeks\u0026#34;].values fvc = data.loc[idx, \u0026#34;fvc\u0026#34;].values if p == patient_ids[0]: ax[i].scatter(weeks, fvc) az.plot_hdi(weeks, fvc_mean[idx].T, color=\u0026#34;C0\u0026#34;, ax=ax[i]) ax[i].plot(weeks, fvc_mean[idx].mean(axis=1), color=\u0026#34;C0\u0026#34;) ax[i].set_xlabel(\u0026#34;weeks\u0026#34;) ax[i].set_ylabel(\u0026#34;fvc\u0026#34;) ax[i].set_title(f\u0026#34;patient {p}\u0026#34;) plot_new_patient(preds, new_data, [39, 176]) Although identical data was used for both patients, the variability increased consideribly for patient 176. However, the mean predictions for both patients appear to be almost identical. Now, lets construct a new patient with different clinical data and see how the predictions change. We will select 10 time of follow up visits at random, and set the smoking_status = \u0026quot;Currently smokes\u0026quot;.\nnew_data.loc[new_data[\u0026#34;patient\u0026#34;] == 176, \u0026#34;smoking_status\u0026#34;] = \u0026#34;Currently smokes\u0026#34; weeks = np.random.choice(sorted(model.data.weeks.unique()), size=10) new_data.loc[new_data[\u0026#34;patient\u0026#34;] == 176, \u0026#34;weeks\u0026#34;] = weeks new_data .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } If we were to keep the default value of sample_new_groups=False, the following error would be raised: ValueError: There are new groups for the factors ('patient',) and 'sample_new_groups' is False. Thus, we set sample_new_groups=True and obtain predictions for the new patient.\npreds = model.predict( idata, kind=\u0026#34;mean\u0026#34;, data=new_data, sample_new_groups=True, inplace=False ) plot_new_patient(preds, new_data, [39, 176]) With smoking_status = \u0026quot;Currently smokes\u0026quot;, and the time of follow up visit randomly selected, we can see that the intercept is slightly higher, and it appears that the slope is steeper for this new patient. Again, the variability is much higher for patient 176, and in particular, where there are fewer fvc measurements.\nPredict new patients with interpret The interpret sub-package in Bambi allows us to easily interpret the predictions for new patients. In particular, using bmb.interpret.comparisons, we can compare the predictions made for a new patient and an existing similar patient. Below, we will compare the predictions made for patient 176 and patient 39. We will use the same clinical data for both patients as we did in the first exampe above.\ntime_of_follow_up = list(new_data.query(\u0026#34;patient == 39\u0026#34;)[\u0026#34;weeks\u0026#34;].values) time_of_follow_up [0.35507246376811596, 0.37681159420289856, 0.391304347826087, 0.4057971014492754, 0.42028985507246375, 0.45652173913043476, 0.5434782608695652, 0.6376811594202898, 0.7463768115942029, 0.7753623188405797] fig, ax = bmb.interpret.plot_comparisons( model, idata, contrast={\u0026#34;patient\u0026#34;: [39, 176]}, conditional={\u0026#34;weeks\u0026#34;: time_of_follow_up, \u0026#34;smoking_status\u0026#34;: \u0026#34;Ex-smoker\u0026#34;}, sample_new_groups=True, fig_kwargs={\u0026#34;figsize\u0026#34;: (7, 3)} ) plt.title(\u0026#34;Difference in predictions for patient 176 vs 39\u0026#34;); Referring to the plots where patient 39 and 176 use identical data, the mean fvc predictions \u0026ldquo;look\u0026rdquo; about the same. When this comparison is made quantitatively using the comparisons function, we can see that mean fvc measurements are slightly below 0.0, and have a constant slope across weeks indicating there is a slight difference in mean fvc measurements between the two patients.\nSummary In this notebook, it was shown how predictions at multiple levels and for unseen groups are possible with hierarchical models. To utilize this feature of hierarchical models, Bambi first updates the design matrix to include the new group. Then, predictions are made for the new group by sampling from the posterior draws of a randomly selected existing group.\nTo predict new groups in Bambi, you can either: (1) create a dataset with new groups and pass it to the model.predict() method while specifying sample_new_groups=True, or (2) use the functions comparisons or slopes in the interpret sub-package with sample_new_groups=True to compare predictions or slopes for new groups and existing groups.\n%load_ext watermark %watermark -n -u -v -iv -w Last updated: Tue Oct 10 2023 Python implementation: CPython Python version : 3.11.0 IPython version : 8.13.2 matplotlib: 3.7.1 arviz : 0.16.1 pandas : 2.1.0 numpy : 1.24.2 bambi : 0.13.0.dev0 Watermark: 2.3.1 ","permalink":"https://gstechschulte.github.io/posts/bambi-predict-new-groups/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003ch1 id=\"predict-new-groups\"\u003ePredict New Groups\u003c/h1\u003e\n\u003cp\u003eIn Bambi, it is possible to perform predictions on new, unseen, groups of data that were not in the observed data used to fit the model with the argument \u003ccode\u003esample_new_groups\u003c/code\u003e in the \u003ccode\u003emodel.predict()\u003c/code\u003e method. This is useful in the context of hierarchical modeling, where groups are assumed to be a sample from a larger group.\u003c/p\u003e\n\u003cp\u003eThis blog post is a copy of the zero inflated models documentation I wrote for \u003ca href=\"https://bambinos.github.io/bambi/\"\u003eBambi\u003c/a\u003e. The original post can be found \u003ca href=\"https://bambinos.github.io/bambi/notebooks/predict_new_groups.html\"\u003ehere\u003c/a\u003e.\u003c/p\u003e","title":"Predict New Groups with Hierarchical Models in Bambi"},{"content":" #| code-fold: true import arviz as az import matplotlib.pyplot as plt from matplotlib.lines import Line2D import numpy as np import pandas as pd import warnings import bambi as bmb warnings.filterwarnings(\u0026#34;ignore\u0026#34;, category=FutureWarning) WARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions. Ordinal Regression This blog post is a copy of the ordinal models documentation I wrote for Bambi. The original post can be found here.\nIn some scenarios, the response variable is discrete, like a count, and ordered. Common examples of such data come from questionnaires where the respondent is asked to rate a product, service, or experience on a scale. This scale is often referred to as a Likert scale. For example, a five-level Likert scale could be:\n1 = Strongly disagree 2 = Disagree 3 = Neither agree nor disagree 4 = Agree 5 = Strongly agree The result is a set of ordered categories where each category has an associated numeric value (1-5). However, you can\u0026rsquo;t compute a meaningful difference between the categories. Moreover, the response variable can also be a count where meaningful differences can be computed. For example, a restaurant can be rated on a scale of 1-5 stars where 1 is the worst and 5 is the best. Yes, you can compute the difference between 1 and 2 stars, but it is often treated as ordinal in an applied setting.\nOrdinal data presents three challenges when modelling:\nUnlike a count, the differences in the values are not necessarily equidistant or meaningful. For example, computing the difference between \u0026ldquo;Strongly disagree\u0026rdquo; and \u0026ldquo;Disagree\u0026rdquo;. Or, in the case of the restaurant rating, it may be much harder for a restuarant to go from 4 to 5 stars than from 2 to 3 stars. The distribution of ordinal responses may be nonnormal as the response is not continuous; particularly if larger response levels are infrequently chosen compared to lower ones. The variances of the unobserved variables that underlie the observed ordered category may differ between the category, time points, etc. Thus, treating ordered categories as continuous is not appropriate. To this extent, Bambi supports two classes of ordinal regression models: (1) cumulative, and (2) sequential. Below, it is demonstrated how to fit these two models using Bambi to overcome the challenges of ordered category response data.\nCumulative model A cumulative model assumes that the observed ordinal variable $Y$ originates from the \u0026ldquo;categorization\u0026rdquo; of a latent continuous variable $Z$. To model the categorization process, the model assumes that there are $K$ thresholds (or cutpoints) $\\tau_k$ that partition $Z$ into $K+1$ observable, ordered categories of $Y$. The subscript $k$ in $\\tau_k$ is an index that associates that threshold to a particular category $k$. For example, if the response has three categories such as \u0026ldquo;disagree\u0026rdquo;, \u0026ldquo;neither agree nor disagree\u0026rdquo;, and \u0026ldquo;agree\u0026rdquo;, then there are two thresholds $\\tau_1$ and $\\tau_2$ that partition $Z$ into $K+1 = 3$ categories. Additionally, if we assume $Z$ to have a certain distribution (e.g., Normal) with a cumulative distribution function $F$, the probability of $Y$ being equal to category $k$ is\n$$P(Y = k) = F(\\tau_k) - F(\\tau_{k-1})$$\nwhere $F(\\tau)$ is a cumulative probability. For example, suppose we are interested in the probability of each category stated above, and have two thresholds $\\tau_1 = -1, \\tau_2 = 1$ for the three categories. Additionally, if we assume $Z$ to be normally distributed with $\\sigma = 1$ and a cumulative distribution function $\\Phi$ then\n$$P(Y = 1) = \\Phi(\\tau_1) = \\Phi(-1)$$\n$$P(Y = 2) = \\Phi(\\tau_2) - \\Phi(\\tau_1) = \\Phi(1) - \\Phi(-1)$$\n$$P(Y = 3) = 1 - \\Phi(\\tau_2) = 1 - \\Phi(1)$$\nBut how to set the values of the thresholds? By default, Bambi uses a Normal distribution with a grid of evenly spaced $\\mu$ that depends on the number of response levels as the prior for the thresholds. Additionally, since the thresholds need to be orderd, Bambi applies a transformation to the values such that the order is preserved. Furthermore, the model specification for ordinal regression typically transforms the cumulative probabilities using the log-cumulative-odds (logit) transformation. Therefore, the learned parameters for the thresholds $\\tau$ will be logits.\nLastly, as each $F(\\tau)$ implies a cumulative probability for each category, the largest response level always has a cumulative probability of 1. Thus, we effectively do not need a parameter for it due to the law of total probability. For example, for three response values, we only need two thresholds as two thresholds partition $Z$ into $K+1$ categories.\nThe moral intuition dataset To illustrate an cumulative ordinal model, we will model data from a series of experiments conducted by philsophers (this example comes from Richard McElreath\u0026rsquo;s Statistical Rethinking). The experiments aim to collect empirical evidence relevant to debates about moral intuition, the forms of reasoning through which people develop judgments about the moral goodness and badness of actions.\nIn the dataset there are 12 columns and 9930 rows, comprising data for 331 unique individuals. The response we are interested in response, is an integer from 1 to 7 indicating how morally permissible the participant found the action to be taken (or not) in the story. The predictors are as follows:\naction: a factor with levels 0 and 1 where 1 indicates that the story contained \u0026ldquo;harm caused by action is morally worse than equivalent harm caused by omission\u0026rdquo;. intention: a factor with levels 0 and 1 where 1 indicates that the story contained \u0026ldquo;harm intended as the means to a goal is morally worse than equivalent harm foreseen as the side effect of a goal\u0026rdquo;. contact: a factor with levels 0 and 1 where 1 indicates that the story contained \u0026ldquo;using physical contact to cause harm to a victim is morally worse than causing equivalent harm to a victim without using physical contact\u0026rdquo;. trolly = pd.read_csv(\u0026#34;https://raw.githubusercontent.com/rmcelreath/rethinking/master/data/Trolley.csv\u0026#34;, sep=\u0026#34;;\u0026#34;) trolly = trolly[[\u0026#34;response\u0026#34;, \u0026#34;action\u0026#34;, \u0026#34;intention\u0026#34;, \u0026#34;contact\u0026#34;]] trolly[\u0026#34;action\u0026#34;] = pd.Categorical(trolly[\u0026#34;action\u0026#34;], ordered=False) trolly[\u0026#34;intention\u0026#34;] = pd.Categorical(trolly[\u0026#34;intention\u0026#34;], ordered=False) trolly[\u0026#34;contact\u0026#34;] = pd.Categorical(trolly[\u0026#34;contact\u0026#34;], ordered=False) trolly[\u0026#34;response\u0026#34;] = pd.Categorical(trolly[\u0026#34;response\u0026#34;], ordered=True) # 7 ordered categories from 1-7 trolly.response.unique() [4, 3, 5, 2, 1, 7, 6] Categories (7, int64): [1 \u0026lt; 2 \u0026lt; 3 \u0026lt; 4 \u0026lt; 5 \u0026lt; 6 \u0026lt; 7] Intercept only model Before we fit a model with predictors, let\u0026rsquo;s attempt to recover the parameters of an ordinal model using only the thresholds to get a feel for the cumulative family. Traditionally, in Bambi if we wanted to recover the parameters of the likelihood, we would use an intercept only model and write the formula as response ~ 1 where 1 indicates to include the intercept. However, in the case of ordinal regression, the thresholds \u0026ldquo;take the place\u0026rdquo; of the intercept. Thus, we can write the formula as response ~ 0 to indicate that we do not want to include an intercept. To fit a cumulative ordinal model, we pass family=\u0026quot;cumulative\u0026quot;. To compare the thresholds only model, we compute the empirical log-cumulative-odds of the categories directly from the data below and generate a bar plot of the response probabilities.\npr_k = trolly.response.value_counts().sort_index().values / trolly.shape[0] cum_pr_k = np.cumsum(pr_k) logit_func = lambda x: np.log(x / (1 - x)) cum_logit = logit_func(cum_pr_k) cum_logit /var/folders/rl/y69t95y51g90tvd6gjzzs59h0000gn/T/ipykernel_22293/1548491577.py:3: RuntimeWarning: invalid value encountered in log logit_func = lambda x: np.log(x / (1 - x)) array([-1.91609116, -1.26660559, -0.718634 , 0.24778573, 0.88986365, 1.76938091, nan]) plt.figure(figsize=(7, 3)) plt.bar(np.arange(1, 8), pr_k) plt.ylabel(\u0026#34;Probability\u0026#34;) plt.xlabel(\u0026#34;Response\u0026#34;) plt.title(\u0026#34;Empirical probability of each response category\u0026#34;); model = bmb.Model(\u0026#34;response ~ 0\u0026#34;, data=trolly, family=\u0026#34;cumulative\u0026#34;) idata = model.fit(random_seed=1234) Below, the components of the model are outputed. Notice how the thresholds are a grid of six values ranging from -2 to 2.\nmodel Formula: response ~ 0 + action + intention + contact + action:intention + contact:intention Family: cumulative Link: p = logit Observations: 9930 Priors: target = p Common-level effects action ~ Normal(mu: 0.0, sigma: 5.045) intention ~ Normal(mu: 0.0, sigma: 5.0111) contact ~ Normal(mu: 0.0, sigma: 6.25) action:intention ~ Normal(mu: 0.0, sigma: 6.7082) contact:intention ~ Normal(mu: 0.0, sigma: 8.3333) Auxiliary parameters threshold ~ Normal(mu: [-2. -1.2 -0.4 0.4 1.2 2. ], sigma: 1.0, transform: ordered) ------ * To see a plot of the priors call the .plot_priors() method. * To see a summary or plot of the posterior pass the object returned by .fit() to az.summary() or az.plot_trace() az.summary(idata) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Viewing the summary dataframe, we see a total of six response_threshold coefficients. Why six? Remember, we get the last parameter for free. Since there are seven categories, we only need six cutpoints. The index (using zero based indexing) of the response_threshold indicates the category that the threshold is associated with. Comparing to the empirical log-cumulative-odds computation above, the mean of the posterior distribution for each category is close to the empirical value.\nAs the the log cumulative link is used, we need to apply the inverse of the logit function to transform back to cumulative probabilities. Below, we plot the cumulative probabilities for each category.\nexpit_func = lambda x: 1 / (1 + np.exp(-x)) cumprobs = expit_func(idata.posterior.response_threshold).mean((\u0026#34;chain\u0026#34;, \u0026#34;draw\u0026#34;)) cumprobs = np.append(cumprobs, 1) plt.figure(figsize=(7, 3)) plt.plot(sorted(trolly.response.unique()), cumprobs, marker=\u0026#39;o\u0026#39;) plt.ylabel(\u0026#34;Cumulative probability\u0026#34;) plt.xlabel(\u0026#34;Response category\u0026#34;) plt.title(\u0026#34;Cumulative probabilities of response categories\u0026#34;); fig, ax = plt.subplots(figsize=(7, 3)) for i in range(6): outcome = expit_func(idata.posterior.response_threshold).sel(response_threshold_dim=i).to_numpy().flatten() ax.hist(outcome, bins=15, alpha=0.5, label=f\u0026#34;Category: {i}\u0026#34;) ax.set_xlabel(\u0026#34;Probability\u0026#34;) ax.set_ylabel(\u0026#34;Count\u0026#34;) ax.set_title(\u0026#34;Cumulative Probability by Category\u0026#34;) ax.legend(bbox_to_anchor=(1.04, 1), loc=\u0026#34;upper left\u0026#34;); We can take the derivative of the cumulative probabilities to get the posterior probabilities for each category. Notice how the posterior probabilities in the barplot below are close to the empirical probabilities in barplot above.\n# derivative ddx = np.diff(cumprobs) probs = np.insert(ddx, 0, cumprobs[0]) plt.figure(figsize=(7, 3)) plt.bar(sorted(trolly.response.unique()), probs) plt.ylabel(\u0026#34;Probability\u0026#34;) plt.xlabel(\u0026#34;Response category\u0026#34;) plt.title(\u0026#34;Posterior Probability of each response category\u0026#34;); Notice in the plots above, the jump in probability from category 3 to 4. Additionally, the estimates of the coefficients is precise for each category. Now that we have an understanding how the cumulative link function is applied to produce ordered cumulative outcomes, we will add predictors to the model.\nAdding predictors In the cumulative model described above, adding predictors was explicitly left out. In this section, it is described how predictors are added to ordinal cumulative models. When adding predictor variables, what we would like is for any predictor, as it increases, predictions are moved progressively (increased) through the categories in sequence. A linear regression is formed for $Z$ by adding a predictor term $\\eta$\n$$\\eta = \\beta_1 x_1 + \\beta_2 x_2 +, . . ., \\beta_n x_n$$\nNotice how similar this looks to an ordinary linear model. However, there is no intercept or error term. This is because the intercept is replaced by the threshold $\\tau$ and the error term $\\epsilon$ is added seperately to obtain\n$$Z = \\eta + \\epsilon$$\nPutting the predictor term together with the thresholds and cumulative distribution function, we obtain the probability of $Y$ being equal to a category $k$ as\n$$Pr(Y = k | \\eta) = F(\\tau_k - \\eta) - F(\\tau_{k-1} - \\eta)$$\nThe same predictor term $\\eta$ is subtracted from each threshold because if we decrease the log-cumulative-odds of every outcome value $k$ below the maximum, this shifts probability mass upwards towards higher outcome values. Thus, positive $\\beta$ values correspond to increasing $x$, which is associated with an increase in the mean response $Y$. The parameters to be estimated from the model are the thresholds $\\tau$ and the predictor terms $\\eta$ coefficients.\nTo add predictors for ordinal models in Bambi, we continue to use the formula interface.\nmodel = bmb.Model( \u0026#34;response ~ 0 + action + intention + contact + action:intention + contact:intention\u0026#34;, data=trolly, family=\u0026#34;cumulative\u0026#34; ) idata = model.fit(random_seed=1234) In the summary dataframe below, we only select the predictor variables as the thresholds are not of interest at the moment.\nIn the summary dataframe below, we only select the predictor variables as the cutpoints are not of interest at the moment.\naz.summary( idata, var_names=[\u0026#34;action\u0026#34;, \u0026#34;intention\u0026#34;, \u0026#34;contact\u0026#34;, \u0026#34;action:intention\u0026#34;, \u0026#34;contact:intention\u0026#34;] ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The posterior distribution of the slopes are all negative indicating that each of these story features reduces the rating—the acceptability of the story. Below, a forest plot is used to make this insight more clear.\naz.plot_forest( idata, combined=True, var_names=[\u0026#34;action\u0026#34;, \u0026#34;intention\u0026#34;, \u0026#34;contact\u0026#34;, \u0026#34;action:intention\u0026#34;, \u0026#34;contact:intention\u0026#34;], figsize=(7, 3), textsize=11 ); Again, we can plot the cumulative probability of each category. Compared to the same plot above, notice how most of the category probabilities have been shifted to the left. Additionally, there is more uncertainty for category 3, 4, and 5.\nfig, ax = plt.subplots(figsize=(7, 3)) for i in range(6): outcome = expit_func(idata.posterior.response_threshold).sel(response_threshold_dim=i).to_numpy().flatten() ax.hist(outcome, bins=15, alpha=0.5, label=f\u0026#34;Category: {i}\u0026#34;) ax.set_xlabel(\u0026#34;Probability\u0026#34;) ax.set_ylabel(\u0026#34;Count\u0026#34;) ax.set_title(\u0026#34;Cumulative Probability by Category\u0026#34;) ax.legend(bbox_to_anchor=(1.04, 1), loc=\u0026#34;upper left\u0026#34;); Posterior predictive distribution To get a sense of how well the ordinal model fits the data, we can plot samples from the posterior predictive distribution. To plot the samples, a utility function is defined below to assist in the plotting of discrete values.\ndef adjust_lightness(color, amount=0.5): import matplotlib.colors as mc import colorsys try: c = mc.cnames[color] except: c = color c = colorsys.rgb_to_hls(*mc.to_rgb(c)) return colorsys.hls_to_rgb(c[0], c[1] * amount, c[2]) def plot_ppc_discrete(idata, bins, ax): def add_discrete_bands(x, lower, upper, ax, **kwargs): for i, (l, u) in enumerate(zip(lower, upper)): s = slice(i, i + 2) ax.fill_between(x[s], [l, l], [u, u], **kwargs) var_name = list(idata.observed_data.data_vars)[0] y_obs = idata.observed_data[var_name].to_numpy() counts_list = [] for draw_values in az.extract(idata, \u0026#34;posterior_predictive\u0026#34;)[var_name].to_numpy().T: counts, _ = np.histogram(draw_values, bins=bins) counts_list.append(counts) counts_arr = np.stack(counts_list) qts_90 = np.quantile(counts_arr, (0.05, 0.95), axis=0) qts_70 = np.quantile(counts_arr, (0.15, 0.85), axis=0) qts_50 = np.quantile(counts_arr, (0.25, 0.75), axis=0) qts_30 = np.quantile(counts_arr, (0.35, 0.65), axis=0) median = np.quantile(counts_arr, 0.5, axis=0) colors = [adjust_lightness(\u0026#34;C0\u0026#34;, x) for x in [1.8, 1.6, 1.4, 1.2, 0.9]] add_discrete_bands(bins, qts_90[0], qts_90[1], ax=ax, color=colors[0]) add_discrete_bands(bins, qts_70[0], qts_70[1], ax=ax, color=colors[1]) add_discrete_bands(bins, qts_50[0], qts_50[1], ax=ax, color=colors[2]) add_discrete_bands(bins, qts_30[0], qts_30[1], ax=ax, color=colors[3]) ax.step(bins[:-1], median, color=colors[4], lw=2, where=\u0026#34;post\u0026#34;) ax.hist(y_obs, bins=bins, histtype=\u0026#34;step\u0026#34;, lw=2, color=\u0026#34;black\u0026#34;, align=\u0026#34;mid\u0026#34;) handles = [ Line2D([], [], label=\u0026#34;Observed data\u0026#34;, color=\u0026#34;black\u0026#34;, lw=2), Line2D([], [], label=\u0026#34;Posterior predictive median\u0026#34;, color=colors[4], lw=2) ] ax.legend(handles=handles) return ax idata_pps = model.predict(idata=idata, kind=\u0026#34;pps\u0026#34;, inplace=False) bins = np.arange(7) fig, ax = plt.subplots(figsize=(7, 3)) ax = plot_ppc_discrete(idata_pps, bins, ax) ax.set_xlabel(\u0026#34;Response category\u0026#34;) ax.set_ylabel(\u0026#34;Count\u0026#34;) ax.set_title(\u0026#34;Cumulative model - Posterior Predictive Distribution\u0026#34;); Sequential Model For some ordinal variables, the assumption of a single underlying continuous variable (as in cumulative models) may not be appropriate. If the response can be understood as being the result of a sequential process, such that a higher response category is possible only after all lower categories are achieved, then a sequential model may be more appropriate than a cumulative model.\nSequential models assume that for every category $k$ there is a latent continuous variable $Z$ that determines the transition between categories $k$ and $k+1$. Now, a threshold $\\tau$ belongs to each latent process. If there are 3 categories, then there are 3 latent processes. If $Z_k$ is greater than the threshold $\\tau_k$, the sequential process continues, otherwise it stops at category $k$. As with the cumulative model, we assume a distribution for $Z_k$ with a cumulative distribution function $F$.\nAs an example, lets suppose we are interested in modeling the probability a boxer makes it to round 3. This implies that the particular boxer in question survived round 1 $Z_1 \u0026gt; \\tau_1$ , 2 $Z_2 \u0026gt; \\tau_2$, and 3 $Z_3 \u0026gt; \\tau_3$. This can be written as\n$$Pr(Y = 3) = (1 - P(Z_1 \\leq \\tau_1)) * (1 - P(Z_2 \\leq \\tau_2)) * P(Z_3 \\leq \\tau_3)$$\nAs in the cumulative model above, if we assume $Y$ to be normally distributed with the thresholds $\\tau_1 = -1, \\tau_2 = 0, \\tau_3 = 1$ and cumulative distribution function $\\Phi$ then\n$$Pr(Y = 3) = (1 - \\Phi(\\tau_1)) * (1 - \\Phi(\\tau_2)) * \\Phi(\\tau_3)$$\nTo add predictors to this sequential model, we follow the same specification in the Adding Predictors section above. Thus, the sequential model with predictor terms becomes\n$$P(Y = k) = F(\\tau_k - \\eta) * \\prod_{j=1}^{k-1}{(1 - F(\\tau_j - \\eta))}$$\nThus, the probability that $Y$ is equal to category $k$ is equal to the probability that it did not fall in one of the former categories $1: k-1$ multiplied by the probability that the sequential process stopped at $k$ rather than continuing past it.\nHuman resources attrition dataset To illustrate an sequential model with a stopping ratio link function, we will use data from the IBM human resources employee attrition and performance dataset. The original dataset contains 1470 rows and 35 columns. However, our goal is to model the total working years of employees using age as a predictor. This data lends itself to a sequential model as the response, total working years, is a sequential process. In order to have 10 years of working experience, it is necessarily true that the employee had 9 years of working experience. Additionally, age is choosen as a predictor as it is positively correlated with total working years.\nattrition = pd.read_csv(\u0026#34;data/hr_employee_attrition.tsv.txt\u0026#34;, sep=\u0026#34;\\t\u0026#34;) attrition = attrition[attrition[\u0026#34;Attrition\u0026#34;] == \u0026#34;No\u0026#34;] attrition[\u0026#34;YearsAtCompany\u0026#34;] = pd.Categorical(attrition[\u0026#34;YearsAtCompany\u0026#34;], ordered=True) attrition[[\u0026#34;YearsAtCompany\u0026#34;, \u0026#34;Age\u0026#34;]].head() .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Below, the empirical probabilities of the response categories are computed. Employees are most likely to stay at the company between 1 and 10 years.\npr_k = attrition.YearsAtCompany.value_counts().sort_index().values / attrition.shape[0] plt.figure(figsize=(7, 3)) plt.bar(np.arange(0, 36), pr_k) plt.xlabel(\u0026#34;Response category\u0026#34;) plt.ylabel(\u0026#34;Probability\u0026#34;) plt.title(\u0026#34;Empirical probability of each response category\u0026#34;); Default prior of thresholds Before we fit the sequential model, it\u0026rsquo;s worth mentioning that the default priors for the thresholds in a sequential model are different than the cumulative model. In the cumulative model, the default prior for the thresholds is a Normal distribution with a grid of evenly spaced $\\mu$ where an ordered transformation is applied to ensure the ordering of the values. However, in the sequential model, the ordering of the thresholds does not matter. Thus, the default prior for the thresholds is a Normal distribution with a zero $\\mu$ vector of length $k - 1$ where $k$ is the number of response levels. Refer to the getting started docs if you need a refresher on priors in Bambi.\nSubsequently, fitting a sequential model is similar to fitting a cumulative model. The only difference is that we pass family=\u0026quot;sratio\u0026quot; to the bambi.Model constructor.\nsequence_model = bmb.Model( \u0026#34;YearsAtCompany ~ 0 + TotalWorkingYears\u0026#34;, data=attrition, family=\u0026#34;sratio\u0026#34; ) sequence_idata = sequence_model.fit(random_seed=1234) sequence_model Formula: YearsAtCompany ~ 0 + TotalWorkingYears Family: sratio Link: p = logit Observations: 1233 Priors: target = p Common-level effects TotalWorkingYears ~ Normal(mu: 0.0, sigma: 0.3223) Auxiliary parameters threshold ~ Normal(mu: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], sigma: 1.0) az.summary(sequence_idata) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The coefficients are still on the logits scale, so we need to apply the inverse of the logit function to transform back to probabilities. Below, we plot the probabilities for each category.\nprobs = expit_func(sequence_idata.posterior.YearsAtCompany_threshold).mean((\u0026#34;chain\u0026#34;, \u0026#34;draw\u0026#34;)) probs = np.append(probs, 1) plt.figure(figsize=(7, 3)) plt.plot(sorted(attrition.YearsAtCompany.unique()), probs, marker=\u0026#39;o\u0026#39;) plt.ylabel(\u0026#34;Probability\u0026#34;) plt.xlabel(\u0026#34;Response category\u0026#34;); This plot can seem confusing at first. Remember, the sequential model is a product of probabilities, i.e., the probability that $Y$ is equal to category $k$ is equal to the probability that it did not fall in one of the former categories $1: k-1$ multiplied by the probability that the sequential process stopped at $k$. Thus, the probability of category 5 is the probability that the sequential process did not fall in 0, 1, 2, 3, or 4 multiplied by the probability that the sequential process stopped at 5. This makes sense why the probability of category 36 is 1. There is no category after 36, so once you multiply all of the previous probabilities with the current category, you get 1. This is the reason for the \u0026ldquo;cumulative-like\u0026rdquo; shape of the plot. But if the coefficients were truly cumulative, the probability could not decreases as $k$ increases.\nPosterior predictive samples Again, using the posterior predictive samples, we can visualize the model fit against the observed data. In the case of the sequential model, the model does an alright job of capturing the observed frequencies of the categories. For pedagogical purposes, this fit is sufficient.\nidata_pps = model.predict(idata=idata, kind=\u0026#34;pps\u0026#34;, inplace=False) bins = np.arange(35) fig, ax = plt.subplots(figsize=(7, 3)) ax = plot_ppc_discrete(idata_pps, bins, ax) ax.set_xlabel(\u0026#34;Response category\u0026#34;) ax.set_ylabel(\u0026#34;Count\u0026#34;) ax.set_title(\u0026#34;Sequential model - Posterior Predictive Distribution\u0026#34;); Summary This notebook demonstrated how to fit cumulative and sequential ordinal regression models using Bambi. Cumulative models focus on modeling the cumulative probabilities of an ordinal outcome variable taking on values up to and including a certain category, whereas a sequential model focuses on modeling the probability that an ordinal outcome variable stops at a particular category, rather than continuing to higher categories. To achieve this, both models assume that the reponse variable originates from a categorization of a latent continuous variable $Z$. However, the cumulative model assumes that there are $K$ thresholds $\\tau_k$ that partition $Z$ into $K+1$ observable, ordered categories of $Y$. The sequential model assumes that for every category $k$ there is a latent continuous variable $Z$ that determines the transition between categories $k$ and $k+1$; thus, a threshold $\\tau$ belongs to each latent process.\nCumulative models can be used in situations where the outcome variable is on the Likert scale, and you are interested in understanding the impact of predictors on the probability of reaching or exceeding specific categories. Sequential models are particularly useful when you are interested in understanding the predictors that influence the decision to stop at a specific response level. It\u0026rsquo;s well-suited for analyzing data where categories represent stages, and the focus is on the transitions between these stages.\n%load_ext watermark %watermark -n -u -v -iv -w Last updated: Fri Sep 15 2023 Python implementation: CPython Python version : 3.11.0 IPython version : 8.13.2 bambi : 0.13.0.dev0 arviz : 0.15.1 numpy : 1.24.2 pandas : 2.0.1 matplotlib: 3.7.1 Watermark: 2.3.1 ","permalink":"https://gstechschulte.github.io/posts/bambi-ordinal-models/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e#| code-fold: true\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e arviz \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e az\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e matplotlib.pyplot \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e plt\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e matplotlib.lines \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e Line2D\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e numpy \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e np\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e pandas \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e pd\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e warnings\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e bambi \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e bmb\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ewarnings\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003efilterwarnings(\u003cspan style=\"color:#e6db74\"\u003e\u0026#34;ignore\u0026#34;\u003c/span\u003e, category\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#a6e22e\"\u003eFutureWarning\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cpre\u003e\u003ccode\u003eWARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions.\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1 id=\"ordinal-regression\"\u003eOrdinal Regression\u003c/h1\u003e\n\u003cp\u003eThis blog post is a copy of the ordinal models documentation I wrote for \u003ca href=\"https://bambinos.github.io/bambi/\"\u003eBambi\u003c/a\u003e. The original post can be found \u003ca href=\"https://bambinos.github.io/bambi/notebooks/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eIn some scenarios, the response variable is discrete, like a count, and ordered. Common examples of such data come from questionnaires where the respondent is asked to rate a product, service, or experience on a scale. This scale is often referred to as a \u003ca href=\"https://en.wikipedia.org/wiki/Likert_scale\"\u003eLikert scale\u003c/a\u003e. For example, a five-level Likert scale could be:\u003c/p\u003e","title":"Ordinal Models in Bambi"},{"content":" #| code-fold: true import arviz as az import matplotlib.pyplot as plt from matplotlib.lines import Line2D import numpy as np import pandas as pd import scipy.stats as stats import seaborn as sns import warnings import bambi as bmb warnings.simplefilter(action=\u0026#39;ignore\u0026#39;, category=FutureWarning) WARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions. Zero inflated models This blog post is a copy of the zero inflated models documentation I wrote for Bambi. The original post can be found here.\nIn this notebook, we will describe zero inflated outcomes and why the data generating process behind these outcomes requires a special class of generalized linear models: zero-inflated Poisson (ZIP) and hurdle Poisson. Subsequently, we will describe and implement each model using a set of zero-inflated data from ecology. Along the way, we will also use the interpret sub-package to interpret the predictions and parameters of the models.\nZero inflated outcomes Sometimes, an observation is not generated from a single process, but from a mixture of processes. Whenever there is a mixture of processes generating an observation, a mixture model may be more appropriate. A mixture model uses more than one probability distribution to model the data. Count data are more susceptible to needing a mixture model as it is common to have a large number of zeros and values greater than zero. A zero means \u0026ldquo;nothing happened\u0026rdquo;, and this can be either because the rate of events is low, or because the process that generates the events was never \u0026ldquo;triggered\u0026rdquo;. For example, in health service utilization data (the number of times a patient used a service during a given time period), a large number of zeros represents patients with no utilization during the time period. However, some patients do use a service which is a result of some \u0026ldquo;triggered process\u0026rdquo;.\nThere are two popular classes of models for modeling zero-inflated data: (1) ZIP, and (2) hurdle Poisson. First, the ZIP model is described and how to implement it in Bambi is outlined. Subsequently, the hurdle Poisson model and how to implement it is outlined thereafter.\nZero inflated poisson To model zero-inflated outcomes, the ZIP model uses a distribution that mixes two data generating processes. The first process generates zeros, and the second process uses a Poisson distribution to generate counts (of which some may be zero). The result of this mixture is a distribution that can be described as\n$$P(Y=0) = (1 - \\psi) + \\psi e^{-\\mu}$$\n$$P(Y=y_i) = \\psi \\frac{e^{-\\mu} \\mu_{i}^y}{y_{i}!} \\ \\text{for} \\ y_i = 1, 2, 3,\u0026hellip;,n$$\nwhere $y_i$ is the outcome, $\\mu$ is the mean of the Poisson process where $\\mu \\ge 0$, and $\\psi$ is the probability of the Poisson process where $0 \\lt \\psi \\lt 1$. To understand how these two processes are \u0026ldquo;mixed\u0026rdquo;, let\u0026rsquo;s simulate some data using the two process equations above (taken from the PyMC docs).\nx = np.arange(0, 22) psis = [0.7, 0.4] mus = [10, 4] plt.figure(figsize=(7, 3)) for psi, mu in zip(psis, mus): pmf = stats.poisson.pmf(x, mu) pmf[0] = (1 - psi) + pmf[0] # 1.) generate zeros pmf[1:] = psi * pmf[1:] # 2.) generate counts pmf /= pmf.sum() # normalize to get probabilities plt.plot(x, pmf, \u0026#39;-o\u0026#39;, label=\u0026#39;$\\\\psi$ = {}, $\\\\mu$ = {}\u0026#39;.format(psi, mu)) plt.title(\u0026#34;Zero Inflated Poisson Process\u0026#34;) plt.xlabel(\u0026#39;x\u0026#39;, fontsize=12) plt.ylabel(\u0026#39;f(x)\u0026#39;, fontsize=12) plt.legend(loc=1) plt.show() Notice how the blue line, corresponding to a higher $\\psi$ and $\\mu$, has a higher rate of counts and less zeros. Additionally, the inline comments above describe the first and second process generating the data.\nZIP regression model The equations above only describe the ZIP distribution. However, predictors can be added to make this a regression model. Suppose we have a response variable $Y$, which represents the number of events that occur during a time period, and $p$ predictors $X_1, X_2, \u0026hellip;, X_p$. We can model the parameters of the ZIP distribution as a linear combination of the predictors.\n$$Y_i \\sim \\text{ZIPoisson}(\\mu_i, \\psi_i)$$\n$$g(\\mu_i) = \\beta_0 + \\beta_1 X_{1i}+,\u0026hellip;,+\\beta_p X_{pi}$$\n$$h(\\psi_i) = \\alpha_0 + \\alpha_1 X_{1i}+,\u0026hellip;,+\\alpha_p X_{pi}$$\nwhere $g$ and $h$ are the link functions for each parameter. Bambi, by default, uses the log link for $g$ and the logit link for $h$. Notice how there are two linear models and two link functions: one for each parameter in the $\\text{ZIPoisson}$. The parameters of the linear model differ, because any predictor such as $X$ may be associated differently with each part of the mixture. Actually, you don\u0026rsquo;t even need to use the same predictors in both linear models—but this beyond the scope of this notebook.\nThe fish dataset To demonstrate the ZIP regression model, we model and predict how many fish are caught by visitors at a state park using survey data. Many visitors catch zero fish, either because they did not fish at all, or because they were unlucky. The dataset contains data on 250 groups that went to a state park to fish. Each group was questioned about how many fish they caught (count), how many children were in the group (child), how many people were in the group (persons), if they used a live bait (livebait) and whether or not they brought a camper to the park (camper).\nfish_data = pd.read_stata(\u0026#34;http://www.stata-press.com/data/r11/fish.dta\u0026#34;) cols = [\u0026#34;count\u0026#34;, \u0026#34;livebait\u0026#34;, \u0026#34;camper\u0026#34;, \u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;] fish_data = fish_data[cols] fish_data[\u0026#34;livebait\u0026#34;] = pd.Categorical(fish_data[\u0026#34;livebait\u0026#34;]) fish_data[\u0026#34;camper\u0026#34;] = pd.Categorical(fish_data[\u0026#34;camper\u0026#34;]) fish_data = fish_data[fish_data[\u0026#34;count\u0026#34;] \u0026lt; 60] # remove outliers fish_data.head() .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } # Excess zeros, and skewed count plt.figure(figsize=(7, 3)) sns.histplot(fish_data[\u0026#34;count\u0026#34;], discrete=True) plt.xlabel(\u0026#34;Number of Fish Caught\u0026#34;); To fit a ZIP regression model, we pass family=zero_inflated_poisson to the bmb.Model constructor.\nzip_model = bmb.Model( \u0026#34;count ~ livebait + camper + persons + child\u0026#34;, fish_data, family=\u0026#39;zero_inflated_poisson\u0026#39; ) zip_idata = zip_model.fit( draws=1000, target_accept=0.95, random_seed=1234, chains=4 ) Auto-assigning NUTS sampler... Initializing NUTS using jitter+adapt_diag... Multiprocess sampling (4 chains in 4 jobs) NUTS: [count_psi, Intercept, livebait, camper, persons, child] Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 4 seconds. Lets take a look at the model components. Why is there only one linear model and link function defined for $\\mu$. Where is the linear model and link function for $\\psi$? By default, the \u0026ldquo;main\u0026rdquo; (or first) formula is defined for the parent parameter; in this case $\\mu$. Since we didn\u0026rsquo;t pass an additional formula for the non-parent parameter $\\psi$, $\\psi$ was never modeled as a function of the predictors as explained above. If we want to model both $\\mu$ and $\\psi$ as a function of the predictor, we need to expicitly pass two formulas.\nzip_model Formula: count ~ livebait + camper + persons + child Family: zero_inflated_poisson Link: mu = log Observations: 248 Priors: target = mu Common-level effects Intercept ~ Normal(mu: 0.0, sigma: 9.5283) livebait ~ Normal(mu: 0.0, sigma: 7.2685) camper ~ Normal(mu: 0.0, sigma: 5.0733) persons ~ Normal(mu: 0.0, sigma: 2.2583) child ~ Normal(mu: 0.0, sigma: 2.9419) Auxiliary parameters psi ~ Beta(alpha: 2.0, beta: 2.0) ------ * To see a plot of the priors call the .plot_priors() method. * To see a summary or plot of the posterior pass the object returned by .fit() to az.summary() or az.plot_trace() formula = bmb.Formula( \u0026#34;count ~ livebait + camper + persons + child\u0026#34;, # parent parameter mu \u0026#34;psi ~ livebait + camper + persons + child\u0026#34; # non-parent parameter psi ) zip_model = bmb.Model( formula, fish_data, family=\u0026#39;zero_inflated_poisson\u0026#39; ) zip_idata = zip_model.fit( draws=1000, target_accept=0.95, random_seed=1234, chains=4 ) Auto-assigning NUTS sampler... Initializing NUTS using jitter+adapt_diag... Multiprocess sampling (4 chains in 4 jobs) NUTS: [Intercept, livebait, camper, persons, child, psi_Intercept, psi_livebait, psi_camper, psi_persons, psi_child] Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 6 seconds. zip_model Formula: count ~ livebait + camper + persons + child psi ~ livebait + camper + persons + child Family: zero_inflated_poisson Link: mu = log psi = logit Observations: 248 Priors: target = mu Common-level effects Intercept ~ Normal(mu: 0.0, sigma: 9.5283) livebait ~ Normal(mu: 0.0, sigma: 7.2685) camper ~ Normal(mu: 0.0, sigma: 5.0733) persons ~ Normal(mu: 0.0, sigma: 2.2583) child ~ Normal(mu: 0.0, sigma: 2.9419) target = psi Common-level effects psi_Intercept ~ Normal(mu: 0.0, sigma: 1.0) psi_livebait ~ Normal(mu: 0.0, sigma: 1.0) psi_camper ~ Normal(mu: 0.0, sigma: 1.0) psi_persons ~ Normal(mu: 0.0, sigma: 1.0) psi_child ~ Normal(mu: 0.0, sigma: 1.0) ------ * To see a plot of the priors call the .plot_priors() method. * To see a summary or plot of the posterior pass the object returned by .fit() to az.summary() or az.plot_trace() Now, both $\\mu$ and $\\psi$ are defined as a function of a linear combination of the predictors. Additionally, we can see that the log and logit link functions are defined for $\\mu$ and $\\psi$, respectively.\nzip_model.graph() Since each parameter has a different link function, and each parameter has a different meaning, we must be careful on how the coefficients are interpreted. Coefficients without the substring \u0026ldquo;psi\u0026rdquo; correspond to the $\\mu$ parameter (the mean of the Poisson process) and are on the log scale. Coefficients with the substring \u0026ldquo;psi\u0026rdquo; correspond to the $\\psi$ parameter (this can be thought of as the log-odds of non-zero data) and are on the logit scale. Interpreting these coefficients can be easier with the interpret sub-package. Below, we will show how to use this sub-package to interpret the coefficients conditional on a set of the predictors.\naz.summary( zip_idata, var_names=[\u0026#34;Intercept\u0026#34;, \u0026#34;livebait\u0026#34;, \u0026#34;camper\u0026#34;, \u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;], filter_vars=\u0026#34;like\u0026#34; ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Interpret model parameters Since we have fit a distributional model, we can leverage the plot_predictions() function in the interpret sub-package to visualize how the $\\text{ZIPoisson}$ parameters $\\mu$ and $\\psi$ vary as a covariate changes.\nfig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10, 3)) bmb.interpret.plot_predictions( zip_model, zip_idata, covariates=\u0026#34;persons\u0026#34;, ax=ax[0] ) ax[0].set_ylabel(\u0026#34;mu (fish count)\u0026#34;) ax[0].set_title(\u0026#34;$\\\\mu$ as a function of persons\u0026#34;) bmb.interpret.plot_predictions( zip_model, zip_idata, covariates=\u0026#34;persons\u0026#34;, target=\u0026#34;psi\u0026#34;, ax=ax[1] ) ax[1].set_title(\u0026#34;$\\\\psi$ as a function of persons\u0026#34;); Interpreting the left plot (the $\\mu$ parameter) as the number of people in a group fishing increases, so does the number of fish caught. The right plot (the $\\psi$ parameter) shows that as the number of people in a group fishing increases, the probability of the Poisson process increases. One interpretation of this is that as the number of people in a group increases, the probability of catching no fish decreases.\nPosterior predictive distribution Lastly, lets plot the posterior predictive distribution against the observed data to see how well the model fits the data. To plot the samples, a utility function is defined below to assist in the plotting of discrete values.\ndef adjust_lightness(color, amount=0.5): import matplotlib.colors as mc import colorsys try: c = mc.cnames[color] except: c = color c = colorsys.rgb_to_hls(*mc.to_rgb(c)) return colorsys.hls_to_rgb(c[0], c[1] * amount, c[2]) def plot_ppc_discrete(idata, bins, ax): def add_discrete_bands(x, lower, upper, ax, **kwargs): for i, (l, u) in enumerate(zip(lower, upper)): s = slice(i, i + 2) ax.fill_between(x[s], [l, l], [u, u], **kwargs) var_name = list(idata.observed_data.data_vars)[0] y_obs = idata.observed_data[var_name].to_numpy() counts_list = [] for draw_values in az.extract(idata, \u0026#34;posterior_predictive\u0026#34;)[var_name].to_numpy().T: counts, _ = np.histogram(draw_values, bins=bins) counts_list.append(counts) counts_arr = np.stack(counts_list) qts_90 = np.quantile(counts_arr, (0.05, 0.95), axis=0) qts_70 = np.quantile(counts_arr, (0.15, 0.85), axis=0) qts_50 = np.quantile(counts_arr, (0.25, 0.75), axis=0) qts_30 = np.quantile(counts_arr, (0.35, 0.65), axis=0) median = np.quantile(counts_arr, 0.5, axis=0) colors = [adjust_lightness(\u0026#34;C0\u0026#34;, x) for x in [1.8, 1.6, 1.4, 1.2, 0.9]] add_discrete_bands(bins, qts_90[0], qts_90[1], ax=ax, color=colors[0]) add_discrete_bands(bins, qts_70[0], qts_70[1], ax=ax, color=colors[1]) add_discrete_bands(bins, qts_50[0], qts_50[1], ax=ax, color=colors[2]) add_discrete_bands(bins, qts_30[0], qts_30[1], ax=ax, color=colors[3]) ax.step(bins[:-1], median, color=colors[4], lw=2, where=\u0026#34;post\u0026#34;) ax.hist(y_obs, bins=bins, histtype=\u0026#34;step\u0026#34;, lw=2, color=\u0026#34;black\u0026#34;, align=\u0026#34;mid\u0026#34;) handles = [ Line2D([], [], label=\u0026#34;Observed data\u0026#34;, color=\u0026#34;black\u0026#34;, lw=2), Line2D([], [], label=\u0026#34;Posterior predictive median\u0026#34;, color=colors[4], lw=2) ] ax.legend(handles=handles) return ax zip_pps = zip_model.predict(idata=zip_idata, kind=\u0026#34;pps\u0026#34;, inplace=False) bins = np.arange(39) fig, ax = plt.subplots(figsize=(7, 3)) ax = plot_ppc_discrete(zip_pps, bins, ax) ax.set_xlabel(\u0026#34;Number of Fish Caught\u0026#34;) ax.set_ylabel(\u0026#34;Count\u0026#34;) ax.set_title(\u0026#34;ZIP model - Posterior Predictive Distribution\u0026#34;); The model captures the number of zeros accurately. However, the model seems to slightly underestimate the counts 1 and 2. Nonetheless, the plot shows that the model captures the overall distribution of counts reasonably well.\nHurdle poisson Both ZIP and hurdle models both use two processes to generate data. The two models differ in their conceptualization of how the zeros are generated. In $\\text{ZIPoisson}$, the zeroes can come from any of the processes, while in the hurdle Poisson they come only from one of the processes. Thus, a hurdle model assumes zero and positive values are generated from two independent processes. In the hurdle model, there are two components: (1) a \u0026ldquo;structural\u0026rdquo; process such as a binary model for modeling whether the response variable is zero or not, and (2) a process using a truncated model such as a truncated Poisson for modeling the counts. The result of these two components is a distribution that can be described as\n$$P(Y=0) = 1 - \\psi$$\n$$P(Y=y_i) = \\psi \\frac{e^{-\\mu_i}\\mu_{i}^{y_i} / y_i!}{1 - e^{-\\mu_i}} \\ \\text{for} \\ y_i = 1, 2, 3,\u0026hellip;,n$$\nwhere $y_i$ is the outcome, $\\mu$ is the mean of the Poisson process where $\\mu \\ge 0$, and $\\psi$ is the probability of the Poisson process where $0 \\lt \\psi \\lt 1$. The numerator of the second equation is the Poisson probability mass function, and the denominator is one minus the Poisson cumulative distribution function. This is a lot to digest. Again, let\u0026rsquo;s simulate some data to understand how data is generated from this process.\nx = np.arange(0, 22) psis = [0.7, 0.4] mus = [10, 4] plt.figure(figsize=(7, 3)) for psi, mu in zip(psis, mus): pmf = stats.poisson.pmf(x, mu) # pmf evaluated at x given mu cdf = stats.poisson.cdf(0, mu) # cdf evaluated at 0 given mu pmf[0] = 1 - psi # 1.) generate zeros pmf[1:] = (psi * pmf[1:]) / (1 - cdf) # 2.) generate counts pmf /= pmf.sum() # normalize to get probabilities plt.plot(x, pmf, \u0026#39;-o\u0026#39;, label=\u0026#39;$\\\\psi$ = {}, $\\\\mu$ = {}\u0026#39;.format(psi, mu)) plt.title(\u0026#34;Hurdle Poisson Process\u0026#34;) plt.xlabel(\u0026#39;x\u0026#39;, fontsize=12) plt.ylabel(\u0026#39;f(x)\u0026#39;, fontsize=12) plt.legend(loc=1) plt.show() The differences between the ZIP and hurdle models are subtle. Notice how in the code for the hurdle Poisson process, the zero counts are generate by (1 - psi) versus (1 - psi) + pmf[0] for the ZIP process. Additionally, the positive observations are generated by the process (psi * pmf[1:]) / (1 - cdf) where the numerator is a vector of probabilities for positive counts scaled by $\\psi$ and the denominator uses the Poisson cumulative distribution function to evaluate the probability a count is greater than 0.\nHurdle regression model To add predictors in the hurdle model, we follow the same specification as in the ZIP regression model section since both models have the same structure. The only difference is that the hurdle model uses a truncated Poisson distribution instead of a ZIP distribution. Right away, we will model both the parent and non-parent parameter as a function of the predictors.\nhurdle_formula = bmb.Formula( \u0026#34;count ~ livebait + camper + persons + child\u0026#34;, # parent parameter mu \u0026#34;psi ~ livebait + camper + persons + child\u0026#34; # non-parent parameter psi ) hurdle_model = bmb.Model( hurdle_formula, fish_data, family=\u0026#39;hurdle_poisson\u0026#39; ) hurdle_idata = hurdle_model.fit( draws=1000, target_accept=0.95, random_seed=1234, chains=4 ) Auto-assigning NUTS sampler... Initializing NUTS using jitter+adapt_diag... Multiprocess sampling (4 chains in 4 jobs) NUTS: [Intercept, livebait, camper, persons, child, psi_Intercept, psi_livebait, psi_camper, psi_persons, psi_child] Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 6 seconds. hurdle_model Formula: count ~ livebait + camper + persons + child psi ~ livebait + camper + persons + child Family: hurdle_poisson Link: mu = log psi = logit Observations: 248 Priors: target = mu Common-level effects Intercept ~ Normal(mu: 0.0, sigma: 9.5283) livebait ~ Normal(mu: 0.0, sigma: 7.2685) camper ~ Normal(mu: 0.0, sigma: 5.0733) persons ~ Normal(mu: 0.0, sigma: 2.2583) child ~ Normal(mu: 0.0, sigma: 2.9419) target = psi Common-level effects psi_Intercept ~ Normal(mu: 0.0, sigma: 1.0) psi_livebait ~ Normal(mu: 0.0, sigma: 1.0) psi_camper ~ Normal(mu: 0.0, sigma: 1.0) psi_persons ~ Normal(mu: 0.0, sigma: 1.0) psi_child ~ Normal(mu: 0.0, sigma: 1.0) ------ * To see a plot of the priors call the .plot_priors() method. * To see a summary or plot of the posterior pass the object returned by .fit() to az.summary() or az.plot_trace() hurdle_model.graph() As the same link functions are used for ZIP and Hurdle model, the coefficients can be interpreted in a similar manner.\naz.summary( hurdle_idata, var_names=[\u0026#34;Intercept\u0026#34;, \u0026#34;livebait\u0026#34;, \u0026#34;camper\u0026#34;, \u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;], filter_vars=\u0026#34;like\u0026#34; ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Posterior predictive samples As with the ZIP model above, we plot the posterior predictive distribution against the observed data to see how well the model fits the data.\nhurdle_pps = hurdle_model.predict(idata=hurdle_idata, kind=\u0026#34;pps\u0026#34;, inplace=False) bins = np.arange(39) fig, ax = plt.subplots(figsize=(7, 3)) ax = plot_ppc_discrete(hurdle_pps, bins, ax) ax.set_xlabel(\u0026#34;Number of Fish Caught\u0026#34;) ax.set_ylabel(\u0026#34;Count\u0026#34;) ax.set_title(\u0026#34;Hurdle Model - Posterior Predictive Distribution\u0026#34;); The plot looks similar to the ZIP model above. Nonetheless, the plot shows that the model captures the overall distribution of counts reasonably well.\nSummary In this notebook, two classes of models (ZIP and hurdle Poisson) for modeling zero-inflated data were presented and implemented in Bambi. The difference of the data generating process between the two models differ in how zeros are generated. The ZIP model uses a distribution that mixes two data generating processes. The first process generates zeros, and the second process uses a Poisson distribution to generate counts (of which some may be zero). The hurdle Poisson also uses two data generating processes, but doesn\u0026rsquo;t \u0026ldquo;mix\u0026rdquo; them. A process is used for generating zeros such as a binary model for modeling whether the response variable is zero or not, and a second process for modeling the counts. These two proceses are independent of each other.\nThe dataset used to demonstrate the two models had a large number of zeros. These zeros appeared because the group doesn\u0026rsquo;t fish, or because they fished, but caught zero fish. Because zeros could be generated due to two different reasons, the ZIP model, which allows zeros to be generated from a mixture of processes, seems to be more appropriate for this datset.\n%load_ext watermark %watermark -n -u -v -iv -w Last updated: Mon Sep 25 2023 Python implementation: CPython Python version : 3.11.0 IPython version : 8.13.2 seaborn : 0.12.2 numpy : 1.24.2 scipy : 1.11.2 bambi : 0.13.0.dev0 matplotlib: 3.7.1 arviz : 0.16.1 pandas : 2.1.0 Watermark: 2.3.1 ","permalink":"https://gstechschulte.github.io/posts/bambi-zip-models/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e#| code-fold: true\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e arviz \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e az\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e matplotlib.pyplot \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e plt\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003efrom\u003c/span\u003e matplotlib.lines \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e Line2D\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e numpy \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e np\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e pandas \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e pd\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e scipy.stats \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e stats\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e seaborn \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e sns\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e warnings\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e bambi \u003cspan style=\"color:#66d9ef\"\u003eas\u003c/span\u003e bmb\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ewarnings\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003esimplefilter(action\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#e6db74\"\u003e\u0026#39;ignore\u0026#39;\u003c/span\u003e, category\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e\u003cspan style=\"color:#a6e22e\"\u003eFutureWarning\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cpre\u003e\u003ccode\u003eWARNING (pytensor.tensor.blas): Using NumPy C-API based implementation for BLAS functions.\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch1 id=\"zero-inflated-models\"\u003eZero inflated models\u003c/h1\u003e\n\u003cp\u003eThis blog post is a copy of the zero inflated models documentation I wrote for \u003ca href=\"https://bambinos.github.io/bambi/\"\u003eBambi\u003c/a\u003e. The original post can be found \u003ca href=\"https://bambinos.github.io/bambi/notebooks/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e","title":"Zero Inflated Models in Bambi"},{"content":" My project \u0026ldquo;Better tools to interpret complex Bambi regression models\u0026rdquo; was completed under the organization of NumFOCUS, and mentors Tomás Capretto and Osvaldo Martin. Before I describe the project, objectives, and work completed, I would like to thank my mentors Tomás and Osvaldo for their precious time and support throughout the summer. They were always available and timely in communicating over Slack and GitHub, and provided valuable feedback during code reviews. Additionally, I would like to thank NumFOCUS and the Google Summer of Code (GSoC) program for providing the opportunity to work on such an open source project over the summer. It has been an invaluable experience, and I look forward to contributing to open source projects in the future.\nProject Description Bayesian modeling has increased significantly in academia and industry over the past years thanks to the development of high quality and user friendly open source probabilistic programming languages (PPL) in Python and R. Of these is Bambi, a Python library built on top of the PyMC PPL, that makes it easy to specify complex generalized linear multilevel models using a formula notation similar to those found in R. However, as the model building portion of the Bayesian workflow becomes easier, the interpretation of these models has not.\nTo aid in model interpretability, Bambi (before this project) had a sub-package plots that supported conditional adjusted predictions plots. The original objective was to extend upon the existing plotting functionality of conditional adjusted predictions by supporting the plotting of posterior predictive samples, and to provide the additional plotting functions predictive comparisons and predictive slopes. However, after discussion with my mentors, and taking inspiration from marginaleffects, it was decided that in addition to the plotting functions, the plots sub-package should also have functions that allow the user to return the dataframe used for plotting. For example, calling plot_slopes() would plot the slopes, and calling slopes() would return the dataframe used to plot the slopes. To this extent, the plots sub-package was renamed to interpret to better reflect all of the supported functionality. These three features allow Bambi modelers to compute and interpret three different \u0026ldquo;quantities of interest\u0026rdquo; in a more automatic and effective manner.\nThus, the three main deliverables of this project were to add (or enhance) the following functions in the interpret sub-package. Additionally, for each feature, tests and documentation need to be added:\nSupport posterior predictive samples (pps) in plot_predictions() Write tests Add documentation Add comparisons() and plot_comparisons() Write tests Add documentation Add slopes() and plot_slopes() Write tests Add documentation Work Completed All three main deliverables (and their associated sub-deliverables) were completed (merged into the main branch of the Bambi repository) on time. In the table below, the name of the deliverable, link to the feature\u0026rsquo;s pull request (PR), and link to the documentation are provided.\nDeliverable Feature PR Documentation PR Allow plot_predictions to plot posterior predictive samples PR #668 PR #670 Add comparisons and plot_comparisons PR #684 PR #695 Add slopes and plot_slopes PR #699 PR #701 In order to quickly obtain a sense of the work completed over the GSoC program, it is probably best to view the documentation PRs. The documentation PRs consist of a Jupyter notebook that demonstrates: (1) why the new feature is useful in interpreting GLMs, (2) a brief overview of how the function computes the quantity of interest based on the user\u0026rsquo;s inputs, and (3) a demonstration of the function using different GLMs and datasets.\nFuture Work As the PRs have been merged upstream, some Bambi users have already been utilizing the new sub-package. The feedback has been positive, and a GitHub issue has been opened to request additional functionality. Going forward, I will continue to contribute to Bambi, maintain the interpret sub-package, and interact with the Bambi community.\n","permalink":"https://gstechschulte.github.io/posts/gsoc-final-report/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"bambi-logo.png\" alt=\"alt\"  /\u003e\n\u003c/p\u003e\n\u003cp\u003eMy project \u0026ldquo;Better tools to interpret complex Bambi regression models\u0026rdquo; was completed under the organization of NumFOCUS, and mentors Tomás Capretto and Osvaldo Martin. Before I describe the project, objectives, and work completed, I would like to thank my mentors Tomás and Osvaldo for their precious time and support throughout the summer. They were always available and timely in communicating over Slack and GitHub, and provided valuable feedback during code reviews. Additionally, I would like to thank NumFOCUS and the Google Summer of Code (GSoC) program for providing the opportunity to work on such an open source project over the summer. It has been an invaluable experience, and I look forward to contributing to open source projects in the future.\u003c/p\u003e","title":"Google Summer of Code - Final Report"},{"content":" It is currently the beginning of week ten of Google Summer of Code 2023. According to the original deliverables table outlined in my proposal, the goal was to have opened a draft PR for the basic functionality of the plot_slopes. Subsequently, week 11 was reserved to further develop the plot_slopes function, and to write tests and a notebook for the documentation, respectively.\nHowever, at the beginning of week ten, I have a PR open with the majority of the functionality that marginaleffects has for slopes. In addition, I also exposed the slopes function, added tests, and have a PR open for the documentation.\nBelow is the documentation for slopes and plot_slopes in Bambi, and is a culmination of the work completed in weeks five through nine.\nPlot Slopes Bambi\u0026rsquo;s sub-package interpret features a set of functions to help interpret complex regression models. The sub-package is inspired by the R package marginaleffects. In this notebook we will discuss two functions slopes and plot_slopes. These two functions allow the modeler to easier interpret slopes, either by a inspecting a summary output or plotting them.\nBelow, it is described why estimating the slope of the prediction function is useful in interpreting generalized linear models (GLMs), how this methodology is implemented in Bambi, and how to use slopes and plot_slopes. It is assumed that the reader is familiar with the basics of GLMs. If not, refer to the Bambi Basic Building Blocks example.\nInterpretation of Regression Coefficients Assuming we have fit a linear regression model of the form\n$$y = \\beta_0 + \\beta_1 x_1 + \\beta_2 x_2 + \\dots + \\beta_k x_k + \\epsilon$$\nthe \u0026ldquo;safest\u0026rdquo; interpretation of the regression coefficients $\\beta$ is as a comparison between two groups of items that differ by $1$ in the relevant predictor variable $x_i$ while being identical in all the other predictors. Formally, the predicted difference between two items $i$ and $j$ that differ by an amount $n$ on predictor $k$, but are identical on all other predictors, the predicted difference is $y_i - y_j$ is $\\beta_kx$, on average.\nHowever, once we move away from a regression model with a Gaussian response, the identity function, and no interaction terms, the interpretation of the coefficients are not as straightforward. For example, in a logistic regression model, the coefficients are on a different scale and are measured in logits (log odds), not probabilities or percentage points. Thus, you cannot interpret the coefficents as a \u0026ldquo;one unit increase in $x_k$ is associated with an $n$ percentage point decrease in $y$\u0026rdquo;. First, the logits must be converted to the probability scale. Secondly, a one unit change in $x_k$ may produce a larger or smaller change in the outcome, depending upon how far away from zero the logits are.\nslopes and plot_slopes, by default, computes quantities of interest on the response scale for GLMs. For example, for a logistic regression model, this is the probability scale, and for a Poisson regression model, this is the count scale.\nInterpreting interaction effects Specifying interactions in a regression model is a way of allowing parameters to be conditional on certain aspects of the data. By contrast, for a model with no interactions, the parameters are not conditional and thus, the value of one parameter is not dependent on the value of another covariate. However, once interactions exist, multiple parameters are always in play at the same time. Additionally, interactions can be specified for either categorical, continuous, or both types of covariates. Thus, making the interpretation of the parameters more difficult.\nWith GLMs, every covariate essentially interacts with itself because of the link function. To demonstrate parameters interacting with themselves, consider the mean of a Gaussian linear model with an identity link function\n$$\\mu = \\alpha + \\beta x$$\nwhere the rate of change in $\\mu$ with respect to $x$ is just $\\beta$, i.e., the rate of change is constant no matter what the value of $x$ is. But when we consider GLMs with link functions used to map outputs to exponential family distribution parameters, calculating the derivative of the mean output $\\mu$ with respect to the predictor is not as straightforward as in the Gaussian linear model. For example, computing the rate of change in a binomial probability $p$ with respect to $x$\n$$p = \\frac{exp(\\alpha + \\beta x)}{1 + exp(\\alpha + \\beta x)}$$\nAnd taking the derivative of $p$ with respect to $x$ yields\n$$\\frac{\\partial p}{\\partial x} = \\frac{\\beta}{2(1 + cosh(\\alpha + \\beta x))}$$\nSince $x$ appears in the derivative, the impact of a change in $x$ depends upon $x$, i.e., an interaction with itself even though no interaction term was specified in the model.Thus, visualizing the rate of change in the mean response with respect to a covariate $x$ becomes a useful tool in interpreting GLMs.\nAverage Predictive Slopes Here, we adopt the notation from Chapter 14.4 of Regression and Other Stories to first describe average predictive differences which is essential to computing slopes, and then secondly, average predictive slopes. Assume we have fit a Bambi model predicting an outcome $Y$ based on inputs $X$ and parameters $\\theta$. Consider the following scalar inputs:\n$$w: \\text{the input of interest}$$ $$c: \\text{all the other inputs}$$ $$X = (w, c)$$\nIn contrast to comparisons, for slopes we are interested in comparing $w^{\\text{value}}$ to $w^{\\text{value}+\\epsilon}$ (perhaps age = 60 and 60.0001 respectively) with all other inputs $c$ held constant. The predictive difference in the outcome changing only $w$ is:\n$$\\text{average predictive difference} = \\mathbb{E}(y|w^{\\text{value}}, c, \\theta) - \\mathbb{E}(y|w^{\\text{value}+\\epsilon}, c, \\theta)$$\nSelecting $w$ and $w^{\\text{value}+\\epsilon}$ and averaging over all other inputs $c$ in the data gives you a new \u0026ldquo;hypothetical\u0026rdquo; dataset and corresponds to counting all pairs of transitions of $(w^\\text{value})$ to $(w^{\\text{value}+\\epsilon})$, i.e., differences in $w$ with $c$ held constant. The difference between these two terms is the average predictive difference.\nHowever, to obtain the slope estimate, we need to take the above formula and divide by $\\epsilon$ to obtain the average predictive slope:\n$$\\text{average predictive slope} = \\frac{\\mathbb{E}(y|w^{\\text{value}}, c, \\theta) - \\mathbb{E}(y|w^{\\text{value}+\\epsilon}, c, \\theta)}{\\epsilon}$$\nComputing Slopes The objective of slopes and plot_slopes is to compute the rate of change (slope) in the mean of the response $y$ with respect to a small change $\\epsilon$ in the predictor $x$ conditional on other covariates $c$ specified in the model. $w$ is specified by the user and the original value is either provided by the user, else a default value (the mean) is computed by Bambi. The values for the other covariates $c$ specified in the model can be determined under the following three scenarios:\nuser provided values a grid of equally spaced and central values empirical distribution (original data used to fit the model) In the case of (1) and (2) above, Bambi assembles all pairwise combinations (transitions) of $w$ and $c$ into a new \u0026ldquo;hypothetical\u0026rdquo; dataset. In (3), Bambi uses the original $c$, and adds a small amount $\\epsilon$ to each unit of observation\u0026rsquo;s $w$. In each scenario, predictions are made on the data using the fitted model. Once the predictions are made, comparisons are computed using the posterior samples by taking the difference in the predicted outcome for each pair of transitions and dividing by $\\epsilon$. The average of these slopes is the average predictive slopes.\nFor variables $w$ with a string or categorical data type, the comparisons function is called to compute the expected difference in group means. Please refer to the comparisons documentation for more details.\nBelow, we present several examples showing how to use Bambi to perform these computations for us, and to return either a summary dataframe, or a visualization of the results.\nimport arviz as az import pandas as pd import bambi as bmb Logistic Regression To demonstrate slopes and plot_slopes, we will use the well switching dataset to model the probability a household in Bangladesh switches water wells. The data are for an area of Arahazar Upazila, Bangladesh. The researchers labelled each well with its level of arsenic and an indication of whether the well was “safe” or “unsafe”. Those using unsafe wells were encouraged to switch. After several years, it was determined whether each household using an unsafe well had changed its well. The data contains $3020$ observations on the following five variables:\nswitch: a factor with levels no and yes indicating whether the household switched to a new well arsenic: the level of arsenic in the old well (measured in micrograms per liter) dist: the distance to the nearest safe well (measured in meters) assoc: a factor with levels no and yes indicating whether the household is a member of an arsenic education group educ: years of education of the household head First, a logistic regression model with no interactions is fit to the data. Subsequently, to demonstrate the benefits of plot_slopes in interpreting interactions, we will fit a logistic regression model with an interaction term.\ndata = pd.read_csv(\u0026#34;http://www.stat.columbia.edu/~gelman/arm/examples/arsenic/wells.dat\u0026#34;, sep=\u0026#34; \u0026#34;) data[\u0026#34;switch\u0026#34;] = pd.Categorical(data[\u0026#34;switch\u0026#34;]) data[\u0026#34;dist100\u0026#34;] = data[\u0026#34;dist\u0026#34;] / 100 data[\u0026#34;educ4\u0026#34;] = data[\u0026#34;educ\u0026#34;] / 4 data.head() .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } well_model = bmb.Model( \u0026#34;switch ~ dist100 + arsenic + educ4\u0026#34;, data, family=\u0026#34;bernoulli\u0026#34; ) well_idata = well_model.fit( draws=1000, target_accept=0.95, random_seed=1234, chains=4 ) User provided values First, an example of scenario 1 (user provided values) is given below. In both plot_slopes and slopes, $w$ and $c$ are represented by wrt (with respect to) and conditional, respectively. The modeler has the ability to pass their own values for wrt and conditional by using a dictionary where the key-value pairs are the covariate and value(s) of interest.\nFor example, if we wanted to compute the slope of the probability of switching wells for a typical arsenic value of $1.3$ conditional on a range of dist and educ values, we would pass the following dictionary in the code block below. By default, for $w$, Bambi compares $w^\\text{value}$ to $w^{\\text{value} + \\epsilon}$ where $\\epsilon =$ 1e-4. However, the value for $\\epsilon$ can be changed by passing a value to the argument eps.\nThus, in this example, $w^\\text{value} = 1.3$ and $w^{\\text{value} + \\epsilon} = 1.3001$. The user is not limited to passing a list for the values. A np.array can also be used. Furthermore, Bambi by default, maps the order of the dict keys to the main, group, and panel of the matplotlib figure. Below, since dist100 is the first key, this is used for the x-axis, and educ4 is used for the group (color). If a third key was passed, it would be used for the panel (facet).\nfig, ax = bmb.interpret.plot_slopes( well_model, well_idata, wrt={\u0026#34;arsenic\u0026#34;: 1.3}, conditional={\u0026#34;dist100\u0026#34;: [0.20, 0.50, 0.80], \u0026#34;educ4\u0026#34;: [1.00, 1.20, 2.00]}, ) fig.set_size_inches(7, 3) fig.axes[0].set_ylabel(\u0026#34;Slope of Well Switching Probability\u0026#34;); The plot above shows that, for example, conditional on dist100 $= 0.2$ and educ4 $= 1.0$ a unit increase in arsenic is associated with households being $11$% less likely to switch wells. Notice that even though we fit a logistic regression model where the coefficients are on the log-odds scale, the slopes function returns the slope on the probability scale. Thus, we can interpret the y-axis (slope) as the expected change in the probability of switching wells for a unit increase in arsenic conditional on the specified covariates.\nslopes can be called directly to view a summary dataframe that includes the term name, estimate type (discussed in detail in the interpreting coefficients as an elasticity section), values $w$ used to compute the estimate, the specified conditional covariates $c$, and the expected slope of the outcome with the uncertainty interval (by default the $94$% highest density interval is computed).\nbmb.interpret.slopes( well_model, well_idata, wrt={\u0026#34;arsenic\u0026#34;: 1.5}, conditional={ \u0026#34;dist100\u0026#34;: [0.20, 0.50, 0.80], \u0026#34;educ4\u0026#34;: [1.00, 1.20, 2.00] } ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Since all covariates used to fit the model were also specified to compute the slopes, no default value is used for unspecified covariates. A default value is computed for the unspecified covariates because in order to peform predictions, Bambi is expecting a value for each covariate used to fit the model. Additionally, with GLM models, average predictive slopes are conditional in the sense that the estimate depends on the values of all the covariates in the model. Thus, for unspecified covariates, slopes and plot_slopes computes a default value (mean or mode based on the data type of the covariate). Each row in the summary dataframe is read as \u0026ldquo;the slope (or rate of change) of the probability of switching wells with respect to a small change in $w$ conditional on $c$ is $y$\u0026rdquo;.\nMultiple slope values Users can also compute slopes on multiple values for wrt. For example, if we want to compute the slope of $y$ with respect to arsenic $= 1.5$, $2.0$, and $2.5$, simply pass a list or numpy array as the dictionary values for wrt. Keeping the conditional covariate and values the same, the following slope estimates are computed below.\nmultiple_values = bmb.interpret.slopes( well_model, well_idata, wrt={\u0026#34;arsenic\u0026#34;: [1.5, 2.0, 2.5]}, conditional={ \u0026#34;dist100\u0026#34;: [0.20, 0.50, 0.80], \u0026#34;educ4\u0026#34;: [1.00, 1.20, 2.00] } ) multiple_values.head(6) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The output above is essentially the same as the summary dataframe when we only passed one value to wrt. However, now each element (value) in the list gets a small amount $\\epsilon$ added to it, and the slope is calculated for each of these values.\nConditional slopes As stated in the interpreting interaction effects section, interpreting coefficients of multiple interaction terms can be difficult and cumbersome. Thus, plot_slopes provides an effective way to visualize the conditional slopes of the interaction effects. Below, we will use the same well switching dataset, but with interaction terms. Specifically, one interaction is added between dist100 and educ4, and another between arsenic and educ4.\nwell_model_interact = bmb.Model( \u0026#34;switch ~ dist100 + arsenic + educ4 + dist100:educ4 + arsenic:educ4\u0026#34;, data, family=\u0026#34;bernoulli\u0026#34; ) well_idata_interact = well_model_interact.fit( draws=1000, target_accept=0.95, random_seed=1234, chains=4 ) # summary of coefficients az.summary(well_idata_interact) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The coefficients of the linear model are shown in the table above. The interaction coefficents indicate the slope varies in a continuous fashion with the continuous variable.\nA negative value for arsenic:dist100 indicates that the \u0026ldquo;effect\u0026rdquo; of arsenic on the outcome is less negative as distance from the well increases. Similarly, a negative value for arsenic:educ4 indicates that the \u0026ldquo;effect\u0026rdquo; of arsenic on the outcome is more negative as education increases. Remember, these coefficients are still on the logit scale. Furthermore, as more variables and interaction terms are added to the model, interpreting these coefficients becomes more difficult.\nThus, lets use plot_slopes to visually see how the slope changes with respect to arsenic conditional on dist100 and educ4 changing. Notice in the code block below how parameters are passed to the subplot_kwargs and fig_kwargs arguments. At times, it can be useful to pass specific group and panel arguments to aid in the interpretation of the plot. Therefore, subplot_kwargs allows the user to manipulate the plotting by passing a dictionary where the keys are {\u0026quot;main\u0026quot;: ..., \u0026quot;group\u0026quot;: ..., \u0026quot;panel\u0026quot;: ...} and the values are the names of the covariates to be plotted. fig_kwargs are figure level key word arguments such as figsize and sharey.\nfig, ax = bmb.interpret.plot_slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, conditional=[\u0026#34;dist100\u0026#34;, \u0026#34;educ4\u0026#34;], subplot_kwargs={\u0026#34;main\u0026#34;: \u0026#34;dist100\u0026#34;, \u0026#34;group\u0026#34;: \u0026#34;educ4\u0026#34;, \u0026#34;panel\u0026#34;: \u0026#34;educ4\u0026#34;}, fig_kwargs={\u0026#34;figsize\u0026#34;: (16, 4), \u0026#34;sharey\u0026#34;: True}, legend=False ) With interaction terms now defined, it can be seen how the slope of the outcome with respect to arsenic differ depending on the value of educ4. Especially in the case of educ4 $= 4.25$, the slope is more \u0026ldquo;constant\u0026rdquo;, but with greater uncertainty. Lets compare this with the model that does not include any interaction terms.\nfig, ax = bmb.interpret.plot_slopes( well_model, well_idata, wrt=\u0026#34;arsenic\u0026#34;, conditional=[\u0026#34;dist100\u0026#34;, \u0026#34;educ4\u0026#34;], subplot_kwargs={\u0026#34;main\u0026#34;: \u0026#34;dist100\u0026#34;, \u0026#34;group\u0026#34;: \u0026#34;educ4\u0026#34;, \u0026#34;panel\u0026#34;: \u0026#34;educ4\u0026#34;}, fig_kwargs={\u0026#34;figsize\u0026#34;: (16, 4), \u0026#34;sharey\u0026#34;: True}, legend=False ) For the non-interaction model, conditional on a range of values for educ4 and dist100, the slopes of the outcome are nearly identical.\nUnit level slopes Evaluating average predictive slopes at central values for the conditional covariates $c$ can be problematic when the inputs have a large variance since no single central value (mean, median, etc.) is representative of the covariate. This is especially true when $c$ exhibits bi or multimodality. Thus, it may be desireable to use the empirical distribution of $c$ to compute the predictive slopes, and then average over a specific or set of covariates to obtain average slopes. To achieve unit level slopes, do not pass a parameter into conditional and or specify None.\nunit_level = bmb.interpret.slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, conditional=None ) # empirical distribution print(unit_level.shape[0] == well_model_interact.data.shape[0]) unit_level.head(10) True .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } well_model_interact.data.head(10) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Above, unit_level is the slopes summary dataframe and well_model_interact.data is the empirical data used to fit the model. Notice how the values for $c$ are identical in both dataframes. However, for $w$, the values are the original $w$ value plus $\\epsilon$. Thus, the estimate value represents the instantaneous rate of change for that unit of observation. However, these unit level slopes are difficult to interpret since each row may have a different slope estimate. Therefore, it is useful to average over (marginalize) the estimates to summarize the unit level predictive slopes.\nMarginalizing over covariates Since the empirical distrubution is used for computing the average predictive slopes, the same number of rows ($3020$) is returned as the data used to fit the model. To average over a covariate, use the average_by argument. If True is passed, then slopes averages over all covariates. Else, if a single or list of covariates are passed, then slopes averages by the covariates passed.\nbmb.interpret.slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, conditional=None, average_by=True ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } The code block above is equivalent to taking the mean of the estimate and uncertainty columns. For example:\nunit_level[[\u0026#34;estimate\u0026#34;, \u0026#34;lower_3.0%\u0026#34;, \u0026#34;upper_97.0%\u0026#34;]].mean() estimate -0.111342 lower_3.0% -0.134846 upper_97.0% -0.088171 dtype: float64 Average by subgroups Averaging over all covariates may not be desired, and you would rather average by a group or specific covariate. To perform averaging by subgroups, users can pass a single or list of covariates to average_by to average over specific covariates. For example, if we wanted to average by educ4:\n# average by educ4 bmb.interpret.slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, conditional=None, average_by=\u0026#34;educ4\u0026#34; ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } # average by both educ4 and dist100 bmb.interpret.slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, conditional=None, average_by=[\u0026#34;educ4\u0026#34;, \u0026#34;dist100\u0026#34;] ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } It is still possible to use plot_slopes when passing an argument to average_by. In the plot below, the empirical distribution is used to compute unit level slopes with respect to arsenic and then averaged over educ4 to obtain the average predictive slopes.\nfig, ax = bmb.interpret.plot_slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, conditional=None, average_by=\u0026#34;educ4\u0026#34; ) fig.set_size_inches(7, 3) Interpreting coefficients as an elasticity In some fields, such as economics, it is useful to interpret the results of a regression model in terms of an elasticity (a percent change in $x$ is associated with a percent change in $y$) or semi-elasticity (a unit change in $x$ is associated with a percent change in $y$, or vice versa). Typically, this is achieved by fitting a model where either the outcome and or the covariates are log-transformed. However, since the log transformation is performed by the modeler, to compute elasticities for slopes and plot_slopes, Bambi \u0026ldquo;post-processes\u0026rdquo; the predictions to compute the elasticities. Below, it is shown the possible elasticity arguments and how they are computed for slopes and plot_slopes:\neyex: a percentage point increase in $x_1$ is associated with an $n$ percentage point increase in $y$ $$\\frac{\\partial \\hat{y}}{\\partial x_1} * \\frac{x_1}{\\hat{y}}$$\neydx: a unit increase in $x_1$ is associated with an $n$ percentage point increase in $y$ $$\\frac{\\partial \\hat{y}}{\\partial x_1} * \\frac{1}{\\hat{y}}$$\ndyex: a percentage point increase in $x_1$ is associated with an $n$ unit increase in $y$ $$\\frac{\\partial \\hat{y}}{\\partial x_1} * x_1$$\nBelow, each code cell shows the same model, and wrt and conditional argument, but with a different elasticity (slope) argument. By default, dydx (a derivative with no post-processing) is used.\nbmb.interpret.slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, slope=\u0026#34;eyex\u0026#34;, conditional=None, average_by=True ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } bmb.interpret.slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, slope=\u0026#34;eydx\u0026#34;, conditional=None, average_by=True ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } bmb.interpret.slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, slope=\u0026#34;dyex\u0026#34;, conditional=None, average_by=True ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } slope is also an argument for plot_slopes. Below, we visualize the elasticity with respect to arsenic conditional on a range of dist100 and educ4 values (notice this is the same plot as in the conditional slopes section).\nfig, ax = bmb.interpret.plot_slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;arsenic\u0026#34;, conditional=[\u0026#34;dist100\u0026#34;, \u0026#34;educ4\u0026#34;], slope=\u0026#34;eyex\u0026#34;, subplot_kwargs={\u0026#34;main\u0026#34;: \u0026#34;dist100\u0026#34;, \u0026#34;group\u0026#34;: \u0026#34;educ4\u0026#34;, \u0026#34;panel\u0026#34;: \u0026#34;educ4\u0026#34;}, fig_kwargs={\u0026#34;figsize\u0026#34;: (16, 4), \u0026#34;sharey\u0026#34;: True}, legend=False ) Categorical covariates As mentioned in the computing slopes section, if you pass a variable with a string or categorical data type, the comparisons function will be called to compute the expected difference in group means. Here, we fit the same interaction model as above, albeit, by specifying educ4 as an ordinal data type.\ndata = pd.read_csv(\u0026#34;http://www.stat.columbia.edu/~gelman/arm/examples/arsenic/wells.dat\u0026#34;, sep=\u0026#34; \u0026#34;) data[\u0026#34;switch\u0026#34;] = pd.Categorical(data[\u0026#34;switch\u0026#34;]) data[\u0026#34;dist100\u0026#34;] = data[\u0026#34;dist\u0026#34;] / 100 data[\u0026#34;educ4\u0026#34;] = pd.Categorical(data[\u0026#34;educ\u0026#34;] / 4, ordered=True) well_model_interact = bmb.Model( \u0026#34;switch ~ dist100 + arsenic + educ4 + dist100:educ4 + arsenic:educ4\u0026#34;, data, family=\u0026#34;bernoulli\u0026#34; ) well_idata_interact = well_model_interact.fit( draws=1000, target_accept=0.95, random_seed=1234, chains=4 ) fig, ax = bmb.interpret.plot_slopes( well_model_interact, well_idata_interact, wrt=\u0026#34;educ4\u0026#34;, conditional=\u0026#34;dist100\u0026#34;, average_by=\u0026#34;dist100\u0026#34; ) fig.set_size_inches(7, 3) As the model was fit with educ4 as a categorical data type, Bambi recognized this, and calls comparisons to compute the differences between each level of educ4. As educ4 contains many category levels, a covariate must be passed to average_by in order to perform plotting. Below, we can see this plot is equivalent to plot_comparisons.\nfig, ax = bmb.interpret.plot_comparisons( well_model_interact, well_idata_interact, contrast=\u0026#34;educ4\u0026#34;, conditional=\u0026#34;dist100\u0026#34;, average_by=\u0026#34;dist100\u0026#34; ) fig.set_size_inches(7, 3) However, computing the predictive difference between each educ4 level may not be desired. Thus, in plot_slopes, as in plot_comparisons, if wrt is a categorical or string data type, it is possible to specify the wrt values. For example, if we wanted to compute the expected difference in probability of switching wells for when educ4 is $4$ versus $1$ conditional on a range of dist100 and arsenic values, we would pass the following dictionary in the code block below. Please refer to the comparisons documentation for more details.\nfig, ax = bmb.interpret.plot_slopes( well_model_interact, well_idata_interact, wrt={\u0026#34;educ4\u0026#34;: [1, 4]}, conditional=\u0026#34;dist100\u0026#34;, average_by=\u0026#34;dist100\u0026#34; ) fig.set_size_inches(7, 3) %load_ext watermark %watermark -n -u -v -iv -w Last updated: Wed Aug 16 2023 Python implementation: CPython Python version : 3.11.0 IPython version : 8.13.2 pandas: 2.0.1 arviz : 0.15.1 bambi : 0.10.0.dev0 Watermark: 2.3.1 ","permalink":"https://gstechschulte.github.io/posts/bambi-slopes/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003cp\u003eIt is currently the beginning of week ten of Google Summer of Code 2023. According to the original deliverables table outlined in my proposal, the goal was to have opened a draft PR for the basic functionality of the \u003ccode\u003eplot_slopes\u003c/code\u003e. Subsequently, week 11 was reserved to further develop the \u003ccode\u003eplot_slopes\u003c/code\u003e function, and to write tests and a notebook for the documentation, respectively.\u003c/p\u003e\n\u003cp\u003eHowever, at the beginning of week ten, I have a \u003ca href=\"https://github.com/bambinos/bambi/pull/699\"\u003ePR\u003c/a\u003e open with the majority of the functionality that \u003ca href=\"https://vincentarelbundock.github.io/marginaleffects/\"\u003emarginaleffects\u003c/a\u003e has for \u003ccode\u003eslopes\u003c/code\u003e. In addition, I also exposed the \u003ccode\u003eslopes\u003c/code\u003e function, added tests, and have a \u003ca href=\"https://github.com/bambinos/bambi/pull/701\"\u003ePR\u003c/a\u003e open for the documentation.\u003c/p\u003e","title":"Google Summer of Code - Average Predictive Slopes"},{"content":" It is currently the end of week five of Google Summer of Code 2023. According to the original deliverables table outlined in my proposal, the goal was to have opened a draft PR for the core functionality of the plot_comparisons. Subsequently, week six and seven were to be spent further developing the plot_comparisons function, and writing tests and a demo notebook for the documentation, respectively. However, at the end of week five, I have a PR open with the majority of the functionality that marginaleffects has. In addition, I also exposed the comparisons function, added tests (which can and will be improved), and have started on documentation.\nPlot Comparisons comparisons and plot_comparisons are a part of Bambi\u0026rsquo;s sub-package interpret that feature a set of functions used to interpret complex regression models. This sub-package is inspired by the R package marginaleffects. These two functions allow the modeler to compare the predictions made by a model for different contrast and covariate values. Below, it is described why comparing predictions is useful in interpreting generalized linear models (GLMs), how this methodology is implemented in Bambi, and how to use comparisons and plot_comparisons. It is assumed that the reader is familiar with the basics of GLMs. If not, refer to the Bambi Basic Building Blocks example.\nDue to the link function in a GLM, there are typically three quantities of interest to interpret:\nthe linear predictor $\\eta = X\\beta$ where $X$ is an $n$ x $p$ matrix of explanatory variables. the mean $\\mu = g^{-1}(\\eta)$ where the link function $g(\\cdot)$ relates the linear predictor to the mean of the outcome variable $\\mu = g^{-1}(\\eta) = g^{-1}(X\\beta)$ the response variable $Y \\sim \\mathcal{D}(\\mu, \\theta)$ where $\\mu$ is the mean parameter and $\\theta$ is (possibly) a vector that contains all the other \u0026ldquo;auxillary\u0026rdquo; parameters of the distribution. Often, with GLMs, $\\eta$ is linear in the parameters, but nonlinear in relation of inputs to the outcome $Y$ due to the link function $g$. Thus, as modelers, we are usually more interested in interpreting (2) and (3). For example, in logistic regression, the linear predictor is on the log-odds scale, but the quantity of interest is on the probability scale. In Poisson regression, the linear predictor is on the log-scale, but the response variable is on the count scale. Referring back to logistic regression, a specified difference in one of the $x$ variables does not correspond to a constant difference in the the probability of the outcome.\nIt is often helpful with GLMs, for the modeler and audience, to have a summary that gives the expected difference in the outcome corresponding to a unit difference in each of the input variables. Thus, the goal of comparisons and plot_comparisons is to provide the modeler with a summary and visualization of the average predicted difference.\nAverage Predictive Differences Here, we adopt the notation from Chapter 14.4 of Regression and Other Stories to describe average predictive differences. Assume we have fit a Bambi model predicting an outcome $Y$ based on inputs $X$ and parameters $\\theta$. Consider the following scalar inputs:\n$$w: \\text{the input of interest}$$ $$c: \\text{all the other inputs}$$ $$X = (w, c)$$\nSuppose for the input of interest, we are interested in comparing $w^{\\text{high}}$ to $w^{\\text{low}}$ (perhaps age = $60$ and $40$ respectively) with all other inputs $c$ held constant. The predictive difference in the outcome changing only $w$ is:\n$$\\text{average predictive difference} = \\mathbb{E}(y|w^{\\text{high}}, c, \\theta) - \\mathbb{E}(y|w^{\\text{low}}, c, \\theta)$$\nSelecting the maximum and minimum values of $w$ and averaging over all other inputs $c$ in the data gives you a new \u0026ldquo;hypothetical\u0026rdquo; dataset and corresponds to counting all pairs of transitions of $(w^\\text{low})$ to $(w^\\text{high})$, i.e., differences in $w$ with $c$ held constant. The difference between these two terms is the average predictive difference.\nComputing Average Predictive Differences The objective of comparisons and plot_comparisons is to compute the expected difference in the outcome corresponding to three different scenarios for $w$ and $c$ where $w$ is either provided by the user, else a default value is computed by Bambi (described in the default values section). The three scenarios are:\nuser provided values for $c$. a grid of equally spaced and central values for $c$. empirical distribution (original data used to fit the model) for $c$. In the case of (1) and (2) above, Bambi assembles all pairwise combinations (transitions) of $w$ and $c$ into a new \u0026ldquo;hypothetical\u0026rdquo; dataset. In (3), Bambi uses the original $c$, but replaces $w$ with the user provided value or the default value computed by Bambi. In each scenario, predictions are made on the data using the fitted model. Once the predictions are made, comparisons are computed using the posterior samples by taking the difference in the predicted outcome for each pair of transitions. The average of these differences is the average predictive difference.\nThus, the goal of comparisons and plot_comparisons is to provide the modeler with a summary and visualization of the average predictive difference. Below, we demonstrate how to compute and plot average predictive differences with comparisons and plot_comparions using several examples.\nimport arviz as az import numpy as np import pandas as pd import bambi as bmb Zero Inflated Poisson We model and predict how many fish are caught by visitors at a state park using survey data. Many visitors catch zero fish, either because they did not fish at all, or because they were unlucky. We would like to explicitly model this bimodal behavior (zero versus non-zero) using a Zero Inflated Poisson model, and to compare how different inputs of interest $w$ and other covariate values $c$ are associated with the number of fish caught. The dataset contains data on 250 groups that went to a state park to fish. Each group was questioned about how many fish they caught (count), how many children were in the group (child), how many people were in the group (persons), if they used a live bait and whether or not they brought a camper to the park (camper).\nfish_data = pd.read_stata(\u0026#34;http://www.stata-press.com/data/r11/fish.dta\u0026#34;) cols = [\u0026#34;count\u0026#34;, \u0026#34;livebait\u0026#34;, \u0026#34;camper\u0026#34;, \u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;] fish_data = fish_data[cols] fish_data[\u0026#34;livebait\u0026#34;] = pd.Categorical(fish_data[\u0026#34;livebait\u0026#34;]) fish_data[\u0026#34;camper\u0026#34;] = pd.Categorical(fish_data[\u0026#34;camper\u0026#34;]) fish_model = bmb.Model( \u0026#34;count ~ livebait + camper + persons + child\u0026#34;, fish_data, family=\u0026#39;zero_inflated_poisson\u0026#39; ) fish_idata = fish_model.fit( draws=1000, target_accept=0.95, random_seed=1234, chains=4 ) User Provided Values First, an example of scenario 1 (user provided values) is given below. In both plot_comparisons and comparisons, $w$ and $c$ are represented by contrast and conditional, respectively. The modeler has the ability to pass their own values for contrast and conditional by using a dictionary where the key-value pairs are the covariate and value(s) of interest. For example, if we wanted to compare the number of fish caught for $4$ versus $1$ persons conditional on a range of child and livebait values, we would pass the following dictionary in the code block below. By default, for $w$, Bambi compares $w^\\text{high}$ to $w^\\text{low}$. Thus, in this example, $w^\\text{high}$ = 4 and $w^\\text{low}$ = 1. The user is not limited to passing a list for the values. A np.array can also be used. Furthermore, Bambi by default, maps the order of the dict keys to the main, group, and panel of the matplotlib figure. Below, since child is the first key, this is used for the x-axis, and livebait is used for the group (color). If a third key was passed, it would be used for the panel (facet).\nfig, ax = bmb.interpret.plot_comparisons( model=fish_model, idata=fish_idata, contrast={\u0026#34;persons\u0026#34;: [1, 4]}, conditional={\u0026#34;child\u0026#34;: [0, 1, 2], \u0026#34;livebait\u0026#34;: [0, 1]}, ) fig.set_size_inches(7, 3) The plot above shows that, comparing $4$ to $1$ persons given $0$ children and using livebait, the expected difference is about $26$ fish. When not using livebait, the expected difference decreases substantially to about $5$ fish. Using livebait with a group of people is associated with a much larger expected difference in the number of fish caught.\ncomparisons can be called to view a summary dataframe that includes the term $w$ and its contrast, the specified conditional covariate, and the expected difference in the outcome with the uncertainty interval (by default the 94% highest density interval is computed).\nbmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast={\u0026#34;persons\u0026#34;: [1, 4]}, conditional={\u0026#34;child\u0026#34;: [0, 1, 2], \u0026#34;livebait\u0026#34;: [0, 1]}, ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } But why is camper also in the summary dataframe? This is because in order to peform predictions, Bambi is expecting a value for each covariate used to fit the model. Additionally, with GLM models, average predictive comparisons are conditional in the sense that the estimate depends on the values of all the covariates in the model. Thus, for unspecified covariates, comparisons and plot_comparisons computes a default value (mean or mode based on the data type of the covariate). Thus, $c$ = child, livebait, camper. Each row in the summary dataframe is read as \u0026ldquo;comparing $4$ to $1$ persons conditional on $c$, the expected difference in the outcome is $y$.\u0026rdquo;\nMultiple contrast values Users can also perform comparisons on multiple contrast values. For example, if we wanted to compare the number of fish caught between $(1, 2)$, $(1, 4)$, and $(2, 4)$ persons conditional on a range of values for child and livebait.\nmultiple_values = bmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast={\u0026#34;persons\u0026#34;: [1, 2, 4]}, conditional={\u0026#34;child\u0026#34;: [0, 1, 2], \u0026#34;livebait\u0026#34;: [0, 1]} ) multiple_values .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Notice how the contrast $w$ varies while the covariates $c$ are held constant. Currently, however, plotting multiple contrast values can be difficult to interpret since the contrast is \u0026ldquo;abstracted\u0026rdquo; away onto the y-axis. Thus, it would be difficult to interpret which portion of the plot corresponds to which contrast value. Therefore, it is currently recommended that if you want to plot multiple contrast values, call comparisons directly to obtain the summary dataframe and plot the results yourself.\nDefault contrast and conditional values Now, we move onto scenario 2 described above (grid of equally spaced and central values) in computing average predictive comparisons. You are not required to pass values for contrast and conditional. If you do not pass values, Bambi will compute default values for you. Below, it is described how these default values are computed.\nThe default value for contrast is a centered difference at the mean for a contrast variable with a numeric dtype, and unique levels for a contrast varaible with a categorical dtype. For example, if the modeler is interested in the comparison of a $5$ unit increase in $w$ where $w$ is a numeric variable, Bambi computes the mean and then subtracts and adds $2.5$ units to the mean to obtain a centered difference. By default, if no value is passed for the contrast covariate, Bambi computes a one unit centered difference at the mean. For example, if only contrast=\u0026quot;persons\u0026quot; is passed, then $\\pm$ $0.5$ is applied to the mean of persons. If $w$ is a categorical variable, Bambi computes and returns the unique levels. For example, if $w$ has levels [\u0026ldquo;high scool\u0026rdquo;, \u0026ldquo;vocational\u0026rdquo;, \u0026ldquo;university\u0026rdquo;], Bambi computes and returns the unique values of this variable.\nThe default values for conditional are more involved. Currently, by default, if a dict or list is passed to conditional, Bambi uses the ordering (keys if dict and elements if list) to determine which covariate to use as the main, group (color), and panel (facet) variable. This is the same logic used in plot_comparisons described above. Subsequently, the default values used for the conditional covariates depend on their ordering and dtype. Below, the psuedocode used for computing default values covariates passed to conditional is outlined:\nif v == \u0026#34;main\u0026#34;: if v == numeric: return np.linspace(v.min(), v.max(), 50) elif v == categorical: return np.unique(v) elif v == \u0026#34;group\u0026#34;: if v == numeric: return np.quantile(v, np.linspace(0, 1, 5)) elif v == categorical: return np.unique(v) elif v == \u0026#34;panel\u0026#34;: if v == numeric: return np.quantile(v, np.linspace(0, 1, 5)) elif v == categorical: return np.unique(v) Thus, letting Bambi compute default values for conditional is equivalent to creating a hypothetical \u0026ldquo;data grid\u0026rdquo; of new values. Lets say we are interested in comparing the number of fish caught for the contrast livebait conditional on persons and child. This time, lets call comparisons first to gain an understanding of the data generating the plot.\ncontrast_df = bmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=[\u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;], ) contrast_df.head(10) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } As livebait was encoded as a categorical dtype, Bambi returned the unique levels of $[0, 1]$ for the contrast. persons and child were passed as the first and second element and thus act as the main and group variables, respectively. It can be see from the output above, that an equally spaced grid was used to compute the values for persons, whereas a quantile based grid was used for child. Furthermore, as camper was unspecified, the mode was used as the default value. Lets go ahead and plot the commparisons.\nfig, ax = bmb.interpret.plot_comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=[\u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;], ) fig.set_size_inches(7, 3) The plot shows us that the expected differences in fish caught comparing a group of people who use livebait and no livebait is not only conditional on the number of persons, but also children. However, the plotted comparisons for child = $3$ is difficult to interpret on a single plot. Thus, it can be useful to pass specific group and panel arguments to aid in the interpretation of the plot. Therefore, subplot_kwargs allows the user to manipulate the plotting by passing a dictionary where the keys are {\u0026quot;main\u0026quot;: ..., \u0026quot;group\u0026quot;: ..., \u0026quot;panel\u0026quot;: ...} and the values are the names of the covariates to be plotted. Below, we plot the same comparisons as above, but this time we specify group and panel to both be child.\nfig, ax = bmb.interpret.plot_comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=[\u0026#34;persons\u0026#34;, \u0026#34;child\u0026#34;], subplot_kwargs={\u0026#34;main\u0026#34;: \u0026#34;persons\u0026#34;, \u0026#34;group\u0026#34;: \u0026#34;child\u0026#34;, \u0026#34;panel\u0026#34;: \u0026#34;child\u0026#34;}, fig_kwargs={\u0026#34;figsize\u0026#34;:(12, 3), \u0026#34;sharey\u0026#34;: True}, legend=False ) Unit level contrasts Evaluating average predictive comparisons at central values for the conditional covariates $c$ can be problematic when the inputs have a large variance since no single central value (mean, median, etc.) is representative of the covariate. This is especially true when $c$ exhibits bi or multimodality. Thus, it may be desireable to use the empirical distribution of $c$ to compute the predictive comparisons, and then average over a specific or set of covariates to obtain the average predictive comparisons. To achieve unit level contrasts, do not pass a parameter into conditional and or specify None.\nunit_level = bmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=None, ) # empirical distribution print(unit_level.shape[0] == fish_model.data.shape[0]) unit_level.head(10) True .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } # empirical (observed) data used to fit the model fish_model.data.head(10) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Above, unit_level is the comparisons summary dataframe and fish_model.data is the empirical data. Notice how the values for $c$ are identical in both dataframes. However, for $w$, the values are different. However, these unit level contrasts are difficult to interpret as each row corresponds to that unit\u0026rsquo;s contrast. Therefore, it is useful to average over (marginalize) the estimates to summarize the unit level predictive comparisons.\nMarginalizing over covariates Since the empirical distrubution is used for computing the average predictive comparisons, the same number of rows (250) is returned as the data used to fit the model. To average over a covariate, use the average_by argument. If True is passed, then comparisons averages over all covariates. Else, if a single or list of covariates are passed, then comparisons averages by the covariates passed.\n# marginalize over all covariates bmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=None, average_by=True ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } Passing True to average_by averages over all covariates and is equivalent to taking the mean of the estimate and uncertainty columns. For example:\nunit_level = bmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=None, ) unit_level[[\u0026#34;estimate\u0026#34;, \u0026#34;lower_3.0%\u0026#34;, \u0026#34;upper_97.0%\u0026#34;]].mean() estimate 3.649691 lower_3.0% 2.956185 upper_97.0% 4.333621 dtype: float64 Average by subgroups Averaging over all covariates may not be desired, and you would rather average by a group or specific covariate. To perform averaging by subgroups, users can pass a single or list of covariates to average_by to average over specific covariates. For example, if we wanted to average by persons:\n# average by number of persons bmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=None, average_by=\u0026#34;persons\u0026#34; ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } # average by number of persons and camper by passing a list bmb.interpret.comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=None, average_by=[\u0026#34;persons\u0026#34;, \u0026#34;camper\u0026#34;] ) .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } It is still possible to use plot_comparisons when passing an argument to average_by. In the plot below, the empirical distribution is used to compute unit level contrasts for livebait and then averaged over persons to obtain the average predictive comparisons. The plot below is similar to the second plot in this notebook. The differences being that: (1) a pairwise transition grid is defined for the second plot above, whereas the empirical distribution is used in the plot below, and (2) in the plot below, we marginalized over the other covariates in the model (thus the reason for not having a camper or child group and panel, and a reduction in the uncertainty interval).\nfig, ax = bmb.interpret.plot_comparisons( model=fish_model, idata=fish_idata, contrast=\u0026#34;livebait\u0026#34;, conditional=None, average_by=\u0026#34;persons\u0026#34; ) fig.set_size_inches(7, 3) Logistic Regression To showcase an additional functionality of comparisons and plot_comparisons, we fit a logistic regression model to the titanic dataset with interaction terms to model the probability of survival. The titanic dataset gives the values of four categorical attributes for each of the 2201 people on board the Titanic when it struck an iceberg and sank. The attributes are social class (first class, second class, third class, crewmember), age, sex (0 = female, 1 = male), and whether or not the person survived (0 = deceased, 1 = survived).\ndat = pd.read_csv(\u0026#34;https://vincentarelbundock.github.io/Rdatasets/csv/Stat2Data/Titanic.csv\u0026#34;, index_col=0) dat[\u0026#34;PClass\u0026#34;] = dat[\u0026#34;PClass\u0026#34;].str.replace(\u0026#34;[st, nd, rd]\u0026#34;, \u0026#34;\u0026#34;, regex=True) dat[\u0026#34;PClass\u0026#34;] = dat[\u0026#34;PClass\u0026#34;].str.replace(\u0026#34;*\u0026#34;, \u0026#34;0\u0026#34;).astype(int) dat[\u0026#34;PClass\u0026#34;] = dat[\u0026#34;PClass\u0026#34;].replace(0, np.nan) dat[\u0026#34;PClass\u0026#34;] = pd.Categorical(dat[\u0026#34;PClass\u0026#34;], ordered=True) dat[\u0026#34;SexCode\u0026#34;] = pd.Categorical(dat[\u0026#34;SexCode\u0026#34;], ordered=True) dat = dat.dropna(axis=0, how=\u0026#34;any\u0026#34;) titanic_model = bmb.Model( \u0026#34;Survived ~ PClass * SexCode * Age\u0026#34;, data=dat, family=\u0026#34;bernoulli\u0026#34; ) titanic_idata = titanic_model.fit(draws=1000, target_accept=0.95, random_seed=1234) Comparison types comparisons and plot_comparisons also allow you to specify the type of comparison to be computed. By default, a difference is used. However, it is also possible to take the ratio where comparisons would then become average predictive ratios. To achieve this, pass \u0026quot;ratio\u0026quot; into the argument comparison_type. Using different comparison types offers a way to produce alternative insights; especially when there are interaction terms as the value of one covariate depends on the value of the other covariate.\nfig, ax = bmb.interpret.plot_comparisons( model=titanic_model, idata=titanic_idata, contrast={\u0026#34;PClass\u0026#34;: [1, 3]}, conditional=[\u0026#34;Age\u0026#34;, \u0026#34;SexCode\u0026#34;], comparison_type=\u0026#34;ratio\u0026#34;, subplot_kwargs={\u0026#34;main\u0026#34;: \u0026#34;Age\u0026#34;, \u0026#34;group\u0026#34;: \u0026#34;SexCode\u0026#34;, \u0026#34;panel\u0026#34;: \u0026#34;SexCode\u0026#34;}, fig_kwargs={\u0026#34;figsize\u0026#34;:(12, 3), \u0026#34;sharey\u0026#34;: True}, legend=False ) The left panel shows that the ratio of the probability of survival comparing PClass $3$ to $1$ conditional on Age is non-constant. Whereas the right panel shows an approximately constant ratio in the probability of survival comparing PClass $3$ to $1$ conditional on Age.\nConclusion Overall, solid progress has been made with comparisons and plot_comparisons to give the Bambi modeller insights into GLMs. One of the most difficult aspects to program is the building of the contrast dataframe (the output above) as it requires tedious shape handling to ensure the predictive comparisons are \u0026ldquo;mapped\u0026rdquo; to the correct contrast and conditional values. Writing better tests to ensure shapes are correct is something I am also working on. Additionally, there are still \u0026ldquo;nice to have\u0026rdquo; features such as:\ncross-contrasts comparisons other than the predictive difference or ratios (e.g., adjusted risk ratio) ","permalink":"https://gstechschulte.github.io/posts/bambi-comparisons/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003cp\u003eIt is currently the end of week five of Google Summer of Code 2023. According to the original deliverables table outlined in my proposal, the goal was to have opened a draft PR for the core functionality of the \u003ccode\u003eplot_comparisons\u003c/code\u003e. Subsequently, week six and seven were to be spent further developing the \u003ccode\u003eplot_comparisons\u003c/code\u003e function, and writing tests and a demo notebook for the documentation, respectively. However, at the end of week five, I have a PR open with the majority of the functionality that \u003ca href=\"https://vincentarelbundock.github.io/marginaleffects/\"\u003emarginaleffects\u003c/a\u003e has. In addition, I also exposed the \u003ccode\u003ecomparisons\u003c/code\u003e function, added tests (which can and will be improved), and have started on documentation.\u003c/p\u003e","title":"Google Summer of Code - Average Predictive Comparisons"},{"content":" A variant of the Metropolis-Hastings (MH) algorithm that uses clever proposals and is therefore more efficient (you can get a good approximate of the posterior with far fewer samples) is Gibbs sampling. A problem with MH is the need to choose the proposal distribution, and the fact that the acceptance rate may be low.\nThe improvement arises from adaptive proposals in which the distribution of proposed parameter values adjusts itself intelligently, depending upon the parameter values at the moment. This dependence upon the parameters at that moment is an exploitation of conditional independence properties of a graphical model to automatically create a good proposal, with acceptance probability equal to one.\nMain Idea Suppose we have a 3-dimensional joint distribution. Estimating this joint distribution is much harder than a 1-dimensional distribution. Subsequently, sampling is also harder in $\\mathbb{R}^3$. In Gibbs sampling, you condition each variable on the values of all the other variables in the distribution. For example, if we have $D=3$ variables:\n$$x_{1}^{s+1} \\sim p(x_1 | x_2^s,x_3^s)$$\n$$x_{2}^{s+1} \\sim p(x_2 | x_1^{s+1},x_3^s)$$\n$$x_{3}^{s+1} \\sim p(x_3 | x_1^{s+1},x_2^{s+1})$$\nwhere $x_1, x_2,\u0026hellip;,x_n$ are variable $1, 2,\u0026hellip;,n$, respectively. By conditioning on the values of the other variables, sampling from the conditional distribution is much easier than the joint. Because of the exploitation of conditional independence properties of graphical models, the Gibbs algorithm can readily generalize to $D$ variables.\nGibbs Sampling - Bayesian Gaussian Mixture Model Here, the Gibbs sampling algorithm is implemented in PyTorch for a 2-dimensional Bayesian Gaussian Mixture Model (GMM). The Bayesian GMM is given by:\n$$p(z = k, x | \\theta) = \\pi_k \\mathcal{N}(x|\\mu_k, \\sum_k)$$\nwhere the parameters $\\theta$ are known and implemented using PyTorch\u0026rsquo;s MixtureSameFamily distribution class. This class implements a batch of mixture distributions where all components are from different parameterizations of the same distribution type (Normal distributions in this example). It is then parameterized by a Categorical distribution over $k$ components.\nFor a GMM, the full conditional distributions are:\n$$p(x|z = k, \\theta) = \\mathcal{N}(x|\\mu_k, \\sum_k)$$\nThis conditional distribution reads; \u0026ldquo;the probability of data $x$ given component $k$ parameterized by $\\theta$ is distributed according to a Normal distribution with a mean vector $\\mu$ and covariance $\\sum$ according to component $k$\u0026rdquo;.\n$$p(z = k | x) = \\frac{\\pi_k \\mathcal{N}(x|\\mu_k, \\sum_k)}{\\sum_{k\u0026rsquo;} \\mathcal{N}(x|\\mu_{k\u0026rsquo;}, \\sum_{k\u0026rsquo;})}$$\nThis conditional distribution is given by Bayes rule and reads; \u0026ldquo;the probability of component $k$ given we observe some data $x$ is equal to the prior probability $\\pi$ of component $k$ times the likelihood of data $x$ being distributed according to a Normal distribution with a mean vector $\\mu$ and covariance $\\sum$ according to component $k$\u0026rdquo; over the total probability of components $k$.\nWith the conditional distributions defined, the Gibbs sampling algorithm can be implemented. Below is the full code to reproduce the results and each main step is outlined below the code block.\n#| echo: false import numpy as np import torch from torch import distributions as dist import matplotlib.pyplot as plt import argparse import warnings warnings.filterwarnings(\u0026#34;ignore\u0026#34;) #| code-fold: true def plot_gibbs(trace_hist, probs, scales, mus, n_iters, n_eval=500): mix = dist.Categorical(probs=probs) comp = dist.Independent(dist.Normal(loc=mus, scale=scales), 1) norm_mixture = dist.MixtureSameFamily(mix, comp) x = torch.arange(-1, 2, 0.01) y = torch.arange(-1, 2, 0.01) X, Y = torch.meshgrid(x, y) Z = torch.dstack((X, Y)) probs_z = torch.exp(norm_mixture.log_prob(Z)) fig = plt.figure(figsize=(12, 5)) plt.contourf(X, Y, probs_z, levels=15) plt.scatter(trace_hist[:, 0], trace_hist[:, 1], alpha=0.25, color=\u0026#39;red\u0026#39;) plt.xlim(-1, 2) plt.ylim(-1, 2) plt.colorbar() plt.xlabel(xlabel=\u0026#39;$X$\u0026#39;) plt.ylabel(ylabel=\u0026#39;$Y$\u0026#39;) plt.title(\u0026#39;Gibbs Sampling for a Mixture of 2d Gaussians\u0026#39;) plt.show() def gibbs_sampler(x0, z0, kv, probs, mu, scale, n_iterations, rng_key=None): \u0026#34;\u0026#34;\u0026#34; implements the gibbs sampling algorithm for known params. of a 2d GMM \u0026#34;\u0026#34;\u0026#34; x_current = x0 z_current = z0 x_samples = torch.zeros(n_iterations, 2) z_samples = torch.zeros(n_iterations) for n in range(1, n_iterations): # p(Z = k | X = x) probs_z = torch.exp( dist.Independent( dist.Normal(loc=mu, scale=scale), 1).log_prob(x_current)) # p(Z = k) * p(Z = k | X = x) probs_z *= probs # denom. of Bayes probs_z = probs_z / torch.sum(probs_z) # indexing component Z = k z_current = kv[-1] if probs_z[-1] \u0026gt; probs[0] else kv[0] # draw new sample X conditioned on Z = k x_current = dist.Normal(loc=mu[z_current], scale=scale[z_current]).sample() x_samples[n] = x_current z_samples[n] = z_current return x_samples, z_samples def main(args): # initial sample values x0 = torch.randn((2,)) # initial component values z0 = torch.randint(0, 2, (2,)) # for indexing kv = np.arange(2) # defining prior mixture probability p(Z = k) mixture_probs = torch.tensor([0.4, 0.6]) # defining mu vector and covariance matrix mus = torch.randn(2, 2) scales = torch.rand(2, 2) x_samples, z_samples = gibbs_sampler( x0, z0, kv, mixture_probs, mus, scales, args.iters ) plot_gibbs(x_samples, mixture_probs, scales, mus, args.iters) parser = argparse.ArgumentParser(description=\u0026#39;rw-mh\u0026#39;) parser.add_argument(\u0026#39;--iters\u0026#39;, type=int, default=1000) args = parser.parse_args(\u0026#34;\u0026#34;) main(args) Explanation of Code The main steps to implement the Gibbs sampling algorithm:\nThe main() function defines the initial values for the sample, component, and mixture distribution.\nThe gibbs_sampler() function first sets the values of x_current and z_current for current data point $x$ and component $z$, respectively. To analyze the trace history of the sampler, x_samples and z_samples are empty lists.\nIn the for loop, the conditional $p(z = k \\vert x)$ is computed using x_current, i.e., given we have observed datum $x$, what is the log probability of component $Z = k$.\nprobs_z is then multiplied by the prior probability of the mixture components.\nThen, the denominator for the conditional $p(z = k \\vert x)$ is computed by dividing probs_z by the total probability.\nSince there are only two components in this GMM, we use logic to determine which component is most likely with the x_current. As zero-based indexing is used, the components are $k = 0, 1$. Therefore, if the probability of $k=1 \u0026gt; k=0$, then use index$=1$, else index$=0$.\nz_current defines the component using zero-based indexing. Thus, indexing mu and scale by the current, most likely component, new samples are drawn according to the conditional Normal distribution.\nData and component samples are appended for analyzing the trace history.\nLimitations Although Gibbs sampling can generalize to $D$ variables, the algorithm becomes inefficient as it tends to get stuck in small regions of the posterior for, potentially, a long number of iterations. This isn\u0026rsquo;t because of the large number of variables, but rather, because models with many parameters tend to have regions of high correlation in the posterior. High correlation between parameters means a narrow ridge of probability combinations, resulting in the sampler getting \u0026ldquo;stuck\u0026rdquo;.\n","permalink":"https://gstechschulte.github.io/posts/gibbs-sampler/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003cp\u003eA variant of the Metropolis-Hastings (MH) algorithm that uses clever proposals and is therefore more efficient (you can get a good approximate of the posterior with far fewer samples) is Gibbs sampling. A problem with MH is the need to choose the proposal distribution, and the fact that the acceptance rate may be low.\u003c/p\u003e\n\u003cp\u003eThe improvement arises from adaptive proposals in which the distribution of proposed parameter values adjusts itself intelligently, \u003cstrong\u003edepending upon the parameter values at the moment\u003c/strong\u003e. This dependence upon the parameters at that moment is an exploitation of conditional independence properties of a graphical model to automatically create a good proposal, with acceptance probability equal to one.\u003c/p\u003e","title":"Gibbs Sampler From Scratch"},{"content":" Main Idea Metropolis-Hastings (MH) is one of the simplest kinds of MCMC algorithms. The idea with MH is that at each step, we propose to move from the current state $x$ to a new state $x\u0026rsquo;$ with probability $q(x\u0026rsquo;|x)$, where $q$ is the proposal distribution. The user is free to choose the proposal distribution and the choice of the proposal is dependent on the form of the target distribution. Once a proposal has been made to move to $x\u0026rsquo;$, we then decide whether to accept or reject the proposal according to some rule. If the proposal is accepted, the new state is $x\u0026rsquo;$, else the new state is the same as the current state $x$.\nProposals can be symmetric and asymmetric. In the case of symmetric proposals $q(x\u0026rsquo;|x) = q(x|x\u0026rsquo;)$, the acceptance probability is given by the rule:\n$$A = min(1, \\frac{p^(x\u0026rsquo;)}{p^(x)})$$\nThe fraction is a ratio between the probabilities of the proposed state $x\u0026rsquo;$ and current state $x$. If $x\u0026rsquo;$ is more probable than $x$, the ratio is $\u0026gt; 1$, and we move to the proposed state. However, if $x\u0026rsquo;$ is less probable, we may still move there, depending on the relative probabilities. If the relative probabilities are similar, we may code exploration into the algorithm such that they go in the opposite direction. This helps with the greediness of the original algorithm—only moving to more probable states.\nThe Algorithm Initialize $x^0$\nfor $s = 0, 1, 2, 3, \u0026hellip;$ do:\nDefine $x = x^s$ Sample $x\u0026rsquo; \\sim q(x\u0026rsquo;|x)$ where $q$ is the user\u0026rsquo;s proposal distribution Compute the acceptance probability given by: $p_a(x_{t+1}|x_i) = min(1, \\frac{p(x_{i+1})q(x_i | x_{i+1})}{p(x_i)q(x_{i+1}|x_I)})$ Compute $A = min(1, \\alpha)$\nSample $u \\sim U(0, 1)$\nSet new sample to: $x^{s+1} = \\left{ \\begin{array}{ll} x\u0026rsquo; \u0026amp; \\quad \\text{if} \\quad u \\leq A(\\text{accept}) \\ x \u0026amp; \\quad \\text{if} \\quad x \u0026gt; A(\\text{reject}) \\ \\end{array} \\right.$\nRandom Walk Metropolis-Hastings The random walk metropolis-hastings (RWMH) corresponds to MH with a Gaussian propsal distribution of the form:\n$$q(x\u0026rsquo;|x) = \\mathcal{N}(x\u0026rsquo;|x, \\tau^2 I)$$\nBelow, I implement the RWMH for sampling from a 1-dimenensional mixture of Gaussians (implemented using the MixtureSameFamily PyTorch class) with the following parameters:\n$\\mu = -20, 20$ Mixture component probability $= 0.3, 0.7$ $\\sum = 10, 10$ #| echo: false import torch from torch import distributions as dist import matplotlib.pyplot as plt import argparse import warnings warnings.filterwarnings(\u0026#34;ignore\u0026#34;) #| code-fold: true def plot(distribution, trace_history, xmin, xmax, n_iterations, n_evals=500): x_evals = torch.linspace(xmin, xmax, n_evals) evals = torch.exp(distribution.log_prob(x_evals)) fig = plt.figure(figsize=(12, 4)) ax = fig.add_subplot(1, 2, 1) ax.plot(torch.arange(n_iterations), trace_history) ax.set_xlabel(\u0026#39;Iterations\u0026#39;) ax.set_ylabel(\u0026#39;Sampled Value\u0026#39;) ax = fig.add_subplot(1, 2, 2, projection=\u0026#39;3d\u0026#39;) ax.plot(torch.arange(n_iterations), trace_history) ax.plot(torch.zeros(n_evals), x_evals, evals) ax.set_xlabel(\u0026#39;Iterations\u0026#39;) ax.set_ylabel(\u0026#39;Sampled Value\u0026#39;) fig.suptitle(\u0026#39;Random Walk Metropolis-Hastings\u0026#39;) plt.show() def true_distribution(mixture_probs, mus, scales): return dist.MixtureSameFamily( mixture_distribution=dist.Categorical(probs=mixture_probs), component_distribution=dist.Normal(loc=mus, scale=scales) ) def metropolis_hasting(x0, tau, mixture, n_iterations, rng_key=None): \u0026#34;\u0026#34;\u0026#34; implements the random walk metropolis-hasting algorithm \u0026#34;\u0026#34;\u0026#34; x_current = x0 x_samples = torch.zeros(n_iterations) x_samples[0] = x_current cnt_acceptance = 0 for n in range(1, n_iterations): # datum of proposed state x\u0026#39; x_candidate = x_current + tau * dist.Normal(loc=0, scale=1).sample() # probs. of proposed state x\u0026#39; p_candidate = torch.exp(mixture.log_prob(x_candidate)) # probs. of current state x p_current = torch.exp(mixture.log_prob(x_current)) # acceptance formula alpha = p_candidate / p_current probs_accept = min(1, alpha) # sample u ~ U(0, 1) u = dist.Uniform(0, 1).sample() if u \u0026gt;= probs_accept: x_current = x_current else: x_current = x_candidate cnt_acceptance += 1 x_samples[n] = x_current acceptence_ratio = cnt_acceptance / n_iterations print(\u0026#39;---- statistics ----\u0026#39;) print(f\u0026#39;acceptance rate = {acceptence_ratio}\u0026#39;) return x_samples def main(args): # initial parameter value x0 = torch.tensor(20.) # mixture dist. parameters mixture_probs = torch.tensor([0.3, 0.7]) mus = torch.tensor([-20., 20.]) scales = torch.tensor([10., 10.]) n_iters = args.iters tau = torch.tensor(args.tau) mixture_distribution = true_distribution(mixture_probs, mus, scales) x_samples = metropolis_hasting(x0, tau, mixture_distribution, n_iters) plot(mixture_distribution, x_samples, -100, 100, n_iters) parser = argparse.ArgumentParser(description=\u0026#39;rw-mh\u0026#39;) parser.add_argument(\u0026#39;--iters\u0026#39;, type=int, default=1000) parser.add_argument(\u0026#39;--tau\u0026#39;, type=float, default=8.) args = parser.parse_args(\u0026#34;\u0026#34;) main(args) ---- statistics ---- acceptance rate = 0.803 Results The mixture distribution can be tricky to sample from as it is has more than one model, i.e., it is a bimodal distribution. However, we can see that the RWMH spends time sampling from both component distributions, albeit, the distribution with the higher probability more. Due to the random search based perturbations (random walk), the sampler seems to randomly jump from component to component, showing that the chain is not sticky. Additionally, the acceptance rate is $0.803$ indicating that about $80%$ of new proposals were accepted.\n","permalink":"https://gstechschulte.github.io/posts/metropolis-hastings-sampler/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003ch3 id=\"main-idea\"\u003eMain Idea\u003c/h3\u003e\n\u003cp\u003eMetropolis-Hastings (MH) is one of the simplest kinds of MCMC algorithms. The idea with MH is that at each step, we propose to move from the current state $x$ to a new state $x\u0026rsquo;$ with probability $q(x\u0026rsquo;|x)$, where $q$ is the \u003cstrong\u003eproposal distribution\u003c/strong\u003e. The user is free to choose the proposal distribution and the choice of the proposal is dependent on the form of the target distribution. Once a proposal has been made to move to $x\u0026rsquo;$, we then decide whether to \u003cstrong\u003eaccept\u003c/strong\u003e or \u003cstrong\u003ereject\u003c/strong\u003e the proposal according to some rule. If the proposal is accepted, the new state is $x\u0026rsquo;$, else the new state is the same as the current state $x$.\u003c/p\u003e","title":"Metropolis Hastings Sampler From Scratch"},{"content":" Inference In the probabilistic approach to machine learning, all unknown quantities—predictions about the future, hidden states of a system, or parameters of a model—are treated as random variables, and endowed with probability distributions. The process of inference corresponds to computing the posterior distribution over these quantities, conditioning on whatever data is available. Given that the posterior is a probability distribution, we can draw samples from it. The samples in this case are parameter values. The Bayesian formalism treats parameter distributions as the degrees of relative plausibility, i.e., if this parameter is chosen, how likely is the data to have arisen? We use Bayes\u0026rsquo; rule for this process of inference. Let $h$ represent the uknown variables and $D$ the known variables, i.e., the data. Given a likelihood $p(D|h)$ and a prior $p(h)$, we can compute the posterior $p(h|D)$ using Bayes\u0026rsquo; rule:\n$$p(h|D) = \\frac{p(D|h)p(h)}{p(D)}$$\nThe main problem is the $p(D)$ in the demoninator. $p(D)$ is a normalization constant and ensures the probability distribution sums to 1. When the number of unknown variables $h$ is large, computing $p(D)$ requires a high dimensional integral of the form:\n$$p(D) = \\int p(D|h)p(h)dh$$\nThe integral is needed to convert the unnormalized joint probability of some parameter value $p(h, D)$ to a normalized probability $p(h|D)$. This also allows us to take into account all the other plausible values of $h$ that could have generated the data. There are three ways for computing the posterior:\nAnalytical Solution Grid Approximation Approximate Inference Many problems are complex and require a model where computing the posterior distribution using a grid of parameters or in exact mathematical form is not feasible (or possible). Therefore, you adopt the approximate inference / sampling approach. The sampling approach has a major benefit. Working with samples transforms a problem in calculus $\\rightarrow$ into a problem of data summary $\\rightarrow$ into a frequency format problem. An integral in a typical Bayesian context is just the total probability in some interval. Once you have samples from the probability distribution, it’s just a matter of counting values in the interval. Therefore, once you fit a model to the data using some sampling algorithm, then interpreting the model is a matter of interpreting the frequency of parameter samples (though this is easier said than done).\nTo gain a better conceptual understanding of algorithmic techniques for computing (approximate) posteriors, I will be diving deeper into the main inference algorithms over the next couple of posts.\nMonte Carlo Approximation As discussed above, it is often difficult to compute the posterior distribution analytically. In this example, suppose $x$ is a random variable, and $y = f(x)$ is some function of $x$. Here, $y$ is our target distribution (think the posterior). Instead of computing $p(y)$ analytically, it is possible to draw a large number of samples from $p(x)$, and then use these samples to approximate $p(y)$.\nIf $x$ is distributed uniformly in an interval between $-1, 1$ and $y = f(x) = x^2$, we can approximate $p(y)$ by drawing samples from $p(x)$. By using a large number of samples, a good approximation can be computed.\n#| code-fold: true def plot_mc(x_samples, probs_x, true_y, pdf_y, approx_y): fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(10, 4)) ax[0].set_title(\u0026#39;True Uniform Distribution\u0026#39;) ax[0].plot(x_samples, probs_x) ax[0].set_xlabel(\u0026#39;$x$\u0026#39;) ax[0].set_ylabel(\u0026#39;$p(x)$\u0026#39;) ax[1].set_title(\u0026#39;True $y$ PDF\u0026#39;) ax[1].plot(true_y, pdf_y) ax[1].set_xlabel(\u0026#39;$y$\u0026#39;) ax[1].set_ylabel(\u0026#39;$p(y)$\u0026#39;) ax[2].set_title(\u0026#39;Approximated $y$ PDF\u0026#39;) ax[2].hist(approx_y, bins=30, density=True) ax[2].set_xlabel(\u0026#39;$y$\u0026#39;) ax[2].set_ylabel(\u0026#39;$p(y)$\u0026#39;) plt.tight_layout() plt.show() def main(): square_func = lambda x: x**2 # True p(x) lower, upper = -1, 1 x_samples = np.linspace(lower, upper, 200) # Analytical solution probs_x = 1 / (upper - lower) * np.ones(len(x_samples)) # p(X = x) true_y = square_func(x_samples) # true output y pdf_y = 1 / (2 * np.sqrt(true_y + 1e-2)) # true pdf of output y # Approximation p(y) uniform_samples = np.random.uniform(-1, 1, 1000) # sample from Uniform approx_y = square_func(uniform_samples) # approx. output y plot_mc(x_samples, probs_x, true_y, pdf_y, approx_y) main() Explanation of Code First, define the squaring function $f(x)$ as square_func x_samples is an array of 200 samples in the interval $[-1, 1]$ The probability of each element in x_samples: $p(X=x)$ is computed Compute true_y using the known x_samples Compute the empirical probability density of the output true_y. Draw 1000 samples from a Uniform distribution Use these samples to approximate approx_y the empirical probability density ","permalink":"https://gstechschulte.github.io/posts/monte-carlo-approximation/","summary":"\u003c!-- raw HTML omitted --\u003e\n\u003ch2 id=\"inference\"\u003eInference\u003c/h2\u003e\n\u003cp\u003eIn the probabilistic approach to machine learning, all unknown quantities—predictions about the future, hidden states of a system, or parameters of a model—are treated as random variables, and endowed with probability distributions. The process of inference corresponds to computing the posterior distribution over these quantities, conditioning on whatever data is available. Given that the posterior is a probability distribution, we can draw samples from it. The samples in this case are parameter values. The Bayesian formalism treats parameter distributions as the degrees of relative plausibility, i.e., if this parameter is chosen, how likely is the data to have arisen? We use Bayes\u0026rsquo; rule for this process of inference. Let $h$ represent the uknown variables and $D$ the known variables, i.e., the data. Given a likelihood $p(D|h)$ and a prior $p(h)$, we can compute the posterior $p(h|D)$ using Bayes\u0026rsquo; rule:\u003c/p\u003e","title":"Monte Carlo Approximation"},{"content":"We don\u0026rsquo;t know the real posterior so we are going to choose a distribution $Q(\\theta)$ from a family of distributions $Q^*$ that are easy to work with and parameterized by $\\theta$. The approximate distribution should be as close as possible to the true posterior. This closeness is measured using KL-Divergence. If we have the joint $p(x, z)$ where $x$ is some observed data, the goal is to perform inference: given what we have observed, what can we infer about the latent states?, i.e , we want the posterior.\nRecall Bayes theorem:\n$$p(z | x) = \\frac{p(x|z)p(z)}{p(x)}$$\nThe problem is the marginal $p(x = D)$ as this could require a hundred, thousand, . . .dimensional integral:\n$$p(x) = \\int_{z_0},\u0026hellip;,\\int_{z_{D-1}}p(x, z)dz_0,\u0026hellip;,d_{z_D{-1}}$$\nIf we want the full posterior and can\u0026rsquo;t compute the marginal, then what\u0026rsquo;s the solution? Surrogate posterior. We want to approximate the true posterior using some known distribution:\n$$q(z) \\approx p(z|X=D)$$\nwhere $\\approx$ can mean you want the approximated posterior to be \u0026ldquo;as good as possible\u0026rdquo;. Using variational inference, the objective is to minimize the distance between the surrogate $q(z)$ and the true posterior $p(x)$ using KL-Divergence:\n$$q^*(z) = argmin_{q(z) \\in Q} (KL(q(z) || p(z|x=D)))$$\nwhere $Q$ is a more \u0026ldquo;simple\u0026rdquo; distribution. We can restate the KL-divergence as the expectation:\n$$KL(q(z) || p(z|D)) = \\mathbb{E_{z \\sim q(z)}}[log \\frac{q(z)}{p(z|D)}]$$\nwhich, taking the expectation over $z$, is equivalent to integration:\n$$\\int_{z_0}, . . .,\\int_{z_{D-1}}q(z)log\\frac{q(z)}{p(z|D)}d_{z_0},\u0026hellip;,d_{z_{D-1}}$$\nBut, sadly we don\u0026rsquo;t have $p(z \\vert D)$ as this is the posterior! We only have the joint. Solution? Recall our KL-divergence:\n$$KL(q(z) || p(z|D))$$\nWe can rearrange the terms inside the $log$ so that we can actually compute something:\n$$\\int_{z}q(z)log(\\frac{q(z)p(D)}{p(z, D)})dz$$\nWe only have the joint. Not the posterior; nor the marginal. We know from Bayes rule that we can express the posterior in terms of the joint $p(z, D)$ divided by the marginal $p(x=D)$:\n$$p(z|D) = \\frac{p(Z, D)}{p(D)}$$\nWe plug this inside of the $log$:\n$$\\int_{z}q(z)log(\\frac{q(z)p(D)}{p(z, D)})dz$$\nHowever, the problem now is that we have reformulated our problem into another quantity that we don\u0026rsquo;t have, i.e., the marginal $p(D)$. But we can put the quantity that we don\u0026rsquo;t have outside of the $log$ to form two separate integrals.\n$$\\int_z q(z)log(\\frac{q(z)}{p(z, D)})dz + \\int_zq(z)log(p(D)dz$$\nThis is a valid rearrangement because of the properties of logarithms. In this case, the numerator is a product, so this turns into a sum of the second integral. What do we see in these two terms? We see an expectation over the quantity $\\frac{q(z)}{p(z, D)}$ and another expectation over $p(D)$. Rewriting in terms of expectation:\n$$\\mathbb{E_{z{\\sim q(z)}}}[log(\\frac{q(z)}{p(z, D)})] + \\mathbb{E_{z \\sim q(z)}}[log(p(D))]$$\nThe right term contains information we know—the functional form of the surrogate $q(z)$ and the joint $p(z, D)$ (in the form of a directed graphical model). We still don\u0026rsquo;t have access to $p(D)$ on the right side, but this is a constant quantity. The expectation of a quantity that does not contain $z$ is just whatever the expectation was taken over. Because of this, we can again rearrange:\n$$-\\mathbb{E_{z \\sim q(z)}}[log \\frac{p(z, D)}{q(z)}]+log (p(D))$$\nThe minus sign is a result of the \u0026ldquo;swapping\u0026rdquo; of the numerator and denominator and is required to make it a valid change. Looking at this, the left side is a function dependent on $q$. In shorthand form, we can call this $\\mathcal{L(q)}$. Our KL-divergence is:\n$$KL = \\mathcal{-L(q)} + \\underbrace{log(p(D))}_\\textrm{evidence}$$\nwhere $p(D)$ is a value between $[0, 1]$ and this value is called the evidence which is the log probability of the data. If we apply the $log$ to something between $[0, 1]$ then this value will be negative. This value is also constant since we have observed the dataset and thus does not change.\n$KL$ is the distance (between the posterior and the surrogate) so it must be something positive. If the $KL$ is positive and the evidence is negative, then in order to fulfill this equation, $\\mathcal{L}$ must also be negative (negative times a negative is a positive). The $\\mathcal{L}$ should be smaller than the evidence, and thus it is called the lower bound of the evidence $\\rightarrow$ Evidence Lower Bound (ELBO).\nAgain, ELBO is defined as: $\\mathcal{L} = \\mathbb{E_{z \\sim q(z)}}[log(\\frac{p(z, D)}{q(z)})]$ and is important to note that the ELBO is equal to the evidence if and only if the KL-divergence between the surrogate and the true posterior is $0$:\n$$\\mathcal{L(q)} = log(p(D)) \\textrm{ i.f.f. } KL(q(z)||p(z|D))=0$$\n","permalink":"https://gstechschulte.github.io/posts/variational-inference/","summary":"\u003cp\u003eWe don\u0026rsquo;t know the real posterior so we are going to choose a distribution $Q(\\theta)$ from a family of distributions $Q^*$ that are \u003cstrong\u003eeasy to work with\u003c/strong\u003e and parameterized by $\\theta$. The approximate distribution should be \u003cem\u003eas close as possible\u003c/em\u003e to the true posterior. This closeness is measured using KL-Divergence. If we have the joint $p(x, z)$ where $x$ is some observed data, the goal is to perform inference: given what we have observed, what can we infer about the latent states?, i.e , we want the posterior.\u003c/p\u003e","title":"Variational Inference - Evidence Lower Bound"},{"content":"Modernity and Abstraction \u0026lsquo;Programmers\u0026rsquo;, loosely speaking, in some form or another have always been developing software to automate tedious and repetitive tasks. Rightly so, as this is one of the tasks computers are designed to perform. As science and technology progresses, and gets more technological, there is a growing seperation between the maker and the user. This is one of the negative externalities of modernism - we enjoy the benefits of a more advanced and technologically adept society, but fewer and fewer people understand the inner workings. Andrej Karpathy has a jokingly short paragraph in his blog on the matter, \u0026ldquo;A courageous developer has taken the burden of understanding query strings, urls, GET/POST requests, HTTP connections, and so on from you and largely hidden the complexity behind a few lines of code. This is what we are now familiar with and expect\u0026rdquo;.\nEver since the rise of machine learning and data science, the industry was bound to develop no and or low code products for everything between data cleaning, processing, and annotation, to the implementation of models. This is an example of one of the highest forms of abstraction - you don\u0026rsquo;t even need to be able to code. Knowledge of the problem at hand and some simple intuition into which model may provide a low validation error is almost all you need. A lower level form of abstraction is through the use of application programming interfaces (APIs) such as scikit-learn, PyTorch, etc. Using these APIs requires more technical skillsets, knowledge of first principles, and a deeper understanding of the problem than the no-code substitutes stated above. Lastly, we have the \u0026lsquo;bare bones\u0026rsquo; implementation of algorithms - what most people usually call, \u0026ldquo;programming things from scratch\u0026rdquo;. But even then, good luck writing your favorite model without using numpy, scipy, or JAX.\nAs teams of couragous developers and organizations seek to create products to help make everyday routines a commodity and more productive, it can be harder to learn technical methods from first principles as they have been abstracted away. There\u0026rsquo;s no need, nor the time, to start and go writing everything from scratch, but having a solid intuition into what is going on under the hood can allow you to uniquely solve complex problems, debug more efficiently, contribute to open source software, etc.\nSwitching Costs, Dependency and Open Source Goods At least the two lower levels of abstractions usually rely on open-source software, whereas the no-code alternative is typically (at the moment) proprietary technology in the form of machine learning as a service and or platform. [Edit 14.09.2021: H20.ai, HuggingFace, MakeML, CreateML, Google Cloud AutoML are all open-source services or platforms in the low or no code ML space] Open-source gives you the biggest flexibility that, if the space again changes, you can move things. Otherwise, you find yourself locked into a technology stack the way you were locked in to technologies from the ’80s and ’90s and 2000s.\nBeing locked into IT components can have serious opportunity costs. For example, switching from Mac to a Windows based PC involves not only the hardware costs of the computer itself, but also involves purchasing of a whole new library of software, and even more importantly, learning how to use a brand new system. When you, or an organization, decides to go the propriety route, these switching costs can be very high, and users may find themselves experiencing lock-in; a situation where the cost of changing to a different system is so high that switching is virtually inconcievable.\nOf course, the producers of this hardware / sofware love the fact that you have an inelastic demand curve - a rise in prices won\u0026rsquo;t affect demand much as switching costs are high. In summary, as machine learning platforms didn\u0026rsquo;t really exist ~15 years ago, they will more than likely change quite a bit and you are dependent on the producer of continually adapting to industry trends and technological advancements while at the same time, staying flexible to cater to your edge cases when you need it.\nLeverage and Building Technology It would be a full time job to stay present and up to date on everything being released in the space of \u0026ldquo;machine learning\u0026rdquo;, but also to be knowledgeable of first princples and have skillsets to use the technologies is in another class of its own. Before AWS, APIs, open source, etc., as an organization or startup, it was likely the question, \u0026ldquo;we need amazing technical people who can build it all\u0026rdquo;. Now, with the increase and rise of PaaS, SaaS, open source librarys and tooling, the question shifts from a question of \u0026ldquo;building\u0026rdquo;, to a question of \u0026ldquo;leveraging existing tools\u0026rdquo;. How long before no / low code gets good enough before we are asking, \u0026ldquo;why did we not build this with no-code tools?\u0026rdquo;.\nThe highest form of leverage for a company is to develop and build out difficult and new technology. No-code, if developed right (and is ideally open-source), can still provide leverage and value-add, but any advantage just becomes table stakes. This is what will distinguish great teams vs. good teams; non-linear returns will continue to be through building out proprietery and difficult technology.\n","permalink":"https://gstechschulte.github.io/posts/no-code-building-technology/2021-08-10-no-code-dependency-and-building-technology/","summary":"\u003ch2 id=\"modernity-and-abstraction\"\u003eModernity and Abstraction\u003c/h2\u003e\n\u003cp\u003e\u0026lsquo;Programmers\u0026rsquo;, loosely speaking, in some form or another have always been developing software to automate tedious and repetitive tasks. Rightly so, as this is one of the tasks computers are designed to perform. As science and technology progresses, and gets more technological, there is a growing seperation between the maker and the user. This is one of the negative externalities of modernism - we enjoy the benefits of a more advanced and technologically adept society, but fewer and fewer people understand the inner workings. Andrej Karpathy has a jokingly short paragraph in his \u003ca href=\"https://karpathy.github.io/2019/04/25/recipe/\"\u003eblog\u003c/a\u003e on the matter, \u0026ldquo;A courageous developer has taken the burden of understanding query strings, urls, GET/POST requests, HTTP connections, and so on from you and largely hidden the complexity behind a few lines of code. This is what we are now familiar with and expect\u0026rdquo;.\u003c/p\u003e","title":"No Code, Dependency, and Building Technology"}]