Shadow demands for unobservable goods: an intertemporal incomplete demand approach

We consider two group of goods–goods of interest and all other goods. The following notations are used: \({\mathbf{x}}_{t}\) is an n vector of the consumption levels for the goods of interest at period t, with the corresponding price vector denoted by \({\mathbf{p}}{}_{t};\) \({\mathbf{z}}_{t}\) is an m vector of the consumption levels of all other goods at period t, with the corresponding price vector denoted by \({\mathbf{q}}_{t} .\) Total expenditure, \(M_{t} ,\) is allocated to \({\mathbf{x}}_{t}\) and \({\mathbf{z}}_{t}\) at period t, and the budget constraint is \({\mathbf{p}}_{t}{\prime} {\mathbf{x}}_{t} + {\mathbf{q}}_{t}{\prime} {\mathbf{z}}_{t} \le M_{t}\) (adding-up condition).

Given a direct utility function, \(u({\mathbf{x}}_{t} , \, {\mathbf{z}}_{t} )\), which is continuous, increasing, and quasi-concave in \(({\mathbf{x}}_{t} ,{\mathbf{z}}_{t} ),\) the indirect utility function, \(v{(}{\mathbf{p}}{}_{t}{, }{\mathbf{q}}{}_{t}{, }M_{t} {)}\), is defined as

This indirect utility function is well defined as a description of the consumer’s within-period preferences under the following regularity conditions: It is continuous in \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} ,M_{t} ),\) decreasing in \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} )\), increasing in \(M_{t}\), quasi-convex in \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} )\), homogeneous of degree zero in \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} )\) and \(M_{t}\) (Deaton and Muellbauer 1980a). The direct utility function, \(u({\mathbf{x}}_{t} , \, {\mathbf{z}}_{t} )\), can be recovered from the indirect utility function as the solution to the minimization problem:

Application of Roy’s identity to the indirect utility function (1) yields the system of demand functions for \({\mathbf{x}}_{t}\) and \({\mathbf{z}}_{t}\):

These demand functions for x and z have well-known properties, with consumption expenditure loosely referred to as income (Deaton and Muellbauer 1980a). They are homogenous of degree zero in \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} )\) and \(M_{t}\), the matrix of the Slutsky terms is symmetric, negative semidefinite, and the adding-up condition hold for x and z.

In the above discussion, \({\mathbf{x}}_{t}\) and \({\mathbf{z}}_{t}\) are assumed to be observable, in that their prices and quantities are directly observed. Hence, the demands for \({\mathbf{x}}_{t}\) and \({\mathbf{z}}_{t}\) described in (3) and (4) constitute a complete demand system. An incomplete demand system is a subset of the demand functions in a complete demand system. In this demand system, while the demands for \({\mathbf{x}}_{t}\) are observable, the demands for \({\mathbf{z}}_{t}\) are not, in that their prices are observed but the quantities not. Accordingly, there are no explicit demand functions for \({\mathbf{z}}_{t}\) as described in (4), and only part of the consumer’s budget is allocated to the consumption of \({\mathbf{x}}_{t} .\) This indicates that the adding-up condition for the budget constraint holds for \({\mathbf{x}}_{t}\) only as an inequality restriction, that is, \({\mathbf{p}}_{t}{\prime} {\mathbf{x}}_{t} \le M_{t} .\) This has an important implication for integrability, namely the recoverability of the underlying preferences from the estimated demand functions satisfying the required regularity conditions. Strong integrability, which ensures the existence of a well-defined indirect, and hence direct, utility function across the goods of interest and other goods, is difficult to establish because of insufficient information on unobservable goods (Epstein 1982).⁷ In particular, since the demands for \({\mathbf{z}}_{t}\) are unobservable, estimation of the demands for \({\mathbf{x}}_{t}\) only⁸ does not provide sufficient information to identify all the parameters characterizing the indirect utility function \(v({\mathbf{p}}_{t} ,\,{\mathbf{q}}_{t} ,\,M_{t} )\) and hence to recover the direct utility function \(u({\mathbf{x}}_{t} , \, {\mathbf{z}}_{t} )\), as discussed with (4) for the complete demand system.

To circumvent the aforesaid issue, LaFrance and Hanemann (1989), instead, propose the concept of weak integrability that takes full advantage of all the information present for the observable goods of interest but is flexible for the unknown information for the unobservable goods. Weak integrability has provided the framework for empirical analysis of incomplete demand systems (Beghin et al. 2004; von Haefen 2010; Zhen et al. 2014). In this subsection, we summarize LaFrance and Hanemann’s (1989) main results of the weakly integrable incomplete demand model as pertaining to our analysis, and draw some implications for our approach to be discussed in the next subsection.

Although individual elements of \({\mathbf{z}}_{t}\) are not observable, total expenditure on \({\mathbf{z}}_{t}\) \((s_{t} ),\) defined by \(s_{t} \equiv {\mathbf{q}}_{t}{\prime} {\mathbf{z}}_{t} = M_{t} - {\mathbf{p}}_{t}{\prime} {\mathbf{x}}_{t} > 0,\) is readily observable. This variable, indeed, plays a pivotal role in the weakly integrable incomplete demand model, and is utilized in a two-step optimization procedure.⁹ A quasi-utility function is first defined as

This function has the same properties of the direct utility function \(u({\mathbf{x}}_{t} , \, {\mathbf{z}}_{t} )\) conditional on \({\mathbf{q}}_{t}\) by treating \(s_{t}\) as a composite commodity with its price normalized to one. Given (5), a quasi-indirect utility function is then defined as

This function has the same properties of the indirect utility function (1). As with the complete demand system, the quasi-utility function (5) is dual to the quasi-indirect utility function (6), which provides the theoretical underpinnings for weak integrability of the incomplete demand system.

Applying Roy’s identity to (6), we obtain the demands for \({\mathbf{x}}_{t} :\)which implies the total expenditure on unobservable goods, \(s_{t} ,\) can be treated as a demand for a composite good expressed by

Using available data on \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} ,M_{t} )\) together with \({\mathbf{x}}_{t}\) and \(s_{t} ,\) the demands for \({\mathbf{x}}_{t}\) and hence \(s_{t}\) can be estimated.

The quasi-indirect utility function (6) is related to the true indirect utility function (1) via the identity (Epstein 1982; LaFrance and Hanemann 1989)

The difference between Eqs. (1) and (6) is due to information loss with the incomplete demand system, which is associated with the structure of \(\psi [{\mathbf{q}}_{t} ,\phi (\,{\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} ,M_{t} )]\) with respect to \({\mathbf{q}}_{t} .\) In particular, the direct and hence indirect utility functions are associated with the demands for \({\mathbf{x}}_{t}\) and \({\mathbf{z}}_{t}\) described in (3) and (4), while the quasi-utility and hence quasi-indirect utility functions are based on the demands for \({\mathbf{x}}_{t}\) only. Clearly, there is an information loss with the incomplete demand system, and, as LaFrance and Hanemann (1989) recognize, this information loss underlying weak integrability makes the incomplete demand system difficult to establish strong integrability, which is more desirable to achieve.

Weak integrability entails some parameter restrictions on the incomplete demand system (7) such as linear in parameters. LaFrance (1990) proposes several parametric forms to represent such incomplete demand systems (see also von Haefen 2002). He notes, however, the adopted parametric demand models are restrictive, in that they are not derived from flexible representations of consumer preferences that can be rationalized by utility maximization. More importantly, the primary concern in incomplete demand analysis is to estimate the demands for observed goods \({\mathbf{x}}_{t} ,\) with little or no attention paid to unobservable other goods \({\mathbf{z}}_{t} ;\) there is no information about individual demands for \({\mathbf{z}}_{t} .\)

Although the demands for \({\mathbf{z}}_{t}\) are unobservable, as argued in the Introduction, there are the implicit quantities associated with \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} )\) and \(M_{t}\) that can be indirectly measured using available data on observable and unobservable goods. Then the demand functions for \({\mathbf{z}}_{t}\) in (4) can be treated as forming the shadow demands for unobservable goods. These shadow demands for unobservable goods together with the demands for observable goods allows us to recover the underlying preferences and hence to establish strong integrability, resolving the key issue in incomplete demand analysis. To show this, we present an intertemporal approach, which allows us to address other problems of incomplete demand analysis as well. The key to this approach is to treat total consumption expenditure on \({\mathbf{x}}_{t}\) and \({\mathbf{z}}_{t} ,\)\(M_{t}\), specified in the indirect utility function (1), as endogenous. In existing studies on incomplete demand systems, and on consumer demands in general, \(M_{t}\) is treated as exogenously given and is left unexplained. However, to the extent that the consumer chooses expenditure to optimally allocate wealth across periods, consumption expenditure is not exogenously given, but is endogenously determined in the consumer’s optimization problem.¹⁰

In formulating an intertemporal optimization problem by endogenizing consumption expenditure, it is important to note that the direct, and hence indirect, utility function in (1) is ordinal. Thus, the intra-temporal allocation of consumption across goods as captured by the demand functions (3) and (4) is invariant to a monotonic transformation of the utility function (1). However, the intertemporal allocation of consumption, which underlies our approach to the incomplete demand system, is invariant with respect to a linear transformation of the utility function, but not to other transformations of this function. For intertemporal preferences, we therefore take a Box–Cox form for \(v{(}{\mathbf{p}}{}_{t}{, }{\mathbf{q}}{}_{t}{, }M_{t} {)}\):where \(\zeta\) is a Box–Cox parameter, with the marginal utility of \(M_{t}\) given bywhere \(v_{M} ({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} ,M_{t} ) \equiv \partial \nu ({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} ,M_{t} )/\partial M_{t}^{{}} .\) While the indirect utility function (3) as a representation of within-period preferences is well defined with its regularity conditions discussed above, we assume that the Box–Cox utility function (10) is continuous, increasing, and, more importantly, strictly concave in \(M_{t}\) for a given \({\mathbf{p}}_{t}^{{}}\) and \({\mathbf{q}}_{t} .\) Concavity of the Box–Cox utility function requires that \(\partial^{2} U_{t}^{{}} /\partial M_{t}^{2} < 0,\) where

This condition ensures the existence of a solution to the intertemporal optimization problem and implies that the necessary conditions are indeed sufficient.

With the transformed indirect utility function (10), the consumer’s optimization problem is to choose \(M_{s}\) for \(s \ge t\) so as to maximizewhere ρ is the constant rate of the consumer’s time preference, subject to the intertemporal finance or budget constraint:where \(A_{s}\) is the value of financial assets at the end of period s to be carried into the next period, \(r_{s}\) is the nominal interest rate on assets that can be both bought and sold between periods s and s + 1, and \(Y_{s}\) is labor income at period s.¹¹ The expectation operator E_t is taken over future variables, using information available at the beginning of period t. We assume that the consumer replans continuously when solving the above stochastic dynamic optimization problem. In other words, the consumer updates his plans continuously by reoptimizing the intertemporal problem at every period s, s ≥ t, with the new information he has. This means that the calendar time τ solution for \(M_{\tau }\) should be the successive time t solution for \(M_{t}\) as planning time s evolves through time, with the present always being time t.

For empirical analysis then, only the first-order conditions necessary for the above intertemporal optimization problem at the initial point in time (\(s = t\)) are relevant. Utilizing the Lagrange method, they are given byandwhere \(\lambda_{t}\) is the Lagrange multiplier associated with the asset accumulation constraint (14) known at time t, which measures the marginal utility of wealth. Equation (15) indicates that the marginal utility of wealth is equated, at the optimum, to the marginal utility of consumption. Equation (16) is known as the Euler equation in consumption/saving studies, which describes an intertemporal allocation of wealth or consumption (Hall 1978; Ludvigson and Paxson 2001). While it is derived with the indirect utility function (1) under general conditions, previous studies on consumption/saving derive it using real consumption with restrictive utility functions based on homothetic preferences.

For empirical analysis, it is more convenient to work with (16) in a ratio form represented by

Using (15), this equation can be rewritten as

Equation (17b) plays a central role in our intertemporal analysis. Notably, it contains expressions for the indirect utility function, \(\nu ({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} ,M_{t} )\) and \(\nu ({\mathbf{p}}_{t + 1}^{{}} ,{\mathbf{q}}_{t + 1} ,M_{t + 1} )\), in two adjacent periods. With the specification of a functional form for the indirect utility function, joint estimation of the demands for \({\mathbf{x}}_{t}\) in (3) with the Euler Eq. (17b), using observed data on \(({\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t} ,M_{t} )\), allows us to estimate the shadow demands for unobservable goods \({\mathbf{z}}_{t}^{{}}\) in (4). Importantly, this provides sufficient information necessary to identify the underlying preferences as represented by the indirect utility function (1) (see Sect. 3.2 for a further discussion) and hence using the result in (2), to recover the direct utility function \(u({\mathbf{x}}_{t}^{{}} ,{\mathbf{z}}_{t}^{{}} )\) and hence to establish strong integrability of the incomplete demand system.

Knowledge of the indirect utility function permits the calculation of exact measures for the welfare effects of changes in the prices of unobservable goods (see Sect. 5.4 for an empirical analysis). These measures can be considered more relevant than those derived from the quasi-indirect utility function (6) based on weak integrability (LaFrance and Hanemann 1989). Hence, we do not have to rely, in empirical analysis, on weak integrability based on the demands for observable goods. This is the gist of the intertemporal approach to the analysis of incomplete demand systems, i.e., the intertemporal incomplete demand model composed of the observable demand system (3) and the shadow demand system (4) together with the Euler Eq. (17).

Empirical analysis requires the specification of an appropriate parametric form for the indirect utility function (1). To properly characterize consumer behavior, however, the chosen functional form should be flexible while satisfying the requisite regularity conditions for within-period, as well as intertemporal preferences. The PIGLOG (Price Independent Generalized Logarithmic) form, popularized by Deaton and Muellbauer’s (1980b) Almost Ideal Demand System (AIDS), is widely used in applied demand analysis. In this study, we have adopted Cooper and McLaren’s (1992) MPIGLOG (Modified Price Independent Generalized Logarithmic) form to characterize the indirect utility function. This form is a generalization of Deaton and Muellbauer’s (1980b) PIGLOG form specified with the AIDS. Our choice for the MPIGLOG form is motivated by its property that regularity, at a base level of real consumption and at all higher levels of real consumption, can be ensured if parameter values satisfy certain simple inequality restrictions (Cooper and McLaren 1996; McLaren and Wong 2009). Other functional forms, such as AIDS or Translog, are only assured to be regular locally, in a region around a data point. Since we also analyze intertemporal consumption behavior, it is important that regularity be satisfied over a large range of values of consumption. The MPIGLOG form also gives a tractable Euler equation for consumption. In essence, the MPIGLOG form is parsimonious but, being effectively globally regular, provides an attractive compromise in tradeoff between regularity and flexibility. This form, however, is based on a complete demand system and does not allow for unobservable goods.

With the MPIGLOG specification allowing for unobservable goods, the indirect utility function (1) is of the following form:where \(\eta\) is a nonnegative parameter with \(0 \le \eta \le 1,\) and \(P_{A} \left( {{\mathbf{p}}_{t} ,{\mathbf{q}}_{t} } \right)\) and \(P_{B} \left( {{\mathbf{p}}_{t} ,{\mathbf{q}}_{t} } \right)\) are price indexes that are positive, increasing, homogeneous of degree one, and concave in (p_t, q_t). To complete the specification in (18), we assume that the price indexes take the forms:for ρ_A > 0, and \(\alpha_{i}^{x} \ge 0,\) \(\alpha_{j}^{z} \ge 0,\)\(\beta_{i}^{x} \ge 0,\) and \(\upbeta _{j}^{z} \ge 0\) ∀ j, j, where \({\text{P}}_{A} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right)\) is a constant elasticity of substitution (CES) function and \({\text{P}}_{B} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right)\) is the product of two Cobb–Douglas functions.

It should be noted that the MPIGLOG specification of the indirect utility function in (18) exploits the rank of a demand system, the number of independent Engel curves, which is equivalent to the number of functions of prices in the indirect utility function (Lewbel 1991). Regularity of the indirect utility function depends on the properties of these functions of prices, and we can only prove regularity if these functions of prices are proper price indices or regular unit cost functions (Cooper and McLaren 1996; McLaren and Wong 2009). The MPIGLOG indirect utility function (18) is of rank 2 with the two price functions or indexes, \({\text{P}}_{A} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right)\) and \({\text{P}}_{B} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right),\) and characterizes nonhomothetic within-period preferences and places no a priori restrictions on the underlying consumer preferences.¹²

Given (18) and (19), we obtain the following derivatives:andwithwhere \(R_{t} \equiv \ln \left[ {\frac{{M_{t}^{{}} }}{{P_{A} \left( {{\mathbf{p}}_{t} ,{\mathbf{q}}_{t} } \right)}}} \right],\) \(E_{{_{Ait} }}^{x} \equiv \frac{{\partial \ln {\text{P}}_{A} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right)}}{{\partial \ln p_{it} }} = \frac{{\alpha_{i}^{x} p_{it}^{{\rho_{A} }} }}{{\sum\limits_{i = 1}^{n} {\alpha_{i}^{x} p_{it}^{{\rho_{A} }} } + \sum\limits_{j = 1}^{m} {\alpha_{j}^{z} q_{jt}^{{\rho_{A} }} } }}\), \(E_{{_{Ajt} }}^{z} \equiv \frac{{\partial \ln {\text{P}}_{A} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right)}}{{\partial \ln q_{jt} }}\)\(= \frac{{\alpha_{j}^{z} q_{jt}^{{\rho_{A} }} }}{{\sum\limits_{i = 1}^{n} {\alpha_{i}^{x} p_{it}^{{\rho_{A} }} } + \sum\limits_{j = 1}^{m} {\alpha_{j}^{z} q_{jt}^{{\rho_{A} }} } }}.\)

Expressions (20) and (22) can be used to derive the demand functions for observable goods x_t via Roy’s identity (3). In a budget or expenditure share form, we havewhere \(W_{it}^{x} \equiv p_{it/} x_{it} /M_{t}\) is the share of the ith \((i = 1,...,n)\) observable good in total expenditure.

Likewise, from (21) and (22), the shadow demand functions, in a budget share form, for unobservable goods z_t can be derived aswhere \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{W}_{ij}^{z} \equiv q_{jt/} \overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{z}_{jt} /M_{t}\) is the shadow share of the jth \((j = 1,...,m)\) unobservable good in total expenditure. The budget share equations in (24) and (25) are derived under nonhomothetic within-period preferences. If within-period preferences are homothetic, η = 0 and budget shares or Engel curves for observable and unobservable goods are independent of income or expenditure, yielding unitary expenditure elasticities for them. While Engel curves are linear in log income with PIGLOG (AIDS), MPIGLOG produces nonlinearity of Engel curves with nonzero η. The adding-up condition for the budget constraint implies that (24) and (25) satisfy that \(\sum\nolimits_{i = 1}^{n} {W_{it}^{x} + }\)\(\sum\nolimits_{j = 1}^{m} {\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{W}_{jt}^{z} } = 1.\) Appendix A provides a discussion of the price and expenditure elasticities derived from the above two budget share Eqs. (24) and (25).

Given (18), we can derive Euler Eq. (17b). Under rational expectations, it is written aswhere \(\frac{{\partial \nu ({\mathbf{p}}_{s}^{{}} ,{\mathbf{q}}_{s} ,M_{s}^{{}} )}}{{\partial M_{s}^{{}} }}\)(s = t and t + 1) is defined in (22), \(\nu ({\mathbf{p}}_{s}^{{}} ,{\mathbf{q}}_{s} ,M_{s}^{{}} )\) (s = t and t + 1) is defined in (18), and \(\varepsilon_{t + 1}\) is an expectation error at time t + 1 with \(E_{t} [\varepsilon_{t + 1} ] = 0\) and \(Var_{t} \left( {\varepsilon_{t + 1} } \right) = \sigma_{t + 1}^{2} .\) In our analysis, the error variance is assumed to be constant, i.e., \(Var_{t} \left( {\varepsilon_{t + 1} } \right) = \sigma_{{}}^{2} ,\) because the representation of the Euler equation in the form of (17), rather than (16), should provide a more favorable disposition toward constant variance.

With the MPIGLOG indirect utility function (18), it is easy to check the regularity conditions for within-period and intertemporal preferences. Provided that the conditions on the two price indexes in (19) are satisfied over the region \(M_{t}^{{}} > P_{A} \left( {{\mathbf{p}}_{t} ,{\mathbf{q}}_{t} } \right)\) for \(0 < \eta \, \le \, 1,\) then (18) will be effectively globally regular (Cooper and McLaren 1996; McLaren and Wong 2009). For intertemporal regularity with respect to \(M_{t}^{{}} ,\) for \(\zeta > 0,\) monotonicity of the Box–Cox utility function (10), i.e., \(\partial U_{t}^{{}} /\partial M_{t}^{{}} > 0,\) is satisfied if \(\partial \nu ({\mathbf{p}}_{s}^{{}} ,{\mathbf{q}}_{s} ,M_{s}^{{}} )/\partial M_{t}^{{}} > 0\) in (22). Concavity of the Box–Cox utility function, discussed in (12), is satisfied if \(\partial^{2} \nu (M_{t}^{{}} ,\,\,{\mathbf{p}}_{t}^{{}} ,{\mathbf{q}}_{t}^{{}} )/\partial M_{t}^{2} < 0\) in (23) for \(\zeta > 0.\)

For empirical analysis, the relevant equation system to analyze the shadow demands for unobservable goods comprises the budget share equations for observed goods in (24), the shadow demands for unobservable goods in (25), and the Euler equation for consumption (26). It is, however, important to note that the budget share equations for observed goods do not have all information to identify the parameters characterizing the shadow demands for unobservable goods, such as \(E_{{_{Ajt} }}^{z}\) and \(\upbeta _{j}^{z}\) ∀ j. This is the essential problem underlying the analysis of the incomplete demand system. In the intertemporal model, however, this problem is resolved by utilizing the Euler equation in estimation. This equation contains an expression for the indirect utility function represented by the MIPGLOG form in (18) that has all information to identify the parameters of the demands for observable goods as well as the shadow demands for unobservable goods.

Estimation of an incomplete demand system is drastically different from that of a complete demand system. For the complete demand system, there is a full set of available data on x_t and z_t. Therefore, the budget share equations for x_t and z_t in (24) and (25) can be jointly estimated to identify the parameters of the MPIGLOG indirect utility function (18) by imposing the adding-up condition that the sum of the budget shares for x_t and z_t is equal to one (Kim et al. 2024). However, this is not the case with estimation of an incomplete demand system. For the incomplete demand system, we have data on the budget shares for x_t only, and, because of the unobservable nature of z_t, we cannot include the budget share equations for z_t (24) in estimation. This means that the sum of the included budget shares for x_t in (24) is not equal to unity, i.e., \(\sum\nolimits_{i = 1}^{n} {W_{it}^{x} } \ne 1\), and we do not drop one budget share equation for x_t to satisfy the adding-up condition. The adding-up condition applies to both x_t and z_t, and has to be imposed in the model by parameter restrictions without including the budget shares for z_t. In our estimation, we include the Euler Eq. (26) and jointly estimate the system of the budget share equations for x_t (24), together with the Euler equation by allowing for the unique feature of the adding-up condition associated with unobservable goods. Inclusion of the Euler equation increases the efficiency in estimation with more degrees of freedom. More importantly, as discussed above, the Euler equation contains necessary information missing in the budget share equations for x_t to identify the parameters of the MPIGLOG indirect utility function (18) and hence to infer the shadow demands for unobservable goods in (25).

To estimate the systems of the budget share equations for x_t and the Euler equation, we add the stochastic error terms to the budget share equations for x_t (24) to allow for errors of optimization or taste shocks, which are uncorrelated with exogenous variables. The error in the Euler Eq. (26) is, by the nature of the first-order condition (17), uncorrelated with all information dated at time t. However, not all of the variables in (24) and (26) are strictly exogenous. In particular, total consumption \(M_{t}\) is endogenously determined in the consumer’s optimization problem (see Sect. 2.2) and hence is correlated with the error terms. Also, there are variables dated t + 1, such as \({\mathbf{p}}_{t + 1}^{{}} ,{\mathbf{q}}_{t + 1} ,M_{t + 1}^{{}}\), and \(r_{t + 1}\), and the use of the realized values causes them to be correlated with the error terms. These facts suggest an instrumental variables approach to estimate the model.

In this study, we use the iterative three-stage least square (3SLS) procedure to estimate the model. As is common with estimation of the budget share equations, autocorrelation was clearly a problem in our initial estimation. We therefore introduce the first-order autoregressive scheme based on an order N parameterization of the autocovariance matrix using the full information maximum likelihood algorithm of Moschini and Moro (1994). To estimate the model, we consider the following set of instruments: a constant, the time trend, and the time trend squared, the first-order lags of all potentially nonexogenous variables excluding any current variables appearing in the equation system.

To summarize the estimation methodology, the budget share equations for x_t in (24) and the Euler Eq. (26) form a joint estimating system with the adding-up condition imposed. This system is estimated to obtain the required parameter estimates with 3SLS allowing for serial correlation, using chosen instruments because the variables—the right-hand side variables in (24) and the left-hand side variables in (26)—are not strictly exogenous. Then, with the estimated parameters, we check for a model fit by calculating R² and the J test of overidentifying restrictions to assess the relevance of the instruments.

Tables 1, 2 report joint estimation results for incomplete and complete demand systems, with Table 1 displaying model summary statistics and Table 2 presenting the parameter estimates. To ensure the estimated model is properly behaved, the regularity conditions for intra-temporal and intertemporal preferences, as discussed in Sect. 3.1, are checked and are satisfied for both incomplete and complete demand systems, with the inequality \(M_{t}^{{}} > P_{A} \left( {{\mathbf{p}}_{t} ,{\mathbf{q}}_{t} } \right)\) holding, at every sample period. Table 1 shows that the J test for overidentifying restrictions for both incomplete and complete demand systems cannot reject the null hypothesis that the moment conditions for instruments are valid. This provides strong evidence for the relevance of the chosen instruments in our estimation. For both systems, the general fit of the budget share system, measured by the generalized R² for instrumental variables regressions (Pesaran and Smith 1994), is quite good. Autocorrelation diagnostic for both systems, revealed in the LM (Lagrange multiplier) χ² statistics, suggests that serial correlation in the error terms is no longer a concern for most goods.¹⁵

Table 2 presents the model parameter estimates for both incomplete and complete demand systems. The symbols a^x’s and a^z’s represent the estimated parameters that describe the price index \({\text{P}}_{A} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right)\), and \(\beta_{{}}^{x}\)’s and \(\beta_{{}}^{z}\)’s are the estimated parameters for the price index \({\text{P}}_{B} \left( {{\mathbf{p}}_{t} , \, {\mathbf{q}}_{t} } \right).\) Accordingly, these estimated parameters generate the two price indexes, \(P_{A} \left( {{\mathbf{p}}_{t} ,{\mathbf{q}}_{t} } \right)\) and \(P_{B} \left( {{\mathbf{p}}_{t} ,{\mathbf{q}}_{t} } \right),\) which form the MPIGLOG indirect utility function (18). They are used to estimate shadow demands for unobservable goods discussed in Sect. 5.2, expenditure and price elasticities for incomplete as well as complete demand systems presented in Sect. 5.3, and welfare effects of price changes analyzed in Sect. 5.4. Looking at the incomplete demand system, the parameter estimates are reasonable in that they satisfy the regularity conditions for the price indices, and most of them are significant at conventional significance levels. The estimated value of η (0.188) is significantly different from zero at conventional significance levels, providing clear evidence against homotheticity for within-period preferences.¹⁶ For the complete demand system, although the parameter estimates show some different values from those of the incomplete demand system, the two sets of estimates are, in large part, not substantially far off. In fact, given that they are expressed in decimals, they can be considered comparable. This result is very encouraging for our incomplete shadow demand model for unobservable goods because, in the absence of available quantity data for some goods, it provides sufficient information to adequately analyze observed consumer behavior in relation to the complete demand model. We investigate this finding further in the succeeding discussions below.

Using the parameter estimates of the incomplete demand system in Tables 2, we can generate the fitted shadow quantity series for the three unobservable goods considered in our analysis: health services (z₁), other nondurables and services (z₂), and durable goods (z₃). These series are derived from the shadow budget share equations in (25) as

To see whether the estimated shadow quantity series \(\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{z}_{jt}^{{}}\) under-, or over-, estimate the actual quantity series \(z_{jt}^{{}}\) in the data, Fig. 1 plots the time paths of the two quantity series for the three unobservable goods. The quantity indexes can be considered real expenditure on these goods. Looking at the actual quantity series, other goods and durable goods show an increasing trend over time, while health services show a stable pattern within the sample period. A comparison of the actual and shadow quantity series reveals that there is no measurable difference between them for durable goods. However, the shadow quantity index appears to underestimate between them for durable goods. However, the shadow quantity index appears to underestimate the actual quantity series for health services; on the other hand, the shadow quantity index appears to overestimate the actual quantity series for other goods.

While these results are informative, they do not provide conclusive evidence whether the two series are statistically different. Accordingly, we conduct two formal tests—t and \(\chi^{2}\) tests—to substantiate them. For the t test, the null hypothesis is no significant difference between the actual and shadow quantity series for each unobservable variable at each period. The \(\chi^{2}\) test is a goodness of fit test, and the null hypothesis is no significant difference between the observed and shadow quantity series for each unobservable variable for the whole sample period.

The two test results are summarized in Table 3 with the relevant test statistics. The t test reveals that there is no significant difference between the actual and shadow quantity series for health services and others at conventional significance levels, but for durable goods, there is some evidence for the difference between the two series for some sample years. These results are also corroborated by the \(\chi^{2}\) test. It confirms the t test for health services and others, but for durable goods, there is a significant difference between the actual and shadow quantity series for the whole sample period. We found that for durable goods, the standard error for some years (1996–2009) are surprisingly small implying that the t ratios and hence the \(\chi^{2}\) test statistics are surprising high in that period. These results suggest that, while there is no information loss in treating the quantities of health services and others as unobservable, it is not so for durable goods.

Price and income or expenditure elasticities of goods and services are important summary measures characterizing consumer behavior. The derivations of these elasticities are presented in Appendix A. Table 4 displays the estimated expenditure and price elasticities for the incomplete and complete demand systems evaluated at the sample means of the variables, using the parameter estimates of Table 2. The complete demand system is based on the premise that health services (z₁), other nondurables and services (z₂), and durable goods (z₃) are observable, but the incomplete demand system is not. The estimated expenditure and price elasticities are significant at conventional significance levels for both incomplete and complete demand systems. Overall, there is no essential difference in the two elasticities between the incomplete and complete demand systems. Given that there is no substantial difference found between the parameter estimates for the incomplete and complete demand systems presented in Table 2, this is as expected. All of the goods are found to be price-inelastic for both incomplete and complete demand systems. There is no clear pattern for the estimated expenditure elasticities, but alcoholic beverages, tobacco, and narcotics (x₂) and health services (z₁) are weakly income-elastic, especially for the incomplete demand system.

There are studies estimating income and price elasticities of demand for French consumers (see Appendix B for a literature review). However, they do not consider durable goods and use different groupings of nondurable goods. These results are, thus, not directly comparable to the ones reported in Table 3. Yet, some of those findings for the estimated expenditure elasticities are noteworthy. Health services or health care are typically considered a necessity. The finding of health services as a luxury, though not strong, for both incomplete and complete demand systems means that the share of income spent on health expenditures rises with income, and it is believed that this is a consequence of rising income or living standards (Hall and Jones 2007). Durable goods are found to be a necessity for both incomplete and complete demand systems. This result is markedly different from previous finding that durable goods are luxuries (see Clements et al. 2020). Traditional demand studies, however, estimate the demand for durable goods in a flow form, but our analysis is based on a stock form with the user cost. There are also studies for the United States based on a stock form finding durable goods as luxuries, but they are limited in scope and analysis with restrictive utility functions (Mankiw 1985). There might be some expensive or large durable goods that are luxuries, but most of them are small and can be treated as necessities.

It is important to note that treating health services, other goods, and durable goods as unobservable, in large part, does not produce much difference in their expenditure and price elasticities from those estimated by treating them as observable. This result is different from what we found from Table 3 where there is no information loss in treating the quantities of health services and others as unobservable, but it is not so for durable goods. Thus, while the results for health services and others are corroborated with the demand elasticities, they are not born out for durable goods. However, the expenditure and price elasticities are more relevant to understand consumer behavior. Our finding here, then, suggests that even though quantity data are not readily available or not directly observed for health services, other goods, and durable goods, the estimated incomplete shadow demand model provides sufficient information about them to enable us to understand consumer behavior.

Since we can recover the indirect utility function (18) from estimation of the demands for observable goods (24) and the Euler Eq. (26), we can utilize it to assess the impacts of changes in the prices of unobservable goods and alternative economic policies on the welfare of the consumer, using exact measures of the compensating and equivalent variations (Deaton and Muellbauer 1980a).

The consumer currently faces total expenditure denoted by \(M_{t}^{0}\) and the prices of unobservable goods denoted by \({\mathbf{q}}_{t}^{0}\). Suppose now that there is a change in policy or circumstances that cause the prices to \({\mathbf{q}}_{t}^{1} ,\) with no change in total expenditure. The compensating variation (CV) is defined byfor given \({\mathbf{p}}_{t} .\) The CV is the amount of additional money necessary for the consumer to enjoy the utility level \(u_{t}^{0}\) while facing \({\mathbf{q}}_{t}^{1}\) with \(M_{t}^{0} .\) A positive value for CV indicates that the consumer is worse off when facing \({\mathbf{q}}_{t}^{1}\) than they were with \({\mathbf{q}}_{t}^{0}\).

The equivalent variation (EV) is defined byfor given \({\mathbf{p}}_{t} .\) The EV is the amount of additional money that would enable the consumer to maintain the new utility level \(u^{1}\) while facing \({\mathbf{q}}_{t}^{0}\) with \(M_{t}^{0} .\) As with the CV, a positive EV value indicates that the consumer is made worse off under \({\mathbf{q}}_{t}^{1}\) than under \({\mathbf{q}}_{t}^{0} .\)¹⁷

The MPIGLOG indirect utility function (18) is highly nonlinear in \(M_{t} ,\) so it is not feasible to derive a closed form for the CV in (28) and the EV in (29). We, therefore, rely on a numerical method. This is appropriate because the indirect utility function \(\nu ({\mathbf{p}}_{t}^{{}} , \, {\mathbf{q}}_{t} ,M_{t}^{{}} )\) is strictly increasing in M_t, as can be seen from \(\partial \nu ({\mathbf{p}}_{s}^{{}} ,{\mathbf{q}}_{s} ,M_{s}^{{}} )/\partial M_{t}^{{}} > 0\) in (22) for the MPIGLOG indirect utility function, which allows us to numerically solve (28) and (29) for the CV and the EV, respectively, using (18). To carry out the calculations of the CV and the EV, we utilize the GAUSS program language (the NLSYS module of GAUSS version 11.0).

Table 5 presents the welfare estimates, as measured by the CV and the EV, from increases in period, for the incomplete and complete demand systems using the parameter estimates in Table 2. To calculate them, we assume an arbitrary 10% increase in the prices of food and beverages, water and fuel, health services, and durable goods in each year. The following comments are in order. First, overall, there is no substantial difference in the welfare estimates between the incomplete and complete demand systems. Although the incomplete demand system tends to overestimate the welfare estimates for durable goods relative to the complete demand system, the difference cannot be considered drastically substantial. Second, the estimated CVs are greater than the EVs. This is expected because of nonnegligible income effects (see Willig 1976), as indicated by significant nonzero expenditure elasticities found in Table 3. However, the numerical differences between the CV and EV estimates are rather small, amounting no more than €3 in all cases. Third, the estimated CV and EV indicate that French consumers are made worse off after the increase in prices. During the sample period, on average, when we look at the CV for the complete demand model, the largest welfare loss they experience comes from the increases in the prices of food and beverages, while the smallest welfare loss is from the increases in the prices of water and fuel. Fourth, in general, there is an increase in welfare loss over time, but as a percentage of total expenditure (%CV and %EV), it remains about the same except for durable goods. It is however, important to note that French consumers suffer the most from the increase in the price of food in recent years. Lastly, when the effects of all selected price increases are aggregated, we find that the average CV and EV estimates range from €424.4 to €445.9, which is about 4.1–4.5% of per capita total consumption expenditure.