Goal setting and signposting in a rasch-based recommender system to promote household energy conservation

To help consumers overcome the barriers posed by choice difficulties and information deficiencies, recommender systems could be of added value. These are systems that present users with items that are predicted to be relevant to them (Lü et al., 2012), and are used to make sense of large quantities of data and a multitude of options (Lü et al., 2012). They are employed in various domains ranging from music (e.g., Spotify), movies, and series (Netflix), to travel, health, nutrition, and fitness applications. These systems can employ a one-size-fits-all approach (i.e., showing the most popular content overall), or personalize recommendations based on user characteristics, preferences, and/or behaviour. Recommender systems can suggest similar items based on item characteristics, so-called content-based filtering, or recommend items based on preferences of similar users, so-called collaborative filtering, (e.g., Koren et al., 2021), or a combination of both. While they provide many opportunities, they are also accompanied by various challenges; for example, tailored systems need a certain amount of user data to draw from, referred to as the cold start problem (Lika et al., 2014), preferences of users might change over time, and trade-offs between diversity and accuracy must be made to optimize these systems (Lü et al., 2012). Additionally, user interface characteristics might influence their effectiveness (Lü et al., 2012), and different users might respond differently to different systems (Knijnenburg et al., 2011). Small changes in these interfaces, such as user goals and different attribute translations that we examine in this paper, can change system outcomes.

Digital technologies to encourage energy-saving behaviour are typically embedded in theories of persuasion (Adaji & Adisa, 2022). To this end, various persuasive methods have been employed (Schultz, 2014; Warren et al., 2017). For example, one-size-fits-all messages or campaigns might highlight benefits to the environment or financial benefits of reducing energy usage or other sustainable behaviours. Doing so might increase effectiveness, for example when matched to personal values (Van den Broek et al., 2017). However, these persuasive messages are typically not tailored to the current behaviour or past preferences of an individual recipient.

Recommender systems in the energy-saving domain are generally based on the premise that behaviour can be altered, often through persuasive methods, e.g., through reward or cue-based systems aimed at habit changes (Alsalemi et al., 2019). From social judgment theory, we know that persuasive messages are most effective if they are within the so-called ‘latitude of acceptance’; a certain range of ‘attitudes’ close enough to one’s own that someone is likely to be receptive to them (Eagly & Telaak, 1972). If persuasive messages convey attitudes that are too far removed from one’s current stance, they are regarded as being in the ‘latitude of rejection’ and are less likely to be effective (Eagly & Telaak, 1972). The ‘widths’ of these ranges differ per person, as do current attitudes, and therefore, energy-saving recommendations might also need to differ per person to be effective. In order to recommend someone with suitable energy-saving measures, it is therefore important to know what someone’s current attitude towards energy-saving is.

One way of determining this is simply looking at the number of energy-saving behaviours someone has already adopted. This is consistent with methods used in recommender system research, which tap into historical data from its users (Jannach et al., 2010). Someone who already performs a large number of measures is considered to have a positive attitude towards energy saving. The psychometric Rasch model is based on this premise, and more specifically, based on the idea that energy-saving behaviour falls on a spectrum of low to high difficulty measures, which is in turn tied to low and high energy-saving ability levels.

The use of Rasch is embedded in an attitude paradigm called ‘Campbell’s Paradigm’ (Kaiser et al., 2010), which aims to mitigate the attitude-behaviour gap. It assumes attitude and behaviour are two sides of the same coin (Kaiser et al., 2010), proposing a stochastic, ‘logit’ relation between individuals’ attitudes and measures’ difficulty when predicting whether a measure will be performed. An attitude becomes apparent by the increasingly difficult behavioural steps that an individual takes towards attaining an attitudinal goal, such as energy conservation. This is modelled on a one-dimensional scale that captures both persons and measures on their ability and difficulty (Kaiser et al., 2010; Starke & Willemsen, 2024). The scale assumes measures are more difficult if they are performed by fewer people, while a person who performs more measures is considered more able, having a stronger ability or attitude. The probability P that an individual n performs a certain measure i can be calculated using the following equation:

In which θ is the individual’s ability, and δ is the measure’s behavioural costs. This formula results in item-characteristic curves (ICC’s) that depend on item difficulty, as can be seen in Fig. 1, indicating the likelihood (Y-axis) that an individual with a certain ability/attitude (X-axis), performs a certain item (different curves). When ability and difficulty match, the probability of engagement is 50%.

Full size image

Because Rasch allows for sorting people and measures on the same one-dimensional scale, this approach seems especially suitable for tailored energy-saving advice and recommender systems; after a user’s ability is determined, they can now be given meaningful recommendations to curb their energy usage. For tailored advice, the Rasch model helps to explore the trade-off between a recommended measure’s novelty and feasibility for a particular user (Starke et al., 2020).

Using the Rasch model as a tailoring algorithm, a recommender system was found to be more effective than a non-personalized system, in terms of user evaluations (Starke et al., 2020). However, it did not necessarily make users choose the measures with higher energy savings. Additional studies that aimed to support higher energy savings through nudges, e.g., by presenting ‘algorithm fit scores’, social norms, and smart saving scores (Bams, 2018; Starke et al., 2017, 2021), affected which measures were chosen, but did not lead to higher savings. Given that we can influence decisions in such a system, the next step is to determine how we can help people choose more efficient recommendations, such that overall savings do increase. One hypothesized problem is that kWh savings and their magnitude (e.g., how much is 100 kWh) are unclear. Attari et al. (2010) show that people tend to underestimate the effectiveness of conservation measures; participants in their study thought that curtailment measures (e.g., turn off the lights after leaving a room) were most effective for saving energy, as opposed to efficiency measures (e.g., insulate a cavity wall), which in reality tend to be higher in kWh savings. In line with these inaccurate expectations of kWh savings for household energy-saving measures, it might also be difficult for people to accurately judge what a reasonable amount of savings would be when using the recommender system, with respect to choosing a limited number of measures. Moreover, users in earlier Rasch recommender studies by Starke et al. (2020), might not have been committed or have had a specific goal in mind and thus opted for only few measures.

To mitigate the issues from earlier studies and to encourage higher savings in this study, we examine two different mechanisms in our novel energy recommender system. First, we investigate the effectiveness of including goal-setting functionality in our recommender system, which aims to mitigate erroneous kWh savings perceptions. In addition, we investigate value-based motivational frames, through signposting, by expressing environmental benefits in terms of other attributes. Signposting refers to attribute translations (e.g., kWh, Euro) in line with user values, with the aim of activating certain user objectives (Ungemach et al., 2018). We will discuss these aspects in more detail below. In doing so, we address the following overall research question:

According to Locke and Latham (2002), the mere presence of goals increases performance, and an adequate balance between goal difficulty and feasibility leads to higher satisfaction. A goal that is assigned by an authority figure will furthermore strengthen the belief that this goal is reachable (Locke & Latham, 1990). At the same time, feeling personally responsible for reaching a goal, leads to higher feelings of competence (Eccles & Wigfield, 2002). Goals that are autonomously set furthermore lead to higher achievement (Koestner et al., 2008). Therefore, incorporating goal setting into an energy-saving recommender system might help users to achieve higher savings than they otherwise would.

Goal setting has shown promising, yet varying effects on energy use. Abrahamse et al. (2007), who consider direct energy usage (e.g., not energy use through travel), found a significant effect of goal setting and education on energy usage. Instructing households to reduce their energy consumption by 5%, along with performance feedback and education, resulted in an energy reduction of 5.1% on average over the course of a 5-month study. In a similar vein, Harding and Hsiaw (2014) asked their users to select energy-saving measures from a list of recommendations. They found that users who committed to reduce their current energy use between 0 and 15%, achieved the highest energy reductions at around 11%. Users who either chose no items, or whose item-choice resulted in overly optimistic goals, only saved between 1 and 1.5% on average. Interestingly, many users tended to over-commit, with 40% of them opting for an overly optimistic goal between 15 and 50%, and 12% of users opting for an even higher goal.

The previous research indicates that goal setting can affect the adoption of energy-saving measures. The goal-setting theory furthermore states that self-chosen goals ensure a degree of autonomy. However, some direction might be needed to inform users what would be a reasonable goal for them. Following this, we presented users in the goal condition with three saving goals (an easy, moderate, and difficult goal) such that they had adequate guidance yet still had a degree of autonomy. We expected that this would lead to higher savings and higher system satisfaction than having no goals:

Energy-saving metrics are not restricted to kWh. They can be presented in different units, such as CO2 emission reductions and monetary savings. The understanding and expected consequences of measures can depend on how the presented information is framed. Stadelmann and Schubert (2018) studied the effect of displaying household appliance energy usage as either monetary or kWh costs in an online shop, e.g., the energy usage of fridges and washing machines. They found no difference between these labels: both labels were effective in increasing the sales of energy-efficient appliances. However, they did not distinguish between different kinds of customers who might be affected differently by different labels.

In a study by Ungemach et al. (2018), it was observed that people with stronger environmental concern, as measured by the NEP scale (New Environmental Paradigm), were more likely to pick an energy-efficient car when presented with information on its CO2 emissions per mile, as compared to its fuel usage per mile. Thus, the same exact information being presented with different units, resulted in different behaviour for different users. As a user could obtain the presented information by a simple scale translation, there seems to be no immediate reason for this effect. However, according to Ungemach et al. (2018), such attribute translations might activate different objectives for users with different personal values. This mechanism is referred to as ‘signposting’; the presentation of information in line with user values, helps to guide users towards their personal objectives, hence the name ‘signpost’. Ungemach et al. (2018) state that signposting effects are distinct from priming and framing effects, in the sense that they do not rely on valence shifts, but rather on a seemingly neutral conversion of different units. Furthermore, the effects of the study could not be explained by a knowledge gap.

As we can present the saving metrics in our energy recommender system with various units, e.g., Euro, CO2, or kWh savings, we are interested in the effect of these different signposts for users with different values. Following the study by Ungemach et al. (2018), we think that CO2 emission metrics might resonate more strongly with users who have stronger environmental concern. Similarly, monetary savings might resonate more with users who attach more importance to money, although this has not been studied before. We thus expect that a CO2 signpost will motivate users with a high level of environmental concern to save more energy. We also expect that the monetary (Euro) signpost will increase savings for users who place more importance on money, and that the kWh signpost will work equally well for everyone, as this has no direct apparent link to personal values. We also expect various effects on user experience, which we will elaborate on in the next sections.

We do note that a similar study involving signposting and goal setting has previously been conducted by Brandsma and Blasch (2019), in which they found no effect of attribute translation on overall willingness to conserve energy. However, they did find differences between users with several different value orientations for their willingness to save energy. Those with stronger biospheric values were more motivated to save energy, while those with more egoistic values were less willing to reduce their energy usage, except when savings were displayed as monetary savings. However, the study by Brandsma and Blasch (2019) only considered a single energy-saving measure: turning off stand-by appliances on a daily basis. Furthermore, participants were only asked to imagine setting themselves a goal and then say if they were willing to perform this action. Given that our current study differs substantially (with over 130 different saving measures, and persuasion towards real behaviour change), we hypothesised that these system manipulations might lead to different outcomes in our system.

Platforms that use recommender systems are usually evaluated on behavioural outcomes (Jannach et al., 2010), but several platforms also evaluate perception and experience aspects (Knijnenburg et al., 2014). Beyond achieving energy savings, users of an energy conservation platform should also be willing to re-use it (Geelen et al., 2019; Starke et al., 2017). Previous recommender studies show that the effects of system manipulations on user behaviour (i.e., amount of energy savings chosen) and user experiences (e.g., satisfaction) are mediated by user perceptions (Knijnenburg et al., 2014). For example, in a different energy-saving recommender study, the interaction effect between preference elicitation method complexity (i.e., the method of asking what a user likes and wants) and domain knowledge of users affected how satisfied users were and, in turn, how many energy savings they selected (Knijnenburg et al., 2014).

Such mediated relations are described by the user-centric evaluation framework by Knijnenburg et al., (2012, 2015). They show that system outcomes in recommender systems (e.g., choice satisfaction, kWh savings), are often explained by changes in subjective perceptions of the system itself (‘subjective system aspects’, such as choice difficulty and system satisfaction). In a study on the use of social norms in an energy recommender system (Starke et al., 2021), all effects of various social norms on number of chosen savings and choice satisfaction, were mediated by changes in perceived feasibility (a subjective system aspect). In a study on effective user interface designs, the effects of interface variations on choice satisfaction, were mediated by changes in perceived effort and perceived support (Starke et al., 2017). Although in this study, there were also some direct effects from interface variations on measure choices, which were not explained by changes in subjective system aspects.

In the current study, we want to evaluate the effects of goal setting and signposting on system outcomes, and if these effects exist, evaluate whether they can be explained by changes in how users perceive the system. We will explain this in detail in the conceptual framework that we will discuss in the next section.

Following the user-centric evaluation framework (Knijnenburg & Willemsen, 2015), we are interested in several system outcomes, as well as several subjective system aspects. For system outcome variables, these are the ways in which people evaluate their experiences of using the system (i.e., user experience), including the extent to which people are satisfied with their chosen measures (choice satisfaction), and the extent to which they now feel confident in saving energy (energy-saving self-efficacy). These also include more behavioural outcomes, such as the total selected savings (kWh savings, as an interaction outcome).

Following the framework's setup (Knijnenburg & Willemsen, 2015), we formulate a number of expectations. See Fig. 2 for an overview. We expect that the effect of our objective system aspects (goal setting and signposting) on the experience (self-efficacy and choice satisfaction) and behavioural outcome variables (selected kWh savings), will be mediated by subjective perceptions of the system. These include the difficulty users experience in choosing between measures (choice difficulty), the extent to which they feel the system supports their saving goals (goal support), the extent to which users are satisfied with the system (system satisfaction), and lastly, the extent to which users believe that the presented measures are feasible for them (perceived feasibility).

Full size image

For example, presenting information in a way that is relevant to the user (i.e., signposting in line with user values), might aid the decision-making process, thus reducing choice difficulty and possibly improving goal support. We expect that such subjective perceptions of the system could lead to higher savings and improved user experiences. A similar trend was also seen in Starke et al., 2021. We furthermore think that goal setting could improve user experience by providing a clear behavioural target. Goals that are realistic and achievable for a user, might lead to higher system satisfaction, and in turn (following the framework), might lead to higher savings.

Lastly, increased savings might lead to increased choice satisfaction, similarly to the findings of Knijnenburg et al., 2014. Figure 2 depicts these hypotheses, in a model that broadly follows the user-centric recommender evaluation framework.

Figure 2 also presents additional paths, which follow the general route of objective system aspects to system outcomes via subjective system aspects, but of which the precise relationships are not based on earlier studies or theory (e.g., because we introduced our own measure of ‘goal support’, or our system differs from earlier ones in terms of system manipulations). Following the Framework for Evaluating Recommender Systems (FEVR) by Zangerle and Bauer (2022), and specifically their distinction between confirmatory and exploratory evaluations of recommender systems, we note that our current study, and especially the SEM model part of this study, has more of an exploratory character rather than a confirmatory one. The possible relationships between objective system aspects and eventual system outcomes might be explained through multiple possible pathways, however, not all of these pathways can be theoretically supported due to a lack of prior work.

We summarize this conceptual framework and our approach with the following (exploratory) hypothesis:

In total, 212 people participated in the initial study (202 after outlier removal), consisting of 86 males, 112 females, and 4 other/undefined, with a mean age of 30.5 and a median age of 25, with 75% of participants being younger than 32. 82% obtained at least some college education, while only 23% were homeowners. Out of all participants, 22% lived in a semi-detached house, 26% in a terraced house, 33% in an apartment, and 17% in a room. Most participants did not know or did not want to disclose the energy label of their house (56%), and otherwise, 10% indicated energy level A, 9% indicated B, 12% indicated C, 4% indicated D and the remaining 9% indicated energy label E or below.

We recruited participants at least 18 years old, living in the Netherlands, as the saving measures were tuned to the Dutch context. We excluded those below 18 as they might have particularly little say in household energy usage. 117 participants were recruited via our university departments’ participant database. This database consists of (former) students and older (non-student) adults. The mean age of this group was 31.3 (95% CI = 28.1. SD = 34.5). 94 participants were recruited via the online Prolific database, which attracted a relatively young demographic to the study, with a mean age of 29.9 (95% CI = 28.05, SD = 31.84). We had one external participant who joined because someone sent them an invite in the final screen. Although we aimed our recruitment to contain mostly adult participants that would be able to engage in energy saving measures given their living situation, the sampling still resulted in a somewhat young sample, though still substantial older than the typical student sample and for most in living situations for which our energy savings were relevant.

The follow-up study was completed by 170 returning participants (160 after outlier removal), with a comparable demographic distribution. Studies were run between May and June 2023. Participants were paid ~ €3,- for completing the initial study, and ~ €4,- total if they also completed the follow-up, exact amount depending on participant database.

Our experiment was subject to a 2 × 3 between-subject design (Table 1). First, participants were either asked to set a goal or not. Second, there were three signpost conditions, presenting attributes and goals in terms of either kWh savings, monetary (€) savings, or annual kg CO2 reductions.

The study used a newly developed online recommender system, based on earlier work (Starke et al., 2021), backed by a database of 135 energy-saving measures, obtained from Bams (2018). These measures ranged from ‘easy’ ones, e.g., ‘Cook with pots & pans the same size as the heating element’, to more difficult ones, e.g., ‘Install a centralized temperature system with zone controls & thermostats’, to ‘install a mini windmill’ as the most difficult measure. To be able to provide ability-tailored advice, users indicated for 19 measures (randomly selected from the entire range of measures) whether they performed them or not, referred to as ‘(number of) current items’ in Table 3. This allowed us to calculate the ability of the user (which ranged between -2.7 and 3.6 logits for the entire sample). Measures that did not apply to a user’s housing situation were not used in the ability (i.e., attitude) calculation.

Afterwards, participants in the goal condition could select an accessible (600 kWh), moderate (1200 kWh) or challenging (1800 kWh) saving goal, based on the mean savings (1200 kWh) and median savings (660 kWh, for those who selected at least some amount of savings in the study by Bams (2018)). This goal was displayed with the signpost a participant got assigned to (kWh/Euro/CO2). For the monetary goal, we used the weighted average of Dutch kWh and/m³ prices (€0.30/kWh), resulting in saving goals of 180, 360 and 540 Euros. For the CO2 goals, we used an average of 0.25 kg CO2/kWh, resulting in saving goals of 150, 300 and 450 kg CO2. All other signposts were also shown as additional information. An example of a goal is given in Fig. 3 (600 kWh).

Full size image

Next, participants were given instructions on how to use the interface with screenshots (Refer to Oonk, 2023, Fig. 6). They were asked to pick items they intended to perform in the future, to remove items they already performed (after which new ones would appear at the bottom of the list) and were shown how to obtain more information on measures (by hovering over the images or clicking the 'more info' buttons). An example of the interface is given in Fig. 4.

Full size image

Afterwards, all participants were presented 20 energy-saving recommendations closest to their ability level (sorted by absolute distance) to choose from. (We refer to the total number of chosen items as ‘# chosen items in Table 3, and the total amount of savings (e.g., in total kWh or total monetary value), as chosen savings throughout this paper.) The measures were accompanied by highlighted saving metrics as seen on the right in Fig. 5, which were translated as either a kWh, monetary, or CO2 metric depending on the signpost condition the participant was in.

Full size image

For exact conversion formulas between these, refer to Oonk (2023). The total selected savings were shown at the top of the screen, as a proportion of the saving goal if applicable (as depicted in Fig. 5a and Fig. 5b). The recommender interface can be tested at: www.besparingshulp.nl/demo.

Following the choice task, participants were surveyed about their perceptions and experiences with the system, with the questions shown in Table 2. We used items from Willemsen et al. (2016) to measure choice difficulty, items from Starke et al. (2017) to measure perceived feasibility, items from Willemsen et al. (2016) to measure choice satisfaction, and items from Knijnenburg et al., (2012, 2014) and Starke et al. (2017) to measure system satisfaction. Goal support questions were largely original items, measuring how well the system supports the pursuit of energy saving.

Additionally, we used the revised New Environmental Paradigm (NEP) (α: 0.81 for both the initial study and follow-up study) by Dunlap et al. (2000) to measure environmental concern, the Money Importance Scale (IMS) by Franzen and Mader (2022) (α: 0.77 for both samples) and the environmental self-efficacy scale by Lee and Tanusia (2016) (α: 0.87 & 0.85, AVE: 0.60 & 0.57 for the initial and follow-up study respectively). For the full overview of questions (before factor analyses), refer to Oonk (2023). To verify the proposed constructs of our theoretical model, we performed an exploratory factor analysis (EFA) and then CFA and SEM using the mPlus software package (Muthén & Muthén, 2023). Items with low factor loadings (below 0.4) or high cross-loadings in the EFA (main loading of a factor should be twice as high as the loadings on another factors) or in CFA/SEM, items with several high (> 20) Mod indices with other items (Several goal support, choice satisfaction, and system satisfaction questions) were removed from the subsequent CFA and SEM analyses. In our SEM analysis, Perceived feasibility had a too high correlation with choice difficulty (correlation above 1, indicating in MPlus that these factors cannot be discriminated in the SEM model). The predictors of perceived feasibility were weaker than those of choice difficulty. Additionally, our system manipulations were more focused on helping users choose (by emphasizing reasonable saving targets and providing personally meaningful information through signposts), rather than influence perceived feasibility; After all, users chose their own measures (and their own goal, when applicable), and thus partially determined themselves how feasible their final set of measures would be. Therefore, we included choice difficulty in our final SEM model rather than perceived feasibility. From the EFA, we determined that goal support questions 3, 4, and 5 with system satisfaction question 2, measured a single construct. To simplify the analysis somewhat, we used the unweighted computed scores for the money importance scale (IMS) and new environmental paradigm (NEP), rather than using the individual questions for the CFA/SEM. This was done because these were existing and validated scales, and their statements were not directly related to the system itself. Participants could save their selected measures and received an email with an overview one week after the study. After four weeks, participants were asked to join the follow-up study to indicate which measures they ended up performing. We compared the differences in savings between goal conditions with rank-sum tests, the interaction effect between values and signposts with several (robust) multiple regressions, and the effect of subjective system aspects with a structural equation model (SEM).

We first compared the savings between the goal and no-goal conditions. We hypothesized [H1] that users in the goal condition would obtain more savings than users in the no-goal condition. The energy savings for the initial (main) study and the follow-up survey across these conditions are depicted in Fig. 8. On average, surprisingly, participants chose less energy savings in the goal condition (M = 2166 kWh; SD = 2624) than in the no-goal condition (M = 3880 kWh; SD = 5452). A rank-sum test indicated that this difference was, however, not significant. Four weeks later, in the follow-up survey, the difference was reversed, with goal-condition savings being higher on average (M = 340 kWh; SD = 565) compared to the no-goal condition (M = 294 kWh; SD = 516). However, once again, this difference was not significant (p = 0.46).

Full size image

We furthermore compared the savings between the three goal amounts, as graphed below in Fig. 9. This was not part of any hypothesis but might still give some insight into what extent people are motivated to reach their self-chosen goal. What stands out is that in the 600-kWh condition, there are more participants with very low chosen savings. This might indicate that those without much motivation to save energy, chose this goal.

Full size image

Figure 10 compares the chosen savings per signpost condition. We hypothesized [H2] that signposts in line with user values, would result in higher savings than signposts not in line with user values. First, we compare the savings in each condition, irrespective of user values. We found that people chose significant less savings in the CO2 condition in the initial study (M = 1782 kWh; SD = 2554), compared to the kWh (M = 3652 kWh, SD = 5049) and Euro (M = 3620 kWh, SD = 4843) conditions (χ² = 10.11, p < 0.001). After four weeks, participants saved on average 347 kWh (SD = 507), 334 kWh (SD = 573) 268 kWh (SD = 538) in the CO2, kWh, and Euro conditions, respectively. An Analysis of Variance (ANOVA) showed no significant difference between these three signpost conditions in the follow-up study, nor when a quadratic transformation was performed.

Full size image

To test for the signposting effects [H2], we inspected how a user’s values (NEP) affected behavioural and user experience outcomes, using moderated regression. See Fig. 11. We found a positive interaction effect between the NEP score and a CO2 signpost, where a higher NEP score led to higher choice satisfaction (β = 0.30, p < 0.01) and higher self-efficacy (β = 0.35, p < 0.05), compared to the downward-sloped effect of the kWh signpost baseline. The latter is depicted by the orange lines in Fig. 11a and b. However, this was mostly due to a downward slope of the kWh signpost, as depicted by the blue lines in Fig. 11a and b. For choice difficulty, the interaction effect of NEP and CO2 was negative (β = −0.55, p < 0.01, Fig. 11c). Again, the kWh baseline differed between low and high NEP scores more strongly than the CO2 signpost did. Additionally, the main effects of the CO2 signpost on choice satisfaction (β = −0.59, p < 0.01) and self-efficacy (β = −0.45, p < 0.05) were negative (see the regression outputs in Table 4). Therefore, the beneficial effects of a CO2 signpost only applied to a minority with very high NEP scores, while the kWh signpost seems beneficial for a larger group of people lower on the NEP scale. We did not find such interaction effects on actual energy savings in the initial and follow-up studies (Fig. 11d, depicts the savings in the initial study). We neither found any interaction effects between signposts and the Importance of Money (IMS) score on any of these outcome variables.

Full size image

Finally, we hypothesized that the effects of system manipulations (goal setting and signposting) on system outcomes, would be mediated by changes in subjective system aspects [H3]. To examine these mediated effects between changes in the interface, user perceptions and choices, and system evaluation, we constructed a structural equation model (SEM) in Mplus (Muthén & Muthén, 2023). The SEM model had a good fit: χ2(202) = 255.00, p < 0.01, RMSE = 0.040, 90%-CI = [0.025,0.052], CFI = 0.981, TLI = 0.979. The fit for the follow-up SEM model was also good, and their results are merged in Fig. 11, depicting only significant paths. The system satisfaction variables were only measured in the initial study, and their effects remained very similar in the follow-up. The paths from Euro signpost were no longer significant in the follow-up study. For separate SEM graphs of the initial and follow-up study, refer to Oonk (2023; p. 68, Fig. 18) and Oonk (2023; p. 88, Fig. 28).

We found several correlations between signposts and latent variables, some of which were mediated by pro-environmental values (NEP score). We found no direct effects from signposts and interactions on outcome variables (Savings, choice satisfaction, and self-efficacy), nor did we find an effect of the Importance of Money Score (IMS score) on savings or any of the other factors. Discriminant validity of choice satisfaction (AVE 0.57) is not maintained, because choice satisfaction was better explained by choice difficulty (β = −0.81, p < 0.001) than by its own questions. Choice difficulty, as per the user-centric evaluation framework by Knijnenburg and Willemsen (2015), is considered to be a subjective system aspect, and the questions we used measure user perceptions, whereas choice satisfaction is usually referred to as an experience outcome and the questions we used are more evaluative. Moreover, they show different (and logical) relations to the other related concepts (self efficacy and goal support). Therefore, merging these into one construct would not be entirely logical, and it would make it harder to gain insight into the relationship between these categories of latent variables and other concepts in our model.

From the model in Fig. 12, we observe that all possible effects from signposts and values on these outcome variables were mediated by choice difficulty, and all effects on savings were mediated by changes in goal support. There were no direct effects from any of the objective system aspects nor the personal characteristics (NEP), on outcome variables. We expected that all effects on outcome variables (savings, choice satisfaction and energy-saving self-efficacy), would be mediated by changes in subjective system aspects. While the main effect of a CO2 signposts (compared to a kWh signpost) increased choice difficulty (β = 1.42, p < 0.001), CO2 signposts moderated by higher NEP scores decreased choice difficulty (β = −0.55, p < 0.01). Effects on savings were indeed furthermore mediated by goal support; the reduction of choice difficulty, led to increased experienced goal support (β = −0.38, p < 0.001). This increased goal support, then led to higher chosen savings in the initial study (β = 0.49, p < 0.001), and higher actual savings in the follow-up study (β = 0.71, p < 0.001). The total indirect effect of the interaction CO2 x NEP score on (initial) savings via this route was β = 0.10 (p < 0.01), meaning that increasing NEP score led to slightly higher savings in the CO2 condition as compared to the kWh condition, and this was caused by a decrease in choice difficulty and increase in goal support. This is in line with hypothesis 2a, but something we did not observe in the regression in Sect. 3.3, possibly because more variables (IMS scores and goal conditions) were included there. However, if we would correct for this double testing, this interaction effect would still be significant (p < 0.025).

Full size image

Next to a mediation of subjective system aspects on savings, we expected similar mediation effects on experience outcomes (energy-saving self-efficacy and choice satisfaction). Indeed, a decrease in choice difficulty led an increase in self-efficacy (β = −0.38, p < 0.01). The total indirect effect from the interaction NEP x CO2 on self-efficacy, including the 3 other paths via goal support, savings and choice satisfaction, was β = −0.97 (p < 0.01). Increased goal support also led to higher choice satisfaction (β = 0.57, p < 0.001). Lastly, increased selected savings led to an increase in choice satisfaction (β = 0.20, p < 0.01), which in turn led to a higher energy-saving self-efficacy (β = 0.29, p < 0.001). This all shows that indeed, these experience outcome variables were also mediated by changes in subjective system evaluations, similarly as seen with the savings, with the most important mediators being choice difficulty and goal support.

As choice satisfaction and self-efficacy are outcome variables that we measured only in the initial study, the pathways from actual savings in the follow-up study, to these outcome variables, are not depicted in the figure. They can be found in Oonk (2023).

We have observed various indirect effects of system characteristics on subjective perceptions of the system and have observed that these subjective perceptions influenced experience and behaviour outcomes. We have furthermore observed that decreased choice difficulty led to increased goal support (both subjective measures), and that increased goal support led to higher savings and higher choice satisfaction. Both increased choice satisfaction, and decreased choice difficulty, led to higher energy-saving self-efficacy. We also saw that increased savings led to increased choice satisfaction, which was also observed in the study by Knijnenburg et al. (2014).

We have not observed significant differences in kWh savings or user experiences between the goal and no-goal conditions, in contrast to previous non-recommender studies in an energy-saving context (Abrahamse et al., 2007; Harding & Hsiaw, 2014). We argue that the selection of energy-saving measures by users could have been experienced as a goal-setting task in itself (i.e., the total selected measures constitute a saving goal), diminishing the effect of overarching goal setting. Furthermore, the personalized Rasch system already aids users in finding ability-matched saving measures, which could minimize the effects of goal setting in terms of decision support. Additionally, goal setting in an energy-saving context is often more effective when paired with feedback (Abrahamse et al., 2005), which was lacking in our session-based system. Goal setting within a system that can be used continuously over a longer time, might be more effective. Although participants received a link to their chosen measures approximately one week after participation, the goal was no longer visible after the initial study. This might have made the goal setting less salient and less impactful on the behaviour of participants.

Lastly, participants selected on average 3057 kWh in savings, as compared to 1200 kWh in a previous study (Bams, 2018), surpassing the levels of the goals we suggested. This might have been due to a misinterpretation of the task for some participants: Several participants asked why there was not a ’I do not want to do this’ button, and 33 participants clicked on either ’I’m already doing this’ or ’I will do this’ for all items in the list. They most likely thought that they had to click on either of those buttons for every recommendation in the list, while this was not required (they were allowed to ‘ignore’ recommendations entirely, though we should have made this clearer in the instructions). This happened 13 times in the goal condition, and 20 times in the no-goal condition. This misinterpretation might have had a stronger effect in the no-goal condition, due to the lack of a realistic saving target. For the signpost conditions, this happened 12 times in the Euro condition, 14 times in the kWh condition, and 7 times in the CO2 condition. Furthermore, we had 14 participants (across conditions), click on all but 1 or 2 items in the list, who might have had the same misunderstanding. Excluding these 33 or 47 participants would bring the average chosen kWh savings down to 2257 kWh or 2073 kWh respectively.

Furthermore, we have compared the effects of three different signposts (kWh, CO2, and Euro) on user energy-saving behaviour and user experience. We have found that the CO2 signpost has led to lower chosen savings in the initial study, irrespective of value orientation. However, there was no significant difference in achieved savings after four weeks. It could be that the kWh and Euro signposts motivated people to choose more measures, but this did not lead to a higher propensity to perform these measures. We have observed that users facing a CO2 signpost reported better user experience outcomes at the upper end of the NEP scale, compared to users facing a kWh signpost. This was, however, caused by the kWh signpost being less compatible with increasing NEP score, rather than the CO2 signpost working better with increasing NEP scores.

This contrasts with the findings of Ungemach et al. (2018), where a CO2 signpost led to more efficient car choices with increasing NEP scores. This car comparison task was merely a choice task and did not call for real-world action. Similarly to the NEP scale, such hypothetical measures might not always correlate with real-world behaviour. Previous studies have shown that a higher level of environmental concern (NEP), does not lead to an increase in energy savings (Urban & Ščasný, 2012). The energy recommender system we used, calls for an actual change in behaviour, and might therefore have yielded different results.

Although not the main aim of our current research, the eventual aim of the platform is to help people save energy. Participants achieved on average (316 kWh) of predicted yearly savings with the measures implemented after just four weeks. This might increase over time, as it does not include the items that a user says they have started implementing (M = 349.8 kWh; SD = 1080.3) or are still planning to do in the future (M = 1314.8; SD = 2852.8), only the ones they are already doing consistently or have fully implemented after four weeks.

Although we did not obtain data on the energy usage of our participants, we can try to put this 316 kWh into perspective; The average yearly energy usage of a Dutch household was 2500 kWh and 820 m³ gas total in 2023 (Centraal Bureau voor de Statistiek (CBS), 2024). Converting gas savings to kWh savings (0.102 m³ gas per kWh) in the same way we did in our system (Refer to Oonk, 2023), this would be 2500 kWh + 820/0.102 = 10,539 kWh equivalent energy usage on average. Although 316 kWh is then a relatively small percentage of the total amount of energy used (2.99%), it is already more than 10% of the electricity usage. We also note that out of the 10 most popular completed measures, 7 were electricity based, though the few gas-based measures generally had higher savings. Most measures that save substantial gas-usage like insulation, need more time to be implemented. We think that these predicted yearly kWh savings are still a worthwhile amount, especially for our somewhat younger sample. Additionally, it would be worth exploring how the system can support the not-yet-implemented but intended behaviours over the long-term.

Our study is subject to a few limitations. One notable demographic limitation is that of the sample’s age. Our participants are relatively young, with a mean age of 30.5, of which 75% were below 32 years old. Few of our participants owned homes, meaning that some might not have been able to implement the often more efficient structural changes (as compared to behavioural measures). Therefore, participants might have been motivated to choose measures that they could not actually perform, and perhaps even to report completing them. We did not check whether non-homeowners picked items that would require structural changes, which would be interesting to look at for future research. Actual energy usage data might have been more informative than the self-reporting in our system, though this would require a much longer-term study and might be experienced as impeding on privacy. Additionally, one of the appeals of a Rasch-based system, is precisely that one does not need to provide energy usage data to receive recommendations. Additional research that involves mostly home-owners or where certain items are filtered out for non-homeowners, might provide additional insights and could show clearer differences between conditions.

Furthermore, the list of measures had a large range of savings, from 0 to 8000 kWh. Although we did observe substantial differences in means between groups, these were not significant. This might, to a certain extent, also have been caused by large standard variations in our data. A larger sample size could indicate whether this larger variation is systematic. Making the saving aid publicly available to a broad audience, might make it easier to test certain system manipulations on a larger sample, and give further insight into possible effects. System manipulations would then be done by means of A/B testing, for example (presenting one set of users with one version, and another set of users with another). Participants furthermore had seemingly high levels of environmental concern, which might have influenced our results; 91% of participants agreed (to some extent) to the NEP scale proposition that 'Humans are severely abusing the environment'. Thus, the observed range of NEP scores was perhaps too small to show large effects between groups. We observed an average NEP score of 0.84 with an SD of 0.55 (on a scale of −2 to 2, which would be 3.55 on a scale of 1–5). For comparison, a previous study in 2023 across Germany and Poland reported a mean of 0.695 (or 3.695 on a scale of 1–5), with a standard deviation of 0.542, with a somewhat older sample (Bohdanowicz et al., 2023). This is somewhat comparable to what we observed. Still, involving a wider range of users might lead to different results. The same goes for involving users from different countries: the results we found here are based on a Dutch sample, and the recommendations were also based on earlier research within the Dutch population. Such a system might show different results in different environmental and socio-economic conditions. For future research, it might also be valuable to look at user satisfaction after an extended period of recommender system use and measure implementation. Similarly, it would be interesting to compare self-efficacy before and after using the system, as self-efficacy variation likely exists in the sample already and might influence choices and behaviour up-front. In the current study, we assume that our system influences self-efficacy, but the level of self-efficacy prior to the study might also influence system usage and choice outcomes. Lastly, it might give additional insights to research a similar system that not only involves behaviours and investments within the living arrangements, but also those outside, e.g., green travel choices and purchasing behaviours.

In a broader technological context, recommender systems can be part of a smart home environment. Feedback on one’s current energy use could be an important additional behavioural determinant of energy conservation (Fischer, 2008), particularly when it is instructive about which appliances to turn off (Fensel et al., 2014; Geelen et al., 2019).