Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme

Shoaib Sufi; Caroline Jay

doi:10.12688/f1000research.16231.1

Home Browse Raising the status of software in research: A survey-based evaluation...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme

[version 1; peer review: 3 approved with reservations]

Shoaib Sufi¹, Caroline Jay¹

PUBLISHED 03 Oct 2018

Author details Author details

¹ School of Computer Science, University of Manchester, Manchester, Greater Manchester, M13 9PL, UK

Shoaib Sufi
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing

Caroline Jay
Roles: Data Curation, Formal Analysis, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Research on Research, Policy & Culture gateway.

Abstract

Background: This paper reports the results of an evaluation of the Software Sustainability Institute’s Fellowship Programme, which focused on identifying and categorising the benefits that the fellowship has afforded its recipients, via a series of open questions.
Methods: The evaluation took the form of a survey open to people awarded Fellowships between 2012 and 2016, which asked people to report the effect that the programme had had on them, their institutions, their research domains and their careers.
Results: The results show that the Fellowship plays a wide-ranging role in supporting communities of best practice and skills transfer, and that a significant benefit is the way it has raised the profile of software in research, and those people who develop and advocate for it.
Conclusions: The evaluation of the programme has shown the need to support research software in situ and credit the engineers and researchers who are working in this important area that supports reproducibility, reuse and the integrity of research investments.

Keywords

research software, fellowship

Corresponding author: Shoaib Sufi

Competing interests: The first author, Shoaib Sufi, is employed on the Institute grant, and the second author, Caroline Jay, is a Fellow; both therefore have an interest in the Institute.

Grant information: This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) Grant EP/H043160/1 and EPSRC, BBSRC and ESRC Grant EP/N006410/1 for the UK Software Sustainability Institute.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2018 Sufi S and Jay C. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

How to cite: Sufi S and Jay C. Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme [version 1; peer review: 3 approved with reservations]. F1000Research 2018, 7:1599 (https://doi.org/10.12688/f1000research.16231.1) First published: 03 Oct 2018, 7:1599 (https://doi.org/10.12688/f1000research.16231.1) Latest published: 03 Oct 2018, 7:1599 (https://doi.org/10.12688/f1000research.16231.1)

Introduction

The Fellowship programme¹ run by the UK Software Sustainability Institute (SSI) is a unique package of financial support, networking and advice, which is competitively awarded to members of the research software community. The main goals of the programme are to encourage Fellows to develop their interests in the area of software sustainability and help them to become ambassadors of good software practice in their domains. The programme offers £3000 to support event attendance, workshops, training and other activities to help build awareness, capability and capacity in computational techniques, reproducible research and open science in diverse research domains.

Fellows are selected via an open competition, where candidates are judged by a panel of experts (former Fellows and Institute staff members) in terms of their track record in practising and promoting software sustainability, and the activities they plan to run with the Fellowship award. To promote diversity, funding is allocated to people at different career stages (from PhD student to research leader) and a variety of domains (e.g. Glaciology, Research Software Engineering, Humanities and Astrophysics). The overarching aim of the Fellowship programme is to provide support and recognition to those people promoting sustainable software practices, and advocating for and producing more verifiable, shareable and useful research outputs.

This paper reports the results of a recent survey evaluation of the programme’s effects on its recipients and their wider communities. A thematic analysis of the results shows that the award of a Fellowship had substantial and wide-ranging benefits both for the Fellows themselves, and for their institutions and research domains. The theme that emerged most strongly and consistently was that the Fellowship provided status to both the Fellows themselves and the role of software within research. Respondents reported that current academic culture does not always afford recognition to research software and research software engineers, and that the Fellowship has played a key role in improving the visibility of this ubiquitous yet undervalued component of research methodology.

An earlier version of this article is available on PeerJ as a preprint https://doi.org/10.7287/peerj.preprints.26849v1.

Related work

A number of other fellowship providers have published evaluations of their programmes, including the Humboldt Foundation², the Erwin Schrodinger Fellowships³ and the Union for International Cancer Control (UICC)⁴. These evaluations used a combination of surveys, data held about the fellows (e.g. demographics, subject areas), and in the case of UICC, case studies. The reports are openly available, but do not constitute peer-reviewed research. Here, we take a different approach, treating the evaluation as a research project (for which ethical approval was obtained), asking primarily open questions, and only including data that were obtained via the study. By conducting the work in this way, we aim to contribute empirically to the software sustainability literature, as well as gaining a local understanding of the Fellowship programme’s impact.

Methods

The survey was conducted using the University of Manchester SelectSurvey.NET instance to ensure the data was collected and stored securely. Participants were contacted via email using the all-fellows mailing list; all current and previous Fellows who are still in contact with the Institute are subscribed to this list. The survey was conducted from the 12^th July 2017 to 31^st August 2017. After the initial email there were two reminder emails and we chased two individual Fellows who had only made partial survey entries to see if they would offer complete entries (which the subsequently did).

The initial part of the survey explained what the purpose of this research was and asked for consent from participants. Participants were asked to confirm that they agreed to participate, that they understood that participation was voluntary, that they understood their data would remain confidential, and that they permitted anonymous quotes to be published. They were able to say ‘Yes’ or ‘No’ to any of these questions. All participants included in the analysis answered ‘Yes’ to all of these questions. There was a further question around retention, “I agree to my data being retained indefinitely for further research related to the Fellowship Programme.” All participants bar one answered ‘Yes’ to this question.

The survey then asked Fellows to comment on the benefits of the programme in a number of categories, and to report any negative consequences and suggested improvements (see Table 1). The survey was sent to the entire population of the 2012–2016 Fellows (78 in total). The study received approval from the Computer Science School Panel (ref: 2017-2308-3295) on the delegated authority of the University Research Ethics Committee (UREC), University of Manchester .

Table 1. Survey questions and analysis performed.

Question	Data
Do you think being awarded a Software Sustainability Institute Fellowship has benefitted you?	Forced choice response: Yes/No/Unsure
How has your Fellowship benefitted you?	Free text
How has your Fellowship benefitted your Institution(s)?	Free text
How has your Fellowship benefitted your domain?	Free text
How has your Fellowship benefitted others in ways not already covered?	Free text
Have there been any negative consequences of your Fellowship? If, yes, please specify.	Free text
Do you think being a Fellow has helped to advance your career?	Forced choice response: Yes/No/Unsure
If not already specified, how has being a Fellow helped in your career development?	Free text
How would you improve the Fellowship Programme?	Free text

Fellows were asked to provide information about gender, year in which their Fellowship was awarded, which funding bodies supported their work and their research area. It also asked about their current job role, job role at the time the Fellowship was awarded, and specific research area, but this information is not reported here as the small number of participants means it may be possible to identify individuals from this data.

The free text answers were thematically analyzed in an open coding fashion following established analysis methods⁵: 1) familiarization with data, 2) generating the initial codes, 3) searching for themes, and 4) iteratively reviewing themes. The generated codebook was agreed between the authors.

Results

There was a response rate of 33% (N = 26). Seven fellows from 2016 responded, 8 from 2015, 6 from 2014, 4 from 2013 and 1 from 2012. One of the respondents (Caroline Jay) is an author of this paper, and her results have thus been excluded from the analysis, leaving a total of 25 respondents.

Demographic information

Five respondents were female and 21 were male. Table 2 shows the funding bodies that supported the respondents’ research.

Table 2. Funding sources of the respondents’ research.

The centre column shows the number of respondents listing the body as their primary funder. The right hand column shows the number of respondents listing the body as an additional funder.

Funder	Primary	Secondary
EPSRC	3	3
BBSRC	2	3
NERC	3	1
AHRC	1	3
ESRC	1	2
MRC	1	2
STFC	1	1
Commission/ERC/European Space Agency	3	5
NIH	1	0
Wellcome	0	7
Internal/employer	5	0
Leverhulme	0	3
Any/Various/Other	0	9

Forced choice analysis

In answer to the question, ‘Do you think being awarded a Software Sustainability Institute Fellowship has benefitted you?’ 96% (n = 24) answered ‘yes’. One person answered ‘unsure’ and zero people answered ‘no.’

In answer to the question, ‘Do you think being a Fellow has helped to advance your career?’ 72% (n = 18) answered ‘yes,’ 16% (n = 4) answered ‘no’, and 12% (n = 3) answered ‘unsure.’

Free text analysis

The first author coded the dataset into a number of initial themes. These were grouped into overarching themes by the second author, which were then used as a codebook for the answers to the questions ‘How has the Fellowship benefitted you/your institution(s)/your domain/others?’. The results were checked by the first author for agreement. The emergent themes are described in the bulleted list below.

• Status: giving status and recognition to individuals and organisations for their role in sustaining software, and to sustainable software practices themselves.
• Community/network: organizing/attending events; building professional and personal networks.
• Professional development: improving one’s own skills through undertaking training and improving the skills of others by providing training.
• Resources: obtaining resources for travel and other professional activities.

Table 3 shows the number of respondents who reported a benefit under each theme for the categories that the questions asked about: self, institution, domain and others. In the following sections we explore each of these themes in turn.

Table 3. Number of respondents reporting a benefit under each theme for the various question categories.

Theme	Self	Institution	Domain	Others	Total
Status	20	4	4	3	31
Community/ network	13	10	3	1	27
Prof development – self	11	2	4	0	17
Prof development – others	3	10	9	4	26
Resources	6	3	3	4	16

Status

Across the questions, 31 comments were made in relation to the Fellowship leading to an improvement in “profile and prestige” (R5). The majority of these (twenty responses) were in relation to improving the status of the individual Fellow.

The impact on the Fellows’ status manifested itself in a number of ways, including: giving them recognition as someone who knew about software sustainability and good coding practices; providing a badge which opened doors and allowed them to market themselves; and becoming more appealing as collaborators at the institutional, domain and interdisciplinary level. Four respondents reported that having a Fellow raised the profile of a department or institution. Table 4 illustrates the impact of the Fellowship on status with quotations.

Table 4. Responses illustrating the impact of the fellowship on status.

Question	Response
How has your Fellowship benefitted you?	“The opportunities this has given me are huge in terms of connections and invitations to speak at international conferences and to participate in workshops, review panels etc. All this external work has been particularly noted in my performance reviews and I believe it was crucial in helping to secure a recent promotion.” (R23) “It has given me credibility locally as an expert in good coding practices, open data and code, and publishing… It has made the department recognise my role in facilitating others' research and to be recognised as a pivotal enabler in *1 research.” (R4) “I think it has opened a lot of doors. I almost always tell researchers that I talk to about my links to the SSI, as an indicator of my standing in the wider community. I am fairly sure that I have been invited to at least two major events due to my SSI links, and these have lead to on going research.” (R24) “It helped me build my reputation within the field of ** and to be a leader in sustainable thinking. It helped* get me a position on the ** technical review board”* (R7)
How has the Fellowship benefitted your institution(s)?	“[It provided] strong recognition for my institution's research IT organisation as being a leader in development of research software engineering services.” (R22) “I think most of the benefits to me translate to my institution as well, as it helps me to do my job - status, meeting people, etc.” (R24) “Collaborations that I developed through the fellowship have indirectly benefited my institution by kudos from the research that was produced” (R14) “During my fellowship I have raised awareness of the wider UK RSE community in my institution. My institution has a strong background in research software and has a high level of self sufficiency. I believe that improving the interaction with the wider community will benefit our organisation and others.” (R20)
How has your Fellowship benefitted your domain?	“Probably not my fellowship, but if we consider all fellows in or close to my domain, I think together, we have had a global impact.” (R2) “It has helped to get a better attention of our team by all the other researchers in the university.” (R13)
How has your Fellowship benefitted others in ways not already covered?	“I believe the Fellowship has helped to create a movement of Research Software Engineers, which I believe is helping to give recognition to the importance of good practice in software development within research institutions. Giving recognition to robust software development (including software for data analysis) is crucial to improve the quality and reproducibility of published research.” (R1) “I'm quite well-known in various areas (*), and I believe that my frequent talking about the issues of reproducible software and scientific software in general raised awareness of both these issues as well as the SSI.” (R8)

*** indicates removed to preserve anonymity.

There was evidence that the credibility conveyed by the Fellowship potentially contributed to the Institute’s mission to improve diversity: “Despite getting a PhD partially from a computer science programme, I could see that my skills and knowledge were always at least to some extent dismissed or doubted. I do not want to speculate whether this is due to gender bias or some other prejudice-based process or my own failing at looking professional, but since being elected a SSI fellow I most definitely observed a significant drop in mansplaining.” (R10).

Community/network

Fellows benefitted from joining a community of like-minded individuals and the networking opportunities that arose from this. Respondents made 27 comments in relation to the Fellowship improving their network, 14 of which showed that this benefit went beyond themselves, to improve the software research communities within their institution/domain. R23 said: “The fellowship has been hugely beneficial to me and my career. The contacts and collaborations formed during my fellowship year have led, directly and indirectly, to a huge number of opportunities.” The benefits included increasing confidence; feeling part of the research software community and not an outsider; sharing good practices; being able to identify as a Research Software Engineer (RSE) and supporting their role in formulating an RSE community of practice via the RSE Association (www.rse.ac.uk).

Respondents reported that the Fellowship gave them the mandate to collaborate with different organisations and institutions, as well improving the local networking of those involved with research software. Three Fellows at one institution were able to work together.

Fellows from a single domain expressed that a number of them working with each other across years had had a cumulative effect over time, in effect seeding a hub of researchers/fellows who took sustainability seriously. There was a platform for them to then influence domain specific groups at different institutions increasing the impact and reach of promoting better sustainability practices. Fellows felt motivated to collaborate, form online communities, and contribute to the open source community.

The Fellowship ultimately provided community, friendship and motivation for new ways of doing things. The Fellowship also helped them become better scientists and ambassadors for sustainability issues in their community and thus better recognised. Table 5 illustrates the impact of the Fellowship on community and network with quotations.

Table 5. Responses illustrating the impact of the Fellowship on community/network.

Question	Response
How has your Fellowship benefitted you?	“Huge range of contacts with interesting people” (R14) “Meetings and interactions with other fellows made me better at programming, understanding issues related to software.” (R19) “Great access to national leadership in e-Infrastructure and scientific software, increasing my influence, and enabling me to advance my career.” (R22)
How has the Fellowship benefitted your institution(s)?	“The Fellowship covered the costs for a research engineering expert to visit my laboratory and to provide guidance on various aspects of software development to colleagues. I believe this guidance strongly influenced my colleagues to move to better quality, more collaborative software development practices.” (R1) “Because of my fellowship, I got involved in other computational reproducibility groups, which have benefitted my institution in both publications and expertise brought back.” (R15) “It has added to their portfolio of cross-discipline interaction.” (R12)
How has your Fellowship benefitted your domain?	“It allowed engaging with the community about issues of data sharing standards and good coding practices.” (R19) “I've had the chance to speak about research software sustainability and RSEs to lots of different audiences in the community, other large experimental facilities and to * PhD students at careers events” (R23)*
How has your Fellowship benefitted others in ways not already covered?	“I have set up a code club in my children’s school” (R11) “I could link some people to women in HPC and to the SSI.” (R17) “Friendships with like-minded scientists and scholars, really help me to believe that the a way of doing things is not just a personal idiosyncrasy but a real wave of change across research. The existence of a community of open-science, reproducibility, research software engineering, new science metrics, and post-postdoc career innovation types is great for motivation” (R22)

Professional development

Respondents stated that the Programme had helped them to progress in their careers, either by way of a new job, promotion, or change in direction: “I can map my entire career trajectory from the opportunity that the fellowship gave me. One meeting led to another...” (R11).

In answer to the question, ‘If not already specified, how has being a Fellow helped your career progression?’ three respondents mentioned gaining confidence, three mentioned improving skills, seven mentioned improving their networks, and five mentioned improving their visibility. The programme had a significant effect for R23: “The fellowship, and then all the external collaborations and followed from it, have been directly cited as reasons for giving me top performance ratings over the last three years… Without this community of like-minded people to engage with I'm not sure I'd still be working in the same organisation, or even in research software at all.”

Across the other questions, 17 comments related to professional benefits for the Fellows themselves that included: improving personal knowledge and practices; understanding how much of research is software driven; developing a habit for research related blogging; identifying new areas in their own research fields; and thinking about research software engineering as a career. Fellows increased their confidence in research software development, and they were able to get career, technical and other advice from other Fellows, mentors, institute staff and others they had met at workshops.

The Fellowship awards had an even greater impact on the professional development of others, with 26 comments relating to this altogether. Fellows ran training courses, such as Software Carpentry⁶, spread best practice via workshops, and supported data sharing and reproducibility initiatives. Table 6 illustrates the impact of the Fellowship on professional development with quotations.

Table 6. Responses illustrating the impact of the Fellowship on professional development – self.

Question	Response
How has your Fellowship benefitted you?	“After I got the fellowship, the department (*) has set up a code clinic, where I troubleshoot people's coding issues for half an hour a week. They are also planning hire me in December to give the same Good Coding Practice seminars that I prepared as part of the fellowship, to the department as a seminar series.” (R4) “It has given me a much greater understanding of the roles of software in academia.” (R18)
How has the Fellowship benefitted your institution(s)?	“… .In fact, out of 7 weeks of teaching, one full week was dedicated to software sustainability (although this included things like testing your software). I believe this was the first time that these kind of topic has been more than just mentioned in any ** module.”* (R10) “I was able to deliver the good coding practice seminar to the British ** Association (2017) to an audience* from around the country, of scientists who regularly code as part of their research. I received exceptionally positive feedback from the seminar.” (R4) “Because of the external funding, visiting PhD students were able to attend the Software Carpentry workshop and take their skills back to their University groups.” (R23). “My fellowship supported a Software Carpentry Instructor Training session in my institution. This training will help staff in my organisation create and deliver better development material focused on scientific software development” (R20)
How has your Fellowship benefitted your domain?	“Library Carpentry has had a MASSIVE impact in . * has benefited from 3 fellows (including me).”* (R6) “Fellowship funds were used to hold a computing workshop for early career ** scientists in the UK.”* (R15) “It has afforded people within my research domain the opportunity to learn a range of different software packages for 3D reconstruction for free - which is a really valuable opportunity when many people have limited, say PhD funding.” (R21)
How has your Fellowship benefitted others in ways not already covered?	“I have given two extended workshops on agent-based modelling to ** thanks to the fellowship funding. A* number of students who participated have shifted their research interest into simulation and a vast majority of participants agreed that even if they will not pursue this line of research further they have enough knowledge to be able to critically engage with published models” (R10) “I believe the Fellowship has helped to create a movement of Research Software Engineers, which I believe is helping to give recognition to the importance of good practice in software development within research institutions. Giving recognition to robust software development (including software for data analysis) is crucial to improve the quality and reproducibility of published research.” (R1)

Resources

Fellows used the £3000 award for attending conferences and workshops that they normally would not be able to; organising events; running training; kick-starting an initiative (such as a product, service or approach); and inviting visitors. Although not everyone used the funds: “My position is probably different to many fellows in that I mostly wanted to be a fellow to show support for the SSI and the fellows network/community and to highlight the importance of this area in my institution. Access to funds wasn't a consideration” (R3), across the respondents they supported a wide range of activities, summarised in Table 7.

Table 7. Activities that respondents reported were made possible using the Fellowship award.

Activity	Number of respondents	Response
Attending event	6	“I was able to use the balance of my fellowship funds to attend a conference in the US that I would not normally be able to.” (R24) “Data sharing in ** is still in its infancy, but thanks to support from the Software Sustainability Institute* colleagues in were able to host a datathon to promote reproducibility and data sharing in , as well as to share a ** dataset.”* (R1)
Organising event	9	“Forty and * took part in the Software Carpentry workshop in late 2015 and had training in Python,* Linux command line and Git. This would not have run without the fellowship funding and was very well received.” (R23)
Project funding	3	“I started as a fellow whilst I was a post doc. The fellowship provided me with really useful independent funding to pursue a line of work and interaction that was not covered by my postdoc funding.” (R12)
Hosting visitor	2	“I was able to fund the travel/accommodation for a keynote speaker to a workshop who ended up being absolutely perfect for the event. Without the fellowship funding I don't think I would have been able to secure her trip.” (R12) “The Fellowship covered the costs for a research engineering expert to visit my laboratory and to provide guidance on various aspects of software development to colleagues. I believe this guidance strongly influenced my colleagues to move to better quality, more collaborative software development practices.” (R1)

Negative consequences

In answer to the question, ‘Have there been any negative consequences of your fellowship?’ 14 people said there had not been anything negative, and 7 people did not give an answer. One person commented that they sometimes had to explain that software sustainability was not the same as digital preservation, and that this disappointed the person they were talking to. Three respondents gave lighthearted answers: “I definitely spend more time on Twitter because of you guys!” (R10); spending time “struggling with installing and implementing open source software (just kidding, though it takes time, I thoroughly enjoy learning new things, and it's an investment in the future)” (R11) and “a lack of time to take advantage of all the opportunities – not a bad problem to have!” (R23)

Although the programme itself did not appear to result in negative consequences, R17 commented that their institution “was not interested in [the Fellowship] at all.”

Improvements

In answer to the question, “How would you improve the Fellowship Programme?” six respondents did not make any suggestions. Nine respondents recommended increasing the number/length of events, and one raised an issue around the distance that they were required to travel for an event. One respondent suggested making more significant funds available to Fellows, including providing salary, and two commented that administration of funds could be improved. Three people had suggestions for improving mentoring, including having non-academic mentors, and using existing Fellows as mentors. Two respondents, who had both moved away from the UK, thought it would be good for the Institute to build stronger links internationally. Three respondents suggested having more explicit roles/activities for Fellows over the longer term.

Respondent	How has your Fellowship benefitted you?	How has your Fellowship benefitted your Institution(s)?	How has your Fellowship benefitted your domain?	How has your Fellowship benefitted others in ways not already covered?	Have there been any negative consequences of your Fellowship? If, yes, please specify.	If not already specified, how has being a Fellow helped in your career development?	How would you improve the Fellowship programme?
1	Most importantly it has made me part of a community of scientists with similar interests and who seek to address similar challenges (for example, staying involved in research while focusing on non-traditional academic outputs such as software). It has also helped to develop my technical knowledge and to share this knowledge with others (for example, by supporting my involvement in Software Carpentry, as a participant initially and later as an instructor).	The Fellowship covered the costs for a research engineering expert to visit my laboratory and to provide guidance on various aspects of software development to colleagues. I believe this guidance strongly influenced my colleagues to move to better quality, more collaborative software development practices. The Fellowship also supported a project with colleagues in *** that led to an important *** dataset being made freely available for research.	A large proportion of my work is focused on sharing *** data with researchers around the world, for the benefit of patients. Data sharing in * is still in its infancy, but thanks to support from the Software Sustainability Institute colleagues in * were able to host a datathon to promote reproducibility and data sharing in *, as well as to share a *** dataset.	I believe the Fellowship has helped to create a movement of Research Software Engineers, which I believe is helping to give recognition to the importance of good practice in software development within research institutions. Giving recognition to robust software development (including software for data analysis) is crucial to improve the quality and reproducibility of published research.	Nope!	Taught me new skills- Allowed me to gain career and technical advice from colleagues- Given me confidence to focus on research software development as a career.	- Offer larger Fellowships that would support full time research software engineer positions.- Have a clearer strategy for handling international fellows (shortly after receiving my fellowship I moved to the US and felt that this didn't really fit with the SSI's goals for fellows, but actually I think building SSI recognition internationally could be a very good thing).- Keep up the good work!
2	Funding to attend conferences and a organise workshop.	Don't think so.	Probably not my fellowship, but if we consider all fellows in or close to my domain, I think together, we have had a global impact.		No.	I think being a fellow got me involved in the SSI and through this association and the association with the RSE community at large, my work and efforts gained some visibility. It is however unclear whether this had any effect (positive or negative) on by academic career.	I don't know, but this could be an interesting discussion at (or the night before/after) a CW meeting.
3	My position is probably different to many fellows in that I mostly wanted to be a fellow to show support for the SSI and the fellows network/community and to highlight the importance of this area in my institution. Access to funds wasn't a consideration. The fellowship hasn't directly benefited my career, but I found it useful as a badge to show: I care about this and I have some understanding of this area.	Indirectly through access to the SSI network, although to an extent I had this before applying.	It helped me justifing sharing of some material that otherwise might not have been shared widely.		No negative consequences
4	It has given me credibility locally as an expert in good coding practices, open data and code, and publishing. I have become more established in my department as someone useful for multidisciplinary work, and as a collaborator. It has made the department recognise my role in facilitating others' research and to be recognised as a pivotal enabler in ***** research.Preparing material for the teaching that I've done as part of the fellowhip has also improved my knowledge of the literature on coding practices, and made me aware of some of the unknown factors in the field.	After I got the fellowship, the department (*****) has set up a code clinic, where I troubleshoot people's coding issues for half an hour a week. They are also planning hire me in December to give the same Good Coding Practice seminars that I prepared as part of the fellowship, to the department as a seminar series.	I was able to deliver the good coding practice seminar to the British ***** Association (2017) to an audience from around the country, of scientists who regularly code as part of their research. I received exceptionally positive feedback from the seminar.		No		A way to pre-pay for the course costs. I've been in �1500 of debt for the last 4 months. I had to pay for venue hire, catering, travel costs. Still not yet reimbursed. The required 'together days' (collaboration workshop, fellows' day) were very far away and are very difficult for me to attend (long travel, needing to come up the night before, accommodation, time off etc). Also the location of the Fellows' day was announced very late, and I could not change my rota to get time off work. I also thought the 'interview process' as a whole day, although it was fun, was a bit excessive.
5	Profile and prestige, connection to the network of SSI Fellows.				None.
6	It seed funded a project I work on. It gave me a network of people to interact with. It gave me a distinctive line on my CV. It feels like it has real long tail benefits.	Not sure. Both my previous (library) and current (academic) employer see the Fellowship as a positive thing.	Library Carpentry has had a MASSIVE impact in ***. *** has benefited from 3 fellows (including me).	They see the work of the SSI as something relevant to the humanities.	In some of the domain in which I work, 'software sustainability' is often thought of through the lens of digital preservation. So it has taken some explaining that the latter is not really what the former is about (with occasional disappointment from the person asking!)	It gave me a network of helpful people to interact with and ask advice (especially around my rubbish programming skills). It forced me to go to the Collaborations Workshop hackdays which gave me masses of confidence.	I love the life fellow concept. Perhaps there could be a structured opportunity to reengage with the SSI 2/3 years post fellowship - maybe a rolling advisory board slot for a previous fellow (based on an application or vote) or a fellowship route for doctoral students of previous fellows?
7	It helped me build my reputation within the field of *** and to be a leader in sustainable thinking. I always put my fellowship prominently on job applications. It helped get me a position on the *** technical review board.	I've moved since I had the fellowship. I suspect they've only benefitted tangentially through the advice I can offer on grant applications.	It taught me to think sustainably about my work, and to encourage others to do the same. It means I make decisions for the long-term and I've stopped making some of the mistakes that resulted in rebuilding the wheel or having to deprecate earlier work.	A number of my colleagues in ***** got fellowships after me, on my encouraging them to apply. This has helped create a hub of researchers within the UK who take sustainability seriously.	No	It's that extra thing that demonstrates aptitude and part of a community of best practice. It's as valuable to me as my fellowship in the Higher Education Academy.	Find more tangible ways for past fellows to stay involved and get further support.
8	The Fellowship exposed me to other people facing similar issues. It's been great to expand my personal network.	N/A - I left academia in the year of my Fellowship.	N/A	I'm quite well-known in various areas (ex-academic: ***, *, *, ***), and I believe that my frequent talking about the issues of reproducible software and scientific software in general raised awareness of both these issues as well as the SSI.	No.		Having a non-academic mentor was essential for me. ***** taught me a lot about how stuff works outside University, both from a technical as well as organisational perspective. I'd recommend all SSI fellows to seek both an academic as well as a non-academic mentor.
9	Exposure to wider network of colleagues in computational research; best practice; funding pointers (e.g. Jisc DMP, SSI SMP), training and exposure.	I have tried to be a software sustainability champion here, and others are coming around. ***** here applied to and won a 2016 Fellowship, for example.	Focus on sustainability as a key prerequisite for reproducibility. ***** suffers from a plethora of overlapping and poorly-supported tools so this is a key issue.	n/a	None.	see above. Also, winning a fellowship gave me confidence personally, and demonstrated to others that I was capable of doing it.	More / broader range of follow-on events for Fellows post-award year. ? Fellows minisite? (Slack channel was one attempt but didn't quite seem to take ofF).Some (some) slightly more subject focused meetups -- not _too_ exclusive or narrow, but while talking to *** or *** is fun it isn't as directly useful as say a ***** meetup.
10	I have benefited from the fellowship immensely and it had a significant impact on a number of aspects in my professional life. I will detailed them in points for clarity sake but they are all very much intertwined. 1. Personal confidenceAs a person lacking much formal training in computer science I have found myself frequently doubting my skills or 'usefulness'. Thanks to the fellowship I saw that CS consists of professionals with a varying degree of CS skills and very varied knowledge and even if my skills cannot rival an experience programmer with 20 years of coding experience, I fall squarely into the community. This was a revelation. 2. Access to the SSI networkThanks to the fellowship my CS horizon, defined as the awareness of 'what's out there', has expanded tremendously. I feel like I moved from only learning about new staff filtered through other *** to actually jumping in the very core of the newest developments and actually becoming one of these * others learn from. 3. Recognition in my fieldAs the only * student in my department I have always been perceived as the 'one off' or a bit of an experiment. Becoming a fellow gave me credentials, which I really appreciated. 4. Recognition of my skills Despite getting a PhD partially from a computer science programme, I could see that my skills and knowledge were always at least to some extent dismissed or doubted. I do not want to speculate whether this is due to gender bias or some other prejudice-based process or my own failing at looking professional, but since being elected a SSI fellow I most definitely observed a significant drop in mansplaining. Finally, I have little doubt that the SSI fellowship was a significant argument (of a type: 'she can clearly do CS') thanks to which I have got my current position (head of the * division at the *** Supercomputing Center).	The fellowship is listed as one of my awards on the researchfish thus contributing to the 'impact' of the PhD programme I was on (***). I do not know if this case study will be highlighted in the final report but it could be considered proof that my 'high-risk' funding (i.e., giving CS funding to a *** student) was a success story.	I might have been a 'quiet fellow' but this is because I have been listening, reading and learning a lot. I have repeatedly used this knowledge (or even the awareness) of software best practice, tools and workflows and I have spread it further on. In the last two years I have given at least 5 talks or dedicated fragments of talks to best practice in research software development (including, for example, the need to cite software) and I have incorporate it into a syllabus I have been teaching this year. In fact, out of 7 weeks of teaching, one full week was dedicated to software sustainability (although this included things like testing your software). I believe this was the first time that these kind of topic has been more than just mentioned in any ***** module.	I have given two extended workshops on *** to *** thanks to the fellowship funding. A number of students who participated have shifted their research interest into simulation and a vast majority of participants agreed that even if they will not pursue this line of research further they have enough knowledge to be able to critically engage with published models.	I definitely spend more time on Twitter because of you guys!		The only change I would suggest for your consideration would be to coordinate the programme timing with the collaborations workshop, so that all fellows have to attend it twice - at the beginning of their fellowship and a year later at the end. It would frame the fellowship period nicely with an 'opening' and a 'closing' event and would also ensure that each cohort gets to meet fellows from two different fellowship years.
11	I can map my entire career trajectory from the opportunity that the fellowship gave me. One meeting led to another...plus all the cool software sustainability tech sideof things that I have developed a burning motivation to learn.	It has made me implement software version control, jump into python and open source issues, licensing etc.	I am passionate about the sustainability side and force my collaborators to interact via git / mercurial repositories, and my students too.	I have set up a code club in my childrens school	Spent lots of time struggling with installing and implementing open source software (just kidding, though it takes time, I thoroughly enjoy learning new things, and it's an investment in the future).	I'll write a blog post...	Not sure...it's already pretty good. Provide a mechanism for members to get peer help from or collaborate with other fellows?
12	I started as a fellow whilst I was a post doc. The fellowship provided me with really useful independent funding to pursue a line of work and interaction that was not covered by my postdoc funding.	It has added to their portfolio of cross-discipline interaction.	It has enabled me to broaden my network.	I was able to fund the travel/accommodation for a keynote speaker to a workshop who ended up being absolutely perfect for the event. Without the fellowship funding I don't think I would have been able to secure her trip.		It has given me another example on my CV of independence and leadership.	Perhaps a mid-year fellows event to complement CW?
13	It has increased my network!! It has also helped my CV and my carreer - I'm now an RSE though I'm not just giving up on fighting for Open Science and better recognition in academia for better skills in software.	Through my fellowship, I organised a workshop which helped to advertise my university. Inside the university, I've been called a few times to provide feedback on the tools used internally. If they follow my guidance that will help them too.	I was focusing my fellowship to help any domain, not on only one, and since I become an RSE I've not managed to keep my previous domain too close. As RSE it has helped to get a better attention of our team by all the other researchers in the university.	The workshop counted with ~30 people, I would like to think that they got some good stuff back to their institutions or companies, improving therefore the way they collaborate or create communities in the online world.	too much fame! kidding :) No, there's not been any negative consequence.	More than my fellowship I would say the SSI, via them I found other options where I could work!! So, thanks SSI!! the fellowship has helped too, but mostly you!!!	I don't really think there's much to improve. I could think in some mentoring program on top where new fellows get some help from previous. I know that the fellowship requires some senior person as a mentor but not necessarily from within the program. Maybe get a mentor assigned afterwards help the new people to get to know the previous people better. Also CW is great, but I want it to last longer (a month or two, otherwise we never have time to finish what we start) :O)
14	Huge range of contacts with interesting peopleFunding to attend conferences, run training etcMet people whom I later collaborated with - including someone who provided a �50k grant for some of my researchOpportunity to attend Collaborations Workshop events for free, which have led to interesting outcomes including the development of a novel Python module called ***** Integration into the SSI 'world' - which has led to pursuing various pieces of work with the SSI, and some paid employment with them	I ran Software Carpentry training courses at my institutionCollaborations that I developed through the fellowship have indirectly benefited my institution by kudos from the research that was producedFunding received through someone I met through the fellowship - this funding came to the institution and was used to employ a postdoc.Attending conferences funded through the fellowship - where the institution's name was mentioned	Provided training on software sustainability to young researchers at a domain-specific conferenceTownhall on software sustainability at a major *** conference (***)Blog posts and articles on software sustainability and how it relates to my field	The open-source community, through the development of '*****' at Collaborations Workshop events		By obtaining a network of colleagues and feeling part of a community of people to collaborate with and get advice fromThrough indirectly helping me produce conference presentations and publications - by putting me in contact with people whom I collaborated with these onThrough putting me in contact with people who later provided funding
15	It was an eye-opening experience to really think about how much my research has been driven by software, both that I use and that I create. It helped me think of myself as a coder and not just a ***** It helped prepare me to interact with other computational communities.	Because of my fellowship, I got involved in other computational reproducibility groups, which have benefitted my institution in both publications and expertise brought back. I am now a co-PI on a grant related to brining together *** and *** communities.	Fellowship funds were used to hold a computing workshop for early career *** scientists in the UK. Thanks to the start from the Fellowship, I am now a co-PI on a grant related to brining together * and *** communities.	N/A	No.		I'm a bit of a special case - I got the Fellowship, but then moved away from the UK at the end of my term, so I haven't been able to benefit from any cohort remaining in the UK or being able to participate in future trainings, workshops, etc. I think it would be great to get a similar fellowship going in other countries!!
16	Great network of contacts; funds helped me attend meeting, and develop reproducible research.	*** has had several fellows and we work together. Most recently we hosted the *** workshop.	I am happy to see several fellows also in *****.	-	-		Nothing immediately comes to mind.
17	I met many people and could enlarge my network.	They were not interested in it at all.	I was exposed to big data and HP and software development.	I could link some people to women in HPC and to the SSI.	No	I got into contact with software developers and HPC expert and the Software Carpentry.	It is fine as is.
18	It has given me a much greater understanding of the roles of software in academia.	The RSE movement has been really helpful for our recruitment and retention strategies				It has shown I have been committed to my work, and raised my profile in my institution.	Possibly more opportunities for alumni to meet, but the program itself is fine.
19	Meetings and interactions with other fellows made me better at programming, understanding issues related to software. The fellowship itself allowed me to attend a meeting and organized a series of workshops.	Beyond talks and workshops that were given because of my fellowship (giving the university more outreach), the good practices I learned are now being tough in the department.	It allowed engaging with the community about issues of data sharing standards and good coding practices.	NA	NA	A deeper understanding of issues addressed by the SSI helped me to be a better scientist and to engage better with those issues within my community.	A wrap-up meeting of fellows could be interesting.
20	The fellowship has allowed me to interact with researchers in similar roles across a variety of academic interests. I believe my visibility in my organisation my increased by a small amount.	During my fellowship I have raised awareness of the wider UK RSE community in my institution. My institution has a strong background in research software and has a high level of self sufficiency. I believe that improving the interaction with the wider community will benefit our organisation and others. My fellowship supported a Software Carpentry Instructor Training session in my institution. This training will help staff in my organisation create and deliver better development material focused on scientific software development.					I found it difficult to devote time to my fellowship and I think this meant my fellowship was less effective than I had hoped. The mandatory meetings (I.e. inaugural meeting and Collaboration Workshop) were very good experiences. Physical meetings made it easier to devote time to the fellowship. I think it would be interesting to have a joint project with the fellows in a given year. Meetings or discussions would happen through the year. This could strengthen the network between fellows. The project could be focussed on one of the key aspects of the Software Sustainability Institute. The creation of (speed) blog posts are a useful outcome from a short period of work but a longer more continuous project may provide a higher impact output (e.g. Multidisciplinary papers, or knowledge diffusion).
21	This allowed me to provide training in software to my community, and thus raise my profile (and the profile of the software I co-wrote). Furthermore, through it I have a network of people I can approach with queries on software development. Without it I would have no such network, and miss out on a lot of best practice.	Through both the raised profile thanks to training courses, but also by putting me in touch with a bunch of people in otehr disciplines that also write software. It has thus helped with encourage interdisciplinary work	It has afforded people within my research domain the opportunity to learn a range of different software packages for 3D reconstruction for free - which is a really valuable opportunity when many people have limited, say PhD funding. Furthermore, it has directly led to further software training opportunities throughout the UK, not directly related to, but building on from, the fellowship.				Regular local area meetups for fellows and ex fellows would be really cool!
22	Great access to national leadership in e-Infrastructure and scientific software, increasing my influence, and enabling me to advance my career.	Strong recognition for my institution's research IT organisation as being a leader in development of research software engineering services.	Well, I kinda think we invented a new domain together, so that's pretty awesome!	Friendships with like-minded scientists and scholars, really help me to believe that the a way of doing things is not just a personal idiosyncrasy but a real wave of change across research. The existence of a community of open-science, reproducibility, research software engineering, new science metrics, and post-postdoc career innovation types is great for motivation!			I think it's pretty much perfect. I'd enjoy an annual dinner with all of the fellowship in an Oxbridge college!
23	The fellowship has been hugely beneficial to me and my career. The contacts and collaborations formed during my fellowship year have led, directly and indirectly, to a huge number of opportunities.As a software engineer in an IT department the fellowship funding gave me a chance to do things that were not then part of my role such as organise a large training workshop and speak at a conference. The fellowship came at a crucial time for me as I was making the transition from developer to team leader. I learnt a lot from the contact with like-minded people and the chance to share ideas at events etc. This gave me more confidence to advocate for improvements internally. The biggest ongoing benefit has been my involvement with the UK RSE association (which I was introduced to at CW15). The RSE leaders group has been my most important source of networking and support I am now serving as ***** of the UK RSE association. The opportunities this has given me are huge in terms of connections and invitations to speak at international conferences and to participate in workshops, review panels etc.All this external work has been particularly noted in my performance reviews and I believe it was crucial in helping to secure a recent promotion.	Forty *** and *** took part in the Software Carpentry workshop in late 2015 and had training in Python, Linux command line and Git. This would not have run without the fellowship funding and was very well received. Members of my team who were helpers at the event are now using the training materials to deliver smaller internal workshops on the three topics and bringing in further helpers so we are on the way to having a pool of trainers.The contacts I have made and the events I've attended have been useful in making better decisions and recommendations about research software for the organisation. The knowledge of how research software effort is organised in RSE groups elsewhere is now being applied to expand the capacity of my team to take on projects in partnership with researchers.The organisation has benefited from being at the forefront of the RSE movement in terms of reputation and recruitment.	Because of the external funding, visiting PhD students were able to attend the Software Carpentry workshop and take their skills back to their University groups. I've had the chance to speak about research software sustainability and RSEs to lots of different audiences in the *** community, other large experimental facilities and to *** PhD students at careers events.		Only a lack of time to take advantage of all the opportunities - not a bad problem to have	The fellowship, and then all the external collaborations and followed from it, have been directly cited as reasons for giving me top performance ratings over the last three years. Developing an RSE pool and collaboration with the UK research software community were written into the job description of the new group leader post that I have just applied for and been offered. Without this community of like-minded people to engage with I'm not sure I'd still be working in the same organisation, or even in research software at all. This network has enabled me to progress within the same team/department without fear of getting stuck in my ways.	Several of the things I would have liked have now been done I think - Slack for discussion, better tool for keeping track of the fellowship funds.Maybe some kind of shared training in essential non-technical skills for fellowships like how to plan an event, influencing & persuading etc.
24	I think it has opened a lot of doors. I almost always tell researchers that I talk to about my links to the SSI, as an indicator of my standing in the wider community. I am fairly sure that I have been invited to at least two major events due to my SSI links, and these have lead to ongoing research.	I think being a fellow and leading a group of RSEs goes very well together. I think most of the benefits to me translate to my institution as well, as it helps me to do my job - status, meeting people, etc.		I was able to use the balance of my fellowship funds to attend a conference in the US that I would not normally be able to.	None	Shows national/international standing.	Make it more obviously applicable to non-academic track people.

Dataset 1.SSI Fellowship evaluation 2012–2016 survey free text.

The free text questions and answers for the survey. CSV file. *** indicates removed to preserve anonymity

Limitations

The study focused on the benefits of the Fellowship Programme. We chose to use the word ‘benefit’, rather than ‘impact’, because we wanted people to reflect on the potential positives that came from the Fellowship in the broadest terms. Whilst the authors did not anticipate that the Fellowship would result in negative consequences, and a question checked for these explicitly, the phrasing of the questions could have biased respondents towards seeing the programme in a positive light. The survey only captured the responses of a third of Fellowship holders, so we do not know the experiences of the remaining two thirds.

Conclusion

The survey evaluation provided evidence that the Fellowship programme has played a significant role in supporting and galvanising engaged people in contributing to the domain of research software engineering. The gains in community building, networking, individual status, individual learning and the development of others, leading to long term benefits, initiatives and communities of practice are significant given the modest investment. Seed corn funding approaches are noted as being particularly effective mechanisms of support⁷. The evaluation of the programme has shown the need to support research software in situ and credit the engineers and researchers who are working in this important area that supports reproducibility, reuse and the integrity of research investments.

Data availability

Dataset 1: SSI Fellowship evaluation 2012-2016 survey free text. The free text questions and answers for the survey. CSV file. *** indicates removed to preserve anonymity, 10.5256/f1000research.16231.d218703⁸

The following is a description of the columns in the dataset:

Column	Type	Title / Question
1	Numerical	Respondent
2	Free text	How has your Fellowship benefitted you?
3	Free text	How has your Fellowship benefitted your Institution(s)?
4	Free text	How has your Fellowship benefitted your domain?
5	Free text	How has your Fellowship benefitted others in ways not already covered?
6	Free text	Have there been any negative consequences of your Fellowship? If, yes, please specify.
7	Free text	If not already specified, how has being a Fellow helped in your career development?
8	Free text	How would you improve the Fellowship programme?

Grant information

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

F1000 recommended

References

1. Sufi S: The Software Sustainability Institute Fellowship Programme. CEUR Workshop Proceedings (CEUR-WS.org):WSSSPE: 4th Workshop on Sustainable Software for Science: Practice and Experiences. 1686. Reference Source
2. Technopolis: Evaluation of the Alexander von Humboldt Foundation’s Humboldt Research Fellowship Programme. 2009; [cited 2018 Aug 8]. Reference Source
3. Meyer N, Bührer S: Impact Evaluation of the Erwin Schrödinger Fellowships with Return Phase. Karlsruhe: Fraunhofer Institute for Systems and Innovation Research ISI; 2014; 97. Reference Source
4. Keith N, Jones R, Chow M: Impact Evaluation of the ICRETT, YY & ACSBI Fellowship Schemes. Union for International Cancer Control (UICC); 2013; [cited 2018 Aug 8]. Reference Source
5. Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006 [cited 2018 Apr 11]; 2(4): 77–101. Publisher Full Text
6. Wilson G: Software Carpentry: lessons learned [version 2; referees: 3 approved]. F1000Res. 2016; 3: 62. PubMed Abstract | Publisher Full Text | Free Full Text
7. The Royal Society, British Academy, Royal Academy of Engineering, et al.: Open for business: a nation of global researchers and innovators. [cited 2018 Apr 11]. Reference Source
8. Sufi S, Jay C: Dataset 1 in: Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme. F1000Research. 2018. http://www.doi.org/10.5256/f1000research.16231.d218703

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 03 Oct 2018

Author details Author details

¹ School of Computer Science, University of Manchester, Manchester, Greater Manchester, M13 9PL, UK

Shoaib Sufi
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing

Caroline Jay
Roles: Data Curation, Formal Analysis, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Competing interests

The first author, Shoaib Sufi, is employed on the Institute grant, and the second author, Caroline Jay, is a Fellow; both therefore have an interest in the Institute.

Grant information

This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) Grant EP/H043160/1 and EPSRC, BBSRC and ESRC Grant EP/N006410/1 for the UK Software Sustainability Institute.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (1)

version 1

Published: 03 Oct 2018, 7:1599

https://doi.org/10.12688/f1000research.16231.1

© 2018 Sufi S and Jay C. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Sufi S and Jay C. Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme [version 1; peer review: 3 approved with reservations] F1000Research 2018, 7:1599 (https://doi.org/10.12688/f1000research.16231.1)

NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 03 Oct 2018

Views

Reviewer Report 18 Feb 2019

Lois Curfman McInnes, Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL, USA

Approved with Reservations

https://doi.org/10.5256/f1000research.17726.r40285

18th February 2019: This report was briefly published as an Approved report, and has now been updated to an Approved with Reservations at the request of the reviewer.

This paper evaluates the Fellowship Programme of the Software Sustainability Institute (SSI), which has the primary goals of encouraging Fellows to develop their interests in software sustainability and to become ambassadors of good software practice in their communities.

The paper analyses a survey of people who were Fellows during 2012-2016 (26 respondents of 78 Fellows contacted). The paper’s goals are contributing to literature on software sustainability and understanding the program’s impact. The results of the study include only data obtained from the study itself, which featured open questions. The paper explains the methods of the survey and discusses results, around four themes that emerged from analysis of free-text answers: status, community/network, professional development, and resources.

This paper and the SSI Fellowship Programme overall are of strong interest to international communities who are working to advance software practices as a key element of increasing overall scientific productivity. Overall, the paper is well written and clearly explains the approach and analysis of the survey. The analysis concludes that the Fellowship promotes the status of the role of research software and of the Fellows themselves. A key observation is that the Fellowship promotes community and provides a platform for Fellows to influence their domain-specific communities in advancing practices of research software. The Fellowship also contributes strongly to professional development.

My main criticism of the paper is the implicit assumption that the reader understands the importance of software sustainability and the scope of software practices addressed by the Fellowship Programme. I recommend adding background information about software sustainability and the SSI, including references, in order for the paper to be more effective as a stand-alone document.

While I understand the reason for the authors to exclude information about employment during and after the Programme (preserving anonymity), it would be interesting to explore the changes in employers and roles over time (of all Fellows), and whether the SSI Fellowship Programme influenced that. Also, it would be interesting to explore changes in the software practices and culture of domain-specific communities, to help understand the longer-term impact of the SSI Fellowship Programme.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: high-performance scientific computing, scalable numerical algorithms and software, scientific software ecosystems, software productivity and sustainability

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 24 Jan 2019

Dan Sholler, Berkeley Institute for Data Science, The rOpenSci Project, University of California, Berkeley, Berkeley, CA, USA

Approved with Reservations

https://doi.org/10.5256/f1000research.17726.r42711

The paper presents the results of a survey investigating the impact of the UK Software Sustainability Institute’s (SSI) Fellowship Programme on recipients’ research, institutions, and careers. The authors note that their approach to Programme evaluation is novel in that, unlike other reports on programme outcomes (e.g., the Humboldt Foundation), it is a peer-reviewed study using only the data collected via survey. Therefore, the results contribute empirical findings to the software sustainability literature. The study found that the Programme elevated the perceived importance of software development for research as well as the status of the fellowship awardees. The authors state that the survey’s findings indicate the importance and value of research software and the people who develop it.

The paper also provides a sound overview of the SSI Fellowship’s goals: Supporting research software developers in their own work and in championing research software’s value (e.g., promoting reproducible research and open science). The introduction notes how awardees are selected and how the Programme fosters diversity in career stages and disciplines.

Overall, I support the indexing of this study with one substantial addition (a discussion section) and several minor changes. I provide some details on what might be included in the discussion in Comment 3 below.

The authors clearly state that they wish to contribute to the literature on software sustainability. However, they do not provide any definitions of software sustainability, generally, and the types of work and workers needed to achieve sustainability. These additions are necessary to contextualize the findings within the broader discussion about software sustainability. Although I am sure the authors are familiar with this emerging literature because they have made substantial contributions to it, I provide some references to start with (Crouch et al., 2013¹, Calero et al., 2013², Jiménez et al., 2017³, Katz et al., 2014⁴ and Venters et al., 2014⁵).
The authors could reflect more on why there were far more respondents who identified as male than female. For example, does this set of responses reflect the overall makeup of the Fellowship Programme? If not, why might that be?
The paper lacks a discussion that integrates and synthesizes the discrete findings sections. There are several possible ways forward to develop such a discussion question. The first suggestion is to contextualize the free text themes within the forced choice responses. For example, did the 4 respondents who answered “no” to the question about career advancement thematically respond to free text questions, particularly the one about negative impacts? Because the sample size is small, it may be possible to point out the threads in the responses of the “no” participants vs. the “yes” participants. Another way forward is to put all of the sections into conversation with one another. For example, in the discussion, the authors could discuss why (in relation to the existing literature or public discourse) fellowship awards had a greater impact on professional development of others than on the individuals. The “how” is already present and appreciated—e.g., Software Carpentry workshops. Likewise, the authors might return to statements such as R17’s institution not being interested in the fellowship, and how that relates to the reported benefits to the institution. In sum, some more reflection on the responses and how they relate to the broader discourse around software sustainability and support for research software development would be very much appreciated.
I commend the authors for explicitly stating the themes in the findings section. I have one issue with how the authors collaborated for the qualitative coding of the free text responses: Did the authors use any “test data” or something similar to ensure inter-rater reliability? For a study like this one, I do not think it is important to report a quantitative measure of IRR, but a bit more detail about how authors reached agreement would be helpful and contribute to a perception of validity for the reader. One way of doing this is to briefly describe an example of disagreement between the two authors on a particular theme or instance of a theme and describe how the authors reached a resolution.
What software, if any, was used for qualitative coding of the data?
I thank the authors for providing Table 4 with some illustrative examples of responses. I also appreciate how the authors presented some examples in the text and place them in conversation with one another as you might see in an interview-based study (indeed, the prevalence of free-text questions lends itself nicely to presenting the results in this way).
With regard to the statement “Across the other questions, 17 comments related to professional benefits for the Fellows themselves that included: improving personal knowledge and practices; understanding how much of research is software driven; developing a habit for research related blogging; identifying new areas in their own research fields; and thinking about research software engineering as a career”: Can the authors provide some information about the prevalence of each category of professional benefits? Counts are not necessarily the only way to do this; the authors might add phrasing indicating whether one or more of the categories was more prominent than the others.
In the limitations section, the authors might also note the bias in who would respond to such a survey. In other words, awardees who had a positive experience might be more inclined to respond to a survey about the program’s benefits.

Miscellaneous notes:

The authors might consider moving the sentence “The study received approval from the Computer Science School Panel (ref: 2017-2308-3295) on the delegated authority of the University Research Ethics Committee (UREC), University of Manchester” to the first paragraph under “Methods,” where it seems more appropriate.
Given the experience of the authors in this domain, to what extent do they agree with/disagree with the suggested improvements, and what other improvements do they suggest?

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

References

1. Crouch S, Hong N, Hettrick S, Jackson M, et al.: The Software Sustainability Institute: Changing Research Software Attitudes and Practices. Computing in Science & Engineering. 2013; 15 (6): 74-80 Publisher Full Text
2. Calero C, Moraga MA, Bertoa MF: Towards a Software Product Sustainability Model. arXiv. 2013; 1309 (1640). Reference Source
3. Jiménez RC, Kuzak M, Alhamdoosh M, Barker M, et al.: Four simple recommendations to encourage best practices in research software.F1000Res. 2017; 6. PubMed Abstract | Publisher Full Text
4. Katz DS, Choi ST, Lapp H, Maheshwari K, et al.: Summary of the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1). arXiv. 2014; 1404 (7414). Reference Source
5. Venters C, Lau L, Griffiths M, Holmes V, et al.: The Blind Men and the Elephant: Towards an Empirical Evaluation Framework for Software Sustainability. Journal of Open Research Software. 2014; 2 (1). Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Technology adoption and resistance; digital infrastructure development; digital infrastructure governance; open science organizations; open source research software organizations; information systems; information science; organization science

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 17 Oct 2018

Colin C. Venters, School of Computing and Engineering , University of Huddersfield, Huddersfield, UK

Approved with Reservations

https://doi.org/10.5256/f1000research.17726.r39053

This paper reports the results of a qualitative study into the perceived effectiveness of the Software Sustainability Institute's Fellowship program between 2012-2016; a programme that provides fellows with financial support to attend a range of events to help build awareness in sustainable software engineering "best" practice and processes relevant to research software. The authors suggest that the results of the study reveal that the fellowship programme supports communities of best practice and knowledge transfer in the development of sustainable research software. However, this position is not sufficiently supported by the anecdotal evidence reported in the paper.

This was an extremely interesting paper to read; hopefully the first of many by the authors or the Institute that examine the efficacy of the programme and the impact of the Software Sustainability Institute in transforming the development of research software as a whole. The topic is of significant interest to those working in the field of research software engineering - especially potential fellows - and of general interest to the broader software engineering community as a whole as it pertains to having identified individuals that practice and promote sustainable software engineering "best" practice with a particular focus on research software as a first-class research object. The paper attempts to address a pertinent question regarding the efficacy of the fellowship programme as a mechanism for gathering intelligence about research and software from all disciplines and identifying and communicating good software best practice in a range of different domains.

The paper would benefit enormously from a background section that included the motivation for the creation of the Software Sustainability Institute in the first instance i.e. unsustainability of academic software, and linking this to the study by Hettrick et al. 2014¹ who demonstrated the importance of software in research. In addition, the paper would also benefit from introducing and defining key terms and concepts for readers unfamiliar with the topic of software sustainability and linking these to emerging themes being driven forward by leading researchers and groups in the fields of requirements engineering, software architectures, HCI, and software engineering in general that address software sustainability.

The overall research methodology is described in sufficient detail. It is somewhat surprising and disappointing that critical demographic data has not been reported in the paper including the recipient's job role and their institutions at the time the fellowships were awarded, as this information is already publicly available and free to harvest should anyone be interested in doing so; this would seem to be a major oversight and the rationale in the paper for its exclusion would appear to be rather weak. In addition, the paper would have benefited enormously by including data related to recipient’s track record and how have they promoted software sustainability prior to the fellowship programme as a comparison; this would have provided a baseline to establish how the programme transformed the recipients into ambassadors of good software practice in their domains. However, without having established what good or best practice means in the first instance, any claims are challenging at best to determine.

It is worth noting that not all fellows are research software engineers. As such, some analysis on the range of the recipient's job role would have been interesting since the programme is designed to promote diversity to people at different stages in their career.

The results of their analysis revealed a number of interesting findings. However, the main result revealed a worrying trend in that the primary outcome is simply an improvement in the status of beneficiaries of the fellowship programme. This raises serious questions as to whether the programme has failed to achieve its primary aim as a key driver of the programme is the promotion of sustainable software engineering practice to produce verifiable, shareable and useful research output.

The paper also contains several unsubstantiated statements to support a number of claims regarding the perceived benefits of the programme. The claim by one recipient that the collective fellowship in their domain has had a global impact cannot be substantiated and is at best anecdotal. Similarly, it is unclear how the programme helped recipients become "better" scientists. Against what baseline is this improvement being measured? Without any control mechanism in the study, it is impossible to demonstrate with any certainty that the perceived benefits of the programme could not have been achieved in another way.

It would have been interesting for the study to have considered what better sustainability practice means in practice and identify the range of sustainability issues that the variety of communities face. The use of a distributed version control system and commenting code does not in itself make software sustainable.

Based on the results of this study the primary benefits of the fellowship programme are the sole recipients themselves. The benefits to the broader community outside the echo chamber have yet to be established and provide a fruitful area for further research. Overall this was an interesting study of an exciting fellowship programme that lays the foundation for future assessment of the programme as a whole in order to assess the wider benefits to the broader community at large.

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Hettrick S, Antonioletti M, Carr L, Chue Hong N, et al.: UK Research Software Survey 2014. Zenodo. 2014. Publisher Full Text | Reference Source

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Sustainable Software Engineering

CITE

Report a concern

Author Response 22 Nov 2018

Shoaib Sufi, The Software Sustainability Institute, UK

22 Nov 2018

Author Response

We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer’s comments and suggestions below:

It should be noted that the Fellowship is ... Continue reading We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer’s comments and suggestions below:

It should be noted that the Fellowship is broad in its aim, and the survey aimed to capture its impact in the broadest terms, and not just from the perspective of software sustainability skills and uptake. By way of example, the use of distributed version control and commenting code are necessary conditions for sustainable software but they may not be sufficient conditions for sustainable software. The authors recognise that the path to software sustainability is a journey and the adoption of better practices are way-marks on this journey.

We decided to exclude background/demographic data of respondents as it may have enabled readers to identify individuals. It is true that information about the Fellows is available online; the issue in the current study is linking this to individual responses. Anonymity is an important part of the study methodology as a respondent’s public response may be more guarded than a private response. The dataset show individuals criticising other members of their research community or institution. It would not have been appropriate to include alongside these comments information that could be used to identify them, and would have violated our ethical approval.

The data are valid qualitative responses from a survey, collected using a well-established research methodology, and analysed in a systematic manner, and are thus not purely anecdotal. Whilst it is true that we cannot validate the objective truth of any response, this is true of responses to any survey, and is thus an accepted limitation of the design. Fellows were specifically asked to reflect on their own experience, and the survey thus depends upon self-reporting. There is unfortunately no baseline against which to compare the responses, as we cannot know what they would have done had they not received Fellowship.

The fact that status emerged as a key theme is interesting rather than worrying. Our aim with this research was to systematically examine the self-reported benefits of the Fellowship to individuals and others. This could have resulted in people talking only about the events they had run; the value of the status conferred by the badge of the Fellowship is a result that has wider relevance for the Research Software Engineering Community, as the comments show that software engineering work is still perceived as being of lower value, and the Fellowship is helping to change that. The study did not aim to assess directly whether the Fellowship was leading to more sustainable software, as this is not methodologically feasible. It should be noted that in terms of professional development more responses (26) related to the professional development of others than the Fellows own professional development (17), see Table 3. We see this as evidence of its benefit to the community.

We appreciate the suggestions for improving the background and will add this to a future version of the paper detailing the motivation for the Institute. We will comment on the relationship of this work with that by Hettrick et al. on the importance of Software in Research. Generating a cross walk of software sustainability concerns to emerging themes in other research areas such as those mentioned (requirements engineering, software architectures, HCI, and software engineering) are out of scope for this paper, but might form useful future work.

A study on what better sustainability practices mean and identifying sustainability issues and which communities face them is a very large endeavour and this developing area is the focus of organisations such as the UK Software Sustainability Institute (www.software.ac.uk), WSSSPE (wssspe.researchcomputing.org.uk), URSSI (urssi.us), BSSw (bssw.io) and related sustainability researchers. This is out of scope for this paper. Certainly, portions of this space would make for very interesting studies and we thank the reviewer for his comments and suggestions in this regard.
We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer’s comments and suggestions below:

It should be noted that the Fellowship is broad in its aim, and the survey aimed to capture its impact in the broadest terms, and not just from the perspective of software sustainability skills and uptake. By way of example, the use of distributed version control and commenting code are necessary conditions for sustainable software but they may not be sufficient conditions for sustainable software. The authors recognise that the path to software sustainability is a journey and the adoption of better practices are way-marks on this journey.

We decided to exclude background/demographic data of respondents as it may have enabled readers to identify individuals. It is true that information about the Fellows is available online; the issue in the current study is linking this to individual responses. Anonymity is an important part of the study methodology as a respondent’s public response may be more guarded than a private response. The dataset show individuals criticising other members of their research community or institution. It would not have been appropriate to include alongside these comments information that could be used to identify them, and would have violated our ethical approval.

The data are valid qualitative responses from a survey, collected using a well-established research methodology, and analysed in a systematic manner, and are thus not purely anecdotal. Whilst it is true that we cannot validate the objective truth of any response, this is true of responses to any survey, and is thus an accepted limitation of the design. Fellows were specifically asked to reflect on their own experience, and the survey thus depends upon self-reporting. There is unfortunately no baseline against which to compare the responses, as we cannot know what they would have done had they not received Fellowship.

The fact that status emerged as a key theme is interesting rather than worrying. Our aim with this research was to systematically examine the self-reported benefits of the Fellowship to individuals and others. This could have resulted in people talking only about the events they had run; the value of the status conferred by the badge of the Fellowship is a result that has wider relevance for the Research Software Engineering Community, as the comments show that software engineering work is still perceived as being of lower value, and the Fellowship is helping to change that. The study did not aim to assess directly whether the Fellowship was leading to more sustainable software, as this is not methodologically feasible. It should be noted that in terms of professional development more responses (26) related to the professional development of others than the Fellows own professional development (17), see Table 3. We see this as evidence of its benefit to the community.

We appreciate the suggestions for improving the background and will add this to a future version of the paper detailing the motivation for the Institute. We will comment on the relationship of this work with that by Hettrick et al. on the importance of Software in Research. Generating a cross walk of software sustainability concerns to emerging themes in other research areas such as those mentioned (requirements engineering, software architectures, HCI, and software engineering) are out of scope for this paper, but might form useful future work.

A study on what better sustainability practices mean and identifying sustainability issues and which communities face them is a very large endeavour and this developing area is the focus of organisations such as the UK Software Sustainability Institute (www.software.ac.uk), WSSSPE (wssspe.researchcomputing.org.uk), URSSI (urssi.us), BSSw (bssw.io) and related sustainability researchers. This is out of scope for this paper. Certainly, portions of this space would make for very interesting studies and we thank the reviewer for his comments and suggestions in this regard.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 22 Nov 2018

Shoaib Sufi, The Software Sustainability Institute, UK

22 Nov 2018

Author Response

We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer’s comments and suggestions below:

It should be noted that the Fellowship is ... Continue reading We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer’s comments and suggestions below:

It should be noted that the Fellowship is broad in its aim, and the survey aimed to capture its impact in the broadest terms, and not just from the perspective of software sustainability skills and uptake. By way of example, the use of distributed version control and commenting code are necessary conditions for sustainable software but they may not be sufficient conditions for sustainable software. The authors recognise that the path to software sustainability is a journey and the adoption of better practices are way-marks on this journey.

We decided to exclude background/demographic data of respondents as it may have enabled readers to identify individuals. It is true that information about the Fellows is available online; the issue in the current study is linking this to individual responses. Anonymity is an important part of the study methodology as a respondent’s public response may be more guarded than a private response. The dataset show individuals criticising other members of their research community or institution. It would not have been appropriate to include alongside these comments information that could be used to identify them, and would have violated our ethical approval.

The data are valid qualitative responses from a survey, collected using a well-established research methodology, and analysed in a systematic manner, and are thus not purely anecdotal. Whilst it is true that we cannot validate the objective truth of any response, this is true of responses to any survey, and is thus an accepted limitation of the design. Fellows were specifically asked to reflect on their own experience, and the survey thus depends upon self-reporting. There is unfortunately no baseline against which to compare the responses, as we cannot know what they would have done had they not received Fellowship.

The fact that status emerged as a key theme is interesting rather than worrying. Our aim with this research was to systematically examine the self-reported benefits of the Fellowship to individuals and others. This could have resulted in people talking only about the events they had run; the value of the status conferred by the badge of the Fellowship is a result that has wider relevance for the Research Software Engineering Community, as the comments show that software engineering work is still perceived as being of lower value, and the Fellowship is helping to change that. The study did not aim to assess directly whether the Fellowship was leading to more sustainable software, as this is not methodologically feasible. It should be noted that in terms of professional development more responses (26) related to the professional development of others than the Fellows own professional development (17), see Table 3. We see this as evidence of its benefit to the community.

We appreciate the suggestions for improving the background and will add this to a future version of the paper detailing the motivation for the Institute. We will comment on the relationship of this work with that by Hettrick et al. on the importance of Software in Research. Generating a cross walk of software sustainability concerns to emerging themes in other research areas such as those mentioned (requirements engineering, software architectures, HCI, and software engineering) are out of scope for this paper, but might form useful future work.

A study on what better sustainability practices mean and identifying sustainability issues and which communities face them is a very large endeavour and this developing area is the focus of organisations such as the UK Software Sustainability Institute (www.software.ac.uk), WSSSPE (wssspe.researchcomputing.org.uk), URSSI (urssi.us), BSSw (bssw.io) and related sustainability researchers. This is out of scope for this paper. Certainly, portions of this space would make for very interesting studies and we thank the reviewer for his comments and suggestions in this regard.
We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer’s comments and suggestions below:

It should be noted that the Fellowship is broad in its aim, and the survey aimed to capture its impact in the broadest terms, and not just from the perspective of software sustainability skills and uptake. By way of example, the use of distributed version control and commenting code are necessary conditions for sustainable software but they may not be sufficient conditions for sustainable software. The authors recognise that the path to software sustainability is a journey and the adoption of better practices are way-marks on this journey.

We decided to exclude background/demographic data of respondents as it may have enabled readers to identify individuals. It is true that information about the Fellows is available online; the issue in the current study is linking this to individual responses. Anonymity is an important part of the study methodology as a respondent’s public response may be more guarded than a private response. The dataset show individuals criticising other members of their research community or institution. It would not have been appropriate to include alongside these comments information that could be used to identify them, and would have violated our ethical approval.

The data are valid qualitative responses from a survey, collected using a well-established research methodology, and analysed in a systematic manner, and are thus not purely anecdotal. Whilst it is true that we cannot validate the objective truth of any response, this is true of responses to any survey, and is thus an accepted limitation of the design. Fellows were specifically asked to reflect on their own experience, and the survey thus depends upon self-reporting. There is unfortunately no baseline against which to compare the responses, as we cannot know what they would have done had they not received Fellowship.

The fact that status emerged as a key theme is interesting rather than worrying. Our aim with this research was to systematically examine the self-reported benefits of the Fellowship to individuals and others. This could have resulted in people talking only about the events they had run; the value of the status conferred by the badge of the Fellowship is a result that has wider relevance for the Research Software Engineering Community, as the comments show that software engineering work is still perceived as being of lower value, and the Fellowship is helping to change that. The study did not aim to assess directly whether the Fellowship was leading to more sustainable software, as this is not methodologically feasible. It should be noted that in terms of professional development more responses (26) related to the professional development of others than the Fellows own professional development (17), see Table 3. We see this as evidence of its benefit to the community.

We appreciate the suggestions for improving the background and will add this to a future version of the paper detailing the motivation for the Institute. We will comment on the relationship of this work with that by Hettrick et al. on the importance of Software in Research. Generating a cross walk of software sustainability concerns to emerging themes in other research areas such as those mentioned (requirements engineering, software architectures, HCI, and software engineering) are out of scope for this paper, but might form useful future work.

A study on what better sustainability practices mean and identifying sustainability issues and which communities face them is a very large endeavour and this developing area is the focus of organisations such as the UK Software Sustainability Institute (www.software.ac.uk), WSSSPE (wssspe.researchcomputing.org.uk), URSSI (urssi.us), BSSw (bssw.io) and related sustainability researchers. This is out of scope for this paper. Certainly, portions of this space would make for very interesting studies and we thank the reviewer for his comments and suggestions in this regard.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 03 Oct 2018

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 1 03 Oct 18	read	read	read

Colin C. Venters, University of Huddersfield, Huddersfield, UK
Dan Sholler, University of California, Berkeley, Berkeley, USA
Lois Curfman McInnes, Argonne National Laboratory, Argonne, USA

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

6 Views

18 Feb 2019 | for Version 1

Lois Curfman McInnes, Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, IL, USA

6 Views Cite this report Responses(0)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

high-performance scientific computing, scalable numerical algorithms and software, scientific software ecosystems, software productivity and sustainability

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

9 Views

24 Jan 2019 | for Version 1

Dan Sholler, Berkeley Institute for Data Science, The rOpenSci Project, University of California, Berkeley, Berkeley, CA, USA

9 Views Cite this report Responses(0)

Approved With Reservations

The authors clearly state that they wish to contribute to the literature on software sustainability. However, they do not provide any definitions of software sustainability, generally, and the types of work and workers needed to achieve sustainability. These additions are necessary to contextualize the findings within the broader discussion about software sustainability. Although I am sure the authors are familiar with this emerging literature because they have made substantial contributions to it, I provide some references to start with (Crouch et al., 2013¹, Calero et al., 2013², Jiménez et al., 2017³, Katz et al., 2014⁴ and Venters et al., 2014⁵).
The authors could reflect more on why there were far more respondents who identified as male than female. For example, does this set of responses reflect the overall makeup of the Fellowship Programme? If not, why might that be?
The paper lacks a discussion that integrates and synthesizes the discrete findings sections. There are several possible ways forward to develop such a discussion question. The first suggestion is to contextualize the free text themes within the forced choice responses. For example, did the 4 respondents who answered “no” to the question about career advancement thematically respond to free text questions, particularly the one about negative impacts? Because the sample size is small, it may be possible to point out the threads in the responses of the “no” participants vs. the “yes” participants. Another way forward is to put all of the sections into conversation with one another. For example, in the discussion, the authors could discuss why (in relation to the existing literature or public discourse) fellowship awards had a greater impact on professional development of others than on the individuals. The “how” is already present and appreciated—e.g., Software Carpentry workshops. Likewise, the authors might return to statements such as R17’s institution not being interested in the fellowship, and how that relates to the reported benefits to the institution. In sum, some more reflection on the responses and how they relate to the broader discourse around software sustainability and support for research software development would be very much appreciated.
I commend the authors for explicitly stating the themes in the findings section. I have one issue with how the authors collaborated for the qualitative coding of the free text responses: Did the authors use any “test data” or something similar to ensure inter-rater reliability? For a study like this one, I do not think it is important to report a quantitative measure of IRR, but a bit more detail about how authors reached agreement would be helpful and contribute to a perception of validity for the reader. One way of doing this is to briefly describe an example of disagreement between the two authors on a particular theme or instance of a theme and describe how the authors reached a resolution.
What software, if any, was used for qualitative coding of the data?
I thank the authors for providing Table 4 with some illustrative examples of responses. I also appreciate how the authors presented some examples in the text and place them in conversation with one another as you might see in an interview-based study (indeed, the prevalence of free-text questions lends itself nicely to presenting the results in this way).
With regard to the statement “Across the other questions, 17 comments related to professional benefits for the Fellows themselves that included: improving personal knowledge and practices; understanding how much of research is software driven; developing a habit for research related blogging; identifying new areas in their own research fields; and thinking about research software engineering as a career”: Can the authors provide some information about the prevalence of each category of professional benefits? Counts are not necessarily the only way to do this; the authors might add phrasing indicating whether one or more of the categories was more prominent than the others.
In the limitations section, the authors might also note the bias in who would respond to such a survey. In other words, awardees who had a positive experience might be more inclined to respond to a survey about the program’s benefits.

Miscellaneous notes:

The authors might consider moving the sentence “The study received approval from the Computer Science School Panel (ref: 2017-2308-3295) on the delegated authority of the University Research Ethics Committee (UREC), University of Manchester” to the first paragraph under “Methods,” where it seems more appropriate.
Given the experience of the authors in this domain, to what extent do they agree with/disagree with the suggested improvements, and what other improvements do they suggest?

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

References

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Technology adoption and resistance; digital infrastructure development; digital infrastructure governance; open science organizations; open source research software organizations; information systems; information science; organization science

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

29 Views

17 Oct 2018 | for Version 1

Colin C. Venters, School of Computing and Engineering , University of Huddersfield, Huddersfield, UK

29 Views Cite this report Responses(1)

Approved With Reservations

Is the work clearly and accurately presented and does it cite the current literature?

No
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Hettrick S, Antonioletti M, Carr L, Chue Hong N, et al.: UK Research Software Survey 2014. Zenodo. 2014. Publisher Full Text | Reference Source

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Sustainable Software Engineering

Respond to this report

Responses (1)

Author Response

22 Nov 2018

Shoaib Sufi, The Software Sustainability Institute, UK

We thank the reviewer for his positive and helpful comments. We respond to the main themes in the reviewer’s comments and suggestions below:

It should be noted that the Fellowship is broad in its aim, and the survey aimed to capture its impact in the broadest terms, and not just from the perspective of software sustainability skills and uptake. By way of example, the use of distributed version control and commenting code are necessary conditions for sustainable software but they may not be sufficient conditions for sustainable software. The authors recognise that the path to software sustainability is a journey and the adoption of better practices are way-marks on this journey.

We decided to exclude background/demographic data of respondents as it may have enabled readers to identify individuals. It is true that information about the Fellows is available online; the issue in the current study is linking this to individual responses. Anonymity is an important part of the study methodology as a respondent’s public response may be more guarded than a private response. The dataset show individuals criticising other members of their research community or institution. It would not have been appropriate to include alongside these comments information that could be used to identify them, and would have violated our ethical approval.

The data are valid qualitative responses from a survey, collected using a well-established research methodology, and analysed in a systematic manner, and are thus not purely anecdotal. Whilst it is true that we cannot validate the objective truth of any response, this is true of responses to any survey, and is thus an accepted limitation of the design. Fellows were specifically asked to reflect on their own experience, and the survey thus depends upon self-reporting. There is unfortunately no baseline against which to compare the responses, as we cannot know what they would have done had they not received Fellowship.

The fact that status emerged as a key theme is interesting rather than worrying. Our aim with this research was to systematically examine the self-reported benefits of the Fellowship to individuals and others. This could have resulted in people talking only about the events they had run; the value of the status conferred by the badge of the Fellowship is a result that has wider relevance for the Research Software Engineering Community, as the comments show that software engineering work is still perceived as being of lower value, and the Fellowship is helping to change that. The study did not aim to assess directly whether the Fellowship was leading to more sustainable software, as this is not methodologically feasible. It should be noted that in terms of professional development more responses (26) related to the professional development of others than the Fellows own professional development (17), see Table 3. We see this as evidence of its benefit to the community.

We appreciate the suggestions for improving the background and will add this to a future version of the paper detailing the motivation for the Institute. We will comment on the relationship of this work with that by Hettrick et al. on the importance of Software in Research. Generating a cross walk of software sustainability concerns to emerging themes in other research areas such as those mentioned (requirements engineering, software architectures, HCI, and software engineering) are out of scope for this paper, but might form useful future work.

A study on what better sustainability practices mean and identifying sustainability issues and which communities face them is a very large endeavour and this developing area is the focus of organisations such as the UK Software Sustainability Institute (www.software.ac.uk), WSSSPE (wssspe.researchcomputing.org.uk), URSSI (urssi.us), BSSw (bssw.io) and related sustainability researchers. This is out of scope for this paper. Certainly, portions of this space would make for very interesting studies and we thank the reviewer for his comments and suggestions in this regard.

View more View less

Competing Interests

No competing interests were disclosed.

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

Click here to access the data.

Downloaded data do not display as expected? Download the data (28.96KB)

[1] 1. Sufi S: The Software Sustainability Institute Fellowship Programme. CEUR Workshop Proceedings (CEUR-WS.org):WSSSPE: 4th Workshop on Sustainable Software for Science: Practice and Experiences. 1686. Reference Source

[2] 2. Technopolis: Evaluation of the Alexander von Humboldt Foundation’s Humboldt Research Fellowship Programme. 2009; [cited 2018 Aug 8]. Reference Source

[3] 3. Meyer N, Bührer S: Impact Evaluation of the Erwin Schrödinger Fellowships with Return Phase. Karlsruhe: Fraunhofer Institute for Systems and Innovation Research ISI; 2014; 97. Reference Source

[4] 4. Keith N, Jones R, Chow M: Impact Evaluation of the ICRETT, YY & ACSBI Fellowship Schemes. Union for International Cancer Control (UICC); 2013; [cited 2018 Aug 8]. Reference Source

[5] 5. Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006 [cited 2018 Apr 11]; 2(4): 77–101. Publisher Full Text

[6] 6. Wilson G: Software Carpentry: lessons learned [version 2; referees: 3 approved]. F1000Res. 2016; 3: 62. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. The Royal Society, British Academy, Royal Academy of Engineering, et al.: Open for business: a nation of global researchers and innovators. [cited 2018 Apr 11]. Reference Source

[8] 8. Sufi S, Jay C: Dataset 1 in: Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme. F1000Research. 2018. http://www.doi.org/10.5256/f1000research.16231.d218703

Raising the status of software in research: A survey-based evaluation of the Software Sustainability Institute Fellowship Programme

Abstract

Keywords

Introduction

Related work

Methods

Table 1. Survey questions and analysis performed.

Results

Demographic information

Table 2. Funding sources of the respondents’ research.

Forced choice analysis

Free text analysis

Table 3. Number of respondents reporting a benefit under each theme for the various question categories.

Status

Table 4. Responses illustrating the impact of the fellowship on status.

Community/network

Table 5. Responses illustrating the impact of the Fellowship on community/network.

Professional development

Table 6. Responses illustrating the impact of the Fellowship on professional development – self.

Resources

Table 7. Activities that respondents reported were made possible using the Fellowship award.

Negative consequences

Improvements

Limitations

Conclusion

Data availability

Grant information

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

The problem

How to fix it

Competing Interests Policy

Stay Updated