|
on Sociology of Economics |
| By: | Tom Coupé (University of Canterbury); Thomas Logchies; W. Robert Reed (University of Canterbury) |
| Abstract: | There is a widespread belief that replication studies are less cited than original research. This study introduces three counterfactual approaches for measuring the citations that an author or a journal would have received had they produced a non-replication study. Two of the measures are designed to measure citation incentives from the perspective of authors. One measure focuses on the perspective of journals. We collect data on 428 replications in economics published between 1958 and 2021 and assess whether these are cited less frequently than their matched counterfactuals. We obtain a wide range of estimates. Our preferred estimates use the ratio of citations of a replication to the citations of its matched counterfactuals. Using this measure, we estimate citation penalties as large as 51% and citation benefits as great as 227%. Most replications receive fewer citations than their matched counterfactuals, but a sizable portion, and sometimes even a majority, receive more. Finally, there is some evidence that replications that do not fully support the original study have more favorable citation rates than those that confirm the original study. While our analysis does not produce an unambiguous answer to whether replications receive less citations than their counterfactuals, it does revise the widely held, one-sided view that replications receive fewer citations. |
| Keywords: | Replications, Citations, Incentives, Academic Publishing |
| JEL: | A10 A14 B41 C80 |
| Date: | 2024–12–01 |
| URL: | https://d.repec.org/n?u=RePEc:cbt:econwp:24/18 |
| By: | Costanza Naguib |
| Abstract: | In mid-2016, all journals of the American Economic Association (AEA) stopped including significance stars in their regression tables. This policy aimed to reduce the emphasis on statistical significance and shift focus toward the broader economic importance of research findings. This study examines the impact of this change on p-hacking and publication bias. The findings indicate some reduction in the bunching of the reported test statistics just beyond the 5%-significance threshold in the treated journals after 2016. However, the effect is modest. |
| Keywords: | p-hacking, significance stars, publication bias, difference-in-difference |
| JEL: | A11 A14 C13 |
| Date: | 2024–10 |
| URL: | https://d.repec.org/n?u=RePEc:ube:dpvwib:dp2409 |
| By: | Heyard, Rachel; Pawel, Samuel (University of Zurich); Frese, Joris; Voelkl, Bernhard; Würbel, Hanno (University of Bern); McCann, Sarah; Held, Leonhard; Wever, Kimberley E. PhD (Radboud university medical center); Hartmann, Helena (University Hospital Essen); Townsin, Louise |
| Abstract: | *Background:* Reproducibility is recognized as essential to scientific progress and integrity. Replication studies and large-scale replication projects, aiming to quantify different aspects of reproducibility, have become more common. Since no standardized approach to measuring reproducibility exists, a diverse set of metrics has emerged and a comprehensive overview is needed. *Methods:* We conducted a scoping review to identify large-scale replication projects that used metrics and methodological papers that proposed or discussed metrics. The project list was compiled by the authors. For the methodological papers, we searched Scopus, MedLine, PsycINFO andEconLit. Records were screened in duplicate against predefined inclusion criteria. Demographic information on included records and information on reproducibility metrics used, suggested or discussed was extracted. *Results:* We identified 49 large-scale projects and 97 methodological papers, and extracted 50 metrics. The metrics were characterized based on type (formulas and/or statistical models, frameworks, graphical representations, studies and questionnaires, algorithms), input required, and appropriate application scenarios. Each metric addresses a distinct question. *Conclusions:* Our review provides a comprehensive resource in the form of a “live”, interactive table for future replication teams and meta-researchers, offering support in how to select the most appropriate metrics that are aligned with research questions and project goals. |
| Date: | 2024–11–26 |
| URL: | https://d.repec.org/n?u=RePEc:osf:metaar:apdxk |
| By: | Steve Agnew (University of Canterbury); Paul Bostock; Ellie Kay; Kaylene Sampson; Russell Wordsworth (University of Canterbury) |
| Abstract: | With business studies now an option alongside traditional economics and accounting at NCEA (National Certificate of Educational Achievement) level three in secondary schools, many students interested in business may not have taken economics before university. This study examines whether prior completion of level three NCEA economics predicts higher academic success in an introductory microeconomics course. After controlling for a range of other variables, studying economics at secondary school is found to be significantly correlated with higher academic achievement in a university introductory economics class. These findings may help guide course advice for high school students considering business studies at university. Findings may inform course selection information given to high school students considering studying business at university, as well as assist economics departments in offering more targeted support to students with no prior economics experience. |
| Keywords: | NCEA (National Certificate of Educational Achievement), Level Three economics, First-year university performance, Academic achievement predictors, Introductory microeconomics |
| Date: | 2024–12–01 |
| URL: | https://d.repec.org/n?u=RePEc:cbt:econwp:24/17 |
| By: | Echenique, Federico; He, Kevin |
| Abstract: | We show that adding noise before publishing data effectively screens [Formula: see text]-hacked findings: spurious explanations produced by fitting many statistical models (data mining). Noise creates baits that affect two types of researchers differently. Uninformed [Formula: see text]-hackers, who are fully ignorant of the true mechanism and engage in data mining, often fall for baits. Informed researchers, who start with an ex ante hypothesis, are minimally affected. We show that as the number of observations grows large, dissemination noise asymptotically achieves optimal screening. In a tractable special case where the informed researchers theory can identify the true causal mechanism with very few data, we characterize the optimal level of dissemination noise and highlight the relevant trade-offs. Dissemination noise is a tool that statistical agencies currently use to protect privacy. We argue this existing practice can be repurposed to screen [Formula: see text]-hackers and thus improve research credibility. |
| Keywords: | dissemination noise, p-hacking, privacy, research integrity |
| Date: | 2024–05–21 |
| URL: | https://d.repec.org/n?u=RePEc:cdl:econwp:qt6sm4w1jf |