|
on Sociology of Economics |
| By: | Brodeur, Abel; Cook, Nikolai M.; Hartley, Jonathan S.; Heyes, Anthony |
| Abstract: | Randomized controlled trials (RCTs) are increasingly prominent in economics, with pre-registration and pre-analysis plans (PAPs) promoted as important in ensuring the credibility of findings. We investigate whether these tools reduce the extent of p-hacking and publication bias by collecting and studying the universe of test statistics, 15,992 in total, from RCTs published in 15 leading economics journals from 2018 through 2021. In our primary analysis, we find no meaningful difference in the distribution of test statistics from pre-registered studies, compared to their non-pre-registered counterparts. However, pre-registered studies that have a complete PAP are significantly less p-hacked. These results point to the importance of PAPs, rather than pre-registration in itself, in ensuring credibility. |
| Keywords: | Pre-analysis plan,Pre-registration,p-Hacking,Publication bias,Research credibility |
| JEL: | B41 C13 C40 C93 |
| Date: | 2022 |
| URL: | http://d.repec.org/n?u=RePEc:zbw:glodps:1147&r= |
| By: | Brodeur, Abel; Cook, Nikolai; Heyes, Anthony |
| Abstract: | Amazon's Mechanical Turk is a very widely-used tool in business and economics research, but how trustworthy are results from well-published studies that use it? Analyzing the universe of hypotheses tested on the platform and published in leading journals between 2010 and 2020 we find evidence of widespread p-hacking, publication bias and over-reliance on results from plausibly under-powered studies. Even ignoring questions arising from the characteristics and behaviors of study recruits, the conduct of the research community itself erodes substantially the credibility of these studies' conclusions. The extent of the problems vary across the business, economics, management and marketing research fields (with marketing especially afflicted). The problems are not getting better over time and are much more prevalent than in a comparison set of non-online experiments. We explore correlates of increased credibility. |
| Keywords: | online crowd-sourcing platforms,Amazon Mechanical Turk,p-hacking,publication bias,statistical power,research credibility |
| JEL: | B41 C13 C40 C90 |
| Date: | 2022 |
| URL: | http://d.repec.org/n?u=RePEc:zbw:glodps:1157&r= |
| By: | Misha Teplitskiy; Soya Park; Neil Thompson; David Karger |
| Abstract: | As the academic community debates the future of in-person conferences, it is important to understand how effective they are at diffusing ideas. Most previous attempts to analyze this question have struggled to separate causation from correlation and used potentially biased measures like self-reported learning. Here, we propose a novel approach using scheduling conflicts. When multiple presentations of interest to an attendee are scheduled at the same time, the attendee is less able to see them, on average. If seeing presentations influences future research, then conflicting presentations should influence research less than unconflicting ones. Analyzing conflicts in the personalized schedules of 1960 attendees of 20 computer science conferences reveals that when an attendee is able to see a paper presentation, she is almost twice as likely to cite the paper in her future work. The effect is robust to underlying differences between attendees, papers, and paper authors, and is even larger for a stronger measure of influence -- citing the presented paper multiple times. Given the substantial learning effects of in-person presentations, it will be important to ensure that attempts to turn conferences hybrid or virtual do not imperil knowledge diffusion. |
| Date: | 2022–09 |
| URL: | http://d.repec.org/n?u=RePEc:arx:papers:2209.01175&r= |
| By: | Julian Kolev; Alexis Haughey; Fiona Murray; Scott Stern |
| Abstract: | What is the role of startups within the innovation ecosystem? Since 2000, startups have grown in their share of commercializing research from top U.S. universities; however, prior work has little to say on the particular advantages of startup ventures in the innovation process relative to more traditional alternatives such as academia and established private-sector incumbents. We develop a simple model of startup advantage based on private information held by the initial inventor, and generate predictions related to the value and impact of startup innovation. We then explore these predictions using patents granted within the regional ecosystems of top-25 research universities from 2000 to 2015. Our results show a significant startup advantage in terms of forward citations and outlier-patent rates. Further, startup innovation is both more original and more general than innovation by incumbent firms. Moreover, startups that survive to become “scale-ups” quickly grow to dominate their regional innovation ecosystems. Our findings have important implications for innovation policy. |
| JEL: | L24 L26 M13 O31 O32 O34 |
| Date: | 2022–08 |
| URL: | http://d.repec.org/n?u=RePEc:nbr:nberwo:30362&r= |