April 22, 2022
Many organizations are reimplementing and extending deep learning models from the research community, a process we describe as model reengineering. This task is challenging for diverse reasons, including under-documented reference models, changing requirements, and the cost of implementation and testing. As yet, we know little about the reengineering process and its common challenges. Our goal is to understand the characteristics and challenges of deep learning reengineering. We conducted an exploratory case study of this phenomenon, focusing on the context of computer vision. Our results draw from two data sources: issues in open-source reeengineering projects, and interviews with the leaders of a (student) reengineering team. In the open-source data, we analyzed 310 defects from 25 open-source deep learning projects. Our reengineering team replicated 7 deep learning models over two years; we interviewed 6 leaders to understand their experiences. Our results describe how deep learning-based computer vision techniques are reengineered, analyze the distribution of defects in this process, and discuss challenges and practices. Our findings inform several future directions, including: additional unknown aspects of model reengineering; testing tools for reengineering; and gaps in current approaches for model reuse.