Toward fractal development of data processing-intensive artificial intelligence systems

Given the data-intensive trend in this current artificial intelligence (AI) wave, data processing keeps playing a crucial role in the modern AI systems. In our data processing-intensive AI project (namely ALeRCE, Automatic Learning for the Rapid Classification of Events), we have employed multiple teams to develop the software infrastructure for different data processing tasks independently, to speed up the project implementation. Along with the growth of ALeRCE, however, we increasingly suffered from the tech stack sprawl, due to the frequent changes of development teams and the hasty employment of disruptive technologies. By investigating a fractal pattern in engineering ALeRCE’s software infrastructure, we eventually developed a fractal development methodology to mitigate the tech stack sprawl and to facilitate data processing implementations at different scales. In ALeRCE, the engineering self-similarity is characterized largely by the same input-output requirements and mechanisms and by the same deployment configurations and operations, no matter what granularity the data processing logic or task has. Correspondingly, the unique feature of our fractal development methodology is to enable granularity-independent scaffolding for coding data processing logics and to maximize the deployment consistency. Although the supporting framework of scaffolding might need to be project-specific, the fractal development methodology would be applicable to AI engineering in general.