UIC Database Group

Header bar

2025-02-02 - Paper on efficient model search accepted at SIGMOD 2025

Efficient Model Search for Transfer Learning

In this work [1] lead by HPI Ph.D. student Nils, Boris together with Tilmann from HPI investigate how to speed up the process of model search for transfer learning. Given a target dataset, the goal is to find an suitable base model to use as the starting point for transfer learning. We exploit overlap between fine-tuned versions of a base model to avoid redundant work during the inference step of model search through caching and optimizing the search order to maximize reuse of cached intermediate results and model blocks.

Transfer learning is an effective technique for tuning a deep learning model when training data or computational resources are limited. Instead of training a new model from scratch, the parameters of an existing “base model” are adjusted for a new task. The accuracy of such a fine-tuned model depends on choosing an appropriate base model. Model search automates the selection of such a base model by evaluating the suitability of candidate models for a specific task. This entails inference with each candidate model on task-specific data. With thousands of models available through model stores, the computational cost of model search is a major bottleneck for efficient transfer learning.

In this paper, we develop Alsatian, a novel model search system. Based on the observation that many candidate models overlap to a significant extent and based on a careful bottleneck analysis, we propose optimization techniques that are applicable to many model search frameworks. These optimizations include: (i) splitting models into individual blocks that can be shared across models, (ii) caching of intermediate inference results and model blocks, and (iii) selecting a beneficial search order for models to maximize sharing of cached results. In our evaluation on state-of-the-art deep learning models from computer vision and natural language processing, we show that Alsatian outperforms baselines by up to ~14x.

  1. Alsatian: Optimizing Model Search for Deep Transfer Learning
    Nils Strassenburg, Boris Glavic and Tilmann Rabl
    SIGMOD. 3, 3 (2025) , 127:1–127:27.
    details