Ionic mobility, which determines the rate performance of several applications, such as batteries, is exponentially dependent on the ionic migration barrier (E_m) within solids, a quantity that is difficult to measure experimentally or estimate computationally. Here, we present a graph neural network based architecture that leverages principles of transfer learning to efficiently and accurately predict E_m across a diverse set of materials. Modifying a pre-trained model on bulk material properties and adding suitable modifications to the framework, we fine-tuned our models on a manually-curated literature-derived calculated dataset of 619 E_m data points. Importantly, our best performing fine-tuned models display R^2 scores and mean absolute errors that are ~40-70% better than scratch and classical machine learning models and universal interatomic potentials. Moreover, our best model generalizes well across migration pathways, intercalant compositions, and chemistries and also acts as a robust classification tool (80% accuracy and 82.7% sensitivity in identifying good conductors). Thus, we establish a pathway for discovering novel materials with high ionic mobility as well as to predict data-scarce material properties for different applications.