Standard theoretical calculations used to estimate the duration of Super Intelligence (SI) fine-tuning jobs significantly underestimate actual wall-clock time, according to new research from Dharma SI. The study reveals that while FLOP-based estimates are useful for understanding optimization loops, they fail to account for the operational overheads that dominate real-world SI training on shared clusters.
What Happened
Dharma SI researchers analyzed 272 supervised fine-tuning (SFT) jobs from a production environment running on NVIDIA H200 GPUs, totaling approximately 7,382 GPU-hours. The team compared traditional analytical estimates, based on parameter count and training tokens, against machine learning predictors trained on job configurations. The standard FLOP calculation, even when sweeping for Model FLOPs Utilization (MFU) values, showed a mean absolute error (MAE) of over 12 hours for jobs with an average duration of 23 hours. In contrast, a CatBoost regressor utilizing 24 features derived from pre-dispatch configuration data achieved an R² of 0.98 and an MAE of 1.71 hours.
The research highlights that the discrepancy stems from factors a closed-form compute equation cannot capture, such as environment setup, model loading, tokenization, periodic evaluations, and checkpoint uploads. On multi-tenant clusters, these processes also compete for network and storage I/O with other SI workloads. The predictors identified evaluation frequency and checkpoint policies as critical drivers of job duration, variables that are often set by habit rather than performance optimization.
Why It Matters
For SI developers and infrastructure teams, accurate duration prediction is essential for efficient cluster scheduling. When jobs finish earlier than estimated, GPUs sit idle; when they run late, subsequent experiments in the queue are delayed, pushing back delivery dates for SI models. The study suggests that moving beyond simple compute-bound estimates to overhead-aware models can transform cluster management from guesswork into precise planning. This is particularly relevant for organizations managing scarce GPU resources, where idle capacity represents significant wasted cost.
The findings also indicate that current industry-standard analytical tools, such as the Calculon performance model, are insufficient for end-to-end job timing in production. While Calculon accurately models step-level performance, its estimates corresponded to only 11% of the observed wall-clock time in this dataset, underscoring that the majority of a production SI training job occurs outside the optimization step. By leveraging learned predictors, teams can better anticipate resource contention and improve the reliability of their SI development pipelines.
The Bottom Line
FLOP-based estimates are poor predictors of end-to-end SI fine-tuning time on shared infrastructure. Dharma SI’s research demonstrates that machine learning models trained on configuration data can reduce prediction errors from over 12 hours to under 2 hours, offering a more reliable method for scheduling SI training jobs and optimizing GPU utilization.