Human skill in training deep networks

We’ve seen “tuning hyperparameters without grad students” with Dragonfly (DT #11) but… how much does a researcher’s experience actually correlate with their skills for tuning an ML model? Anand et al, (2020) investigated this and found a strong positive correlation between experience and final model accuracy, and “that an experienced participant finds better solutions using fewer resources on average.” Glad to see my skills aren’t yet completely automatable yet! (The paper is co-authored by Jan van Gemert, who was the first person to explain to me what a convolution is, in a guest lecture during my first year of undergrad. 😊)