Evaluating Bias in Low Resource MLMs: A Comprehensive Study on Measuring Gender Bias in BERT for Bangla

Abstract

This paper investigates gender–profession associations in Bengali BERT family models using a controlled masked language model (MLM) probing framework. We construct a systematic evaluation corpus of 64,800 sentences using a full factorial design over gender-denoting person words, profession terms, and sentence templates. Using prior-normalized log-probability and sigmoid based scoring, we measure how profession context influences the likelihood of gendered terms. Across three models (mBERT, BUET BanglaBERT, and Sagor Sarker BanglaBERT) we observe consistent variations in association strength, with a general tendency toward stronger alignment with male-denoting words, alongside notable exceptions across profession categories. These findings reflect learned distributional patterns rather than explicit rules. We further examine the applicability of an English originated bias measurement framework in Bengali, highlighting challenges in interpretation due to linguistic and data differences. Overall, this work provides a controlled methodology and empirical analysis for studying gender profession associations in low-resource language models.