Crowdsourcing-based Model Testing in Federated Learning

Federated Learning (FL) is a distributed machine learning technique that trains models on local devices to preserve data privacy. In FL, evaluating model quality is crucial for detecting malicious clients and improving model accuracy. However, existing methods typically require a representative public testing dataset on the server, which is often unavailable in practical federated learning scenarios. To address this problem, we propose a novel four-step framework, taking a crowdsourcing approach. The basic idea is to distribute the model to be evaluated as a task to a set of testing clients selected from the original clients pool, who evaluate the model quality using their local datasets. By consolidating these individual evaluations, we obtain the overall model quality. To select a suitable number of testing clients, we propose an exploration-exploitation-based framework. Furthermore, to safeguard against attacks from potential malicious testing clients, we introduce a Correlated Agreement (CA) mechanism. This is achieved by comparing correlations of accuracy among the same set of testing clients (who were selected for the aforementioned evaluation task). Extensive experiments demonstrate the effectiveness of our approach, which yields accuracy comparable to methods that rely on a public testing dataset on the server. Moreover, our approach can identify and filter out dishonest testing clients and thereby ensure model quality even in adversarial settings.