Abstract:The filtering distribution in hidden Markov models evolves according to the law of a mean-field model in state-observation space. The ensemble Kalman filter (EnKF) approximates this mean-field model with an ensemble of interacting particles, employing a Gaussian ansatz for the joint distribution of the state and observation at each observation time. These methods are robust, but the Gaussian ansatz limits accuracy. Here this shortcoming is addressed by using machine learning to map the joint predicted state and observation to the updated state estimate. The derivation of methods from a mean field formulation of the true filtering distribution suggests a single parametrization of the algorithm that can be deployed at different ensemble sizes. And we use a mean field formulation of the ensemble Kalman filter as an inductive bias for our architecture.
To develop this perspective, in which the mean-field limit of the algorithm and finite interacting ensemble particle approximations share a common set of parameters, a novel form of neural operator is introduced, taking probability distributions as input: a measure neural mapping (MNM). A MNM is used to design a novel approach to filtering, the MNM-enhanced ensemble filter (MNMEF), which is defined in both the mean-field limit and for interacting ensemble particle approximations. The ensemble approach uses empirical measures as input to the MNM and is implemented using the set transformer, which is invariant to ensemble permutation and allows for different ensemble sizes. In practice fine-tuning of a small number of parameters, for specific ensemble sizes, further enhances the accuracy of the scheme. The promise of the approach is demonstrated by its superior root-mean-square-error performance relative to leading methods in filtering the Lorenz '96 and Kuramoto-Sivashinsky models.
| Comments: | Accepted by the Journal of Computational Physics |
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG); Systems and Control (eess.SY); Computational Physics (physics.comp-ph) |
| Cite as: | arXiv:2504.17836 [stat.ML] |
| (or arXiv:2504.17836v4 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2504.17836 arXiv-issued DOI via DataCite |
Submission history
From: Bohan Chen [view email]
[v1]
Thu, 24 Apr 2025 17:48:03 UTC (1,133 KB)
[v2]
Tue, 27 May 2025 22:43:54 UTC (1,060 KB)
[v3]
Sat, 6 Dec 2025 00:35:49 UTC (907 KB)
[v4]
Tue, 23 Dec 2025 23:38:19 UTC (907 KB)