CGS-SLAM: Collaborative Gaussian Splatting based SLAM for Multi-Agent Reconstruction

We propose a hybrid decentralized–centralized architecture for light-weight collaborative Gaussian-Splatting SLAM. Each agent independently reconstructs a local map from its own limited sensor suite, employing a decentralized approach while periodically exchanging keyframe encodings through a central server, allowing a dynamic keyframing process. After all agents finish their exploration, the centralized server fuses the individual submaps into a globally consistent map.

In a monocular setup, relying on a constant-velocity assumption between frames is fragile under rapid camera motion or low frame rates. We instead follow Visual-Inertial SLAM: before optimizing the camera pose for each new frame, we compute an initial pose estimate by chaining IMU preintegration between the last tracked frame and the incoming one, which is far more accurate than a motion prior alone.

To recover metric geometry from a single camera, we use a metric monocular depth estimator (MMDE), Depth Pro, at every keyframe. Because it predicts depth at absolute scale, it avoids the temporal drift of relative-depth models and removes the need for scale recovery through post-hoc alignment or batch-based deep bundle adjustment, enabling truly online operation from the very first frame.

Each agent builds its local 3D map from synchronized RGB video and IMU data. During tracking, the Gaussian map is held fixed and the camera pose is optimized against a composite photometric and depth loss; the depth term uses the Pearson correlation between rendered and estimated depth maps, which is more robust to the residual scale drift of the MMDE than an L1 term. During mapping, new keyframes densify the Gaussian representation and a sliding-window bundle adjustment jointly refines poses and Gaussian parameters to prevent catastrophic forgetting.

A frame is promoted to keyframe status through a cascade of criteria: low co-visibility with the last keyframe, sufficient spatial displacement, and, crucially, cross-agent overlap: each agent continuously receives keyframe encodings broadcast by the others, and a high-similarity match against an incoming encoding triggers denser local keyframing in that overlapping region, directly improving the submap alignment described below.

Keyframe selection cascade: co-visibility, spatial displacement, temporal spacing, then cross-agent overlap.

Once an agent finishes exploring, it transmits its local map, including keyframes, Gaussian parameters, and camera intrinsics, to the server. The server identifies co-visible keyframe pairs across submaps using NetVLAD visual descriptors, then feeds candidate pairs (plus temporal neighbors) into VGGT to obtain camera extrinsics. These are combined with the locally tracked, metrically-scaled poses to estimate an initial rigid transformation between submaps, discarding VGGT's relative scale, since our front-end is already metric. This estimate is refined first by minimizing pose error over the neighboring keyframes, then by a residual correction optimized through the differentiable rasterizer against rendered depth. The whole alignment step (NetVLAD matching, VGGT inference, refinement) takes about 5 seconds per submap pair.

We evaluate on two datasets providing synchronized RGB and IMU data: the TUM RGB-D dataset and the UT-MM dataset. Since all UT-MM sequences were captured in the same physical environment (the Anna Hiss Gymnasium, UT Austin), we treat independently-recorded sequences as a synthetic multi-agent benchmark for submap alignment.

All experiments run on a consumer-grade laptop (RTX 2080 Max-Q, i7-10750H) at a sustained 10 FPS for tracking and mapping within ~1 GB of VRAM. Depth Pro is offloaded to a centralized server (RTX 4090, ~169 ms/keyframe). The keyframe-encoding payload is only 65 KB, yielding <130 KB/s uplink per agent, viable even on degraded tactical networks. Final maps weigh between 25–75 MB.

Replacing consumer-grade LiDAR with a modern metric depth estimator does not just remove a hardware requirement: it can improve reconstruction quality. Depth Pro shows reduced linear distortion and fewer sensor artifacts than both a noisy LiDAR sensor and a relative depth estimator (MiDaS) on the same scene.

SLAM papers typically rotate, scale, and translate a predicted trajectory to best fit the ground truth before computing ATE. Here we only align the origin: our metric depth prior gives a trajectory that is already close to ground-truth scale, whereas MM3DGS, which relies on relative depth, drifts to roughly a third of the true size.

Despite using no depth sensor at all, CGS-SLAM matches RGB-D-based 3DGS SLAM methods on TUM RGB-D and clearly outperforms the only other RGB-only 3DGS baseline, MM3DGS, on average.

We outperform the monocular baseline MM3DGS on every sequence, and match RGB-D methods on most, with Magic-SLAM remaining ahead only on fr1/room.

Against a constant-velocity (CV) baseline, IMU pre-integration brings the largest gains on challenging, fast-motion sequences: on Fast-straight, ATE drops from 12.85 cm to 6.07 cm (−53%) while PSNR improves by 2.73 dB. Notably, even our CV variant (12.85 cm) beats MM3DGS with IMU (16.78 cm) on that sequence, suggesting our spatial alignment and map initialization contribute independently of the motion prior.

Treating independently-captured UT-MM sequences as different agents exploring the same environment, our server successfully aligns trajectories, even under significant heading differences, to sub-meter accuracy (translation error < 0.40 m), and scales to more than two agents. Alignment completes in about 5 seconds per submap pair: <1 s NetVLAD matching, 1 s VGGT inference, ~3 s refinement.

Collaboration is online: keyframe encodings are broadcast continuously so agents can detect overlap with one another and locally densify keyframing, deliberately avoiding a tightly-coupled multi-agent bundle adjustment, whose bandwidth requirements are incompatible with degraded-network operation. Disabling this dynamic keyframing degrades global ATE by up to 72%.