EgoSocial (Meta Project Aria research)

The problem

Egocentric wearables capture geometry, motion, and activity, but nothing about the wearer's internal state. Existing datasets treat the wearer as an isolated agent, so a model can describe what someone is doing and still have no idea whether they are comfortable, engaged, or trying to leave the conversation. There is no large-scale dataset that links first-person sensor data to psychologically validated labels during real social interaction.

What I built

EgoSocial is a proposed dataset collected on Meta Aria Gen 2 glasses across three interaction conditions: human-human, human-AI, and human-group. Synchronized RGB, binocular eye tracking, PPG, IMU, and 7-channel spatial audio are aligned with pre-study trait assessments (OCEAN, ERQ, SIAS/SPS), in-the-moment diary annotations, and post-block surveys. On top of that dataset, a JEPA-style self-supervised world model learns latent representations of egocentric experience, and EgoBehave, a personality-conditioned temporal model, predicts emotional state, intent, comfort, and social context from those representations.

What I contributed

I am a co-author on the proposal, working on the research framing, the modeling plan, and the experiment design. The status is honest: the application to the Meta Project Aria program is submitted and under review, so nothing has been collected yet. What exists today is the design, which is the part I care most about getting right, because a dataset with a bad protocol is expensive to discover after the fact.

Why the labels are the hard part

Sensor streams are easy to collect and nearly useless without ground truth about what the wearer was actually experiencing. Post-session surveys alone assume people remember what they felt and when, and self-reports about negative emotion invite social desirability bias. The design answers that with three layers: trait assessments before the session, a marker button for in-the-moment annotation, and short surveys immediately after each block rather than at the end. Each survey construct is chosen to map onto a specific sensor, such as arousal to PPG, dominance to gaze and speaking time, and cognitive load to pupil dilation.

Why Aria specifically

A VR headset blocks eye contact and changes how people behave, which destroys the thing being measured. Conventional wearable cameras drop the physiological channel entirely. Aria Gen 2 is the one platform that combines eye tracking, PPG, IMU, spatial audio, and RGB with research-grade timestamp synchronization in a form factor people can wear through a real conversation.

Separating representation learning from prediction

The JEPA model is trained self-supervised on sensor streams alone, with no labels, to learn what egocentric experience looks like. EgoBehave then runs on those embeddings, conditioned on a per-participant trait vector, to predict internal state. Keeping the two stages separate means the scarce and expensive resource, which is validated psychological labels, is spent only where it is actually needed.