Sensor Context Encoder Challenge

Summary

The challenge asks whether a window of inertial-sensor data can be presented to a frozen language model as a single continuous embedding and read back out as an activity label. I built the required pair of models, a direct sensor classifier and a context-embedding model that shares its encoder, together with a projector design that makes the interface interpretable by construction: the context embedding is constrained to a simplex whose corners are six genuine token embeddings. The two models reach the same macro-F1 on the held-out test subjects (0.94), the shuffled-embedding control collapses to chance level, and the trainable parameter count is 1.7% of the 10M budget. A further set of ablations and probes examines what the language model contributes. The evidence indicates that it functions as a well-conditioned transport channel rather than as a reader of the signal.

Data and evaluation protocol

The dataset is UCI Human Activity Recognition Using Smartphones: 30 subjects, each recorded continuously by a waist-mounted smartphone IMU while performing a scripted sequence of the six target activities. The inputs are the nine windowed inertial signals (total acceleration, body acceleration, and angular velocity per axis) in 128-sample windows at 50 Hz with 50% overlap, giving a 128×9 array per example. The supplied 561-dimensional engineered features are not used, per the challenge rules.

Because the test set should be evaluated exactly once, the first step was to construct a validation set. All splits are subject-wise: every window of a given person falls in exactly one of train, validation, or test, so no person leaks across splits. Validation subjects 17, 25, 26, and 30 were drawn from the 21 training subjects with seed 42, giving 5,800 / 1,552 / 2,947 windows for train / validation / test. The official test split was never consulted during development; for each training seed, the checkpoint with the best validation score is evaluated exactly once on test.

Windows per activity and split

class balance is healthy in every split; y-axis: number of 2.56 s windows

train: 17 subjectsval: subjects 17, 25, 26, 30 (seed 42)test: official, untouched

Windows per subject, colored by split membership

split membership is by whole subject; no person appears in two splits; y-axis: windows

train: 17 subjectsval: subjects 17, 25, 26, 30 (seed 42)test: official, untouched

Before modeling, I verified the recordings by dead reckoning: double integration of example windows from each class, alongside the super-odometry method from my TartanIMU research, rendered in the interactive viewer below (adapted from that project's tooling). The trajectories are consistent with the labeled activities. One caveat surfaced by this inspection: the apparently smooth trajectories of the laying class reflect integration error rather than genuine motion.

tp64 live viewer: one dead-reckoned walking window (interactive)

playback, orbitable 3D trajectory trail, IMU chart panels; requires internet access for the Three.js CDN

Sensor encoder and projector design

Encoder. A 1-D CNN over the raw 128×9 window: three Conv1d stages (9→64, kernel 5, stride 2; 64→128, kernel 5, stride 2; 128→128, kernel 3), each with BatchNorm and GELU, followed by global mean- and max-pooling over time, concatenated into a 256-dimensional feature and passed through a Linear+GELU projection. The trunk holds about 160k parameters, and each unit of the third convolution sees approximately 0.42 s, roughly one step of a stride cycle. Convolution suits this signal because the label is invariant to where in the window a step lands and because the discriminative patterns are short local motifs. Mean-pooling carries the sustained posture and gravity statistics that separate the static classes; max-pooling carries the transient impacts that separate the gait classes.

Inside the shared CNN trunk

slab width ∝ time steps, height ∝ channels; ~160k parameters, identical in every condition

Anchor selection. The projector design starts from an observation about the label phrases themselves: they are poorly separated in the model's embedding space. The phrases "walking upstairs" and "walking downstairs" share their leading token, and the maximum pairwise cosine among the six mean-pooled label phrases is 0.82. The activities that are hardest to disambiguate from the sensor data, the three walking variants, are also the most similar semantically, so the label phrases make a poor basis for the interface.

Label-phrase similarity in SmolLM2 embedding space

mean-pooled token embeddings of the six label phrases, centered cosine

lighter = less similar · 1.0 = identical (centered cosine)

walking → [ walking]; walking upstairs/downstairs → [ walking][ up/down][stairs]; sitting / standing / laying → 1 token each. The stairs pair sits at 0.82.

Instead, I searched the 21,185 single word-like tokens of SmolLM2's vocabulary for one synonym per activity such that the six selections are mutually well separated: grow a synonym island around each activity, then pick one token per island to minimize the worst pairwise cosine while keeping each pick a recognizable synonym.

universe U ← the 21,185 vocab tokens matching " [a-z]{3,}"
E ← tied embedding table, centered on the vocab mean, L2-normalized

for each state s in {walking, upstairs, downstairs, sitting, standing, laying}:
    seeds(s)  ← hand-picked synonyms that survive as single tokens
    c(s)      ← normalize(mean E[seeds(s)])              # island centroid
    island(s) ← tokens t ∈ U owned by c(s) (nearest of the six centroids)
                with cos(E[t], c(s)) ≥ 0.35, capped at 25

candidates(s) ← {t ∈ island(s) : cos(E[t], c(s)) ≥ 0.42}  # stays a synonym
repeat 300 times (random restarts):
    pick one candidate per state; hill-climb single swaps to minimize
    the pair (max, mean) of pairwise cos among the six picks
return the best assignment
  → walking · climb · descending · seated · upright · lying  (max pair 0.24)

The selected anchors, walking, climb, descending, seated, upright, lying, have a maximum pairwise cosine of 0.24.

Chosen anchor-token similarity

the six single-token anchors from the synonym-island search, centered cosine

lighter = less similar · 1.0 = identical (centered cosine)

max off-diagonal similarity 0.24 (vs 0.82 for the original phrases); every anchor is one token.

Projector. A single Linear 256→6 followed by a softmax gives barycentric weights w on the 5-simplex; the context embedding is e = w·A, where A holds the six frozen rows of the LM's embedding table selected above. Because e is a convex combination of genuine token embeddings, it is guaranteed to lie on the LM's input manifold, and the encoder's entire interface to the LM is six numbers.

Language-model path. e replaces <SENSOR> in the mandated prompt's input-embedding sequence; sensor values are never rendered as text. The LM is SmolLM2-360M-Instruct in bf16, fully frozen including its embedding table; gradients flow through it into the projector. A trainable Linear 960→6 head reads the final hidden state after Activity:. The direct classifier is the identical trunk with a Linear 256→6 head, so the comparison isolates the interface rather than the feature extractor.

Architecture: controls vs. simplex

five lanes share one CNN trunk; dashed lane = evaluation-time sensor-dependence check; table: val / test macro-F1 over multiple seeds and trainable parameters

Training setup and trainable parameter count

Both models train with AdamW (learning rate 1e-3 with cosine decay, weight decay 1e-4), batch size 128, 30 epochs, cross-entropy loss, and per-channel z-scoring computed from training-split statistics.

Model	Trainable parameters
Direct classifier (trunk + head)	161,286
Context model (trunk + projector + LM head)	167,052

Both models use about 1.7% of the 10M trainable-parameter budget; the LM's 361.8M parameters are entirely frozen.

Results

Condition	Macro-F1 (test)
Direct sensor classifier	0.9402 ± 0.0154
Context-embedding model	0.9404 ± 0.0155
Context model with shuffled embeddings	0.1652 ± 0.0083

Values are mean ± standard deviation over multiple training seeds, with split seed 42 and shuffle seeds 123+s. One context-model seed collapsed during training and is excluded; the failure is fully visible in the validation curve, so ordinary model selection would reject it without consulting the test set. The shuffled condition permutes complete projected embeddings between test examples, with no retraining.

Interpretation. The frozen-LM interface costs nothing in accuracy: the context model and the direct classifier are statistically indistinguishable on unseen subjects. The shuffled control falls to chance (1/6 ≈ 0.167), so the prediction depends on the matching sensor embedding rather than on prompt priors. Errors concentrate where the sensor physics predicts, on the sitting-versus-standing pair, because a waist-mounted phone measures nearly the same gravity vector in both postures. They do not concentrate where the label tokens are most similar, so the context path is not bottlenecked by label semantics.

Test confusion matrices (seed 0)

rows = true class, columns = predicted, cell = window count; color = share of the row

direct sensor classifier

context-embedding model (simplex)

SIT↔STAND dominates both models; LAYING recall is perfect in both.

Validation macro-F1 by epoch (seed 0)

solid blue = simplex context; dashed = direct classifier; dotted = free 960-d projector (peaks early, then drifts); y-axis clipped at 0.75

Analysis of the simplex weights

A UMAP of the validation windows' simplex weights shows that they do not cluster around their semantically matching corners of the simplex. The assignment the encoder learned is a derangement: the walking windows sit near the descending corner and the walking-downstairs windows near walking. If the assignment of regions to corners is arbitrary, the meaning of the anchors may not matter at all.

UMAP of the validation windows’ simplex weights

1,552 windows · 57 recording runs · n_neighbors 30, seed 42 · trails connect consecutive 50%-overlapping windows of one subject+activity run; the six pure corners are embedded in the same map

WALKINGW-UPW-DOWNSITTINGSTANDINGLAYING◇ anchor corner

A direct test confirms this. Re-running the selection procedure for six maximally separated tokens with meaning ignored yields the, antioxid, agitated, salaries, whe, toggle, a set whose embeddings are mutually near-orthogonal: the largest pairwise cosine among the six is −0.002, meaning every pair sits at or just below zero. Retraining on these anchors gives 0.970 validation macro-F1 over three seeds, against 0.973 for the synonym anchors. Anchor semantics contribute nothing measurable; what the simplex provides is a well-conditioned frozen basis, and geometric separation alone is what conditions it. Consistent with this, learnable corners tie frozen ones, while corners initialized randomly off the token manifold train successfully in only about one seed in three and never migrate toward the token cloud.

PCA of the 1,552 validation weight vectors w (points on the 5-simplex, hence at most five degrees of freedom) gives variance shares of 41% / 31% / 22% / 7% / ~0% by principal direction: three components reach 90% of the variance, four reach 95%, and the fifth direction is essentially unused. The within-class variation is systematic rather than noise: adjacent 50%-overlapping windows sit 3.3× closer in weight space than random same-class, cross-subject pairs. The spare capacity also carries subject identity. A linear probe on the six weights alone identifies which of the four unseen validation subjects produced a window at up to twice chance.

The weight vector w is therefore a coordinate, not a classification. The largest simplex weight matches the true label only about 31% of the time, so w must not be read as class probabilities; the encoder uses the interior of the hull as a coordinate system that the LM and the head decode jointly. The panels below make this visible: a standing window whose weights sit almost entirely on the seated corner is still read correctly by the head, while the LM's own next-token distribution barely reacts to the sensor at all.

Prediction playground: two validation windows

panels from an interactive playground built during exploration to visualize individual predictions; left: the encoder’s simplex weights w over the six anchors, right: the frozen LM’s next-token probability on each anchor word, header: the trained head’s prediction

Role of the language model

The results above raise the question of what the frozen LM actually contributes. Three hypotheses were tested: that the LM is a generic fixed nonlinearity that any random frozen transformer could replace (H0, reservoir); that its pretrained attention moves the sensor embedding's information to the readout position more reliably than random circuits would, with no semantics involved (H1, routing); and that it lifts e into its concept geometry, representing sensor-walking windows like the word walking (H2, semantic mapping). The experiments reject H0 and favor H1 over H2: pretraining provides trained routing and a residual stream that carries information through 32 layers without corrupting it, not comprehension of the signal.

Hypotheses and probe experiments

H0 is rejected. Retraining with the transformer blocks randomly re-initialized and frozen, keeping the pretrained embedding table so that the interface and prompt are identical, reaches 0.617 ± 0.025 validation macro-F1 against approximately 0.97 for the pretrained model. Pretraining is worth roughly 35 points on this task, so the channel is not a generic reservoir.

The evidence favors H1 over H2. Deleting the prompt entirely matches the full prompt on converged seeds (0.978 / 0.967 validation), and replacing it with an unrelated recipe prompt matches as well (0.961 ± 0.014), so the language context is not read; its presence acts only as optimization ballast. A layer-wise linear probe finds the activity fully decodable at the readout position immediately after layer 1 and flat for the remaining 31 layers: the sensor embedding is transported by a few attention heads and then preserved rather than transformed. A logit-lens analysis shows the vocabulary space ignoring the sensor until the prompt-driven "walking" response assembles in the last few layers. Reconciling this with the rejection of H0: pretraining buys fidelity, meaning trained routing and a residual stream that carries information through 32 layers without corrupting it, which a random stack cannot do. It does not buy comprehension.

Three further ablations bound the design space. Replacing the simplex with an unconstrained 960-dimensional projector at the same output radius scores lower (0.914 test) and trains unstably, so the on-manifold constraint is doing real work. Removing the LM entirely while keeping the same six-number bottleneck collapses to 0.63 ± 0.12 test macro-F1, because the softmax saturates without the LM's gradient signal; the channel performs no semantics, but it is load-bearing. A sweep over the number of simplex corners plateaus from K≈4 through K=64, consistent with the PCA finding that about four degrees of freedom are in use, and a single anchor with a sigmoid strength gate already reaches 0.936 with one scalar passing through the LM.

Validation macro-F1 vs number of simplex corners

one seed per point (K=6 points are 3-seed means); K=1 under softmax is a constant embedding (the information-zero floor); ◇ = a single anchor with a sigmoid strength gate

Known limitations

• The anchors are not used semantically: six maximally separated, meaning-free tokens tie the synonym anchors, and the learned assignment of activities to corners is a derangement. Geometric separation of the corners is what matters, not their meaning.
• The simplex weights are a coordinate rather than a classification: the largest weight matches the true label only about 31% of the time, so w must not be interpreted as class probabilities.
• Capacity saturates early: about four of the five degrees of freedom are in use, and the corner-count sweep is flat from K=4 upward, so additional corners contribute nothing on this task.
• The transformer acts as a transport channel: prompt content is irrelevant, anchor meaning is irrelevant, and the layer-wise probe shows the activity decodable after layer 1 and unchanged thereafter. The challenge's experiment therefore cannot establish that the LM understands the sensor; it establishes that messages can be sent through the LM. The channel is nonetheless load-bearing, since the same bottleneck without the LM collapses to 0.63 ± 0.12.
• The representation is not subject-invariant: a probe on the six simplex weights identifies which unseen validation subject is moving at up to twice chance. This subject channel explains part of the validation-to-test gap and constitutes a soft biometric if embeddings ever leave the device.
• The scope is limited to one dataset and one sensor placement, and the LM path costs a 360M-parameter forward pass per window against a 161k-parameter CNN.

Recommendation and further directions

Develop the context-embedding approach further, with a mechanistic focus rather than an accuracy target. The premise of the challenge holds: a sensor can write into a frozen language model as one continuous vector and be read out at parity with a direct classifier, with a sensor-dependence check that passes cleanly and a projector whose six corners are auditable tokens. Every ablation, however, indicates that the model is used as a structured nonlinear function rather than as a reader of the signal. The challenge's underlying question, whether raw sensor values can be understood through tokens as the interface, is therefore answered here only in its connectivity half; the understanding half is where the work should go next, and interpretability techniques, steering vectors, and manifold geometry are the instruments for it. For activity recognition in isolation, the direct classifier remains the better engineering choice, delivering equal accuracy at a small fraction of the inference cost.

• Move the interface off the input tokens: the sensor embedding arrives at the readout position after one layer and is never transformed, so writing e into a mid-depth residual stream, and reading out at mid-depth rather than from the final layer, is the natural next experiment. The layer-wise probes built here, together with steering-vector methods and the manifold-geometry analysis of the simplex, are the instruments for it.
• Train the model to describe its input: whether the LM can understand the sensor becomes answerable if it is trained to say what it sees. A reinforcement-learning setup in the style of Li et al., rewarding descriptions consistent with the sensor window, would turn the channel into an interpreter and give the question a direct test.
• Treat e as a steering vector: the context embedding is a learned steering direction in the LM's input space, so the arithmetic, patching, and attribution tooling from that literature applies to it directly, and the six anchor corners give it an auditable basis.
• Establish safety guarantees: an interface that bypasses text also bypasses text-level guardrails, so before embeddings of this kind are deployed, guarantees are needed about what a continuous input can induce the model to do.