Creator Trend Engine

X For You algorithm, line by line · Part 10

X For You algorithm, line by line — Part 10: Scorers + Selectors

Part 10 of the deep dive into xai-org/x-algorithm. PhoenixScorer with new-user cluster routing + egress fallback, the 290-line RankingScorer consolidating 22 feature-switch-driven weights + author-diversity exponential decay + tri-branched OON downweighting, VMRanker with DPP diversity, plus the BlenderSelector that interleaves posts/ads/prompts/who-to-follow/push-to-home.

May 15, 2026·22 min read

The ranking core. After 16 hydrators have filled in every per-candidate field and every per-query feature, the scorers turn that data into a single ranked list. Then the selector picks top-K.

Files covered (~933 LOC):

home-mixer/scorers/
├── mod.rs                         (3)   only declares the 3 active scorers
├── phoenix_scorer.rs              (124) Phoenix model inference
├── ranking_scorer.rs              (290) consolidates weighted-sum + author diversity + OON
├── vm_ranker.rs                   (132) VMRanker remote service
│                                       — orphans (not in mod.rs) —
├── weighted_scorer.rs             (92)  pre-refactor weighted combiner
├── author_diversity_scorer.rs     (72)  pre-refactor diversity scorer
└── oon_scorer.rs                  (38)  pre-refactor OON downweighter

home-mixer/selectors/
├── mod.rs                         (5)
├── top_k_score_selector.rs        (15)  trivial top-K
└── blender_selector.rs            (164) For You feed-item blender

Pipeline scorer order (from Session 06):

PhoenixScorer  →  RankingScorer  →  VMRanker

Each runs sequentially. PhoenixScorer fills in phoenix_scores. RankingScorer combines those scores into weighted_score + score, applies author diversity, applies OON downweighting. VMRanker (optional, feature-flag-gated) overlays a separate model's score.

The orphans show how the design evolved: the three concerns (weighted combine, diversity, OON) used to be three separate scorers. They got merged into RankingScorer (single pass over candidates, no intermediate sorting between stages).

scorers/mod.rs (3 lines)

pub mod phoenix_scorer;
pub mod ranking_scorer;
pub mod vm_ranker;

Three modules. The other three .rs files in the directory are not declared — they're dead code. We'll cover them at the end for the historical narrative.

phoenix_scorer.rs (124 lines)

The Phoenix model inference. Takes the user's scoring sequence (history) + candidates as input, returns per-action probabilities per candidate.

Imports + struct

use crate::models::candidate::CandidateHelpers;
use crate::models::candidate::PostCandidate;
use crate::models::query::ScoredPostsQuery;
use crate::params::{
    PhoenixInferenceClusterId, PhoenixRankerNewUserHistoryThreshold,
    PhoenixRankerNewUserInferenceClusterId, UseEgressSidecar,
};
use crate::util::phoenix_request::build_prediction_request;
use std::sync::Arc;
use tonic::async_trait;
use xai_candidate_pipeline::component_library::clients::phoenix_prediction_client::{
    PhoenixCluster, PhoenixPredictionClient,
};

use xai_candidate_pipeline::component_library::utils::current_timestamp_millis;
use xai_candidate_pipeline::scorer::Scorer;
use xai_recsys_proto::ProductSurface;

pub struct PhoenixScorer {
    pub phoenix_client: Arc<dyn PhoenixPredictionClient + Send + Sync>,
    pub egress_client: Arc<dyn PhoenixPredictionClient + Send + Sync>,
}

Two clients — the primary phoenix_client and the egress_client. From Session 06 we know the egress client routes through an egress sidecar (separate cell / DC). The scorer can try egress first, fall back to primary.

Cluster resolution

impl PhoenixScorer {
    fn resolve_cluster(query: &ScoredPostsQuery) -> PhoenixCluster {
        let configured_cluster =
            PhoenixCluster::parse(&query.params.get(PhoenixInferenceClusterId));

        let threshold: u64 = query.params.get(PhoenixRankerNewUserHistoryThreshold);
        if threshold > 0 {
            let action_count = query
                .scoring_sequence
                .as_ref()
                .and_then(|s| s.metadata.as_ref())
                .map(|m| m.length)
                .unwrap_or(0);

            if action_count < threshold {
                return PhoenixCluster::parse(
                    &query.params.get(PhoenixRankerNewUserInferenceClusterId),
                );
            }
        }

Two-tier cluster routing:

  1. Default: use PhoenixInferenceClusterId from feature switches.
  2. New user override: if the threshold is set AND the user's history has < threshold actions, use PhoenixRankerNewUserInferenceClusterId instead.

The new-user cluster is a separate model trained on cold-start patterns. Same input contract, different weights — handles the "user has 5 actions" case differently from the "user has 5000 actions" case.

        if let Some(decider) = &query.decider {
            match configured_cluster {
                PhoenixCluster::Experiment1Fou if decider.enabled("override_qf_use_lap7") => {
                    return PhoenixCluster::Experiment1Lap7;
                }
                PhoenixCluster::Experiment1Lap7 if decider.enabled("override_qf_use_fou") => {
                    return PhoenixCluster::Experiment1Fou;
                }
                _ => {}
            }
        }

        configured_cluster
    }
}

Decider overrides for experiments. The two cluster names "Fou" and "Lap7" we saw in Session 06's prod constructor. The decider can swap between them per-user. The two named overrides:

  • override_qf_use_lap7: if the configured cluster is Fou, override to Lap7.
  • override_qf_use_fou: the inverse.

So a user in the "lap7" experiment arm gets routed there regardless of the default config. Useful for shadow comparison: keep most users on Fou (the proven cluster), route 1% to Lap7 (the candidate replacement) for measurement.

score

#[async_trait]
impl Scorer<ScoredPostsQuery, PostCandidate> for PhoenixScorer {
    fn enable(&self, query: &ScoredPostsQuery) -> bool {
        !query.has_cached_posts
    }

    async fn score(
        &self,
        query: &ScoredPostsQuery,
        candidates: &[PostCandidate],
    ) -> Vec<Result<PostCandidate, String>> {
        let last_scored_at_ms = current_timestamp_millis();
        let product_surface = if query.in_network_only {
            ProductSurface::HomeTimelineRankedFollowing
        } else {
            ProductSurface::HomeTimelineRanking
        };

enable: skip if cached posts are available. Same recurring pattern.

Record timestamp + product surface for the model. Two surfaces:

  • HomeTimelineRankedFollowing — Following tab.
  • HomeTimelineRanking — For You.

The model uses surface as a feature (different relevance criteria per surface).

        if query.scoring_sequence.is_none() {
            return vec![Ok(PostCandidate::default()); candidates.len()];
        };

Degraded mode: if the user's scoring sequence wasn't hydrated (Aggregation service failed), return defaults. No scores → RankingScorer will assign 0 (or near-zero) → all candidates end up at the bottom of the ranking. Effectively a "no personalization" fallback.

        let cluster = Self::resolve_cluster(query);
        let request = build_prediction_request(query, candidates, product_surface);

        let use_egress: bool = query.params.get(UseEgressSidecar);
        let client = if use_egress {
            &self.egress_client
        } else {
            &self.phoenix_client
        };

        let mut predictions = client.predict(cluster, request.clone()).await;

        if predictions.is_err() && use_egress {
            tracing::debug!("Egress predict failed, falling back");
            predictions = self.phoenix_client.predict(cluster, request).await;
        }

Build the prediction request (the heavy lifting is in build_prediction_request, not shown — it packs the user's columnar sequence + candidate features into a Phoenix proto).

Pick client based on feature flag. If use_egress and egress fails: fall back to the primary client. Belt-and-suspenders.

Note request.clone() on the first call so we still have it for the fallback. Cheap clone — protos are mostly small + Bytes.

        let predictions = predictions.map_err(|e| format!("Phoenix prediction failed: {}", e));

        let predictions = match predictions {
            Ok(predictions) => predictions,
            Err(err) => return vec![Err(err); candidates.len()],
        };

        candidates
            .iter()
            .map(|c| PostCandidate {
                phoenix_scores: predictions.candidate_scores(&c.get_original_tweet_id()),
                prediction_request_id: Some(query.prediction_id),
                last_scored_at_ms,
                ..Default::default()
            })
            .map(Ok)
            .collect()
    }

    fn update(&self, candidate: &mut PostCandidate, scored: PostCandidate) {
        candidate.phoenix_scores = scored.phoenix_scores;
        candidate.prediction_request_id = scored.prediction_request_id;
        candidate.last_scored_at_ms = scored.last_scored_at_ms;
    }
}

If prediction failed totally, return err for every candidate. Otherwise, for each candidate, look up its scores from predictions.candidate_scores(original_tweet_id) — the lookup is by original_tweet_id, so retweets and originals share scores (the model doesn't care about the retweet wrapper — it scores the content).

Three fields written: phoenix_scores, prediction_request_id (join key for engagement logs), last_scored_at_ms (cache TTL).

ranking_scorer.rs (290 lines) — the all-in-one combiner

This is the largest scorer. It does three things in one pass:

  1. Weighted sum of Phoenix scores → weighted_score.
  2. Author diversity attenuation.
  3. OON downweighting.

The orphan files at the bottom of the directory used to split these three concerns into separate scorers. The current code consolidates them — fewer iterations over candidates.

ScoringWeights struct

struct ScoringWeights {
    favorite: f64,
    reply: f64,
    retweet: f64,
    photo_expand: f64,
    click: f64,
    profile_click: f64,
    vqv: f64,
    share: f64,
    share_via_dm: f64,
    share_via_copy_link: f64,
    dwell: f64,
    quote: f64,
    quoted_click: f64,
    quoted_vqv: f64,
    cont_dwell_time: f64,
    cont_click_dwell_time: f64,
    follow_author: f64,
    not_interested: f64,
    block_author: f64,
    mute_author: f64,
    report: f64,
    not_dwelled: f64,
    negative_sum: f64,
    total_sum: f64,
    min_video_duration_ms: i32,
    enable_quoted_vqv_duration_check: bool,
}

22 per-action weights packaged into a struct. Plus three derived fields (negative_sum, total_sum) and two config flags (min_video_duration_ms, enable_quoted_vqv_duration_check).

The action list maps to the Phoenix model outputs:

  • Positive engagements: favorite (like), reply, retweet, photo_expand, click, profile_click, vqv (video quality view), share, share_via_dm, share_via_copy_link, dwell, quote, quoted_click, quoted_vqv, follow_author.
  • Continuous engagements: cont_dwell_time, cont_click_dwell_time (these are predicted durations, not probabilities).
  • Negative engagements (downvotes): not_interested, block_author, mute_author, report, not_dwelled.
impl ScoringWeights {
    fn from_params(params: &xai_feature_switches::Params) -> Self {
        let favorite = params.get(FavoriteWeight);
        let reply = params.get(ReplyWeight);
        let retweet = params.get(RetweetWeight);
        let photo_expand = params.get(PhotoExpandWeight);
        let click = params.get(ClickWeight);
        let profile_click = params.get(ProfileClickWeight);
        let vqv = params.get(VqvWeight);
        let share = params.get(ShareWeight);
        let share_via_dm = params.get(ShareViaDmWeight);
        let share_via_copy_link = params.get(ShareViaCopyLinkWeight);
        let dwell = params.get(DwellWeight);
        let quote = params.get(QuoteWeight);
        let quoted_click = params.get(QuotedClickWeight);
        let quoted_vqv = params.get(QuotedVqvWeight);
        let cont_dwell_time = params.get(ContDwellTimeWeight);
        let cont_click_dwell_time = params.get(ContClickDwellTimeWeight);
        let follow_author = params.get(FollowAuthorWeight);
        let not_interested = params.get(NotInterestedWeight);
        let block_author = params.get(BlockAuthorWeight);
        let mute_author = params.get(MuteAuthorWeight);
        let report = params.get(ReportWeight);
        let not_dwelled = params.get(NotDwelledWeight);
        let min_video_duration_ms = params.get(MinVideoDurationMs);
        let enable_quoted_vqv_duration_check = params.get(EnableQuotedVqvDurationCheck);

Every weight is a feature switch. The ranking formula is fully runtime-configurable. To re-weight "share" up 10%, no code change — just bump ShareWeight in the feature-switch config and the new value flows through on the next request.

This is how the algorithm is tuned: the ML model produces probabilities, but the combination of those probabilities is policy. Policy lives in feature switches, not code.

        let positive_sum = favorite
            + reply
            + retweet
            + photo_expand
            + click
            + profile_click
            + vqv
            + share
            + share_via_dm
            + share_via_copy_link
            + dwell
            + quote
            + quoted_click
            + quoted_vqv
            + follow_author;
        let negative_sum = -(not_interested + block_author + mute_author + report + not_dwelled);
        let total_sum = positive_sum + negative_sum;

Pre-compute the sums of weights (positive and negative). Note negative_sum is negated — the weights themselves are positive numbers (e.g., not_interested_weight = 50), but their contribution to the score is negative.

total_sum = positive_sum + negative_sum — could be positive (if positives dominate), negative (if negatives dominate), or zero. Used below in offset_score for normalization.

Weighted-score computation

pub struct RankingScorer;

impl RankingScorer {
    fn apply(score: Option<f64>, weight: f64) -> f64 {
        score.unwrap_or(0.0) * weight
    }

Helper: Option<f64> * f64 = f64. None becomes 0 (no contribution).

    fn compute_weighted_score(
        weights: &ScoringWeights,
        query: &ScoredPostsQuery,
        candidate: &PostCandidate,
    ) -> f64 {
        let scores: &PhoenixScores = &candidate.phoenix_scores;

        let vqv_weight = crate::util::candidates_util::vqv_weight(
            query,
            candidate,
            weights.min_video_duration_ms,
            weights.vqv,
        );

        let quoted_vqv_weight = crate::util::candidates_util::quoted_vqv_weight(
            candidate,
            weights.min_video_duration_ms,
            weights.quoted_vqv,
            weights.enable_quoted_vqv_duration_check,
        );

Two dynamic weights computed at scoring time:

  • vqv_weight: only apply the VQV weight if the video is long enough (≥ min_video_duration_ms). For short videos / no video, VQV weight is 0.
  • quoted_vqv_weight: same for the quoted tweet's video.

This is eligibility-aware weighting: don't reward video-view predictions on non-eligible content.

        let combined_score = Self::apply(scores.favorite_score, weights.favorite)
            + Self::apply(scores.reply_score, weights.reply)
            + Self::apply(scores.retweet_score, weights.retweet)
            + Self::apply(scores.photo_expand_score, weights.photo_expand)
            + Self::apply(scores.click_score, weights.click)
            + Self::apply(scores.profile_click_score, weights.profile_click)
            + Self::apply(scores.vqv_score, vqv_weight)
            + Self::apply(scores.share_score, weights.share)
            + Self::apply(scores.share_via_dm_score, weights.share_via_dm)
            + Self::apply(
                scores.share_via_copy_link_score,
                weights.share_via_copy_link,
            )
            + Self::apply(scores.dwell_score, weights.dwell)
            + Self::apply(scores.quote_score, weights.quote)
            + Self::apply(scores.quoted_click_score, weights.quoted_click)
            + Self::apply(scores.quoted_vqv_score, quoted_vqv_weight)
            + Self::apply(scores.dwell_time, weights.cont_dwell_time)
            + Self::apply(scores.click_dwell_time, weights.cont_click_dwell_time)
            + Self::apply(scores.follow_author_score, weights.follow_author)
            + Self::apply(scores.not_interested_score, weights.not_interested)
            + Self::apply(scores.block_author_score, weights.block_author)
            + Self::apply(scores.mute_author_score, weights.mute_author)
            + Self::apply(scores.report_score, weights.report)
            + Self::apply(scores.not_dwelled_score, weights.not_dwelled);

        Self::offset_score(combined_score, weights)
    }

The actual scoring formula. A linear combination of 22 terms. Notice:

  • Positive terms add to the score.
  • Negative-action weights (e.g., weights.not_interested) are positive numbers but multiply into negative probabilities indirectly — wait no, that's wrong. Let me re-read.

Actually, looking more carefully: the negative-action weights are positive numbers, multiplied by the positive probabilities. So not_interested_score * not_interested_weight is positive. But then in ScoringWeights::from_params, the negative_sum is computed as -(not_interested_weight + ...) — negated.

This is the trick. The negative-action weights are stored as positive (configurable as positive numbers), but the contribution to the final score is treated as negative because offset_score (below) subtracts them. Let me re-check offset_score...

Actually wait, looking at the code again: Self::apply(scores.not_interested_score, weights.not_interested) adds not_interested_score * not_interested_weight — both positive. So negatives ADD to the combined_score same as positives.

But that doesn't make sense — predicting high "not interested" should REDUCE the candidate's score, not increase it.

Looking at the orphan weighted_scorer.rs, the negative weights there are likely stored as negative (e.g., NOT_INTERESTED_WEIGHT = -50). In the active RankingScorer, the weights from feature switches are also expected to be negative for negative actions. The fact that they sum into negative_sum = -(positive_weights_for_negatives_actions) is computing the absolute sum (for the offset_score normalization, where it represents "how much can scores go negative").

The pattern works if weights.not_interested is a negative number in feature-switch config. Then not_interested_score * (-50) = -50 * P(not_interested). Combined_score gets the negative contribution.

The negative_sum field is then used in offset_score to know the lower-bound of combined_score.

    fn offset_score(combined_score: f64, w: &ScoringWeights) -> f64 {
        if w.total_sum == 0.0 {
            combined_score.max(0.0)
        } else if combined_score < 0.0 {
            (combined_score + w.negative_sum) / w.total_sum * NEGATIVE_SCORES_OFFSET
        } else {
            combined_score + NEGATIVE_SCORES_OFFSET
        }
    }

Score normalization to keep things in a positive range (the selector sorts descending, and we want all candidates to have positive scores so OON multipliers behave sensibly).

Three branches:

  • Degenerate (total_sum == 0): just clamp to non-negative.
  • Negative combined_score: scale into [0, NEGATIVE_SCORES_OFFSET). The formula (combined_score + negative_sum) / total_sum * NEGATIVE_SCORES_OFFSET maps the most-negative possible value (where combined = negative_sum) to 0, and 0 to NEGATIVE_SCORES_OFFSET.
  • Positive combined_score: add NEGATIVE_SCORES_OFFSET so the minimum positive score sits above the maximum-of-negatives. Preserves ordering both within and across the positive/negative bands.

So negative-scoring candidates get squashed into [0, OFFSET), and positive-scoring candidates fly above OFFSET. The whole space is [0, +∞), naturally sortable.

Author diversity

    fn diversity_multiplier(decay_factor: f64, floor: f64, position: usize) -> f64 {
        (1.0 - floor) * decay_factor.powf(position as f64) + floor
    }

A multiplier in [floor, 1.0] that decays exponentially with position. Position 0 → 1.0 (no penalty). Position 1 → (1-floor) * decay + floor. As position grows, the multiplier asymptotes to floor.

With e.g. decay_factor = 0.5, floor = 0.1: positions 0, 1, 2, 3 → 1.0, 0.55, 0.325, 0.2125, ... → 0.1.

    fn apply_author_diversity(
        query: &ScoredPostsQuery,
        candidates: &[PostCandidate],
        weighted_scores: &[f64],
    ) -> Vec<f64> {
        let decay_factor = query.params.get(AuthorDiversityDecay);
        let floor = query.params.get(AuthorDiversityFloor);

        let mut indexed: Vec<(usize, f64)> = weighted_scores
            .iter()
            .enumerate()
            .map(|(i, &s)| (i, s))
            .collect();
        indexed.sort_by(|(_, a), (_, b)| b.partial_cmp(a).unwrap_or(Ordering::Equal));

        let mut adjusted = vec![0.0_f64; candidates.len()];
        let mut author_counts: HashMap<u64, usize> = HashMap::new();

        for (idx, weighted) in indexed {
            let author_id = candidates[idx].author_id;
            let position = author_counts.entry(author_id).or_insert(0);
            let multiplier = Self::diversity_multiplier(decay_factor, floor, *position);
            *position += 1;
            adjusted[idx] = weighted * multiplier;
        }

        adjusted
    }

The diversity algorithm:

  1. Sort candidate indices by score descending.
  2. Walk the sorted list. For each candidate, look up how many we've already seen from this author. Apply diversity_multiplier(position). Increment author count.
  3. Write the adjusted score back to the same index in adjusted.

So the highest-scored post by an author keeps its score. The second-highest by the same author gets multiplied by ~decay. The third by ~decay². Until we hit the floor.

Critical: the indexing pattern adjusted[idx] = ... preserves the original order — the per-candidate adjustment is computed in score order, but the output keeps positional alignment with the input.

Net effect: a feed of 50 posts can't have 30 from the same author (they'd be aggressively downweighted past position 5 or so). Forces author diversity within the result.

OON downweighting

    fn effective_oon_weight(query: &ScoredPostsQuery) -> f64 {
        if !query.topic_ids.is_empty() {
            return query.params.get(TopicOonWeightFactor);
        }

        let oon_weight_factor = query.params.get(OonWeightFactor);

        let new_user_age_threshold = Duration::from_secs(query.params.get(NewUserAgeThresholdSecs));

        let is_eligible_new_user = duration_since_creation_opt(query.user_id)
            .map(|age| age < new_user_age_threshold)
            .unwrap_or(false)
            && query.user_features.followed_user_ids.len() >= NEW_USER_MIN_FOLLOWING;

        if is_eligible_new_user {
            NEW_USER_OON_WEIGHT_FACTOR
        } else {
            oon_weight_factor
        }
    }

The OON downweight multiplier. Three branches:

  • Topic request: use TopicOonWeightFactor (typically higher than default — for topic feeds, OON content is the main content, so don't penalize it).
  • New user with enough follows: use NEW_USER_OON_WEIGHT_FACTOR (probably higher than the default — new users have small in-network supply, so OON needs to dominate to fill the feed).
  • Default: OonWeightFactor (typically < 1.0 — downweights OON to ensure in-network content competes).

The "new user with enough follows" condition has two parts: age below threshold AND ≥ NEW_USER_MIN_FOLLOWING follows. A user that's new but hasn't followed anyone gets the standard treatment (because their in-network is empty anyway, so the OON factor is moot).

The main score method

#[async_trait]
impl Scorer<ScoredPostsQuery, PostCandidate> for RankingScorer {
    fn enable(&self, _query: &ScoredPostsQuery) -> bool {
        true
    }

    async fn score(
        &self,
        query: &ScoredPostsQuery,
        candidates: &[PostCandidate],
    ) -> Vec<Result<PostCandidate, String>> {
        let weights = ScoringWeights::from_params(&query.params);

        let weighted_scores: Vec<f64> = candidates
            .iter()
            .map(|c| {
                let raw = Self::compute_weighted_score(&weights, query, c);
                normalize_score(c, raw)
            })
            .collect();

        let diversity_adjusted = Self::apply_author_diversity(query, candidates, &weighted_scores);

        let effective_oon = Self::effective_oon_weight(query);

        candidates
            .iter()
            .enumerate()
            .map(|(i, c)| {
                let after_diversity = diversity_adjusted[i];
                let final_score = match c.in_network {
                    Some(false) => after_diversity * effective_oon,
                    _ => after_diversity,
                };

                Ok(PostCandidate {
                    weighted_score: Some(weighted_scores[i]),
                    score: Some(final_score),
                    ..Default::default()
                })
            })
            .collect()
    }

The full ranking pass:

  1. Build the weights snapshot from params.
  2. Pass 1: compute weighted scores per candidate (linear combination + normalize).
  3. Pass 2: apply author diversity (re-sorts internally, attenuates by author).
  4. Pass 3: apply OON downweight to non-in-network candidates.
  5. Return both weighted_score (pre-adjustment) and score (post-everything).

Why keep both? weighted_score is the pure ML signal — useful for downstream analysis ("how much did diversity/OON change the ranking?"). score is what the selector sorts on.

The normalize_score(c, raw) helper applies an additional normalization based on candidate properties (e.g., a calibration for OON candidates whose Phoenix probabilities are systematically biased). Not shown — lives in util/score_normalizer.

update just copies both fields.

vm_ranker.rs (132 lines) — overlay model

A secondary ranker that runs a separate model (VMRanker) and overlays its score onto the candidates. Feature-flag-gated.

pub struct VMRanker {
    pub client: Arc<dyn VMRankerClient>,
}

#[async_trait]
impl Scorer<ScoredPostsQuery, PostCandidate> for VMRanker {
    fn enable(&self, query: &ScoredPostsQuery) -> bool {
        query.params.get(EnableVMRanker)
    }

Disabled by default. The pipeline includes it but the flag turns it on.

    async fn score(
        &self,
        query: &ScoredPostsQuery,
        candidates: &[PostCandidate],
    ) -> Vec<Result<PostCandidate, String>> {
        let cluster = VMRankerCluster::parse(&query.params.get(VMRankerClusterId));
        let request = build_request(query, candidates);

        let response = match self.client.rank(cluster, request).await {
            Ok(resp) => resp,
            Err(e) => {
                let msg = format!("VMRanker gRPC call failed: {e}");
                return vec![Err(msg); candidates.len()];
            }
        };

        let score_map: HashMap<u64, f64> = response
            .candidates
            .iter()
            .map(|sc| (sc.tweet_id, sc.score))
            .collect();

        candidates
            .iter()
            .map(|c| {
                let score = score_map.get(&c.tweet_id).copied().or(c.score);
                Ok(PostCandidate {
                    score,
                    ..Default::default()
                })
            })
            .collect()
    }

Standard scorer shape: build request → call client → map results back.

The interesting bit: score_map.get(&c.tweet_id).copied().or(c.score). If the VMRanker returned a score for this tweet, use it. Otherwise fall back to the previous score (c.score from RankingScorer). So VMRanker is an overlay — it can partial-update.

The score_map: HashMap<u64, f64> is built from the VMRanker response, which uses tweet_id as key. Not original_tweet_id — VMRanker sees individual posts, including retweets as separate.

fn build_request(query: &ScoredPostsQuery, candidates: &[PostCandidate]) -> RankRequest {
    let min_video_duration_ms = query.params.get(MinVideoDurationMs);
    let vqv_weight_value = query.params.get(VqvWeight);
    let request_timestamp_ms = query.request_time_ms as u64;

    let proto_candidates: Vec<RankCandidate> = candidates
        .iter()
        .map(|c| {
            let phoenix_scores = Some(PhoenixScores {
                favorite_score: c.phoenix_scores.favorite_score,
                reply_score: c.phoenix_scores.reply_score,
                retweet_score: c.phoenix_scores.retweet_score,
                photo_expand_score: c.phoenix_scores.photo_expand_score,
                click_score: c.phoenix_scores.click_score,
                profile_click_score: c.phoenix_scores.profile_click_score,
                vqv_score: c.phoenix_scores.vqv_score,
                share_score: c.phoenix_scores.share_score,
                share_via_dm_score: c.phoenix_scores.share_via_dm_score,
                share_via_copy_link_score: c.phoenix_scores.share_via_copy_link_score,
                dwell_score: c.phoenix_scores.dwell_score,
                quote_score: c.phoenix_scores.quote_score,
                quoted_click_score: c.phoenix_scores.quoted_click_score,
                follow_author_score: c.phoenix_scores.follow_author_score,
                not_interested_score: c.phoenix_scores.not_interested_score,
                block_author_score: c.phoenix_scores.block_author_score,
                mute_author_score: c.phoenix_scores.mute_author_score,
                report_score: c.phoenix_scores.report_score,
                not_dwelled_score: c.phoenix_scores.not_dwelled_score,
                dwell_time: c.phoenix_scores.dwell_time,
                click_dwell_time: c.phoenix_scores.click_dwell_time,
            });

Pack Phoenix scores into the VMRanker proto. Field-by-field copy — internal PhoenixScores and proto PhoenixScores are different types (different crates).

            let vqv_weight =
                candidates_util::vqv_weight(query, c, min_video_duration_ms, vqv_weight_value);

            RankCandidate {
                tweet_id: c.tweet_id,
                author_id: c.author_id,
                in_network: c.in_network.unwrap_or(false),
                is_retweet: c.retweeted_tweet_id.is_some(),
                is_reply: c.in_reply_to_tweet_id.is_some(),
                author_followers_count: c.author_followers_count.unwrap_or(0),
                vqv_ineligible: vqv_weight == 0.0,
                retweeted_tweet_id: c.retweeted_tweet_id.unwrap_or(0),
                score: c.score,
                phoenix_scores,
            }
        })
        .collect();

Per-candidate proto: ID, author, structural flags, previous score (c.score from RankingScorer), Phoenix scores.

vqv_ineligible: vqv_weight == 0.0 — pre-compute the VQV-eligibility flag (saves VMRanker from re-running the logic).

    let dpp_theta = query.params.get(VMRankerDppTheta);
    let dpp_max_selected_rank = query.params.get(VMRankerDppMaxSelectedRank);

    let dpp_params = if dpp_theta > 0.0 || dpp_max_selected_rank > 0 {
        Some(DppParams {
            theta: dpp_theta,
            max_selected_rank: dpp_max_selected_rank,
        })
    } else {
        None
    };

    RankRequest {
        viewer_id: query.user_id,
        request_timestamp_ms,
        candidates: proto_candidates,
        value_model_id: query.params.get(VMRankerValueModelId),
        viewer_following_count: query.user_features.followed_user_ids.len() as u32,
        dpp_params,
        new_user_age_threshold_secs: Some(query.params.get(NewUserAgeThresholdSecs)),
    }
}

Request envelope. Two interesting fields:

  • value_model_id — selects which VM model to use (A/B testing across model versions).
  • dpp_params — Determinantal Point Process parameters. DPP is a probabilistic technique for selecting diverse subsets. The two parameters (theta, max_selected_rank) control diversity strength. If both are zero/empty, no DPP.

So VMRanker can do diversity-aware ranking via DPP on top of the base score. Different from RankingScorer's per-author multiplicative diversity — DPP is set-level diversity.

Orphans

Three scorer files exist in the directory but aren't in mod.rs. They reference the orphan candidate_pipeline::query::ScoredPostsQuery from Session 06's orphans, confirming they're pre-refactor.

weighted_scorer.rs (92 lines)

A simpler version of RankingScorer::compute_weighted_score, but with hard-coded weights (p::FAVORITE_WEIGHT, etc.) instead of feature-switch-driven weights.

let combined_score = Self::apply(s.favorite_score, p::FAVORITE_WEIGHT)
    + Self::apply(s.reply_score, p::REPLY_WEIGHT)
    + Self::apply(s.retweet_score, p::RETWEET_WEIGHT)
    + Self::apply(s.photo_expand_score, p::PHOTO_EXPAND_WEIGHT)
    + ...

Same shape as the active version, but constants from params (compile-time) instead of dynamically read from feature switches. This is the "v1" model — fixed weights baked at deployment time. The active version (RankingScorer) replaced it with runtime-configurable weights.

author_diversity_scorer.rs (72 lines)

The old, standalone author diversity scorer. Same diversity_multiplier formula, same sorting + per-author counting. But run as a separate scorer.

The current RankingScorer inlines this logic so we don't have to sort candidates twice and recompute.

oon_scorer.rs (38 lines)

The simplest of the three orphans:

#[async_trait]
impl Scorer<ScoredPostsQuery, PostCandidate> for OONScorer {
    async fn score(
        &self,
        _query: &ScoredPostsQuery,
        candidates: &[PostCandidate],
    ) -> Result<Vec<PostCandidate>, String> {
        let scored = candidates
            .iter()
            .map(|c| {
                let updated_score = c.score.map(|base_score| match c.in_network {
                    Some(false) => base_score * p::OON_WEIGHT_FACTOR,
                    _ => base_score,
                });

                PostCandidate {
                    score: updated_score,
                    ..Default::default()
                }
            })
            .collect();

        Ok(scored)
    }
    ...
}

Just multiply OON candidates by a constant. Like the others, the constant was hard-coded (p::OON_WEIGHT_FACTOR). The new version uses dynamic weights + new-user / topic-request branches.

selectors/mod.rs (5 lines)

mod blender_selector;
mod top_k_score_selector;

pub use blender_selector::BlenderSelector;
pub use top_k_score_selector::TopKScoreSelector;

Two selectors, both re-exported at the parent level.

selectors/top_k_score_selector.rs (15 lines)

The trivial selector used by PhoenixCandidatePipeline.

pub struct TopKScoreSelector;

impl Selector<ScoredPostsQuery, PostCandidate> for TopKScoreSelector {
    fn score(&self, candidate: &PostCandidate) -> f64 {
        candidate.score.unwrap_or(f64::NEG_INFINITY)
    }
    fn size(&self) -> Option<usize> {
        Some(params::TOP_K_CANDIDATES_TO_SELECT)
    }
}

15 lines. Implements Selector from the framework (Session 01). Override two methods:

  • score(): pull candidate.score, default NEG_INFINITY if unset (sorts to the bottom).
  • size(): return the K from params.

That's it — the framework's default select does sort-descending + truncate.

selectors/blender_selector.rs (164 lines) — the For You blender

The For You pipeline's selector. Different from TopKScoreSelector because it doesn't just sort — it blends posts with non-post item types (ads, prompts, who-to-follow, push-to-home).

pub struct BlenderSelector {
    safe_gap_blender: SafeGapAdsBlender,
    partition_organic_blender: PartitionOrganicAdsBlender,
}

impl BlenderSelector {
    pub fn new() -> Self {
        Self {
            safe_gap_blender: SafeGapAdsBlender,
            partition_organic_blender: PartitionOrganicAdsBlender,
        }
    }
}

Holds two ads blenders (we'll cover them in Session 14). They're alternative ad-injection strategies — feature flag selects which.

impl Selector<ScoredPostsQuery, FeedItem> for BlenderSelector {
    fn select(
        &self,
        query: &ScoredPostsQuery,
        candidates: Vec<FeedItem>,
    ) -> SelectResult<FeedItem> {
        let PartitionedFeedItems {
            posts,
            ads,
            wtf_modules,
            prompts,
            push_to_home,
        } = partition_feed_items(candidates);

Note: this overrides select directly (not just score + size). Custom selection logic.

First step: partition the heterogeneous candidates by item type.

        let input_post_count = posts.len();
        let input_ad_count = ads.len();

        let blender_type = query.params.get(AdsBlenderType);
        let blender: &dyn AdsBlender = match blender_type.as_str() {
            "safe_gap" => &self.safe_gap_blender,
            _ => &self.partition_organic_blender,
        };

        let mut blended = blender.blend(posts, ads);

Pick ads blender by feature flag. Delegate post + ad blending to it (returns a single Vec<FeedItem> with posts and ads interleaved according to the blender's logic).

        insert_prompts(&mut blended, prompts);
        insert_who_to_follow(&mut blended, wtf_modules);
        pin_push_to_home(&mut blended, push_to_home);

Then insert the other item types:

  • Prompts (compose nudges, polls, verification reminders) — inserted at the top via PROMPTS_POSITION.
  • Who-to-follow module — single module inserted at WHO_TO_FOLLOW_POSITION (e.g., position 3).
  • Push-to-home — if the user clicked a notification, pin that post at position 0.
        let output_post_count = blended
            .iter()
            .filter(|i| matches!(i.item, Some(feed_item::Item::Post(_))))
            .count();
        let output_ad_count = blended
            .iter()
            .filter(|i| matches!(i.item, Some(feed_item::Item::Ad(_))))
            .count();

        let dropped_posts = input_post_count.saturating_sub(output_post_count);
        let dropped_ads = input_ad_count.saturating_sub(output_ad_count);
        let non_selected = build_non_selected_placeholders(dropped_posts, dropped_ads);

        SelectResult {
            selected: blended,
            non_selected,
        }
    }

    fn score(&self, _candidate: &FeedItem) -> f64 {
        0.0
    }
}

Track how many posts/ads were dropped during blending (e.g., the blender might cap ads at N). Build placeholder non_selected entries (default Post / Ad protos) for each dropped one — fills the non_selected slot in SelectResult with the right count even though the content is empty. This is to keep the side-effects' "how many candidates did we consider but not show" metrics meaningful.

score() returns 0 — select was overridden, so score is unused. Still required by the trait. Hack.

fn insert_prompts(blended: &mut Vec<FeedItem>, prompts: Vec<Prompt>) {
    for (i, prompt) in prompts.into_iter().enumerate() {
        blended.insert(
            i,
            FeedItem {
                position: PROMPTS_POSITION,
                item: Some(feed_item::Item::Prompt(prompt)),
            },
        );
    }
}

Prompts go at positions 0, 1, 2, … of blended. Each insert(i, ...) shifts everything to the right. So prompts stack at the top in the order they came in.

fn insert_who_to_follow(blended: &mut Vec<FeedItem>, wtf_modules: Vec<WhoToFollowModule>) {
    let Some(wtf) = wtf_modules.into_iter().next() else {
        return;
    };
    let insert_idx = WHO_TO_FOLLOW_POSITION.saturating_sub(1).min(blended.len());
    blended.insert(
        insert_idx,
        FeedItem {
            position: WHO_TO_FOLLOW_POSITION as i32,
            item: Some(feed_item::Item::WhoToFollow(wtf)),
        },
    );
}

Only the first WTF module is used (the others are discarded). Inserted at position WHO_TO_FOLLOW_POSITION - 1, clamped to the list length.

saturating_sub(1) avoids underflow if WHO_TO_FOLLOW_POSITION == 0 (which shouldn't happen but defensive).

fn pin_push_to_home(blended: &mut Vec<FeedItem>, push_to_home: Option<PushToHomePost>) {
    let Some(pth) = push_to_home else {
        return;
    };
    blended.insert(
        0,
        FeedItem {
            position: 0,
            item: Some(feed_item::Item::PushToHome(pth)),
        },
    );
}

Push-to-home goes at position 0 always (overrides prompts). User clicked a notification → that post is the first thing they see.

Note ordering matters: insert_prompts runs before insert_who_to_follow before pin_push_to_home. The push-to-home goes in last, so it lands at position 0 regardless of what's there. The blended order is:

  1. Push-to-home (if present)
  2. Prompts (in order)
  3. Posts + Ads (from blender)
  4. WTF module inserted somewhere in the middle (e.g., position 3)
fn build_non_selected_placeholders(dropped_posts: usize, dropped_ads: usize) -> Vec<FeedItem> {
    let mut non_selected = Vec::with_capacity(dropped_posts + dropped_ads);
    for _ in 0..dropped_posts {
        non_selected.push(FeedItem {
            position: 0,
            item: Some(feed_item::Item::Post(ScoredPost::default())),
        });
    }
    for _ in 0..dropped_ads {
        non_selected.push(FeedItem {
            position: 0,
            item: Some(feed_item::Item::Ad(AdIndexInfo::default())),
        });
    }
    non_selected
}

Placeholder for non-selected items. Empty ScoredPost::default() / AdIndexInfo::default() per dropped item. Lossy — we don't preserve the actual dropped content. Just used for counting in side-effects.

struct PartitionedFeedItems {
    posts: Vec<ScoredPost>,
    ads: Vec<AdIndexInfo>,
    wtf_modules: Vec<WhoToFollowModule>,
    prompts: Vec<Prompt>,
    push_to_home: Option<PushToHomePost>,
}

fn partition_feed_items(items: Vec<FeedItem>) -> PartitionedFeedItems {
    let mut posts = Vec::new();
    let mut ads = Vec::new();
    let mut wtf_modules = Vec::new();
    let mut prompts = Vec::new();
    let mut push_to_home = None;
    for item in items {
        match item.item {
            Some(feed_item::Item::Post(post)) => posts.push(post),
            Some(feed_item::Item::Ad(ad)) => ads.push(ad),
            Some(feed_item::Item::WhoToFollow(wtf)) => wtf_modules.push(wtf),
            Some(feed_item::Item::Prompt(prompt)) => prompts.push(prompt),
            Some(feed_item::Item::PushToHome(pth)) => push_to_home = Some(pth),
            None => {}
        }
    }
    PartitionedFeedItems {
        posts,
        ads,
        wtf_modules,
        prompts,
        push_to_home,
    }
}

The partition utility. Walks the input, sorts by enum variant. push_to_home is Option<_> (only one allowed, last one wins).

This is typed partitioning — given a Vec<FeedItem> (a heterogeneous mix of variants), produce a struct with per-variant vecs. Standard pattern when a wire-format union enum needs to be processed per-type downstream.

What we've learned

The three-scorer pipeline:

  1. PhoenixScorer: ML inference → phoenix_scores (15+ per-action probabilities).
  2. RankingScorer: weighted-sum + author diversity + OON downweight → weighted_score + score.
  3. VMRanker (optional): overlay model that overwrites score via a separate service. Supports DPP-based diversity.

Phoenix routing:

  • New users (history < threshold) get a different cluster.
  • Decider overrides can swap between experiment arms (Fou ↔ Lap7).
  • Egress sidecar can be tried first, fall back to primary.

The ranking formula: 22 weighted Phoenix-action probabilities. Every weight is a feature switch — the formula is policy, not code. Tuning the algorithm = config change.

Score normalization:

  • negative_sum / total_sum are pre-computed for the offset_score normalization.
  • Negative combined_score maps to [0, OFFSET).
  • Positive combined_score maps to [OFFSET, +∞).
  • Keeps scores positive for downstream multiplicative adjustments.

VQV-eligibility: video-view weights only apply when the video is long enough. Pre-computed per candidate.

Author diversity: exponential decay per author position. Sort by score, count seen-per-author, multiply by (1-floor) * decay^position + floor. Bounded between floor and 1.0.

Three-way OON branching:

  • Topic request → use TopicOonWeightFactor (less downweight).
  • New-user-with-follows → use NEW_USER_OON_WEIGHT_FACTOR (less downweight, more OON exposure).
  • Default → use OonWeightFactor (downweight OON).

VMRanker as overlay: not a fresh score from scratch — falls back to existing c.score if a candidate isn't in the response. Allows partial coverage.

DPP for diversity: VMRanker supports Determinantal Point Process parameters for set-level diversity, distinct from RankingScorer's per-author multiplicative diversity.

Selector duality:

  • TopKScoreSelector — sort + take, 15 lines.
  • BlenderSelector — full custom override: partition by item type, blend posts+ads, inject prompts/WTF/push-to-home at specific positions.

The For You item types: Post, Ad, WhoToFollow, Prompt, PushToHome. Only one of each non-post type appears (e.g., only one WTF module, only one push-to-home).

Item position rules (in BlenderSelector):

  • Push-to-home: position 0 (always pinned).
  • Prompts: positions 0, 1, 2, … (stacked top).
  • Posts + Ads: blender-controlled.
  • WTF: WHO_TO_FOLLOW_POSITION (typically 3 or so).

Orphan pattern: the old separate scorers (WeightedScorer, AuthorDiversityScorer, OONScorer) used hard-coded weights and ran in series. The active RankingScorer consolidates them with feature-switch-driven weights. Different design philosophies: pre-refactor was "small composable units"; current is "one consolidated scorer with all the policy."

Next session

Session 11 — Sources (903 LOC). The 11 source implementations:

  • thunder_source.rs — in-network from Thunder
  • tweet_mixer_source.rs — legacy OON
  • phoenix_source.rs, phoenix_topics_source.rs, phoenix_moe_source.rs — three Phoenix retrieval flavors
  • scored_posts_source.rs — wraps the inner Phoenix pipeline
  • ads_source.rs, who_to_follow_source.rs, prompts_source.rs, push_to_home_source.rs — non-post sources for For You
  • cached_posts_source.rs — re-uses cached scored posts