Orion

Autonomous content publishing, industry intelligence feeds, and multi-channel SEO curation encounter three fundamental breakdown modes in production:

1. Cross-Outlet Syndication Duplication: When a major industry event occurs (e.g. regulatory audits, executive shifts, logistics expansions), dozens of news outlets publish slight variations of the same underlying press release. Standard keyword-based or hash-based deduplication algorithms fail completely, resulting in audiences being spammed with 4–6 identical stories throughout the day.
2. Asset Degradation & Scraped Formatting Noise: RSS/Atom enclosures are notoriously inconsistent. Many providers include low-resolution thumbnails (100x100), broken HTML markup, tracking pixels, or completely omit primary imagery.
3. Operational Maintenance Drag: Most AI-assisted scrapers rely on delicate cron scripts that silently stall when encountering Telegram API flood limits, upstream Cloudflare blocks, or LLM token spikes, requiring continual human oversight and board grooming.

The core architectural question: Can an unattended backend pipeline continuously poll disparate industry feeds, mathematically eliminate semantic near-duplicates using lightweight on-device dense vector embeddings without incurring remote API costs, heuristically resolve maximum-resolution hero imagery from canonical HTML, and deliver crisp, analysis-driven publications (<900 characters with dedicated "Why it matters" takeaways) 24/7 under native Linux systemd supervision?

Orion operates as an industry-agnostic syndication engine — originally designed and stress-tested across retail, e-commerce, FMCG, and marketplace media streams (e.g. retail.ru, new-retail.ru, Ozon, Wildberries, X5).

1. Local Dense Vector Deduplication Without API Costs

The critical differentiator in Orion is its local embedding pipeline. When an industry event breaks (for example, the Russian anti-monopoly authority auditing food retail markups), different publications release varying headlines:

• Source A: «ФАС проверит обоснованность роста цен на сельдь в рознице»
• Source B: «ФАС проанализирует наценки продовольственных товаров»

Standard TF-IDF or Levenshtein string distances frequently fail because the vocabulary overlap is minimal. Calling an external LLM on every incoming candidate title to ask "Are these two news items identical?" burns significant money and introduces latency.

Orion solves this entirely on-premise inside dedupe.py:

• Model: Loads intfloat/multilingual-e5-small locally via SentenceTransformers (384-dimensional dense embeddings).
• Normalization: All vectors are $L_2$-normalized ($v = vec / ||vec||_2$).
• Matrix Dot Product: Cosine similarity against the rolling index of up to 3,000 previously published articles is executed via a single NumPy matrix multiplication: sims = (self._vecs @ q).astype(np.float32).
• Deterministic Thresholding: If $\max(sims) \ge 0.90$ (calibrated in production up to 0.95), the article is marked as a near-duplicate and dropped.
• Collision Audit Ledger: Every blocked candidate is permanently logged to semantic_skips.csv with timestamp, exact cosine score (e.g. 0.927), candidate title, and reference title, creating full transparency into why a story was rejected.

def is_near_duplicate(self, title_norm: str) -> Tuple[bool, float, Optional[str]]:
    if not self.enabled or not title_norm:
        return False, 0.0, None
    q = self._embed(title_norm)
    if self._vecs is None or self._vecs.shape[0] == 0:
        return False, 0.0, None
    
    # Vectorized cosine similarity across rolling index of up to 3,000 stories
    sims = (self._vecs @ q).astype(np.float32)
    idx = int(np.argmax(sims))
    max_sim = float(sims[idx])
    
    if max_sim >= self.threshold:
        ref = self._titles[idx] if idx < len(self._titles) else None
        return True, max_sim, ref
    return False, max_sim, None

2. Deep-Page Hero Image Resolution & Heuristic Area Scoring

RSS enclosures are notoriously unreliable: feeds often supply 150px square thumbnails, company logos, or completely omit media tags.

Orion implements an intelligent DOM scraper in fetch_best_image_from_article:

• It follows the canonical article link using custom User-Agent headers and SOCKS5 proxy routing.
• It extracts candidate URLs from og:image, og:image:secure_url, twitter:image, <link rel="image_src">, and <img srcset>.
• For HTML <img> tags and URLs containing resolution patterns (e.g. image-1200x630.jpg), a regex parser computes the pixel area (W × H). Candidates are assigned priority scores (e.g. 1,800,000 points for high-definition OpenGraph images vs 200 points for inline thumbnails).
• The highest-scoring asset is dynamically attached to the publication payload, prioritizing the highest-resolution available source asset.

3. Grounded Editorial Contract & Voice Alignment

Autonomous news rewriting easily devolves into generic corporate jargon or hallucinated commentary. Orion prevents this through strict system prompt engineering (prompt.txt):

• Strict Character Budget: Under 900 characters total to fit within mobile instant-view and Telegram photo caption limits without truncation.
• Source-Constrained Output Contract: Source-constrained generation; unsupported facts are prohibited by the output contract. All revenue metrics, executive names, corporate entities, and direct quotes must strictly mirror source text.
• "Why This Matters" Analytical Takeaway: Rather than mere paraphrasing, every brief includes an isolated, high-signal paragraph explaining the strategic operational impact for industry practitioners.
• JSON Output Contract: The model emits structured JSON (title, description, important), which the post-builder validates and formats into semantic HTML with zero markdown artifacts.

4. Production Orchestration: Linux Systemd & Round-Robin Ingestion

Instead of running heavy, long-lived Python processes susceptible to memory leaks or network stalls, Orion is orchestrated entirely by native Linux systemd:

• Systemd Timers: orion-publish.timer executes discrete one-shot runs every 7 to 20 minutes, ensuring memory is completely reclaimed after each cycle.
• Active Window Guard: Evaluates local time against business publishing windows (08:00 to 23:00 Moscow Time), quietly exiting during quiet overnight hours.
• Round-Robin Ingestion: Ingests feeds in interleaved rotation. If Outlet A publishes 20 articles at once while Outlet B publishes 1, the round-robin selector takes 1 from each, preventing feed starvation.
• Resilient Network Handling: Implements exponential backoff for HTTP requests and catches Telegram RetryAfter flood-control exceptions, sleeping for the exact backoff window requested by Telegram before retrying.

Autonomous publishing requires bulletproof edge-case handling to maintain channel trust: