Architecture

PageSieve is designed to be local-first, ergonomic, and reusable. This page provides a high-level overview of the project’s architecture and its core components.

Tech Stack

The extension is built using:

  • TypeScript: Ensuring type safety across the codebase.
  • Svelte: Used for the sidebar and full-page dashboards.
  • Vite: The build tool and development server.
  • Shadcn UI: For building consistent and accessible UI components.
  • XState: For implementing main loop of extension.
  • Zod: Defining and validating scrape configurations and other schemas.

Core Components

  • Sidebar UI (apps/extension/src/ui/sidebar): The main interface for defining scrape configurations.
  • Content Script (apps/extension/src/content.ts): Injected into the webpage to interact with the DOM.
  • Background Script (apps/extension/src/background.ts): Manages messaging between different parts of the extension.
  • Scraping State Machine (packages/core/src/scrapeMachine.ts): State machine controlling the main extraction and pagination loop.
  • DOM Inspector (apps/extension/src/dominspector.ts): Handles the click-based selection of elements.
  • Selector Engine (apps/extension/src/selectorgadget.ts): Implements the CSS selector guessing algorithm. Vendored version of cantino/selectorgadget.
  • CLI Crawlers (apps/cli/src/): Headless runners supporting Playwright for browser rendering and Cheerio for lightweight HTTP extraction.
  • Shared Extractor & Converters (packages/core/src/): Reusable extraction pipeline, data converters (JSON, CSV, YAML), and schema definitions.
  • Data Types (packages/core/src/schema.ts): Central location for Zod schema defining the scrape configuration.
  • Logging System (packages/core/src/logger.ts) and (apps/extension/src/logger.ts): Structured logging using LogTape, supporting relay to local environments.