@storiny/discovery

Source for the Discovery service, responsible for embedding and rendering third-party content on Storiny.

Tech stack

• Framework: Actix Web serves our HTTP endpoints and middlewares
• State & caching: Redis handles rate limiting
• Metadata & DOM parsing: Visdom, html5ever, and markup5ever_rcdom for parsing, querying and safely manipulating HTML DOM nodes
• HTTP client: reqwest for making HTTP requests to oEmbed endpoints and web pages
• Serialization: serde and serde_json for mapping oEmbed specifications into typed Rust structures
• Telemetry: Sentry for tracing, monitoring, and capturing errors in production

URL discovery & oEmbed

The heart of this repository is its embed resolution logic. Whenever a user pastes a link into a story/document, this service determines exactly how to render it:

• as a rich interactive iframe
• a native video
• a photo
• or a standard link card

Under the hood

1. Request & URL decompression: When the client requests an embed from the frontend, they pass an LZ-compressed URL to our /embed/{compressed_url} endpoint. Inside the route handler, we use lz-str to decompress the string back into a valid URL. This simplifies caching behaviour as all of our /embed/ endpoints are heavily cached to prevent third-party rate limits.

2. Provider resolution: We maintain a curated registry of supported oEmbed providers in providers.json file. The decompressed URL is matched against pre compiled Regex schemas to see if it belongs to a known provider (like YouTube, Twitter, Spotify…).

   • If a match is found, we know exactly which endpoint to hit and whether the provider supports light/dark binary themes.
   • If no match is found, we immediately fallback to our custom metadata scraper.

3. oEmbed fetching & parsing: For supported providers, we make a request to their oEmbed endpoint. The JSON response is parsed into our EmbedResponse and EmbedType enums (Photo, Video, Link, Rich).

   • Based on the EmbedType, we parse the returned HTML using visdom, inject our own iframe styling, apply responsive aspect ratio paddings, and safely sanitize the output.
   • The heavily customized HTML (or raw JSON for scripted embeds) is then piped through the sailfish templating engine and returned to the client.

4. Metadata fallback: If the link isn’t supported by an oEmbed provider or if the provider returns a generic Link type, we fall back to get_metadata. This utility fetches the raw HTML of the webpage and extracts opengraph (og:) tags, Twitter card tags, titles and descriptions; returning a JSON metadata payload for rendering link preview cards on the frontend.

Steps to launch a local instance of the service and a list of API endpoints are available in the repository README.