Raxol.Gateway.Pipeline.Transcribe (RaxolGateway v0.2.0)

Copy Markdown View Source

Feed-loop stage that turns a voice media event into a %{text: transcript} event before it is routed.

A platform adapter normalizes a voice note to %{media: %{kind: :voice, ref: ref, ...}} (e.g. Raxol.Telegram.GatewayAdapter for message.voice updates). This stage fetches the audio behind ref, converts it to raw PCM, transcribes it, and hands the transcript on as the ordinary %{text: binary} event every Raxol.Gateway.Handler already understands. Any non-voice event passes through untouched, so the stage composes unconditionally:

with {:ok, route, event} <- MyAdapter.normalize_event(raw),
     {:ok, event} <- Transcribe.run(event, fetch_fn: fetch) do
  Raxol.Gateway.SessionRouter.route(router, route, event)
end

Why a feed-loop stage, not a Session concern

A Raxol.Gateway.Session records the raw inbound event to its :log before the handler sees it, and it processes events one at a time. Running STT inside the session would log opaque media references instead of transcripts and block the per-chat mailbox for the whole recognition run. In the feed loop the logged event is the transcript, and only the feed callback (e.g. the poller's :on_update) waits.

Options

  • :fetch_fn - (media_map -> {:ok, binary} | {:error, term}). Fetches the audio bytes behind the media :ref; platform-specific, so there is no default (a voice event without one is dropped). For Telegram: fn media -> Raxol.Telegram.GatewayAdapter.fetch_media(conn, media) end.

  • :convert_fn - (binary -> {:ok, pcm} | {:error, term}). Converts the fetched audio to raw f32le mono 16kHz PCM (what Raxol.Speech.Recognizer.recognize/1 takes). Default: convert_with_ffmpeg/2 with these options.

  • :recognize_fn - (pcm -> {:ok, text} | {:error, term}). Default: Raxol.Speech.Recognizer.recognize/1 when the optional raxol_speech dependency is loaded and its Recognizer process is running.

  • :max_bytes - drop audio larger than this before converting (default 20MB, the Bot API getFile ceiling). Checked against the media's :size_bytes metadata before fetching and against the fetched binary.
  • :max_duration_s - drop voice notes longer than this (default 300, matching the raxol_speech Listener's recording cap). Checked against the media's :duration_s metadata before fetching, enforced in the default converter (ffmpeg -t), and re-checked against the decoded PCM size. :max_bytes alone cannot bound the decode: f32le mono 16kHz is 64000 bytes/s, so a low-bitrate note under the compressed ceiling would otherwise expand without limit in memory.
  • :ffmpeg_path, :cmd_fn, :tmp_dir - see convert_with_ffmpeg/2.

A stage function that raises or exits (e.g. a recognizer process dying mid-call) is caught and handled as that stage's failure; only a shape summary of the crash reaches the log and telemetry, never the arguments (a GenServer.call exit term embeds the full audio binary).

Failure mode

Fail-open per event: any stage failure (no :fetch_fn, fetch/convert errors, STT unavailable, empty transcript) drops that one voice event with :ignore, a Logger.warning, and [:raxol_gateway, :transcribe, :error] telemetry (metadata %{stage, reason, kind}). Text traffic is unaffected; a dropped voice note never takes the chat down. Successful runs emit [:raxol_gateway, :transcribe, :done] with the audio and transcript sizes. Neither audio bytes nor transcript text ever reach the logs.

Latency

The stage blocks its caller for the whole fetch + convert + recognize run. The first recognition after boot also pays the XLA graph compile (minutes on CPU); give Raxol.Speech.Recognizer a generous :recognize_timeout_ms via Raxol.Speech.Supervisor's :recognizer_opts or warm the serving up front, otherwise every cold call times out, aborts the compile, and the next call starts it over.

Summary

Functions

Default :convert_fn: decode any audio ffmpeg understands to raw f32le mono 16kHz PCM.

Transcribe a voice media event; pass any other event through unchanged.

Types

event()

@type event() :: term()

media()

@type media() :: %{:kind => atom(), :ref => term(), optional(atom()) => term()}

Functions

convert_with_ffmpeg(bytes, opts \\ [])

@spec convert_with_ffmpeg(binary(), keyword()) :: {:ok, binary()} | {:error, term()}

Default :convert_fn: decode any audio ffmpeg understands to raw f32le mono 16kHz PCM.

System.cmd/3 cannot feed stdin, so the input bytes go through a temporary file (deleted afterwards, success or not):

ffmpeg -hide_banner -loglevel error -i <tmp> -t <max_duration_s>       -f f32le -ac 1 -ar 16000 pipe:1

Options

  • :ffmpeg_path - executable path (default System.find_executable("ffmpeg")). Only a binary named ffmpeg that exists on disk is accepted.
  • :cmd_fn - (path, args -> {output_binary, exit_status}) runner override for tests (default System.cmd/3)
  • :tmp_dir - where the temporary input file goes (default System.tmp_dir!())
  • :max_duration_s - decode at most this many seconds (-t, default 300), bounding the PCM held in memory regardless of how small the compressed input is

run(event, opts \\ [])

@spec run(event(), keyword()) :: {:ok, event()} | :ignore

Transcribe a voice media event; pass any other event through unchanged.

Returns {:ok, %{text: transcript, source: :voice}} for a transcribed voice event, {:ok, event} untouched for everything else, and :ignore when a voice event cannot be transcribed (logged, never raised).