{ "title": "SIRDCC Speech Technology WG assessment of current STT technology", "type": "REPORTED", "description": "This secret document from the SIRDCC Speech Technology Working Group provides a technical assessment of Speech-to-Text (STT) capabilities for the Security Service (MI5) and GCHQ. The report acknowledges that while automated transcription of intercepted speech remains highly challenging, the technology offers immediate \"force multiplier\" benefits for specific tasks like identifying spoken telephone numbers or keywords. The Working Group recommends continued investment in bespoke training and a transition toward a common system—specifically the BBN Byblos engine—to facilitate the sharing of classified linguistic models between UK and US agencies.", "long_description": "The Working Group identifies a significant \"performance gap\" between speech recognition on public, cooperative data versus real-world \"intercept\" speech. While systems can achieve high accuracy on news broadcasts, intercepted conversations are often noisy, informal, and regionally accented, leading to high Word Error Rates (WER). The report highlights that state-of-the-art systems at the time, such as BBN's Byblos and IBM's Attila, were heavily biased toward US English due to decades of American-led research. This bias created a distinct disadvantage for UK agencies, necessitating the creation of domestic corpora like NIRAD (Northern Irish Regional Accent Data) to retrain these algorithms for local targets.\n\nThe comparative evaluation of three commercial systems—BBN Byblos, IBM Attila, and SAIL Labs—found that BBN significantly outperformed its competitors on challenging UK intercept data. Byblos achieved a 60% Word Error Rate after bespoke training, compared to 76% for IBM and 101% for SAIL Labs. Despite the high error rate, GCHQ reported successful operational deployments by using \"language-model biasing.\" This technique allows the software to ignore most of the conversation while maintaining a high sensitivity to specific strings, such as Caribbean drug runners reciting telephone numbers or specific high-priority keywords.\n\nA major strategic hurdle identified is the immense cost and time required for \"Bulk Transcription.\" Converting one hour of intercepted audio into the high-quality text needed to train a machine costs approximately £1,500, with a robust model requiring hundreds of hours of such data. Because these linguistic models are not portable between different software engines, the Working Group sees considerable benefit in the UK agencies agreeing to use a common system. By adopting the BBN system used by the NSA, GCHQ hoped to gain access to the NSA's vast archives of SIGINT-trained models, effectively bypassing years of expensive manual transcription and CPU-intensive model building.", "tags": [ "IBM", "MI5", "Security Service", "speech-to-text", "speech recognition", "phonetic search", "transcription", "drug trafficking", "keyword scanning", "speech processing", "voice analytics" ], "document": { "creator": "GCHQ", "date_created": "2009-12-07", "classification": [ "UK SECRET", "STRAP 1" ], "dissemination": [], "banner": "UK SECRET STRAP1", "genre": "report", "programs": [ "BYBLOS" ], "countries": [ "Ireland" ], "people": [] }, "publication": [ { "publisher": "The Intercept", "date_published": "2015-05-05", "title": "How the NSA Converts Spoken Words Into Searchable Text", "link": "https://theintercept.com/2015/05/05/nsa-speech-recognition-snowden-searchable-text/", "reporters": [ "Froomkin, Dan" ], "contributors": [ "Fishman, Andrew" ], "doc_source": "https://theintercept.com/document/2015/05/05/sirdcc-speech-technology-wg-assessment-current-stt-technology/", "doc_url": "https://s3.documentcloud.org/documents/2072016/wg-advice-to-security-service.pdf" } ], "pdf_sha256": "6957e1047faa2eff7d868c911ef21571ce4139214a6cc7eb7fecf162436ca2bd", "pages": 11 }