Data for automatic speech recognition (ASR) training or finetuning takes the form of pairs of audio recordings and target transcripts of that audio.

The audio recordings should reflect the domain to which the ASR is expected to function including background noise and type of recording equipment, but also speaker dialect, accent, gender, race, age and other characteristics that might affect the speech.

In cases where recordings do not reflect the target domain, this should be communicated and will require extra efforts from the ML team in model training and evaluation to ensure performance on the target domain.

The transcripts must be verbatim (or near-verbatim) to be useful for training, fine-tuning and/or evaluating an ASR model. Any syntactic property of the transcripts expected from the delivered ASR model must also be reflected in the target transcripts (e.g. capitalization and punctuation).

Transcripts generated by other highly accurate dictation software vendors can be excellent sources of data, provided they meet the criteria for verbatim or near-verbatim accuracy. This ensures alignment with the expected outputs of the ASR model. Being machine-generated, such transcripts often contain errors making them ill suited for direct use in model training. They remain useful for a number of other use-cases, and can also be corrected in an active learning setup. In short, leveraging such transcripts can often be an effective way to produce high-quality datasets for training or evaluation.

In cases where verbatim transcripts cannot be obtained it might be possible for Corti’s ML team to extract approximately verbatim transcripts from near-verbatim texts such as summaries written from a recorded dictation.

A data sample (excluding the audio file itself) might contain the following parts:

While this data sample is in the JSON format, other text-formats such as CSV are also supported, as long as they comply with the guidelines for the transcript itself (which usually include commas!).

A data sample can also include metadata such as speaker_id, gender, race, age, language, dialect, location, date and generally should include as much metadata as possible. This might be used for training but is always valuable to estimate model performance on subpopulations.

For specialized domains, metadata can also include lists of specialized words. These can help familiarize the model with specialized words that are rare in the data but important for correct understanding and downstream uses of the transcript. A list of words does not need associated audio recordings. If recordings are available they can be included.

For a language where dialects are not a big concern, 50 hours of audio recordings and corresponding verbatim, or near-verbatim, transcripts for 20+ speakers are needed to see useful improvements in transcript quality.

The more specialized the vocabulary, the more audio-transcript pairs are needed to be able to provide equivalent transcription quality.

Examples of specialized domains ranked by specialization (highest to lowest):

For languages with a significant number of strong dialects, diverse speaker representation for each dialect is required.

If the dialects are “fairly similar”, a good starting point would be 5-10 hours per dialect and 5+ speakers per dialect. In the extreme that dialects are very different (almost separate languages), upwards 50 hours, 20 speakers per dialect would be required.

One reason for the data requirements is the disconnect between the written and spoken languages for strong dialects. As an example, Swiss-German has no official written language but is similar in many ways to High-German. In such cases, target transcripts might be in High-German while the speech is Swiss-German which complicates the task imposed on the model.

Some languages lack a written form. For a transcription task to make sense in these cases, a substitute written form must be agreed upon. In some cases, cultural norms might impose a common written language to which transcription could translate, effectively extending the speech recognition task to include translation.

This disconnect between spoken and written forms further increases the required amount of data. An estimate is 100 hours of audio recordings and corresponding verbatim, or near-verbatim, transcripts for 20+ speakers.

Audio files should be provided as individual files. The audio should not be resampled and that it is delivered in the format and encoding in which it was originally recorded. This is true even if audio files come in several different formats. A single file per recording is preferred, i.e. there is also no need to divide any recording into multiple audio files.

Transcripts should be segmented i.e. each audio file should be transcribed into several separate phrases labelled with their start and stop times in the full audio file. Timestamps should be provided at least at the hundred millisecond level in the format H:MM:SS.m where H are hours, MM are minutes (up to 60), SS are seconds (up to 60), and m is hundreds of milliseconds. Transcripts can be provided in different formats, as long as their link to their source audio file remains clear.

To continuously improve the ASR models ability to deal with special dialect words and pronunciations as well as medical terms, additional data may be required. The above amounts should be seen as a starting point.

If you would like more accurate results and would like to help train our models, please contact [email protected] to learn more.