Microphone Text Input Mod User Documention

Architectury 2.x

Introduction

The Microphone Text Input Mod is a Fabric mod designed for Minecraft clients. It provides speech recognition input
functionality and automatically converts spoken words into text chat messages, enhancing the in - game communication
experience.

Features

• Multiple Modes: Supports different operation modes, including AUTO_SEND, RELEASE_KEY_TO_SEND, and
  RELEASE_KEY_TO_INPUT.
• Advanced Configuration: Allows users to adjust advanced parameters related to the Whisper library, such as the
  number of threads, audio context size, and sampling strategy.

Dependencies

| Dependency Name | Fabric | NeoForge |
|------------------|------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| Java | 21 | 21 |
| Fabric API | see fabric.mod.json | ❌ |
| Architectury API | ❌ | see neoforge.mods.toml |
| MidnightLib | see fabric.mod.json | see neoforge.mods.toml |

Configuration

Keybinding

| Keybinding Name | Recognize |
|-----------------|----------------------------------|
| Category | key.category.minecraft.mcmti |
| Translation Key | key.mcmti.recognize |
| Default Key | V |

General Settings

| Setting | Translation Key | Field | Type | Default Value | Description |
|---------------------------|-----------------------------------------|----------------------------------------------------------|----------------------------------------------|------------------------------------------|---------------------------------------------------------------------------------------------|
| GGML Whisper Model | mcmti.midnightconfig.model | me.jaffe2718.mcmti.config.McmtiConfig.model | String | "" | Path to the GGML Whisper model. |
| Language | mcmti.midnightconfig.language | me.jaffe2718.mcmti.config.McmtiConfig.language | String | "en" | Language for speech recognition. |
| Mode | mcmti.midnightconfig.mode | me.jaffe2718.mcmti.config.McmtiConfig.mode | me.jaffe2718.mcmti.config.McmtiConfig.Mode | "RELEASE_KEY_TO_SEND" | Mod's work mode. |
| Record Cycle (ms) | mcmti.midnightconfig.recordCycleMs | me.jaffe2718.mcmti.config.McmtiConfig.recordCycleMs | int | 5000 | Record cycle in milliseconds. |
| Record Buffer Size (byte) | mcmti.midnightconfig.recordBufferSize | me.jaffe2718.mcmti.config.McmtiConfig.recordBufferSize | int | 1024 | Record buffer size in bytes. |
| Prefix | mcmti.midnightconfig.prefix | me.jaffe2718.mcmti.config.McmtiConfig.prefix | String | "⌈Speech Input⌋" | Prefix added to the recognized text. |
| Draft Input | mcmti.midnightconfig.draftInput | me.jaffe2718.mcmti.config.McmtiConfig.draftInput | boolean | false | Enable draft input. If enabled, the recognized text will be shown in the entry as a draft. |
| Encoding Repair | mcmti.midnightconfig.encodingRepair | me.jaffe2718.mcmti.config.McmtiConfig.encodingRepair | boolean | false | Enable encoding repair. |
| Source Encoding | mcmti.midnightconfig.srcEncoding | me.jaffe2718.mcmti.config.McmtiConfig.srcEncoding | String | Charset.defaultCharset().displayName() | Source encoding for text. Applies only if encoding repair is enabled. |
| Destination Encoding | mcmti.midnightconfig.dstEncoding | me.jaffe2718.mcmti.config.McmtiConfig.dstEncoding | String | Charset.defaultCharset().displayName() | Destination encoding for text. Applies only if encoding repair is enabled. |

Advanced Settings

| Setting | Translation Key | Field | Type | Default Value | Description |
|------------------------------|-----------------------------------------------------|----------------------------------------------------------------------|----------------------------------------------------------------|---------------|-------------------------------------------------------------------------------------|
| Enable Advanced Config | mcmti.midnightconfig.advancedConfig | me.jaffe2718.mcmti.config.McmtiConfig.advancedConfig | boolean | false | Enable advanced configuration. |
| nThreads | mcmti.midnightconfig.nThreads | me.jaffe2718.mcmti.config.McmtiConfig.nThreads | int | 0 | Number of threads to use for the operation of the Whisper model. 0 for max cores. |
| audioCtx | mcmti.midnightconfig.audioCtx | me.jaffe2718.mcmti.config.McmtiConfig.audioCtx | int | 0 | Audio context size for the Whisper model. 0 means use default. |
| nMaxTextCtx | mcmti.midnightconfig.nMaxTextCtx | me.jaffe2718.mcmti.config.McmtiConfig.nMaxTextCtx | int | 16384 | Max tokens to use from past text as prompt for the decoder. |
| offsetMs | mcmti.midnightconfig.offsetMs | me.jaffe2718.mcmti.config.McmtiConfig.offsetMs | int | 0 | Offset in ms to start recording from. |
| durationMs | mcmti.midnightconfig.durationMs | me.jaffe2718.mcmti.config.McmtiConfig.durationMs | int | 0 | Audio duration to process in ms. 0 means use default. |
| translate | mcmti.midnightconfig.translate | me.jaffe2718.mcmti.config.McmtiConfig.translate | boolean | false | Translate the text to the default language. |
| noTimestamps | mcmti.midnightconfig.noTimestamps | me.jaffe2718.mcmti.config.McmtiConfig.noTimestamps | boolean | false | Do not generate timestamps. |
| initialPrompt | mcmti.midnightconfig.initialPrompt | me.jaffe2718.mcmti.config.McmtiConfig.initialPrompt | String | "" | Initial text to use as a prompt for the whisper. |
| noContext | mcmti.midnightconfig.noContext | me.jaffe2718.mcmti.config.McmtiConfig.noContext | boolean | true | Do not use past transcription (if any) as initial prompt for the decoder. |
| singleSegment | mcmti.midnightconfig.singleSegment | me.jaffe2718.mcmti.config.McmtiConfig.singleSegment | boolean | false | Force single segment output (useful for streaming). |
| printSpecial | mcmti.midnightconfig.printSpecial | me.jaffe2718.mcmti.config.McmtiConfig.printSpecial | boolean | false | Print special tokens. |
| printProgress | mcmti.midnightconfig.printProgress | me.jaffe2718.mcmti.config.McmtiConfig.printProgress | boolean | true | Print progress information. |
| printRealtime | mcmti.midnightconfig.printRealtime | me.jaffe2718.mcmti.config.McmtiConfig.printRealtime | boolean | false | Print results from within whisper.cpp (avoid it, use callback instead). |
| printTimestamps | mcmti.midnightconfig.printTimestamps | me.jaffe2718.mcmti.config.McmtiConfig.printTimestamps | boolean | true | Print timestamps for each text segment when printing realtime. |
| suppressBlank | mcmti.midnightconfig.suppressBlank | me.jaffe2718.mcmti.config.McmtiConfig.suppressBlank | boolean | true | Decoder option. |
| suppressNonSpeechTokens | mcmti.midnightconfig.suppressNonSpeechTokens | me.jaffe2718.mcmti.config.McmtiConfig.suppressNonSpeechTokens | boolean | true | Tokenizer option. |
| temperature | mcmti.midnightconfig.temperature | me.jaffe2718.mcmti.config.McmtiConfig.temperature | float | 0.0f | Initial decoding temperature. |
| maxInitialTs | mcmti.midnightconfig.maxInitialTs | me.jaffe2718.mcmti.config.McmtiConfig.maxInitialTs | float | 1.0f | Maximum initial timestamp. |
| lengthPenalty | mcmti.midnightconfig.lengthPenalty | me.jaffe2718.mcmti.config.McmtiConfig.lengthPenalty | float | -1.0f | Length penalty. |
| temperatureInc | mcmti.midnightconfig.temperatureInc | me.jaffe2718.mcmti.config.McmtiConfig.temperatureInc | float | 0.4f | Temperature increment. |
| entropyThold | mcmti.midnightconfig.entropyThold | me.jaffe2718.mcmti.config.McmtiConfig.entropyThold | float | 2.4f | Entropy threshold (similar to OpenAI's "compression_ratio_threshold"). |
| logprobThold | mcmti.midnightconfig.logprobThold | me.jaffe2718.mcmti.config.McmtiConfig.logprobThold | float | -1.0f | Log probability threshold. |
| noSpeechThold | mcmti.midnightconfig.noSpeechThold | me.jaffe2718.mcmti.config.McmtiConfig.noSpeechThold | float | 0.6f | No speech threshold. |
| greedyBestOf | mcmti.midnightconfig.greedyBestOf | me.jaffe2718.mcmti.config.McmtiConfig.greedyBestOf | int | -1 | Specific to greedy sampling strategy. |
| beamSearchBeamSize | mcmti.midnightconfig.beamSearchBeamSize | me.jaffe2718.mcmti.config.McmtiConfig.beamSearchBeamSize | int | 2 | Specific to bean search sampling strategy. |
| beamSearchPatience | mcmti.midnightconfig.beamSearchPatience | me.jaffe2718.mcmti.config.McmtiConfig.beamSearchPatience | float | -1.0f | Specific to bean search sampling strategy. |
| grammar | mcmti.midnightconfig.grammar | me.jaffe2718.mcmti.config.McmtiConfig.grammar | String | "" | Grammar file path. Empty means no grammar. |
| grammarPenalty | mcmti.midnightconfig.grammarPenalty | me.jaffe2718.mcmti.config.McmtiConfig.grammarPenalty | float | 100.0f | Penalty for non grammar tokens. |
| whisperSamplingStrategy | mcmti.midnightconfig.whisperSamplingStrategy | me.jaffe2718.mcmti.config.McmtiConfig.whisperSamplingStrategy | io.github.freshsupasulley.whisperjni.WhisperSamplingStrategy | BEAM_SEARCH | The WhisperContext enum to configure whisper's sampling strategy. |
| vad | mcmti.midnightconfig.vad | me.jaffe2718.mcmti.config.McmtiConfig.vad | boolean | false | Enable VAD (Voice Activity Detection). |
| vad__max_speech_duration_s | mcmti.midnightconfig.vad__max_speech_duration_s | me.jaffe2718.mcmti.config.McmtiConfig.vad__max_speech_duration_s | float | 0f | Max duration of a speech segment before forcing a new segment. |
| vad__min_silence_duration_ms | mcmti.midnightconfig.vad__min_silence_duration_ms | me.jaffe2718.mcmti.config.McmtiConfig.vad__min_silence_duration_ms | int | 0 | Min silence duration to consider speech as ended. |
| vad__min_speech_duration_ms | mcmti.midnightconfig.vad__min_speech_duration_ms | me.jaffe2718.mcmti.config.McmtiConfig.vad__min_speech_duration_ms | int | 0 | Min duration for a valid speech segment. |
| vad__samples_overlap | mcmti.midnightconfig.vad__samples_overlap | me.jaffe2718.mcmti.config.McmtiConfig.vad__samples_overlap | float | 0f | Overlap in seconds when copying audio samples from speech segment. |
| vad__speech_pad_ms | mcmti.midnightconfig.vad__speech_pad_ms | me.jaffe2718.mcmti.config.McmtiConfig.vad__speech_pad_ms | int | 0 | Padding added before and after speech segments. |
| vad__threshold | mcmti.midnightconfig.vad__threshold | me.jaffe2718.mcmti.config.McmtiConfig.vad__threshold | float | 0f | Probability threshold to consider as speech. |
| vad_model_path | mcmti.midnightconfig.vad_model_path | me.jaffe2718.mcmti.config.McmtiConfig.vad_model_path | String | "" | Path to the VAD model. Empty means use default. |
| useCustomDynamicLib | mcmti.midnightconfig.useCustomDynamicLib | me.jaffe2718.mcmti.config.McmtiConfig.useCustomDynamicLib | boolean | false | Enable using custom dynamic library for Whisper. |
| customDynamicLibDir | mcmti.midnightconfig.customDynamicLibDir | me.jaffe2718.mcmti.config.McmtiConfig.customDynamicLibDir | String | "" | Custom dynamic link library directory for Whisper. |

• For more info,
  see WhisperFullParams.java

WARNING: Activating the advanced configuration will change the default parameters of the Whisper model configuration,
which will have a critical impact on the speech recognition results. Inappropriate configuration of advanced
parameters can lead to problems such as speech recognition failure, high computer resource usage, and program crashes.
Please use with caution.

Usage

1. Install the mod and download the GGML Whisper model.
2. Configure the mod according to your needs in the configuration menu.
3. Use the keybinding you set (default is V) to start recording and speech recognition.
4. If the mode is set to AUTO_SEND, the recognized text will be automatically sent as a chat message. If set to
   RELEASE_KEY_TO_SEND or RELEASE_KEY_TO_INPUT, follow the corresponding key - release actions.

Use VAD

If you want to use VAD (Voice Activity Detection), you need to download VAD model,
enable the advanced configuration and set the vad_model_path to the path of the VAD model in the configuration menu.

Custom Dynamic Library

1. Download the custom dynamic library from Jaffe2718/whisper-jni and extract the files.
2. Enable the advanced configuration and set the useCustomDynamicLib to true in the configuration menu.
3. Set the customDynamicLibDir to the directory where the custom dynamic library is located in the configuration menu.
4. If you want to use the dynamic library which is supported vulkan, check your check that your computer has drivers and libraries running Vulkan installed.

   vulkaninfo
   

5. If you want to use CUDA, check that your computer has drivers and libraries running CUDA installed.

   nvidia-smi
   

   For Linux, you need to install CUDA Toolkit >= 12.4.0 and configure the environment variables.
   For Windows, if the game crashes, you have to force the game to use the Java >= 25, see Jaffe2718/whisper-jni/v0.5.6

Troubleshooting

• Crash on Startup: Please check the compatibility and version of your use of the dynamic link library. You can disable the advanced configuration and set the useCustomDynamicLib to false to use the default dynamic library (CPU version).
• Audio Input Device Load Failed: Please check if Java has access to the audio input device.
• Whisper Model Load Failed: Make sure the path to the GGML Whisper model is correct.

Contributing

If you'd like to contribute to this project, please feel free to submit issues or pull requests
on GitHub.