Adaptive Tokenizer Frame Granularity Improves Bilingual Voice Assistant Responsiveness and Comprehension in Bilingual Human Machine Dialogues
DOI:
https://doi.org/10.66238/8yhmzb95Keywords:
Tokenization, Voice Assistants, Bilingual Dialogues, Frame Granularity, Computational LinguisticsAbstract
The widespread adoption of voice assistants has fundamentally altered human-computer interaction, necessitating systems capable of processing diverse and complex linguistic inputs. Among the most challenging input types are bilingual dialogues, which frequently feature rapid code-switching and contrasting phonotactic constraints. A critical determinant of voice assistant performance in these scenarios is the tokenizer frame granularity, which dictates the temporal and structural units of speech and text processed by the underlying machine learning models. This comprehensive paper investigates the effects of varying tokenizer frame granularities on the performance, latency, and accuracy of bilingual voice assistant dialogues. By systematically evaluating fine-grained, medium-grained, and coarse-grained tokenization approaches across a substantial corpus of English-Spanish conversational speech, this research elucidates the inherent trade-offs between computational efficiency and boundary-resolution accuracy during code-switching events. The findings indicate that while coarse granularity enhances processing speed and reduces computational overhead, it significantly degrades the identification of mid-sentence language transitions. Conversely, fine-grained tokenization yields superior phonetic alignment and morphological disambiguation at the cost of elevated latency. Through detailed empirical analysis, this study proposes an adaptive granularity framework that dynamically adjusts token and acoustic frame sizes based on real-time linguistic context, optimizing both responsiveness and comprehension in bilingual human-machine interactions.
References
1. Chen, C., Zhang, J., Lakshmikanth, S. K., Fang, Y., Shao, R., Wetzstein, G., ... & Adeli, E. (2025). The language of motion: Unifying verbal and non-verbal language of 3d human motion. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 6200-6211).
2. Xu, X., Tu, W., & Yang, Y. (2023, June). Selector-enhancer: learning dynamic selection of local and non-local attention operation for speech enhancement. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 11, pp. 13853-13860).
3. Zhang, J., Chen, C., Chen, X., Yu, H., Xiang, T., Khan, A. S., ... & Adeli, E. (2026). Vibes: A conversational agent with behaviorally-intelligent 3d virtual body. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 39994-40008).
4. Xu, X., Tu, W., & Yang, Y. (2024). Adaptive selection of local and non-local attention mechanisms for speech enhancement. Neural Networks, 174, 106236.
5. Xu, X., Han, C., Zhang, Y., Tu, W., & Yang, Y. (2024, April). Curricular Contrastive Regularization for Speech Enhancement with Self-Supervised Representations. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 10486-10490). IEEE.
6. Lv, Z., Wang, Q., Lyu, T., & Dong, J. (2026). A Deep Learning-Based Model for User Emotion Recognition and Behavior Guidance Design: EMoGuideNet-An Emotion-Aware Multimodal Framework for Enhancing Intelligent User Interaction. Journal of Organizational and End User Computing (JOEUC), 38(1), 1-30.
7. Dai, S., Wu, Y., Chen, S., Huang, R., & Dannenberg, R. B. (2023, November). SingStyle111: A Multilingual Singing Dataset With Style Transfer. In ISMIR (pp. 765-773).
8. Zhang, H., Chen, J., Wu, D., Li, Y., Zhang, Y., Zhang, X. T., ... & Tian, F. (2026). Duplexsla: A full-duplex spoken language model with synchronized speech, language, and action. arXiv preprint arXiv:2605.20755.
9. Li, A., Sun, Z., Hao, F., Li, X., & Zheng, C. (2025). Neural vocoders as speech enhancers. arXiv preprint arXiv:2501.13465.
10. Li, A., Lei, T., Sun, Z., Chen, R., Yin, E., Li, X., & Zheng, C. (2025). Learning neural vocoder from range-null space decomposition. arXiv preprint arXiv:2507.20731.
11. Li, A., Liu, W., Zheng, C., Fan, C., & Li, X. (2021). Two heads are better than one: A two-stage complex spectral mapping approach for monaural speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 1829-1843.
12. Li, A., Zheng, C., Zhang, L., & Li, X. (2022). Glance and gaze: A collaborative learning framework for single-channel speech enhancement. Applied Acoustics, 187, 108499.
13. Li, A., Liu, W., Zheng, C., & Li, X. (2022, May). Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6487-6491). IEEE.
14. Xu, X., Li, J., Zhang, Y., Tu, W., & Yang, Y. (2025). Spatial information aided speech and noise feature discrimination for Monaural speech enhancement. Expert Systems with Applications, 269, 126349.
15. Xu, Xinmeng, Weiping Tu, and Yuhong Yang. "Pcnn: A lightweight parallel conformer neural network for efficient monaural speech enhancement." arXiv preprint arXiv:2307.15251 (2023).
16. Chen, L. (2026). Beyond external constraints: The missing dimension of ai governance. Available at SSRN 6449738.
17. Xu, X., Wang, Y., Xu, D., Peng, Y., Zhang, C., Jia, J., & Chen, B. (2022, May). Vsegan: Visual speech enhancement generative adversarial network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7308-7311). IEEE.
18. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., et al. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.
19. Dai, L., Li, A., Chi, C., Liang, Y., Li, X., & Zheng, C. (2026). GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks. arXiv preprint arXiv:2601.13758.
20. Zhang, H., Liu, H., Zhang, X., Zhang, Q., Hu, Y., Zhao, J., ... & Chng, E. S. (2025). Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English. arXiv preprint arXiv:2505.17076.
21. Xu, X., Tu, W., Yang, Y., Li, J., & Zhang, Y. (2025). Interactive Target Positive and Negative Features Modeling for Monaural Speech Enhancement. IEEE Transactions on Audio, Speech and Language Processing, 33, 4856-4869.
22. Yu, H., Zhang, J., Chen, C., Xiang, T., Fang, Y., Niebles, J. C., & Adeli, E. (2026, March). Socialgen: Modeling multi-human social interaction with language models. In 2026 International Conference on 3D Vision (3DV) (pp. 1-17). IEEE.
23. Xu, Xinmeng, et al. "Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning." arXiv preprint arXiv:2605.01673 (2026).
24. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.
25. Li, A., Liu, W., Luo, X., Yu, G., Zheng, C., & Li, X. (2021). A simultaneous denoising and dereverberation framework with target decoupling. arXiv preprint arXiv:2106.12743.
26. Xu, X., Tu, W., & Yang, Y. (2025). Efficient audio–visual information fusion using encoding pace synchronization for Audio–Visual Speech Separation. Information Fusion, 115, 102749.
27. Xu, X., Tu, W., & Yang, Y. (2023). CASE-Net: Integrating local and non-local attention operations for speech enhancement. Speech Communication, 148, 31-39.
28. Shneiderman, B. (1983). Direct manipulation: A step beyond programming languages. IEEE Computer, 16(8), 57–69.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Chloe Grant, Amelia Hughes, Henry Carter (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All articles published in the International Journal of Machine Learning and Engineering (IJMLE) are licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits unrestricted use, distribution, reproduction, adaptation, and commercial use in any medium or format, provided the original author(s) and the source are properly credited.
Authors retain the copyright of their work while granting the journal the right of first publication.
To view a copy of this license, please visit:
https://creativecommons.org/licenses/by/4.0/
