In of my recent keynote address in Taipei, I spoke about solving pain points by coding with the help of generative AI.

A night before the talk, I figured that I can create an English to Chinese translator app to provide real time translation to my mainly Chinese speaking audience. I am sure there are something in the market that does the job, but my point is about tinkering and solving pain points (and alleviating the itch to create!).

Thanks to generative AI, a simple tool was born, and I placed it alongside my slides. It picked up my speech through the microphone on my notebook and translated it to Chinese text. You can test it here (while it is still there, before my next tinkering overrides the deployment): https://app-keefellow.pythonanywhere.com/

Sharing the code below for posterity.

The 4 key steps taken in this English to Chinese speech translation app is as follows:

1. Speech Recognition Setup: The app uses the Web Speech API to continuously listen for English speech input, displaying both interim and final results in real-time.

2. Translation Processing: When final speech results are detected, they're sent to a Flask backend endpoint ('/translate') which uses Google Translate API to convert English text to Chinese.

3. Server Configuration: The Flask server handles the translation requests and serves the web interface, with routes for the main page, output page, and the translation API endpoint.

4. User Interface Display: The app presents a clean interface with a "Start/Stop Listening" button and two text boxes that show the original English speech input and its Chinese translation in real-time.

Love to hear ideas from you to see how this can customised.

from flask import Flask, render_template, request, jsonify
import os
import requests
import json

app = Flask(__name__, static_folder='static')

# Translate text using Google Translate API
def translate_text(text, target_lang='zh-CN'):
    try:
        base_url = "https://translate.googleapis.com/translate_a/single"
        params = {
            "client": "gtx",
            "sl": "en",
            "tl": target_lang,
            "dt": "t",
            "q": text
        }

        response = requests.get(base_url, params=params)
        if response.status_code == 200:
            try:
                result = response.json()
                translated_text = ''
                for sentence in result[0]:
                    if sentence[0]:
                        translated_text += sentence[0]
                return translated_text
            except json.JSONDecodeError:
                # Handle case when response is not valid JSON
                return f"Translation Error: Invalid response format"
        else:
            return f"Translation Error: {response.status_code}"
    except Exception as e:
        return f"Translation Error: {str(e)}"

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/output')
def output():
    return render_template('output.html')

@app.route('/translate', methods=['POST'])
def translate():
    try:
        data = request.get_json()
        if not data or 'text' not in data:
            return jsonify({'error': 'No text provided'}), 400

        english_text = data['text']

        # Translate to Chinese
        chinese_text = translate_text(english_text)

        result = {
            'english_text': english_text,
            'chinese_text': chinese_text
        }

        return jsonify(result)
    except Exception as e:
        return jsonify({'error': str(e)}), 500

if __name__ == '__main__':
    # Create directories if they don't exist
    os.makedirs('templates', exist_ok=True)
    os.makedirs('static', exist_ok=True)

    # Create templates
    with open('templates/index.html', 'w') as f:
        f.write('''



    
    
    Speech Recording
    


    
English to Chinese Speech Translator

    

        

            Click "Start Listening" and speak in English. Your speech will be continuously
            translated to Chinese in real-time as you speak.
        

        

            Start Listening
            

                 Listening...
            
        

        


        

            

                
English:
                

            

            

                
Chinese Translation:
                

            
        


    

    
        const recordBtn = document.getElementById('recordBtn');
        const englishText = document.getElementById('englishText');
        const chineseText = document.getElementById('chineseText');
        const recognitionStatus = document.getElementById('recognitionStatus');
        const listeningIndicator = document.getElementById('listeningIndicator');

        let recognition;
        let isListening = false;
        let translationDebounceTimer;

        // Initialize Web Speech API
        function initSpeechRecognition() {
            if ('webkitSpeechRecognition' in window) {
                recognition = new webkitSpeechRecognition();
            } else if ('SpeechRecognition' in window) {
                recognition = new SpeechRecognition();
            } else {
                alert('Your browser does not support speech recognition. Try Chrome or Edge.');
                return false;
            }

            recognition.continuous = true;
            recognition.interimResults = true;
            recognition.lang = 'en-US';

            recognition.onstart = function() {
                isListening = true;
                recordBtn.textContent = 'Stop Listening';
                recordBtn.classList.add('recording');
                listeningIndicator.style.display = 'inline-block';
                recognitionStatus.textContent = '';

                // Reset for new session
                currentSession = '';
                lastFinalResult = '';
            };

            recognition.onend = function() {
                if (isListening) {
                    // If we're still supposed to be listening, restart recognition
                    // This helps with the 60-second limit some browsers have
                    try {
                        recognition.start();
                    } catch (e) {
                        console.error('Failed to restart recognition:', e);
                        stopListening();
                    }
                } else {
                    stopListening();
                }
            };

            recognition.onerror = function(event) {
                console.error('Speech recognition error', event.error);

                if (event.error === 'no-speech') {
                    // This is a common error that happens when no speech is detected
                    // We don't need to show this to the user or stop listening
                    return;
                }

                recognitionStatus.textContent = 'Error: ' + event.error;

                if (event.error === 'network' || event.error === 'service-not-allowed') {
                    stopListening();
                }
            };

            let currentSession = '';
            let lastFinalResult = '';

            recognition.onresult = function(event) {
                let interimTranscript = '';
                let finalTranscript = '';

                // Process all results from this session
                for (let i = 0; i < event.results.length; ++i) {
                    const transcript = event.results[i][0].transcript;

                    if (event.results[i].isFinal) {
                        // Store the latest final result
                        if (i >= event.resultIndex) {
                            lastFinalResult = transcript;
                            // Send only new final results for translation
                            translateText(transcript);
                        }
                    } else if (i >= event.resultIndex) {
                        // Only add interim results from the current recognition
                        interimTranscript += transcript;
                    }
                }

                // Display the text - prioritize showing interim results
                if (interimTranscript) {
                    englishText.innerHTML = '' + interimTranscript + '';
                } else if (lastFinalResult) {
                    englishText.innerHTML = lastFinalResult;
                }
            };

            // When recognition restarts, reset the tracking variables
            recognition.onstart = function() {
                isListening = true;
                recordBtn.textContent = 'Stop Listening';
                recordBtn.classList.add('recording');
                listeningIndicator.style.display = 'inline-block';
                recognitionStatus.textContent = '';

                // Reset for new session
                currentSession = '';
                lastFinalResult = '';
            };

            return true;
        }

        function stopListening() {
            isListening = false;
            if (recognition) {
                try {
                    recognition.stop();
                } catch (e) {
                    console.error('Error stopping recognition:', e);
                }
            }
            recordBtn.textContent = 'Start Listening';
            recordBtn.classList.remove('recording');
            listeningIndicator.style.display = 'none';
        }

        // Toggle listening
        recordBtn.addEventListener('click', function() {
            if (!recognition && !initSpeechRecognition()) {
                return;
            }

            if (isListening) {
                stopListening();
            } else {
                // Clear previous text when starting a new session
                englishText.innerHTML = '';
                try {
                    recognition.start();
                } catch (e) {
                    console.error('Error starting recognition:', e);
                    alert('Could not start speech recognition. Please refresh the page and try again.');
                }
            }
        });



        // Translate text using the Flask API
        async function translateText(text) {
            try {
                if (!text || text.trim() === '') return;

                chineseText.innerHTML = 'Translating...';

                const response = await fetch('/translate', {
                    method: 'POST',
                    headers: {
                        'Content-Type': 'application/json',
                    },
                    body: JSON.stringify({ text: text })
                });

                if (!response.ok) {
                    throw new Error(`Server responded with ${response.status}`);
                }

                const data = await response.json();

                if (data.error) {
                    chineseText.innerHTML = 'Error: ' + data.error + '';
                } else {
                    chineseText.innerHTML = data.chinese_text;
                }
            } catch (error) {
                console.error('Error:', error);
                chineseText.innerHTML = 'Error: ' + error.message + '';
            }
        }

        // Initialize the page
        document.addEventListener('DOMContentLoaded', function() {
            // Try to initialize speech recognition
            initSpeechRecognition();
        });
    </script>
</body>
</html>
        ''')

    app.run(debug=True)