Voice control of iCubSim via calling Google Cloud
=================================================

I. Speech-to-text calling GoogleCloud

Option 1. The mobile phone as a microphone

There is an application for mobile phones with Android (about 7.x and more) 
in the directory apk which is calling Google cloud to convert the recorded 
voice into text, then transmit the text into PC via TCP. 

It is good to understand the network layout for the work with the application.
Your PC is connected to LAN and gets an IP address, e.g., 158.195.28.200
Your mobile would need to be in the same or accessory (hierarchically lower) 
network to see your PC. If we have wifi on the LAN (as in the home we usually have), 
we connect the mobile phone to the same network. However, now we have no wifi, so 
we have established one. We have attached one wifi router to the LAN, and it has the 
address, e.g., 158.195.28.130 (you can ping it by: ping 158.195.28.130)
Behind this router, there is a local network 192.168.2.xxx, the router is 
192.168.2.1, and your mobile phone (please connect it to this wifi) has IP, 
e.g., 192.168.2.22. The router 192.168.2.1 is its gateway so that it can access 
any node in the upper networks: you can browse on the internet and also connect 
to any PC on the LAN 158.195.28.xxx 
Please find your IP address by the ipconfig utility (launched from cmd)

(If you try this exercise at your home or use your notebook in the terminal room, 
it is easier since both your computer and mobile phone use the same wifi. 
Just find the IP address of your PC by the ipconfig utility, and your smartphone 
connected to your wifi can see it. Pay attention to antivirus that can block
communication - apply "pause firewall" tentatively)

How to install the apk into your mobile:
the best way is to employ your connection to the internet and install total commander 
into your mobile via Shop; it is free. Our apk, unfortunately, is not available in 
the Shop (I tried, but its quality is not acceptable for the Shop).
Now connect mobile to PC via USB cable (we can provide) or wireless as you would like 
to download photos and upload *.apk file from directory apk\ver2.
After transfer, use the total commander to find the file on your mobile phone
and click on it and confirm the installation.

Then test that the apk works. Launch telnet\tcp-server.bat, which launches TCP 
server listening on port 7171 on your PC. Now open the apk on your mobile phone. 
Enter your IP and port in the form like 158.195.28.200:7171 and click on the button 
Connect. As a result, "connect" appears in the TCP server window. Then click on the 
button Speak and tell something and then be as quiet as possible to let the mobile 
phone detect that the recording is over (this is the biggest fallacy of the 
application, if somebody uses a drum in the other room, it will never stop
recording). Finally, you should receive the recognized text in the TCP server window.

Your mobile is ready. (Switch off the TCP server; otherwise, it occupies the same 
port you need to use by other tasks in the following.)

Option 2. Microphone is available on your PC

The alternative solution of voice recording and recognition is based on the desktop 
application desktop\recorder\recorder.exe It provides the same service and transmits 
the recognized text to localhost, port 7171

(You need a working microphone on your computer. There is no onboard microphone
in the terminal room, but you can employ the microphone in the usb camera. Power the 
camera from usb and insert audio jack to the headphones connector in the front of
the minitower. Then run application "Dell audio", select the "Microphone" tab, 
and put it "On".) 

II. Text-to-speech (a speaker or headphones needed)

Now let us make the speaker enabled (not required at your home, just in the terminal room): 
Speaker is inbuilt in the monitor, but need to be enabled.
Launch "Sound" and select "DELL S2721QSA" as you default playback device.
Then launch the "Dell Audio" and put Speaker "On". Finnaly adjust volume in 
the "Volume Control".

Open cmd in the directory with the training content and type:
> set ESPEAK_DATA_PATH=.
> espeak "hallo student"
and adjust the sound intensity by the above procedure; about 20 percent would be 
an acceptable value.

Now try the same from Python: launch from cmd (we suppose your working directory 
is e:\users\lesson12)
> pushd e:\users\lesson12
> python speak.py "Hallo student"

III. ChatGPT

See prompt.py for calling ChatGPT.

Ask for the api_key and insert it instead of '...'

Call ChatGPT by the run-prompt.bat

By default, we call ChatGPT with
- command "Remove the object from the table!",
- situation "There is a ball on the table."

that are turned into prompt:

 "You are a robot named iCubSim. You can move with two hands, you can touch "
 "and hit. You can perform the following actions: Smile(True or False), "
 "Sleep(miliseconds), Say(text), Hit(), StandardPosition() and waveWithHand(). "
 "Please answer with json code. Return action parameters as a list. Please "
 "always use the Say action at least one time. Situation: There is a table in "
 "from of you. There is a ball on the table. 'Remove the object from the "
 "table!' Actions:"

we get a response, e.g. (it varies)

{'choices': [{'finish_reason': 'stop',
              'index': 0,
              'logprobs': None,
              'message': {'content': '[\n'
                                     '  {\n'
                                     '    "action": "Say",\n'
                                     '    "parameters": ["I will remove the '
                                     'ball from the table"]\n'
                                     '  },\n'
                                     '  {\n'
                                     '    "action": "StandardPosition",\n'
                                     '    "parameters": []\n'
                                     '  },\n'
                                     '  {\n'
                                     '    "action": "Hit",\n'
                                     '    "parameters": []\n'
                                     '  },\n'
                                     '  {\n'
                                     '    "action": "Say",\n'
                                     '    "parameters": ["The ball has been '
                                     'removed from the table"]\n'
                                     '  },\n'
                                     '  {\n'
                                     '    "action": "Smile",\n'
                                     '    "parameters": ["True"]\n'
                                     '  }\n'
                                     ']',
                          'role': 'assistant'}}],
 'created': 1715692884,
 'id': 'chatcmpl-9OmNornIUgpw7ztnIOzlICKcEnCoz',
 'model': 'gpt-4-0613',
 'object': 'chat.completion',
 'system_fingerprint': None,
 'usage': {'completion_tokens': 104, 'prompt_tokens': 113, 'total_tokens': 217}}
 
from which we can extract the resulted actions are: 

Say(I will remove the ball from the table)
StandardPosition()
Hit()
Say(The ball has been removed from the table)
Smile(True)

Try to modify the command or situation.

Try to modify the robot's description in the prompt. 
(As a result the parsing can fail and may need to be modified.)

IV. Integration

Great, we are ready to program something nice with iCubSim.
At first, launch run-renew.bat to install iCubSim and put it into the standard 
position. Confirm OK in the Qt Motor Robot GUI. 
By R (restore), W, Z, and F, V adjust the viewpoint on the iCub to see its body 
and its face. Anytime later, launch again the run-renew.bat  to establish 
standard positions of the robot motor (including emotional face expression)
(this you cannot launch from cmd until you define PATH according to the bat file)

At first, we provide coordination of speaker with lips of iCub. Since this is 
parallel processing, we employ the agent-space architecture. Before the synthesizer 
starts, we write to Space that the robot is speaking. After the synthesis ends, 
we write there that speaking is over. Then we add agent LipsAgent (see lips.py), 
which reflects this information and moves with lips until the speaking ends.
Launch run-speak-lips.bat and click on the iCub window to listen and to see 
the speaking.
(Unfortunately, it takes some time to switch the espeak synthesizer, so the iCub 
moves lips also a few times after speaking, please ignore it.)

Now open the main.py (launch it only via run-main.bat), which adds two additional 
agents: The first agent - the ReceiverAgent - listens on port 7171 and expects 
text commands, and writes them into Space as data named "text."
The second one - the ListenerAgent (coded directly in the main.py) - process these 
data. At the moment, it obeys only the command "Say X," on which it says X and moves 
its lips during that. Launch the run-main.bat and then open putty on raw TCP on 
localhost and port 7171 and enter some command (if you make a mistake, do not use 
backspace, but press enter and type again) e.g., say: "you are brilliant." 
As a result, iCub should say that. Now launch your mobile application and press 
connect again and say the same; you should get the same result. (ReceiverAgent is 
listening to more clients). When connected, iCub moves its hand; if you say 
"say hello," iCub says "hello." To prevent troubles with lowercase and uppercase
letters, we transform the received command to lowercase letters before processing.

Your task is to add some additional voice commands to the ListenerAgent.

Using regex matching, you can employ the prepared function "match" for text processing.
Regex pattern is a unique object in Python and has the form r'xxxxxx' 
where xxxxxx is the pattern. 
(The pattern frequently employs '\,' so we put 'r' in front of the string to tell to
Python that it is a raw string, so '\' is not interpreted)
Few examples:
r'hallo' corresponds to comparison with string 'hallo'
r'h...o' is matching hallo but also hello or hexyo since . is any character
r'h.*o' is matching hallo but also ho or haaaaaalllllllllllllllloooooo since * means repetition
r'h.+o' is matching hallo but not ho, since + means repetition but at least one time 
you also find some portion of the text; just mark them by ():
r'h(.)llo' not only match hallo and hello but also you can ask for group 0 and you get a or e
this groups are available by the prepared function matched(), so for 'hallo' matched()[0] provides 'a'

However, much better quality, you can employ by calling ChatGPT.

Some hints on what you could implement:
- if you say something nice, iCub smiles, and if something ugly, iCub is angry.
- if you say move with the left hand, iCub shakes the left hand for a short time

