Raspberry Pi Joystick- Eye-gaze & Voice control edition

Hi all,

 I found a Xbox Adaptive Controller project on github that allows the use of bluetooth keyboards & mice, with the help of a Raspberry Pi. The Pi plugs directly in the XAC and the KB & Mouse connect to it over bluetooth, emulating a joystick and buttons. 

After reading about it, I realized the project creator is right here at MMC, @MMC_Milad :slight_smile: https://github.com/milador/RaspberryPi-Joystick Nice work, Milad.

I have an idea for a remix of this project that could help a lot of people get back in to console gaming at a low cost:

  Would it be possible to alter the original code so that the Pi would be able to receive mouse cursor data and global keystrokes of a PC over bluetooth? The Pi would then turn that data in to joystick movements and button presses for the XAC.  I'm guessing the PC would need to be running some sort of software to be able to send that data to the Pi.

This would allow players to use their usual eyetracker or any other mouse-moving adaptive equipment to control a joystick in XAC games.

This would also enable players to use any keystroke software, including voice control and eye-gaze screen regions to press controller buttons.

I’m not sure if the joystick movement of the current build is analog or digital. My idea would be to capture the desktop’s x/y position of the mouse cursor and convert it to a value that fits within the range of an analog joystick.

The middle of the screen would have an adjustable dead-zone. Ideally, the overall size and shape of the aiming area would also be adjustable.

To use this with an eyetracker, you would calibrate the eyetracker and turn on it’s mouse-moving software on windows and afterwards switch the monitor input to the gaming console.

Does any of this sound do-able? I’m still too new to programming to know what the limitations are. :grin:

Thanks,
Jonathan

1 Like

Hi Jonathan,

So your idea is to use PC eyetrackers and send data to RaspberryPi and then from Raspberry Pi to XAC?

You can use a simple python script to read mouse movements and keystrokes on PC and sent them through Bluetooth or USB through RaspberryPi. You would need additional hardware to act as USB/Bluetooth HID.

Options:

  1. PC eyetrackers -> Python code in PC -> Python serial code to output data as serial -> use a bluetooth enabled dev board to convert serial to bluetooth HID - >pair dev board with RaspberryPi ->XAC (Too many parts)

  2. PC eyetrackers -> Python code in PC -> Python code to output data as raw bluetooth packets -> change my xac interface code to accept raw bluetooth packets instead of USB HID data - >pair dev board with RaspberryPi ->XAC (Better option)

  3. PC eyetrackers -> Python code in PC -> Python serial code to output data as serial -> use a TTL to serial cable to connect PC with raspberrypi -> change my xac interface code to accept serial data instead of USB HID data ->XAC (Better option )

  4. PC eyetrackers -> Third party software like Across -> RaspberryPi ->XAC (Better option )

Hope it helps to get you started.

Milad

Milad,

Thanks for your feedback. That Across software look really interesting, that solution wouldn’t require any new coding, would it? I have a Raspberry Pi 3 I can try this on.

Jonathan