A multitouch control for the smartphone is much more similar to traditional remote controlling to an aircraft.
Therefore this was a test to how good a quadrocopter can be controlled with such an App.
In this case two technologies can be distinct. The first is bluetooth as radio link, the second is controlling via a smartphone.
Bluetooth as radio link is really easy to use because it’s buildin within the smartphone and on the aircraft it’s also easy to handle as a serial device. The drawback is a low range. Therefore it’s not perfect for fast aircraft like some sort of planes.
Controlling with a smartphone is nice, because it’s really handy and a lot of people does already have one. The drawback is really bad haptical feedback. It’s harder to handle it without the feeling where the sticks are. But if the thumbs are not lifted, the controlling of a quadrocopter for example, is really good.
Some notes on the multitouch implementation. Like on a simple single touch App it is the same callback method for multitouch
handling, the onTouchEvent method. Within the parameter from the type (class) MotionEvent there is a getAction method which has the information about what touch the event has called, first or second one. It’s not a really a easy to guess api, but it works and within the net are a lot examples for handling it.
A usual Android ™ smartphone includes all necessary technology to use it as a remote control for a quadrocopter.
- Accelerometer and magnetometer to measure the attitude
- Touch sensitive display
The app does only have 300 lines of code without any optimizing. For attitude angle measuring the important methods are SensorManager.getRotationMatrix and SensorManager.getOrientation. With these information the pitch and roll input can be controlled. The position of a finger on the touch display can be determined with the method onTouchEvent(MotionEvent event). This is used to control yaw and thrust.
That’s all to control the QC. But most QC flight controls need some special stick position to arm their motors. For this a button or something similar is needed. If this button is pressed, the special stick positions are send for a couple of seconds to arm the motors. After that the control of the QC is as desribed.
The first tests for video transmission an o UAV was made with Skype ™ which has some drawbacks. Therefore the software has been extended to also transmit video from the Android ™ smartphone to a laptop. Technically this is realized through the preview function on the Android API. Currently 15 frames per second are used. Every frame is converted to a jpeg picture and send as UDP package to the ground station software. This approach has several advantages:
- Frame rate and video quality can be chooses
- Only one app to have UAV control, telemetry data and video downlink
- Video can be mirrored in software if it’s necessary to attach a mirror in front of the smartphone cam.
- Video can be turned
- Should be possible to take pictures or record movies in HD at the same time the preview is transferred.
In the video you see 3 setups. On the first one the WLAN access on the DSL router was used for both, the smartphone and the laptop. The latency was about 0.2 seconds.
On the second one the mobile access (vodafone ™) on the smartphone was used while the laptop still used the WLAN DSL access. This time the latency was about 0.26 seconds. The next mobile tower is only 150 meters away. At this video there a some errors in the pictures which is a bug in the software and has to be fixed.
On the third time both devices used two separate mobile access (vodafone ™) and the latency was about 0.36 seconds which is not too bad for a 3G connection. Especially because of this scenario has to use a kind of “proxy” server for the connection between the two devices.
After controlling the FPV car via Internet the next step was to enhance the system to get a quadrocopter with it into the air.
For this goal the number of channel has to be increased from two to four. Also the software needs some logic in case of a transmission lost on both, the bluetooth connection and also the udp internet connection. In both cases there is implemented a failsave regulation to get the copter to the ground on an well-defined manner.
The fist test was not successful, because the MWC stability mode was not stabile enough for the test and the smartphone mounting was too flexible too see the copter movements early enough. Therefore a DJI Naza was ordered but took some time for delivery (more than a month). This flight control is really stable and adequate for the experiment. To handle the flexible mounting the copter is picked up in traditional manner with a direct sight to it. After a touch down it can more easily becontrolled via fpv.
The test was successful so far. Even with the not really great O2 net connection it is full controllable but also with a delay on the video transmission at about 0.1 up to 1/4 sec. This is also the reason to use a really stable copter at stable flight mode.
Imagine what is possible with a 4G (LTE) connection and an optimized copter that can stay for an hour or more in the air.
- Full HD video RX/TX with LTE
- Multicopter with more than one hour flighttime
I worked on my 3G data link based FPV car with the Android Smartphone. Improved the following things
- Wide angle lense
- “4 channel” RC transmitter for control input generation
- Automatic reconnect function on both sides, Smartphone and Laptop
The first thing is an optional lense to get view angel of nearly 90 degres. That helps a lot to navigate and control a vehicle with first person view. Another effect is, that the cam gets more light from the environment and that improves also the speed of dark-bright adaption.
The second thing was to find something with which it is easier to control a vehicle than with the mouse pad. An old remote from an xufo is used for that. To get the stick positions an Arduino reads the potentiometer values and send it via FTDI interface to the laptop.
The last was, to improve the software on Smartphone and Laptop a bit. The main thing was to introduce a detection if connection is lost on both sides. If this is the case, the smartphone software try to reconnect and the laptop software informs the user that connection is lost and wait for a new connection.
Now the car is much more controllable, even with a poor 3G data link as you can see on the following video:
In the last post the connection between the Android smartphone and an RC Car was established. Now it’s also possible to have a connection from the smartphone to the Internet for both, video streaming and remote controlling (RC). Both is needed for a first person view controlling of a vehicle.
But why controlling a FPV car via Internet? Because theoretically there is no limit about the range except the battery capacity. Normal FPV gear and RC components have without the need of a special radio license in Germany a range about half or one kilometer with clear line of sight between sender and receiver. On ground it’s even less, maybe 100 or 200 meter at best. With a 3G data link it depends on the telecommunication net which is big on most places. That’s the reason for doing a little bit “research” on this topic.
The video can be produced through the on board cam of the smartphone. The cam video can be streamed to the net. I use Skype ™ for this purpose, because it’s fast and easy available for my Galaxy S2 and support video telephony.
The tricky part was the car controlling. I decided to establish the connection from the smartphone to my laptop. This worked well even through the home router. Next part was the protocol. TCP is not the perfect option, because it can be very slow with all the checks in it. UDP is much better, because it doesn’t matter if some of the controlling information don’t reach the car. It’s much more important that – new – controlling information coming fast to the car.
For this the smartphone act as the client. On the laptop site there is a small server software which communicate with the smartphone. This software wait for the initial UDP connection of the smartphone and send after that control information via UDP to the smartphone. The smartphone transmit these information via Bluetooth to the Arduino.
A first test show that this all works together but with a lack of accuracy and some delay.
I think the software part of this approach can be improved for faster and better reactions and also more reliability. This approach also use 3G data link and IPv4. You can imagine how this will be improved if IPv6 and 4G data link is available.