Smooth teleoperator: The rise of the remote controller
Smooth teleoperator: The rise of the remote controller
When an interviewer pressed Postmates VP Ali Kashani last year on whether the company’s use of teleoperation technology was an “admission” that AI alone can’t solve all of the challenges its robots encounter on sidewalks, Kashani swiftly retorted: “That’s a strategy, not an admission.”
Postmates, the on-demand delivery platform Uber is acquiring for MYR 11,225,177,125.51 ( $2.7 billion), is one of a number of companies developing autonomous sidewalk-traversing robots that deliver goods to homes and offices. Underpinning its service are human teleoperators who can step in and guide the robots when required. While AI-driven job loss has been hotly debated in recent years, mounting evidence suggests AI will also create jobs — like teleoperation — and open up the talent pool.
San Francisco-based Postmates has employed the services of Phantom Auto, a company founded in 2017 to build remote communication software that integrates with all manner of unmanned vehicles, from robo-taxis and delivery robots to forklifts and yard trucks. Operators can use the software to monitor fleets or draw a path for a robot to follow. When necessary, they can even take over and control the vehicle directly.
Excitement about an autonomous revolution has given way to “autonomous disillusionment,” with the prospect of fully self-driving cars retreating further into the future, despite impressive advances made over the past decade. Before driverless cars hit the mainstream, companies will need teleoperators to deal with all the “edge” cases on the roads, such as disorderly parking lots, roadworks, or stray animals.
“I’ve been doing autonomous vehicles for a long time, and in 2014 everyone thought that by 2018, 2019, or 2020 that these vehicles were just going to be driving themselves,” Phantom Auto cofounder Elliot Katz told VentureBeat. “And now people realize this is a very complex problem. You need a human in the loop today, and [probably] 40 years from now.”
Teleoperation has already been used to explore the world’s oceans and defuse bombs. Amid the COVID-19 crisis, however, teleoperation — whether taking control of a vehicle remotely or offering indirect “remote assistance” — could take on greater importance, as it minimizes social contact. Teleoperation could also open up roles to an aging or less physically mobile workforce.
“People who would otherwise not have been able to operate a forklift — say, someone with a physical disability or someone who has gotten to an advanced age where their [physical] skills have atrophied a bit … they can now operate a forklift,” Katz said. “That’s something that we didn’t even think of before, but it has been a topic discussed across the board with most of our customers.”
Remote work setups could also lead to “labor arbitrage,” with companies taking advantage of cheaper labor costs in other locales. Phantom Auto’s technology allows anyone to control a robot, taxi, or forklift from thousands of miles away, meaning a warehouse in a premium location can access a remote workforce with lower wage expectations.
“In Silicon Valley, let’s say that you have to pay MYR 83.15 ( $20 )an hour to a forklift operator,” Katz continued. “If you can now hire forklift operators in Kansas, or anywhere for that matter, there’s labor savings and you’re still getting the exact same output.”
Then there are potential safety benefits — in the U.S. alone, the Centers for Disease Control and Prevention (CDC) estimates that around 100 workers are killed and 20,000 are seriously injured in forklift-related incidents each year.
“Operating a forklift involves a lot of risk, as they are picking up and dropping off large pallets, sometimes at great heights — and there can be accidents,” Katz added. “So [by] removing humans from … the warehouse, you’re eliminating that safety risk.”
Truck on
Einride, a Swedish company developing electric autonomous trucks, started hiring for remote truck operators earlier this year and plans to retrain former truck drivers for the roles — although the training program is still a work in progress.
“As this is a brand-new role and the start of a new profession entirely, we are still developing the training regimen for our operators,” said Einride founder and CEO Robert Falck. “Holding a heavy vehicle license is a requirement for the job, however, so the rest is additional training related to the uniqueness of the role and developing the protocols for future remote operators.”
The company, which has raised north of MYR 133,039,136.30 ( $32 million) from big-name backers like Ericsson, is still in the hiring phase. But it is eager to talk about a future in which teleoperators control multiple autonomous trucks from a single remote station. In autonomous mode, the operator monitors what’s going on to make sure everything is running smoothly. But with the tap of a button, they can take control to ensure the truck is safely maneuvered into a parking bay, for example.
Above: Einride teleoperator controlling a truck remotely
If any other vehicle in the fleet requires assistance, a message flashes on the operator’s screen, prompting them to switch screens and take control.
Above: Einride teleoperator switching to control another truck
A few months back, Einride revealed it would start developing human-driven trucks with some of the underlying technology from its fully automated vehicles, including electrification and the telematics hardware that delivers data to its freight mobility platform.
While Einride has said “diversification” was always part of its plan, the move away from purely autonomous vehicles highlights some of the hurdles involved. Teleoperation is close to being a viable mainstream technology, but it will rely on the widespread proliferation of autonomous vehicles, mobile robots, and associated networking technology.
“We are currently using this technology [teleoperation] at customer sites and on public roads in Sweden, notably at the DB Schenker facility outside Jönköping, so it is already on the market,” Falck said. “The biggest challenge will be scaling the service, as that is dependent on the proliferation of 5G on a much wider scale.”
Teleoperation could eventually turn trucking into something approximating a 9-to-5 job. Because drivers would be able to control vehicles from anywhere, they wouldn’t have to put in long days on the road and sleep in motels or truck cabins at night.
“Teleoperation has the potential to be as widespread in the future as truck driving is today, transforming what it means to be a trucker to … a more hospitable profession with more regular hours,” Falck continued.
Third-party service
Israeli startup Ottopia has been building out its teleoperation platform since 2018. It completed work on a minimal viable product (MVP) last year before deploying it commercially with a handful of (undisclosed) paying customers, according to founder and CEO Amit Rosenzweig.
“Our customers are the organizations who develop all sorts of autonomous ground vehicles — delivery robots, forklifts, AGVs (automated guided vehicles), yard trucks, excavators, taxis, freight trucks, combines, and so on,” Rosenzweig said.
Above: Ottopia CEO Amit Rosenzweig
Ottopia develops software that works with most off-the-shelf hardware, including Nvidia, Intel, and ARM architectures. And as with other startups developing teleoperation services, it’s betting many — if not most — companies will prefer to use a third-party teleoperation provider rather than build the infrastructure in-house.
“Apparently, those companies have a lot on their plate already, plus it’s extremely difficult to build a reliable product that can really provide the needed teleoperation KPIs (key performance indicators) — for example, sub 100 milliseconds glass-to-glass latency at a 99.999% video availability,” Rosenzweig continued. “[It takes] many millions of R&D dollars spent on the right engineers and methodology to actually build a reliable teleoperation platform. Just like it’s faster and safer for those companies to just buy a camera or a lidar from a third party, it’s also faster and safer to buy a teleoperation platform from a third party.”
Cellular connectivity is pivotal to Ottopia’s offering, and to those of others in the space. However, Ottopia has previously stated that it isn’t waiting for 5G to come into its own — instead, it’s going all-in on 4G LTE. Although the startup’s team readily admits 5G will enable remote driving that’s “more efficient, at a lower cost” and will “unlock new use cases,” they consider 4G to be good enough for now — though they’ve found this to be a tough sell. Rosenzweig said one of the biggest challenges has been “testing — specifically, proving to ourselves and to our customers and partners that our platform works in a huge variety of cellular network conditions.”
But the company remains undaunted. “We follow a very strict methodology. Over the last 20 months, we have recorded, cleaned, and analyzed more than 3,000 hours of high-fidelity cellular data from multiple countries. That data is used to train our machine learning algorithms to provide superior network performance,” Rosenzweig said.
The teleoperator job itself is not particularly challenging, beyond the skill set a regular driver or operator would have. To carry those skills over, the remote station is usually designed to replicate a vehicle, with steering wheels, brakes, accelerators, and so on.
Above: Ottopia teleoperation station
The amount of time it takes to train someone depends on what it is they’re controlling — but we’re talking days, rather than weeks.
“We’ve trained people, and it is based on experience — [but] around two full days to become fully comfortable,” Rosenzweig said. “That is, assuming you’re starting with a person who used to be a regular driver or forklift operator in their previous job. It’s a bit different when dealing with robo-delivery, because people didn’t have a previous job of driving a robot. Therefore, for delivery robots it could be three to four days to become fully comfortable.”
Teleoperation won’t necessarily provide a “bridge” to full autonomy. While the humble elevator used to have human operators paid to control them and give passengers peace of mind, those jobs are long gone, replaced by buttons and safety mechanisms that connect to the outside world. Rosenzweig sees parallels to autonomous vehicles, but cars are obviously much more complex than elevators, and lawmakers may always require someone able to take over if needed.
“I don’t ever see the regulators saying, ‘We don’t need a backup anymore, this autonomy thing is so solid it will never stop or have an issue,'” he said. “No one will agree to put their family in such an autonomous vehicle if it doesn’t have any human backup whatsoever, even 20 years into the future.”
Connections
Many teleoperation companies have foundations in Israel, including Phantom Auto and Ottopia. According to Rosenzweig, Israel has “strong roots” in all the main technologies needed to build teleoperation technologies, including cybersecurity, video compression, and optimized communication (e.g., forward error correction [FEC], low latency, encryption).
Another Israeli startup, DriveU.Auto, recently raised MYR 16,629,892.04 ( $4 million) after spinning out of LiveU, a renowned specialist in HD video transmission. DriveU.Auto is a teleoperation connectivity platform for autonomous vehicles that focuses on ultra-low latency and “high reliability” across networks. DriveU.Auto CEO Alon Podhurst said customers are already using his company’s technology on public programs, but he declined to divulge names.
“The main challenge today is getting a reliable low-latency link with high video quality from the vehicle to the remote control center,” Podhurst explained. “This is so hard because standard video does not operate well in dynamic conditions of bandwidth and latency. Operating at latencies of less than 100 milliseconds, any capacity issue has an immediate impact on the video, something you would not even notice in a voice call or when downloading a file.”
DriveU.Auto aims to overcome these issues with a dynamic video encoding technology coupled with “cellular bonding,” which achieves higher bandwidth by combining modems. This helps it cope with unpredictable and fast-changing network conditions while supporting high-resolution video, audio, sensor data, and more.
“The bonding solution maximizes the performance of the networks for the specific needs of the teleoperation service,” Podhurst added. “The dynamic encoding provides the best 4K video quality, yet adapts to lower resolution without losing a frame.”
Assistance
Russian tech titan Yandex has been developing self-driving cars for years. As with others in the space, it has remote capabilities to support the development of driverless vehicles, but its focus is on “remote assistance.” Yandex doesn’t plan to enable full teleoperation capabilities in its vehicles due to the inherent technological restrictions.
“We don’t think directly remote-controlling a vehicle in real time can ever be safe enough, as it relies on cellular connection which is almost never 100% stable for long durations,” Yandex self-driving car head Dmitry Polishchuk said. “Thus, we are developing autonomous vehicle technology that will enable a car to safely navigate public roads without the need for a sustained internet connection.”
For this, Yandex said it has developed proprietary remote assistance software that’s optimized for its self-driving system. Yandex anticipates using human intervention only for “corner cases,” where the vehicle can’t decide what course of action to take. In such situations, the car will slow to a halt and send a request for backup. This approach is particularly well-suited for environments where network connectivity is limited, given that it doesn’t require the “same stability and bandwidth as actual remote control,” Polishchuk added.
“This means providing a vehicle with additional information or instructions on demand remotely so it can continue navigating autonomously,” he said. “For example, if a lane is blocked as a result of a road accident and the only way to drive around it involves a forbidden behavior, such as crossing a lane marking, we can send the vehicle a permission to cross the marking for this particular event. The vehicle will then analyze the situation and make the maneuver when it is safe to do so.”
To support this remote assistance approach, Yandex said it’s developing “special protocols” that enable faster data delivery between the vehicle’s sensors and remote operators, providing the operator with all the information they need to assess the road situation. Polishchuk said he believes self-driving technologies will eventually get better at solving corner cases independently and the need for remote assistance will decrease. But he said the vehicles will likely always need remote intervention capabilities.
“The world is very complex and constantly changing,” he said. “We believe autonomous vehicles will always be challenged with new corner cases, which may require some kind of human intervention. They may also experience scenarios in new countries and regions they’ve never dealt with before, which will likely require remote assistance.”
Uber has been another prominent player in the burgeoning driverless car industry, and it’s developing in-house teleoperation technology, combining “proprietary and industry-standard protocols,” according to Jon Thomason, VP of software engineering at Uber’s Advanced Technologies Group (ATG). Like Yandex, Uber is taking a lighter approach to teleoperation — keeping humans “in the loop,” rather than in control.
“Remote vehicle assistance allows the operator to suggest maneuvers and verify that various actions would be safe and effective and allows the operator to monitor the action as it happens, but it’s the autonomy system driving the car,” Thomason stressed. “Our system allows for monitoring in real time, as well as having the ability to notify the operator when human assistance [is required] — typically to unblock trip progression.”
Teleoperators are trained in the autonomous system and remote assistance technology, but they don’t train specifically in steering, braking, or other driving maneuvers.
“We believe that teleoperation, also known as full remote driving, is not a good interim step to autonomy, and [we] are not planning to do it,” Thomason added. “The path we’re pursuing is human-in-the-loop. Assisting the autonomy system with long-tail events that the system cannot handle without assistance will probably be around for a long time.”
Smooth (tele)operator
There has been a flurry of activity across the teleoperation realm in the past year. Voyage, which spun out of Udacity and last year raised MYR 128,881,663.29 ( $31 million) to commercialize community-focused autonomous taxis, recently launched Voyage Telessist, a software and workstation offering for remote operators.
Above: Voyage’s Telessist Pod: A custom-built workstation for remote operators
Electric micromobilty startups Tortoise and Go X recently kicked off a pilot in Georgia that allows customers to beckon an electric scooter through a mobile app. While the scooters have full autonomy built in, the companies are using teleoperation as a bridge until residents become accustomed to seeing riderless scooters.
Postmates rival DoorDash recently snapped up teleoperator startup Scotty Labs, though it has yet to share plans for the acquisition. But given that DoorDash has been piloting autonomous robots for several years already, the possibilities are readily apparent.
All this activity suggests we could be on the cusp of a major industrial shift, one that eases geographic restrictions on the labor pool, enhances safety, and widens the job market for aging or less physically mobile workers. Teleoperation might not be a new concept, but with the proliferation of high-speed internet, it could play a key role in taking autonomous transportation mainstream.
Comments
Post a Comment