This article was published in Unmanned Systems Technology Magazine June/July 2023 issue, written by Rory Jackson, and can be viewed online here: https://www.ust-media.com/ust-magazine/UST050/52/
TOWERS OF STRENGTH
Rory Jackson looks at the workings behind these UAVs designed for inspecting cell towers and other tall, high-value infrastructure
Aerial photogrammetry is one of the longest-running and most common UAS services in the world, and for good reason. Its value proposition comes not only from the usefulness of the data and analytics derived from missions, but also from the lives it can save.
Effectively, any tall infrastructure or geological feature that needs inspecting will bring some risk to its human surveyors, and people are regularly killed each year through accidental falls or other injuries.
The lives saved by UAV photogrammetry stand to go beyond even that: it is widely recognized that many of the devastating wildfires across the western US over the past few years were caused not by forests growing too wild but by critical infrastructure decaying over time and toppling onto dry, flammable tinder. The wildfires destroyed towns and villages, killing many people, particularly in the drier parts of northern California. Such tragedies could have been prevented through prudent maintenance of infrastructure.
Structures such as power poles and cell towers are particularly challenging to survey, even with UAVs. Each tower often requires hundreds of photographs, with resolutions fine enough to identify literally every bolt in the structure — the absence or blurring of even one will cause concern.
US-based Skyfish has built a reputation in recent years for its ability to survey these high-value structures with a consistency and resolution that is rare even in the UAS photogrammetry industry.
It carries out photogrammetry missions using primarily its M6 and M4 multi-rotor UAVs, but its capabilities are supplemented by expertise in technologies that include onboard computing, flight control and GCS design, as well as more obvious areas such as image processing, RTK-GNSS, and high-resolution cameras.
Its choice of components has enabled Skyfish to comply fully with the US NDAA (National Defense Authorization Act) — that is, with no parts imported from unapproved nations — and it has been vetted and certified by the US Army for the quality of its services.
The M6 and M4 are also set to be added to the AUVSI Green UAS List, a standard analogous to the Blue List of companies approved for US Department of Defense-related operations. That will certify that the UAVs meet the highest levels of cybersecurity and NDAA supply chain requirements for providing commercial services to the DoD.
Sony and photogrammetry
The seeds of Skyfish’s specialization in photogrammetry were planted early in CEO Orest Pilskalns’ career, who studied mathematics and physics before working for Lockheed Martin Astronautics as a research scientist, obtaining his PhD in computer science in the process.
“I spent cumulatively about 10 years in academia at Washington State University, with a focus on software engineering and geospatial information systems. It was during that time, in 2014, that I started Skyfish,” Pilskalns says.
“We moved the company to Montana and worked on prototyping for a year, particularly on navigation and mapping software owing to my background in computer science. We were essentially a software company trying to achieve high-end photogrammetry, but it soon hit us that the kinds of 3D models we aimed to create would also need incredibly high-end cameras, and a similar caliber of flight tools for carrying them.”
That led Skyfish to Sony, not only for its high-resolution cameras but also the level of support Sony could offer — which, for all practical purposes, seemed far higher to Pilskalns and his colleagues than that of DJI and other potential partners. Skyfish then began building its own UAVs, and soon had a complete UAS solution, integrating its choices of high-end sensors, IMUs, onboard computers and more.
“We then sought to build up a lot of knowledge about industrial photogrammetry,” Pilskalns says. “Each target for inspection and digital twinning will model differently, whether it’s a cell tower, a dam or a bridge, so we went out into the field with lots of different customers to model varying kinds of infrastructure.
“That was key to understanding how to handle, sharpen and refine our 3D models, how best to integrate Sony’s cameras, and also what kinds of components were best for our aim of producing fine-grain digital twins. As the saying goes, ‘Garbage in, garbage out’, and that applies so strongly to photogrammetry. You need to put everything you can into capturing excellent photos and data in flight, and then understand how to combine it.
“And we’ve been asked, ‘Well I’m using a Sony camera, why don’t I get the models you’re creating?’ We’ve uncovered about 50 little trade secrets along the way, some of them hardware-centric, some of them in the software.”
The company’s resulting specialty in cell tower inspections was showcased in 2022 during a competition arranged by NATE (the National Association of Tower Erectors), a US organization representing cell tower builders, in which UAV companies were invited to show how precise they could generate 3D reconstructions of a given tower.
Key to that was the question of whether any companies (be it through EO, IR, Lidar or other sensors) could provide a model so precise that NATE’s analysts could gauge the thicknesses of metal from it — that is, the metal on electrical boxes, antennas, structural components and so on — down to the sub-millimeter level.
This caliber of photogrammetry was the bar set to determine whether NATE’s members should start switching from human inspectors to UAS inspections. If metal thicknesses can be judged from photography and 3D models, the need for humans to climb the towers and inspect components first-hand is eliminated.
Skyfish emerged as one of few competitors capable of capturing the thicknesses of metals across the target cell tower, not only of pre-existing componentry but also of various small metallic ‘Easter egg’ objects hidden by NATE across the structure.
The M6 and M4
The M6 is a hexacopter measuring 174 cm across and weighing 10.4 kg when empty, with a MTOW of 17.7 kg and a payload capacity of 5.4 kg. The M4 is a quadrotor measuring 129.5 cm across, weighing 3.8 kg when empty and 10.9 kg when fully loaded.
The M6 can fly for 33 minutes and 47 seconds when carrying 1.87 kg (enough for a Sony A7R-IV camera and a Gremsy T3V3 gimbal), or up to 27 minutes and 51 seconds when carrying 4.54 kg. The M4 can achieve 38 minutes of flight with a 2 kg payload, or 53 minutes with 0.61 kg.
Both are fitted with subsystems for optimizing them for photogrammetry. For instance, their navigation systems are RTK-ready, with redundant antennas to ensure consistent coverage (from the UAVs’ side). Also, their autopilots and mission planning software include autonomous behaviors for different flight patterns according to the type of inspection target, including corkscrew-like patterns for mapping around tower-like structures or long narrow patterns for corridor mapping.
Also integrated is Skyfish’s Skymind computer, which provides integration and control of flight, navigation and sensor systems as well as edge computing for the gimbal and motors to ensure a stable and intelligent photography platform in all weather conditions.
Both crafts are battery-powered, have carbon composite hulls and propellers, and are monitored remotely — or controlled if necessary — through Panasonic Toughbook-based GCSs, which are often used to stream live video data from the UAV about the quality of the data it is capturing and its flight condition.
Survey preparation
Assembling and launching the UAVs tend to be straightforward. “That’s been part of the design since early on,” Pilskalns says. “There’s no point in having a great photogrammetry solution if users can’t deploy it in the field.
“One key part of setting up — and disassembly — is that we didn’t want the payload sensor to have to be attached and detached multiple times per day, even when moving from site to site. So the M4 and M6 fold at their arms, fit into cases and can be deployed within a few minutes once the arms are folded back out and the batteries are installed. The boot-up time for all the systems to come online is then between 60 and 90 seconds.”
The pre-flight planning on Skyfish’s GCS includes preparing a cache of local maps for fast and up-to-date georeferencing, as well as SRTM (Shuttle Radar Topography Mission) data for following the terrain’s elevation and inputting points of interest alongside navigational waypoints. The waypoints can also be adjusted for altitude requirements to assist with surveying vertical structures.
Naturally (as these aircraft are photogrammetry vehicles), setting up an RTK base station is a common part of survey prep. Most of the time that means setting up a tripod-type RTK unit nearby, or on a survey point by way of a plumb bob, with Skyfish’s software providing offsets where necessary.
“RTK processing isn’t specifically more valuable for cell towers than other industrial photogrammetry targets,” Pilskalns says. “In terms of geolocation, cell tower operators are already quite confident that they know where their towers are, it’s more the real-time corrections of modeling data that are useful.
“Recently, we flew with a moving baseline RTK system on one of the UAVs. We’re offering that now, essentially as part of a ‘Generation 1.5’ version of our UASs which are to incorporate some newer technologies. But that mounted RTK system is especially useful because, out of all the systems on a UAV that will suffer problems during photogrammetry, the most severely impacted will be the magnetic compass.
“Magnetic fields are easily disturbed by a lot of things, and anyone who’s flown a DJI UAV for photogrammetry will be familiar with the constant warning pings of ‘Reset Compass’, so even though we have back-up systems we’re moving away from relying on magnetic fields and more towards new RTK innovations for ensuring our heading is always correct.”
Gimbal
Capturing sufficient data about a particularly complex cell tower requires around 1000 photos, or just under an hour of total flight time. That, therefore, entails a quick battery swap partway through to enable the M6 or M4 to take off again to gather the remaining imagery.
For stabilizing the camera during a flight, Skyfish works with Gremsy, which builds a customized version of its T3 gimbal for Skyfish. As standard, the gimbal weighs 1.2 kg, measures 288 mm long and 237 mm at its greatest width, and features a heated and temperature-controlled IMU as well as eight connectors in its I/O module. These comprise ports for CAN, COM, SBus, JR, USB, AUX, and HDMI interfacing, as well as a 15–52 V power plug.
More functionality can be added via the AUX port, which has up to nine signal interfaces for controlling devices such as camera triggers. Lastly, Gremsy’s gMotion Controller is typically incorporated for 32-bit stabilization control and up to 2000 motor-actuator corrections per second.
“However, we further modify the gimbal they supply, and once we’ve taken delivery of one we tear the base off it and install another computing module underneath,” Pilskalns explains.
“One of our problems is that we move a lot of data from the camera for computer processing, and HDMI and USB-3 signals can easily get messed up by EMI. So we use some proprietary methods for moving and safeguarding that data. We also co-locate a computing module with the camera, on board the gimbal, to minimize the risk of EMI having an impact on the quality of that data.”
Although Skyfish is satisfied with the performance of the gimbal, it is now working on a next-generation gimbal in-house to further optimize its data collection.
Between versions, Skyfish anticipates carrying over its proprietary algorithms for remotely and autonomously controlling third-party cameras via the onboard computer and gimbal companion computer.
It notes that many other photogrammetry UAVs operate their own standard-issue, OEM cameras without issue, but often suffer all sorts of glitches and malfunctions when integrating a third-party camera — particularly high-end ones from companies such as Sony, which could otherwise give better resolutions and offer other performance-critical features.
Skyfish has worked closely with Sony to optimize its Skymind computer to automate changes of camera settings during missions. And although the Sony Alpha series is typically used (enabling resolutions of up to 61 MP) Skyfish emphasizes that other payloads can be integrated and used, including a range of Lidars such as GeoCue’s Trueview series (which package Riegl and Hesaisolutions), and thermal cameras from Workswell and Teledyne FLIR.
Integrating an entirely new payload sensor is typically achieved in a matter of days, and in some cases same-day delivery has been possible. Skyfish puts this quick turnaround time down to having full control over their onboard computing software stack.
Onboard computing
Skymind runs on AM 335x and AM5729 ARM cores and operates at 100 Hz to achieve fine control over the flight regime. Such tight control over the motors and interfacing with telemetry is key, as otherwise photographs could easily be geotagged incorrectly when the UAV travels at any speed over 3 m/s.
“Within that is our selection of a very high-end MEMS IMU,” Pilskalns adds. “The onboard computer interfaces with that directly to adjust the flight system accordingly.
“Previously, Skymind was built using Texas Instruments processors, but recently we’ve moved to Broadcom chipsets. They’re extremely powerful and fast for systems that are only a few matchboxes in size.”
Skyfish does not currently build its 3D models in real-time, however. It might do so someday, and customers have expressed interest in such a capability, but to do so in less than an hour during flight would require mounting a supercomputer on the M6 or M4.
“For now, we use AWS as the host for our cloud computing and processing the survey data to construct our digital twins,” Pilskalns says. “We might spin up as many as 50 cloud-based engines simultaneously if a customer wants a tower model turned out quickly, by which I mean in less than an hour, but it’s also the most computing-intensive and expensive option.
“It means firing up a lot of high-end processor cores, so if the customer’s need isn’t time-sensitive we can do it over a couple of days at a far lower price.” Skymind is therefore not designed for processing purposes but for intelligent, smooth control of the camera, gimbal and UAV all at once, including autonomous decision-making for maintaining safety during flights and deriving quality photography at the same time.
In particular, Skymind autonomously aims the gimbal and focuses the camera correctly at key infrastructure targets for modeling, while handling key settings for the camera and the gimbal’s motor controllers, and governing these on the basis of incoming image data. It also performs more advanced functions such as real-time geometric projection, to speed up and improve the 3D mapping of objects in the camera’s view.
In addition to optimizing the data collection, Skymind handles autonomous navigation tasks, including the execution of the pre-programmed flight path and monitoring the GNSS and inertial data to ensure precise and correct guidance. Other onboard sensors for obstacle avoidance could also be used, depending on the UAV’s mission and configuration.
“One reason for capturing images at the top of our camera’s resolution capabilities is that we’ve worked with Sony for a long time now on how to engineer Skymind to understand all the environmental factors that could affect image quality, and to respond rapidly in flight by adjusting the UAV to compensate for those factors in the photography,” Pilskalns says.
As well as integrating ARM cores and Broadcom chipsets, the onboard computing currently comprises 19 PCBs (besides the main motherboard), including boards for the smart BMS, for interfacing with the gimbal computer, and other subsystems. The main motherboard carries the flight control systems as well as the radios and processors for RTK-GNSS and data links.
“We constantly monitor how well Skymind handles customers’ tasks,” Pilskalns says. “We release software updates to tune performance, and although we’ll periodically make modified versions, say for customers who want to alpha-test some special capabilities, whole new versions of Skymind tend to be built around a 2-year development cycle. That’s how long it takes before we decide it’s worth investing in switching to next-generation processors and other electronics.
“Between new versions though, we still have a lot of flexibility thanks to controlling the stack. For example, a government agency came to us recently with a specific sensor they wanted on the M6 which no one had ever integrated before, along with an automated return-to-base capability to execute in response to a single motor failure. They also needed very stable flight during that automated return, and all of that needed to be ready for a competition 4 weeks down the line.
“We were able to do it though, and win the competition and the business that followed from it.”
Real-time data
Although the digital twins are not produced in real-time, Skyfish notes that more and more customers want real-time feedback from their photogrammetry.
That is natural, given the need to inspect and repair industrial infrastructure nationwide, as well as the time constraints that asset owners must abide by for customers such as utility service providers. Real-time monitoring and data processing is therefore still a critical part of the company’s UASs.
“For faster real-time processing, one thing we’ll do is install an encoder on the gimbal, which is connected to the camera via an HDMI cable, to process the camera feed and send it to the pilot so that they can always see what the M6 or M4 are seeing, even on a fully autonomous mission,” Pilskalns explains. “It’s just to make sure things are going well, and really high-resolution video is critical for them in that regard.”
This direct connection is vital to protecting the camera data from EMI, particularly around cell towers that will often be emitting microwaves that can easily interfere with highly vulnerable HDMI connections. Once encoded, the stream is exported from the encoder via an Ethernet-type standard with inherently higher error checking than HDMI.
“Figuring out the relationship between the encoder and the radio is key to success in any sort of long-distance operation,” Pilskalns says. Data is coming out of the encoder at a very high rate, and as the signal-to-noise ratio in the data link goes up, the system has to be able to react to that in near real time.
“We have a number of workarounds for that in our stack, but the most important thing is that encoder and radio vendors are aware of each other’s integration compatibilities and incompatibilities, or are even working together on a solution in which their products work together.”
The radios have been co-developed over several years with a couple of suppliers, and Skyfish has also been working with a partner to enable a near real-time data link for direct transfer of survey data from the UAVs to the cloud. With that link, Skyfish anticipates that images can be taken and fully uploaded to the cloud before the next image needs to be taken. That means entire photogrammetry data sets can be uploaded and possibly even processed into an actionable digital twin by the time the UAV lands.
While this will enable in-depth analysis of survey information from anywhere in the world, Skyfish anticipates that its operators and customers in the field will continue using its GCS. Running on the Panasonic Toughbook, its design will be familiar to many, and it features a very bright screen that is key for the effective scrutiny of video data and health parameters coming in from the M6 and M4 when using it outdoors.
“We’re also working on a less expensive GCS for customers with tighter budgets,” Pilskalns says. “It isn’t a Panasonic system, but it will still be NDAA-compliant.”
Data analysis
The production of hi-res digital twins from the photogrammetry data is critical for running various stress, wind and load analyses so that cell tower operators and owners can visualize where maintenance and repairs are imperative.
The digital model ‘product’ is typically constructed after an hour of cloud processing, and increasingly the company is able to produce its 3D models within 45 minutes.
Skyfish says that just a few years ago, it would have taken at least 24 hours to produce models with high-megapixel resolutions. The speed of modeling and analytics now though is such that survey teams may be able to return to cell towers later in the same day to re-inspect them for signs of critical damage or indicators of whether any bird species nesting on the towers are on an endangered list and hence cannot be disturbed.
“While it’s hard to speak directly about the various trade secrets by which we get really detailed digital twins from our photogrammetry, one of them of course is the geotagging,” Pilskalns says. “For instance, understanding angles correctly by extracting and using some very particular metadata from our IMUs and RTK systems is key to building 3D shapes that are accurate to the sub-millimetre level.
“We’re also using metadata, which a lot of people don’t, and it’s also about collecting enough of it — and doing so appropriately — before making sure it’s correlated robustly to the photographs in near real-time. That’s key for photogrammetry as well as straightforward inspections.”
Once the digital twin is constructed, it can be exported into CAD and other types of systems for detailed review. Skyfish’s own solutions include tools for studying measurements, area polygons, volumetric objects, drop lines and slice planes, with machine learning for contextual understanding of which parts and components are distributed throughout the model.
“We have a patent-pending automated system that’s like an MRI in reverse, where we slice up the digital twin, piece it back together, and meanwhile use 2D analysis tools to get an accurate understanding not just of what the objects are but what condition they’re in. That is all without ever doing anything manually, all the way through to the reports we export for customers.
“Then our AI systems work in the background to recognize 3D objects, put bounding boxes around them, and calculate a load of metadata about them that we can send to a CAD program. The CAD then rebuilds the idealized structures in exactly the way and format the infrastructure engineers need them for their analyses, including gauging the thicknesses of metals in pipes, antennas, electronics boxes and so on.”
Recovery
A key design target of the autonomy stack in the M6 and M4 has been configurability of their behaviors at the end of a survey. Pilskalns observes that with most COTS aerial photogrammetry solutions, such as those from DJI, UAVs are limited to autonomously returning to their launch point, which often runs counter to what customers actually want.
“We have many customers who want the UAV to hover at a point, capturing data until every last ounce of battery energy is exhausted, and then land in place,” he says. “Even if you make a smart return-to-home algorithm — and we’ve made a few different ones, with different parameters that can be adjusted ad hoc — photogrammetry customers have wide and varying sets of requirements, so anything’s possible when it comes to recovery.”
Future plans
In addition to working on its various next-generation components for its UAVs and photogrammetry products, Skyfish is expanding its range of customers, particularly towards large organizations in the private sector and the US government. These customers’ applications will be focused primarily on digital twinning of key infrastructure such as telecoms and power utilities, as well as roadworks and other transportation infrastructure.