IoT vendors ignore basic security best practices, CITL research finds

Turning on compile-time security features is easy. So why aren’t more IoT device makers doing so?

Adding flags for security features when building IoT firmware binaries would dramatically improve the security of IoT devices across the board. Almost no one is doing it, and the problem is getting worse, not better, according to new research from the CITL mass fuzzing project.

Cyber ITL is a non-profit Consumer Reports-style security laboratory that has so far automated the fuzzing of more than three million IoT firmware binaries released over the last 15 years. Its results are discouraging.

“It’s very easy to do,” CITL chief scientist Sarah Zatko tells CSO of IoT vendors’ failure to turn on basic compile-time safety features. “There’s no good reason not to do it, and they’re just not bothering.”

“I don’t think they are neglecting to do it on purpose,” she adds. “This isn’t really a case where someone made a conscious decision to exclude these safety features. More like benign neglect. It didn’t occur to someone that this is their job.”

Time for a post-build checklist?

IoT vendors could easily turn on these compile-time safety flags and check for them as part of their release management process. Good build hygiene includes checking to see if there is a more recent version of the compiler and making sure to enable basic security flags like ASLR, DEP and stack guards. While none of these security mitigations are magic, they serve as the airbags and seatbelts of the IoT world. Maybe they won’t prevent a crash, but they might just save your life.

It requires a couple hours of engineering work, one day tops, Zatko says. “If there’s some weird edge case where a particular combination of operating system and chips for some reason, it could become a more complicated issue, but for the most part should be really simple.”

Because the consequences of poor build hygiene compound as we continue to deploy insecure IoT devices, vendors need to start checking what post-build QA testing they are doing, or they may find themselves regulated to do so.

Given the free market has so far failed to encourage such responsible conduct from vendors, one wonders how long before a class break prompts kneejerk bad regulation, as Bruce Schneier fears.

IoT security: Free market or regulation?

The free market has so far failed to create incentives for vendors to ship products with strong cybersecurity, but Zatko is hopeful that large buyers can make a difference. “The free market on its own isn’t doing much,” she admits, “the needle hasn’t moved in 15 years…. If the people making large purchasing decisions started asking questions about build safety, that could influence vendor practices. Enterprise organizations, government organizations buy a lot of IoT products.”

“[Buyers] usually do have a checklist of security questions they ask before entering into a new agreement,” Zatko says. Including build safety questions in that checklist would force the vendor to actually check, and also let vendors know that big buyers care about build safety.

IoT firmware security gotchas

CITL’s research turned up some surprises–cascading points of failure IoT vendors ought to know about. Compilers and firmware toolchains like buildroot are key upstream dependencies that could do a better job of flagging security issues for developers at compile time. Also, MIPS is still a thing and needs special handling.

A lot folks have mistakenly assumed that MIPS is on its way out, but the hardware architecture has made a comeback in the IoT space, and in fact was the most common architecture that CITL encountered during their mass fuzzing exercise. The problem, they discovered, is that not every architecture is created equal from a security point of view.

“A lot of people think that if you take the same source code and move it to a different chip or a different architecture that the same safety and security features will hold,” CITL acting director Tim Carstens says, “but that’s really not true. The whole picture has to be considered when it comes to security.”

Turns out that enabling ASLR and DEP compile-time flags for Linux MIPS builds doesn’t work properly, and it could take years before the problem gets ironed out, deployed and universally patched (if ever). “Linux MIPS binaries have been easier to exploit by way of classic stack overflow attacks for over a decade, and continue to be so according to our examination of toolchain patches,” the CITL report says.

Worse, it seems that there is a lot of thoughtless code reuse happening in the IoT firmware space, with everyone assuming someone else did the security audit. “When we took a look at different products a lot actually had some set of binaries in common,” Carstens tells CSO. “That right there is an indication that many different vendors are using the same frameworks to construct their IoT platforms.”

Sensible safety defaults in the developer user interface–to both compilers and firmware build toolchains–would flow downstream and affect vendors who use those products, and the subsequent millions of people who will use those devices.

“Compilers themselves could provide better reporting on what safety features got implemented, a kind of ‘nutritional facts’ at the end of the compile process,” Zatko says. “It would be very easy for a compiler to provide that transparency and provide better feedback to the developers about what was just produced.”