Preface

Many air conditioner manufacturers nowadays provide units that can connect to Wi-Fi and be controlled with an app. The use case is understandable - turning on your A/C and cooling down the house while you’re on your way home sounds like a dream. However, using the manufacturer’s app is often not as convenient as using more mature home automation software, that might be integrated with voice assistants from Google and Amazon.

So, it shouldn’t be a surprise that when one of us moved into an apartment with a smart A/C controller, hacking it to work with Home Assistant, an open-source home automation software, was prioritized highly. Little did we know that going down this rabbit hole, we would find some glaring security vulnerabilities that exposed the controllers’ users to complete takeover from the internet, amongst other issues.

Central A/C Wi-Fi Controller

The controller in question was one produced by Electra (an Israeli air conditioner manufacturer). It is used to control residential central air conditioner units, and is installed in many new apartments in recent years. Electra provides a specialized app that is used to connect the controller to a Wi-Fi network, and then to control it over the Internet.

The first step, of course, was to search for existing integrations and libraries that would allow interfacing with the controller. One such library existed, but it simply emulated the app to communicate with Electra’s API server, that would contact the A/C controller over the Internet.

While that was an option, we wanted to see whether a tighter integration was possible. Could we access the controller over the local network, both to improve latency and to provide control when the Internet is down?

Local Network Analysis

We started with running a port scanner, nmap, against the controller (which, unsurprisingly, did not return any results). It was clear that the contoller was acting as a client that connected to a remote server (otherwise, port forwarding would be requird - not something the average user would be capable to provision). So we used tcpdump to monitor the network traffic going out of the controller from the moment it was plugged in. We used a router running OpenWRT for this functionality.

Our traffic logs revealed that upon powering up, the controller first resolved the domain alk2da.messaging.internetofthings.ibmcloud.com (the resulting IP was 62.90.149.38). It then pinged that IP for a couple of minutes, sending out a ping request every second or so. Following that, the controller connected to that IP over port 1883, and several seconds later disconnected and connected again over port 8883. That last connection would remain operational indefinitely.

These ports (1883 and 8883) are commonly used by MQTT (for regular and secure connections, respectively). And upon attempting to connect to that server over these ports, we saw that this was, indeed, an MQTT server.

Electra MQTT Server

Our first reaction was optimistic - we would just set up the local DNS server to point to a custom MQTT server, and this would cause the controller to connect to it, instead of using the original server by Electra (remember, at that point security vulnerabilities weren’t on the radar, and local control was our only goal). Then we would explore the messages it sends and receives over MQTT, and control it locally - easy-peasy, right?

Apparently, someone did take security precatuions when developing the controller (though mostly in vain, as we will see later). The controller would refuse to connect to our MQTT server, and based on the connection log on the server side, the issue was with the certificate (or lack thereof). Of course, we tried generating a self-signed certificate for the required domain (alk2da.messaging.internetofthings.ibmcloud.com), but that didn’t help either - the controller used certificate pinning to verify that it was connecting to the correct server. Replacing the MQTT server with a local instance was no longer an optoin (at least, not without patching the firmware to skip the certificate verification). So we attempted the next best thing - connecting to the Electra MQTT server, in the hopes that we could control the A/C from there. Not local, but at least as direct as possible, skipping the API used by the app.

Of course, to connect to the MQTT server, you need a username and password. And the connection on port 8883 is encrypted via TLS, so sniffing that wouldn’t work. We could’ve tried the unsecured connection, but at the time we missed those few packets and concentrated on the encrypted connection. Luckily, getting the username and password proved easy with the help of the app.

Extracting the MQTT password using the app

The pairing procedure of a controller through the app is pretty straightforward:

1. If the controller is not in the pairing state (i.e., it’s already been paired before), reset it by holding down the button on its side for about 15 seconds.
2. Add a new air conditioner in the app - follow the instructions and input the required information (serial number, real SSID to connect to, etc.).
3. When asked, choose the Wi-Fi hotspot opened by the controller (begins with ELRSSID).
4. At this point, the app will connect to the A/C and complete the pairing process.

But what exactly is happening at the last stage? To find out, we opened a hotspot of our own, with a name that looks like a real controller hotspot (i.e., starts with ELRSSID). The app expects the hotspot to have a matching password (same as the SSID, but ELRSSID replaced with ELRPASS). This password can be found out if you choose to troubleshoot the connection via the app - it will tell you to connect to the SSID manually, and will provide you with the matching password.

Once we had the SSID up and running, we fired up a sniffer and attempted pairing with the app. We quickly saw the app was attempting to communicate with the host 192.168.1.1 on port 80. So we started a basic TCP listener on that port, just so the connection succeeds, and found out the HTTP request sent out by the app (for privacy reasons, the real values for SSID name, password, and token have been replaced with dummy ones):

POST / HTTP/1.1
user-agent: Dart/2.10 (dart:io)
content-type: application/x-www-form-urlencoded
accept-encoding: gzip
content-length: 87
host: 192.168.1.1

__SL_P_UST=3&__SL_P_UPW=RealSSIDPassword&__SL_P_UTO=SecretToken&__SL_P_UAP=RealSSIDName

It’s easy to see that the request contains 3 important pieces of information:

a. The name of the SSID for the controller to connect to (in the __SL_P_UAP parameter). b. The password of the SSID above (in the __SL_P_UPW parameter). c. The token used for the MQTT server. This was deduced by further investigation of the pairing process, that showed no other requests were performed by the app - so this token was the only additional piece of information passed by the app to the controller.

Connecting to the MQTT Server

Upon having the token, we also needed to find out the username and the client ID to use for connecting to the MQTT server. These were found out by investigating the domain of the server, alk2da.messaging.internetofthings.ibmcloud.com. You can see that this server seems to be part of the IBM Watson IoT Platform, and its documentation explains that the username should be use-token-auth, while the client ID should be in the format of d:orgId:deviceType:deviceId (note: this platform is deprecated, and it’s possible the above links will not work in the future).

Upon using the above information, together with the token extracted from the application, we used MQTT Explorer to connect to the server, and - lo and behold - saw an enormous amount of data:

We could see hundreds of air conditioner units. For each of them, we could see their IP address (giving an approximate location), as well as detailed state, including the set temperature, operation mode, current temperature, etc. It was a massive privacy issue, but the worse was still to come.

Extracting the firmware

At this point, the security implications of our innocent research were becoming pretty clear, and it was time to take a look at the device’s firmware. And besides, we still wanted to control the device, so knowing what kinds of commands it was listening to was a requirement.

Once we opened up the plastic enclosure of the controller, we saw that it basically had 2 main chips on its board: an ST Microelectronics STM32F030x and a Texas Instruments CC3100.

On the side opposite to these chips, a bunch of headers seemed like perfect candidates for UART/JTAG access. By testing the continuity of the pins on those headers to the different pins on the chips (the pinout can be found within datasheets for the corresponding chips, such as this), we established the purpose of each header. Of interest to us was the JTAG/SWD header, that we promptly hooked up to our trusty TIAO Tumpa, and used OpenOCD with the correct (STM32F0) configuration to dump the firmware from the STM32 chip.

Loading up the firmware into Ghidra is a straightforward process. We knew from the datasheet that this was an ARMv7 chip, and correctly presumed the firmware to be little-endian. For the interested reader, there are many resources available about analyzing ARM firmware files - they explain the presence of the vector table at the beginning of the firmware, the setup of the base memory address (found in the datasheet/programming manual), etc.

Controlling our air conditioner

Our first task after running the initial auto-analysis of the firmware was to find the mechanism for controlling the device over MQTT. At this point, for the uninitiated, it’s worth outlining how MQTT works. This is a simple message queue protocol, where connected clients can subscribe to topics, and publish messages to topics. A message published to a topic will be delivered to clients subscribed to that topic.

When we first connected to the Electra MQTT server, we subscribed to the wildcard # topic, thus receiving messages published on any topic on that server. Most of the messages were published by the air conditioner units - but of interest was the topic iot-2/evt/CMD/fmt/json. That topic seemed to reflect the last command sent by the app to an A/C - but not limited to our app instance or our A/C unit. The message on this topic would be constantly overwritten, as the air conditioners around the country were in use - whenever we would change the temperature on our unit, we would see the message corresponding to our command for a few seconds, and then it would be replaced by another message (probably targeting another unit).

The above CMD topic was very helpful - it allowed us to understand the format of the message used to control the A/C unit (it was a very readable JSON). However, it was clear that the topic was not the one used for controlling the air conditioners - as there was no identifiers that would mark which air conditioner should be controlled by that message. Our assumption was that the command message was published to a topic that was filtered when using wildcards (i.e., you had to subscribe to it explicitly in order to receive its messages), and messages on that topic were mistakingly reflected on this generic topic, as well.

So, while we knew the content of the command message we wanted to send - we still had to find out the topic that the message should be published to. That’s where looking at the firmware code helped. We searched the code for strings containing parts of the above topic, and quickly found the code that subscribes to the relevant topic:

Armed with the actual CMD topic (which contained the client ID of the device, based on its MAC address), we could finally attempt controlling the air conditioner without the app. It worked! Sending a command JSON to the correct topic indeed caused our air conditioner at home to change its operation based on the contents of the JSON message. Bingo! The app was no longer required, and the air conditioner could be integrated into a home automation system using regular MQTT messages. A job well done. There was just one more thing to try…

Controlling other air conditioners

What would happen if we sent the command message to a different topic (that is, use the client ID of a different air conditioner as part of the topic)? We found a colleague who had a similar air conditioner, and asked him for the MAC address (that appears on the controller). Using the MAC address allowed us to know the correct client ID. It’s important to note, that we still used the user ID, client ID, and token from our air conditioner. We sent the command message to the appropriate topic - and our colleague informed us that the air conditioner responded!

Terrific?

Terrifying.

Anyone can take the Electra Smart app, generate credentials for connecting to the Electra MQTT server, connect to it, and start controlling arbitrary air conditioners. Still, could be worse…

Connecting to the MQTT Server… Revisited

At some point during our research, we made a mistake and copied the wrong piece of text to paste as the password for the MQTT server. We realized the mistake, but the login button was already clicked. Surprisingly, the connection worked - the login succeeded, and we were presented with the regular flow of messages from the air conditioners. That was unsettling. So for our next login attempt, we did not provide a password at all… The login succeeded. Issuing commands after that login succeeded, as well.

And so, the actual situation was as following: there was an MQTT server, open to anyone over the Internet, that allowed anonymous logins with permission to control any Electra Smart air conditioner connected to the network. With such brilliant security measures in place, we decided to look further into the device’s firmware and see if we can find any more gems like this.

Hardware Revisions

At the beginning of the research, we ordered a couple more controllers, under the assumption that we might damage the first one during our attempts to extract the firmware (a fear that was eventually unwarranted). As we received the devices, that both looked similar to the first one on the outside, we opened them up to look at the board. Surprisingly, one of the boards looked quite different from the other 2. The first controller, pictured in the previous sections, was marked as Revision 005. The second controller was marked as Revision 004, and looked very similar to Revision 005. However, the 3rd controller was marked as Revision 008, and the chips on the board were quite different. Gone was the STM32F0, replaced by a Chinese GigaDevice GD32F303RCT6. And the CC3100, used simply as a WiFi module and controlled by the STM32 chip in the older revisions, was replaced by another Chinese chip - MXChip EMW110. Going from Western chips to Chinese alternatives sure sounded like cost-cutting. But that could also mean that the firmware would change substantially - after all, these chips aren’t 100% interchangable with the old ones. So the firmware extraction commenced once again.

This time, the firmware extracted from the GD32F3 chip (replacing the STM32F0) proved to be a lot leaner, with quite a bit less functionality. For example, we could not find anything related to the MQTT server. So we investigated the EMW110 chip a bit closer. We found out that it had a UART header conveniently located on the other side of the PCB. We connected it to a terminal, turned on the board, and were greeted with a plethora of information.

From China With Love

The first thing that caught our attention was an HTTP PUT request to a Chinese server, right at the beginning of the boot process (payload JSON formatted for brevity):

PUT /api/device/update_version HTTP/1.1
Host: cn-api.topband-cloud.com
Content-Type: application/json
Connection: Close
Content-Length: 145
Token: <redacted>

{
    "version": {
        "wifi_software": "WIFI2230",
        "wifi_hardware": "XXEMW110",
        "mcu_software": "MCU00004",
        "mcu_hardware": "HARD0000",
        "mcu_protocol": "00000004"
    }
}

Accessing this API requires a token - its generation can be seen earlier in the log:

[3610][Debug: http.c:  80] Token source: product_code=04CNJJ0002product_secret=<redacted>mac_address=<redacted>pass_code=ECSGVSQOGC
[3612][Debug: http.c:  95] Token: <redacted>

We simply used the token used by the device - it seemed pretty static, and did not change between boots. In fact, we later confirmed its generation in the firmware code, and it indeed depends only on the above values (product_code, product_secret, mac_address, and pass_code), which are not changed during the device’s lifetime.

We were worried that there would be some sort of client certificate verification (we’ve seen update servers that worked like that, to prevent unauthenticated firmware download), but luckily, no such measures were in place on this server. The above API endpoint seems to notify the Chinese server, responsible for its firmware updates, about the currently installed version. It seems to simply reflect the version data sent to the server.

So, there is some sort of firmware version management, but we still haven’t seen an actual update in action.

Exploring the subscribed MQTT topics

We then verified that the device communicated over MQTT, just like in previous versions. Indeed it did. However, it connected to a seemingly different server - iot.ecpiot.co.il. In practice, however the DNS resolved to the same IP as the previous server. Due to the mismatch in domain names, TLS errors were ignored (that will be important later).

Upon further inspection of the boot log, we could see that the device subsribed to multiple topics:

[...]
[8764][Debug: cloud.c: 966] Subscribing topic2: iot-2/cmd/OPER_CMD/fmt/json
[...]
[9786][Debug: cloud.c: 990] Subscribing topic3: <redacted: MAC Address of the unit>/update/#
[...]
[10814][Debug: cloud.c:1014] Subscribing topic4: iot-2/cmd/CMD/fmt/json
[...]

The last topic, iot-2/cmd/CMD/fmt/json, was familier (used for controlling the device). The first topic, iot-2/cmd/OPER_CMD/fmt/json, seemed to be similar to the last one (maybe a remnant of older version). But what about the 2nd topic? Its name lookd very intriguing… An update! To understand the format of the message, we had to refer to the firmware itself.

Extracting the firmware… again

With the hardware used for the newer revisions being completely different, the firmware extraction approach had to be adapted, as well. Specifically, based on the boot logs above, it was clear that the communication logic now resided on the WiFi chip, whereas in the older revisions the WiFi chip was only used as a peripheral. Therefore, we had to refer to its datasheet to learn its pinout and JTAG access procedure.

After soldering wires to the correct points on the EMW daughter board, we again used the Tumpa to connect to the board over JTAG, and extract the firmware.

Analyzing the EMW110 firmware

Loading the firmware into Ghidra was a bit more challenging this time. It took multiple attempts, and a deeper dive into the datasheet of the actual SoC on that board (called MOC108), to realize that we were dealing with a much older architecture, ARMv5. Once we loaded the firmware with the correct architecture selected, we could start exploring the code.

Using the relevant strings (such as the debug messages seen while processing commands received over the MQTT topics), we quickly found the functions responsible for handling data over subscribed MQTT topics. At that point it was a matter of simply following the code, identifying the JSON functions used (knowing the expected format of the CMD packets helped immensly with that), and then reconstructing the expectd JSON message describing an update:

{
    "type": "ota",
    "data": {
        "wifi": {
            "wifi_hardware": "XXEMW110",
            "wifi_software": "WIFI2231"
        }
    }
}

Forcing a controller to update

With the update message in hand, we were ready to send it to our controller. And, lo and behold, the logs indicated that it requested an update! The request was done to the following URL (quite similar to the URL used to notify the server of the currently used firmware).

GET /api/device/firmware?device_code=***redacted***&version=WIFI2230&type=wifi&beta=0 HTTP/1.1
Host: cn-api.topband-cloud.com
Connection: Close
Content-Length: 0
Token: ***redacted***

And the response indicated an available update:

{
    "product_code": "04CNJJ0002",
    "version": "wifi2239",
    "type": "wifi",
    "file": "http://file.topband-cloud.com/topband/smartsys/media/attachments/021fb2e0c5aff742e2eb7a61f82052ab/wifi2239.bin",
    "beta": false,
    "md5": "021fb2e0c5aff742e2eb7a61f82052ab",
    "create_date": "2022-08-25 08:48:13",
    "update_date": "2022-08-25 08:48:13",
    "slug": "9645001f-f3cc-459d-8989-c199085e584c",
    "is_active": true,
    "force_update": false
}

It’s important to notice several glaring issues in the above process:

1. The initial connction to the cn-api.topband-cloud.com server is performed over regular, non-encrypted HTTP.
2. The firmware download URL is, once again, served over regular, non-encrypted HTTP.
3. There is no signature verification, the only verification performed is the check that the MD5 sum is correct.

All of the above means that when the controller is connected to a compromised access point, an attacker can push arbitrary firmware to the controller.

Combining the findings

Following is a complete attack vector description, that provides an attacker with full control over newer (revision 007/008) controllers that haven’t been provisioned (such controllers can be found in new apartments and in new A/C installations, before the user connects them to the app).

1. When the A/C controller is not initialized (i.e., has not been connected to a Wi-Fi network, as happens in many homes), an attacker can provision it to connect to a malicious AP.
2. Once connected to the malicous AP, the controller resolves the domain iot.ecpiot.co.il, and connects to it over MQTT without any TLS verification. Therefore the attacker controlling the AP can cause the controller to connect to a malicious MQTT server under their control.
3. After the controller connects to the malicious MQTT server, the attacker can issue an update command, that causes the controller to request an update from the update server.
4. The update server is contacted over HTTP, and so can be spoofed by the attacker, serving a malicious firmware for the consumption of the controller. The controller will download and flash the firmware provided by the attacker, that can contain arbitrary malicious code.

It’s important to reiterate, that any newly installed Electra A/C controller, that hasn’t been provisioned by the user (i.e., connected to a network), is vulnerable to the above attack - which can be easily carried out from the street. Walking around the city next to recently completed buildings reveals many SSIDs indicating such unprovisioned controllers. And while an update could resolve it, it can only occur after connecting the controller to the network, at which point it might have been already compromised. So the only recourse to verify that a controller hasn’t been compromised is a recall.

Disclosure

The vulnerabilities were reported to the Israeli National Cyber Directorate (INCD) on October 30th, 2022. After several requests for status updates, we were informed that the information was finally passed down to Electra (several weeks after the initial report to the INCD). The vulnerabilities were not initially confirmed by Electra (in fact, the company responsible for wrongfully dismissing them was Vayosoft - the subcontractor that developed the software for Electra’s smart air conditioners).

In the end of January, 2023 - almost 3 months after the initial report - we managed to schedule a discussion between us, the INCD, Electra’s CISO, and a representative from Vayosoft. During the discussion, we went over all of the vulnerabilities and explained their impact. Electra’s CISO was very understanding and accommodating. Meanwhile, the representative from Vayosoft tried to downplay every finding. Specifically, he maintained that the revision 008 board that we received was pre-production and not shipped to customers, and that the current software on those boards performs SSL certificate verification and pinning - both statements being blatantly false.

Electra/Vayosoft quickly addressed the most critical issue - the complete lack of authentication/authorization on the MQTT server. Within days, the issue was resolved, and connecting to the MQTT server no longer allowed communication with other air conditioners (and required a proper password). Electra also sent out another board for us to test - based on Vayosoft’s claim that the new software implements SSL pinning. We inspected the board (revision 007, quite indistinguishable from revision 008 that we had before), and saw that nothing was different - the connection was insecure, both to the MQTT server and to the Chinese update server. We informed Electra about our findings, updated the INCD, and the CVEs were issued shortly thereafter, on March 12th, 2023.