Devon Bleibtrey, Author at Auklet

Posts by devonbleibtrey

Python Programming Language

Redefined Python Agent For The Edge

August 28, 2018 Posted by Internet of Things 0 thoughts on “Redefined Python Agent For The Edge”

Ever since we attended PyCon this year, we’ve been hard at work rebuilding our python agent. We took the feedback we received from the event, compiled it with the ongoing input from our beta users and set out to redefine how we manage APM for Python. The result is a more than ten times more efficient agent. To accomplish these gains, we changed how we perform performance analysis, ship data, and serialize packets.


Python Performance Analysis

The most significant amount of feedback we received was that our agent created an unacceptable amount of overhead for almost every user’s app. This is why reducing the impact of our performance monitoring was essential to this release. We had fun and learned a lot doing the necessary refactor of function monitoring and data transmission. In the end, we were able to maintain the accuracy of our performance issue identification while obtaining the 10x improvement we were striving to achieve. Although this was a significant improvement, we continue to work towards having zero impact on customer applications.


Shipping Data From Python with MQTT

We’ve been planning to move to MQTT for a while and took this opportunity to take the plunge. MQTT provides all the functionality we need to transmit data securely and aligns with our roadmap almost perfectly. MQTT allows us to continue to capture unhandled exceptions and poorly performing functions while devices are not remotely accessible. This offline support means, as with our first implementation, we never have to miss a problem.

MQTT also enables us to provide better device identity management. With this new identity management we’re now able to identify the following problems:

  • Troublesome devices whose application is performing worse than the rest of the fleet
  • Regional & environment based performance issues
  • Function performance anomaly detection relating to multiple devices


MessagePack Serialization

In our first iteration of the python agent, we utilized JSON for serialization because of its integrated python support. This out of the box support enabled customers to refrain from adding more third-party packages than they already required. However, over the course of our beta, we chatted with our customers and found that another third party package wasn’t a big deal. Knowing that JSON wasn’t exactly size or performance friendly, and having this input drove us to make the shift.

Based on most of our user’s input we settled on investigating Protocol Buffer and MessagePack. Due to Protocol Buffer’s support from Google, having the most requests, and being slightly more performant we started down the path of integrating it. Our first red flag was the comment on the README that stated “[Python Protobuff] may be more buggy, and it is known to be pretty slow at this time.” We had heard great things about it, so we didn’t want to discount it without at least giving it a try. Unfortunately, after a week of getting it integrated some nasty benchmarks, we decided we should transition over to MessagePack due to these exact issues.

After moving to MessagePack, we couldn’t be happier. MessagePack serialization on average produces a packet that is 33% smaller than the JSON equivalent. This data reduction has been a massive win for our beta users. Paired with some additional modifications to our backend we’ve been able to drastically reduce the amount of data our agents produce. All of this comes with the added benefit of a reduction in the serialization overhead itself. Which in turn reduces the impact Auklet has on customer’s applications.


Automotive Safety Integrity Level B Compliance

Aside from making improvements to the python agent’s performance we also improved the overall test coverage. We undertook this effort to adhere to our internal quality standards, reduce the potential of our agent introducing problems into applications, and to meet the demands of our Automotive customers. With these improvements, we are proud to announce our 0.6.0 release meets all ASIL B requirements. We will maintain this level of integrity throughout all future releases of the Python agent. If there are other compliances you need to meet to use an APM tool, please reach out.


1.0 Release Coming

With these reductions in overhead, data transmission, and issues we’re finally ready to draw our beta to a close. It has been a fantastic ride, and we cannot thank our beta users enough. Everyone has been integral to helping us pinpoint bugs and areas for improvements. We’ve received amazing input on what helps developers solve problems before their customers find them and couldn’t be more thankful. We’ll be releasing version 1.0 of our python agent before the end of September. This transition does not affect our existing beta users or any users that sign up in the interim. It also has no impact on our engagement with our customers. Please keep the feedback coming. We always love hearing your thoughts!

Improving Software Through Process as You Enter the Internet of Things

August 23, 2018 Posted by Internet of Things 0 thoughts on “Improving Software Through Process as You Enter the Internet of Things”

One of the most requested tasks we are called into enterprises for is to help them adopt an agile methodology. Often these organizations are giants in their industry and have reached these positions using a cocktail of waterfallV-model, and self-built processes that have resulted in billions of dollars in revenue. The end products they’ve produced range from sub-par to breathtaking excellence. Therefore the internal push often does not come from the executive level but instead mid-level management who are neck deep in trying to bring software development into the organization to embrace the internet of things revolution. They are not only working to change the methodology but also the culture of the company and transition from, one and done, to a platform that is consistently updated week over week, month over month, and year over year.


Historically automotive, home appliance, aerospace, and other industries that produce products with high price tags have operated in a multi-year long product cycle. Companies often did and still do customer feedback reviews at specific milestones, like 90 days after initial purchase. Outside of these engagements with a sampling of customers, mechanic visits, customer trips to the dealership, or other infrequent touch points, there wasn’t much of a feedback loop. Customers expected this, and due to the logistical, manufacturing, and regulatory wonders that had to be executed to deliver these products, companies developed their workflows around conservative practices that introduced minimal levels of risk from year to year.

Manufacturers focused on mechanical and electrical systems with little to no software integrated into their products. Then once development was completed, aside from recalls, or extremely minor tweaks once the product shipped, it wasn’t updated. It resided with a customer for 5–20 years and never saw a security update, feature enhancement, or optimization. Did they sell without any of these? Yup. Did they get positive reviews from customers? Yup. Did customers expect to pay for ongoing updates and enhancements? Nope.

This is where we often see the disconnect between the two primary segments we interface with at customers who are looking to transition to agile. On one side are mid-level managers and their software teams who often have a background in consumer electronics or web apps and have a culture of building, releasing, breaking, and iterating, continually bringing new features to customers. On the other are upper executives who often have a background from within the organization itself and have come up through a time where mechanical and electrical were king to the customers and the company. They know full well the complications that can come from breaking anything in a regulatory heavy industry and rightly hold concerns ranging from security to recalls. Both sides are intelligent and capable and are trying to make their products the best they can be.

Adopting The New Process

This is where we often enter the fold. Many of our customers believe the tool choices or the process definition will be difficult and it can be. We use our experience from rolling out implementations at various size organizations as well as in our internal development activities to help pick the right tools for the teams or organization we’re assisting. We then assist with training and getting people up to speed on the selected toolchains. Teaching them the discipline necessary to ensure complete traceability, from requirements, to the lines of code that fulfill them and the unit tests that validate them. The real difficulty comes with redefining much of the existing culture, one that is built around old methodologies and compliance requirements. For example, in automotive, there’s a safety rating process that is used to assign a ranking to each system in the vehicle. Automotive Safety Integrity Level B (ASIL B) basically dictates that there must be a well-defined development process (which we cover with our end to end traceability solution) and code must have 100% test coverage.

This creates headaches out of the gate. We understand the reasoning behind ASIL B (and the other requirements to the various ASIL ratings) but far too often find teams that are using this as the guiding light for their culture. If you look at it historically, standards like this make sense. When first bringing software into the vehicle many organizations were attempting to define blanket requirements where no previous standard had existed. So management needs to ensure ASIL B is met to meet certain requirements and of course who doesn’t like seeing 100%. On paper, it just looks good. This butts up against software engineers who laugh at the requirement knowing that 100% test coverage means nothing. Then the advocacy commences and we hear many of the same things:

One Side: We are only being paid/tracked to reach 100% test coverage and obtain ASIL B. If we meet the requirements set by the customer we’re golden.

The Other: 100% test coverage doesn’t mean 100% validation and will not guarantee fewer issues or security holes occur in production. It’s short-sighted to use this as a metric.

Both are valid points. If you’re not paid based on the proper criteria, then it pushes your culture to advocate for the wrong things. Although this can result in short-term gains for a company or an industry, it can have negative long-term consequences. Such as an industry-wide culture that promotes:

  • Writing tests to only test happy path scenarios
  • Writing tests only to the requirements that were defined before kickoff of the project
  • Not preparing to push updates, monitor the performance of your application in production, and iterate on your tests and functionality
  • A higher percentage of manual tests executed by validation teams
  • Higher cost when an issue is found or worse when a recall occurs

Yes, surprisingly all of this relates to being agile. The point is if upper management doesn’t place funding or structure their proposals to customers properly and shift what they are marketing or advocating on, the mid-level management and engineering teams are incentivized not to follow an agile methodology and fall more and more into a waterfall method. An example would be in a proposal stating something like:

  • Shall be ASIL B Compliant

As opposed to:

  • Shall continuously iterate over the lifetime of the project to identify and validate against happy path, malicious inputs, expected outputs, known scenarios, and issues identified through the usage of real customers based on what can be gathered by application performance monitoring solutions.

In the first example, revenue correlates to only achieving 100% coverage for the initial requirements defined in the specification. In the second we shift to an iterative process that will require ongoing funding for the given solution and sets the expectation that there will need to be development beyond the initial launch of the product. If you’re a supplier, making these shifts might mean you need to work with customers to educate them on why doing this will be advantages to them. If you’re a mid-level manager in an organization building software, you may have to coach upper management, purchasing, and sales on these new needs. Otherwise, leadership will continue to see these elements as binary objects that have either been accomplished or not. Remember software is not a widget.

Benefits of Shifting

Once these expectations shift, it becomes much easier to facilitate an agile development process. By incentivizing to prepare for managing product updates over its lifecycle as opposed to when it goes into production, it pushes teams to:

  • Build a platform that can be reused and updated over time rather than throw away code that only meets the existing requirements
  • Create tests for the platform that continuously improve on letting fewer issues or security holes into production
  • Not write tests for the sole purpose of increasing coverage percentages and checking boxes

All of which helps all parties by:

  • Setting the groundwork for customers to be more involved throughout the process rather than defining a fixed scope up front that cannot change without a significant amount of overhead
  • Facilitating a better end product for end users and improving relationships
  • Reducing the need for costly recalls
  • Reducing negative customer reviews


There are always exceptions, and this scenario doesn’t apply to every company. Some are entirely on board and ready to jump in head first. Whether you’re at a company that is working through this now or thinking about doing it in the future and need help with choosing tools, building out pipelines, or need an outsider to help be an unbiased set of eyes we would love to help (our contact). Our goal at Auklet is to help you solve problems with your software and it all starts by creating a robust foundational process. If you’re already agile and need an over the air update solution and APM tool for your continuous integration pipe check out my post on OTA solutions and our edge first APM solution, Auklet.

Developer Environment Vs. Production Environment

Not Your Senior Developer’s Application Performance Monitoring Solution

August 13, 2018 Posted by Internet of Things 0 thoughts on “Not Your Senior Developer’s Application Performance Monitoring Solution”

We all want to make amazing products that our customers love and that work fantastically when new features roll out. Sad to say, no matter how much testing we do there’s always at least a bug or two that make it out into production. It doesn’t matter if you’re a Fortune 500 with million dollar validation systems and teams of engineers pounding away on your IoT application. Maybe you’re a startup where everyone sits in the same room and knows the ins and outs of every line of code; doesn’t matter it’s still going to break eventually. As the systems that we create have more and more touch points and become ever more complex there just isn’t a way to validate every single workflow and situation your precious application is going to run into. Especially as we enter into the world of IoT where development and production environments are about as different as Bubblegum Castle and Westeros.

Developer Environment Vs. Production Environment

Photo Credits: (n.d.). Bubblegum Castle. Retrieved from Adventure Time Wiki Jon Snow GOT. HBO

In one you’re safe, can make modifications instantly, the environment is just how you want it, and nothing really ages.  In the other, you’re constantly under attack, it takes forever to get anything done, you’re at the whim of the environment changing every second, and your devices are continually aging faster than you ever thought possible.

As things become more complex we need an application monitoring tool that doesn’t require us to set performance indices, dig through mountains of data with complex queries, or passively monitor graphs for days. We don’t have time for that. We need to make our customers’ lives better with our excellent products.

Found The Needle In The Haystack

When you combine, let’s call them unique, users with different hardware variations, multiple production software versions, devices spread to every edge of the world (We all know that’s what you want), and a slew of other data points you quickly find that finding the underlying problem causing a bug in IoT is extremely hard to pinpoint. Not only is it hard to pinpoint sometimes it’s almost impossible to even identify it exists. How many times has a customer reported a problem to you and you’ve been unable to replicate the issue on your bench?

This is why Auklet is working to do this analysis for you, on real production data. We take in all this information about your application and learn how it is behaving across your entire fleet. When we spot outliers or odd behavior in a cluster of devices we inform you immediately and provide you with the intel that describes the problem for you. Making it possible to replicate the situation on your bench or identify the specific devices that you want to have come in for analysis.

You Did Something Awesome

After you resolve the issue and deploy a new release, we’ll track those devices, as well as others, and let you know that you fixed the problem. If there’s still room for improvement, we’ll let you know the fixes didn’t quite get the job done and start feeding you the new intelligence so you can take another stab at it. We’re building our solution to be your friend, sure we’ll tell you when there’s an unexpected exception that happens or you need to do some introspection, but we’ll also pump you up when you did something awesome.

Notification Zen

Auklet might be your friend, but it’s not your crazy friend who never stops texting you and messages you just to say “hey how’s it going” at 2 AM on a Wednesday night. We know you need your sleep and that you’re already overloaded by PR requests, chat “emergencies”, and social media “in case you missed it”s. If there’s a major problem popping up across the fleet, we’ll sound the alarm, but we won’t keep notifying you of the same issue just because it happened on your one-millionth unit. Instead, we’ll let you know when we first identify an issue and then give you some periodic updates if we find some new, useful, and different data points we think could help you fix the problem.

We Won’t Hold Leaving Against You

Yeah we’re going to double down on friendship, we’re not going to lock you in and force you to stay like some psycho. We are here to help you make amazing products whenever you need it. We’re not here to lock you into some relationship that you might not end up liking after a couple months. Instead we’ll continue to work tirelessly to find out how we can make Auklet even more of an expert at providing you with the knowledge that makes you and your team even more of an all-star cast. Whether you’re using Auklet or not, let us know what issues you are finding out in the wild. We love teaching Auklet how to find them for you so that you can take one thing off your plate and focus on more important objectives.

Come check us out at or give the team and me a ring.

Internet of Things Devices

Iterating Your Internet of Things

June 7, 2018 Posted by Internet of Things 2 thoughts on “Iterating Your Internet of Things”

The age of IoT is upon us. The last decade has seen the advent of the most complex consumer-facing IoT application out there, your vehicle. We’ve gone from a completely disconnected form of transportation with four wheels and a steering column to an array of sensors and modules that communicate over an ever-growing suite of networks ranging from industry-centric busses like automotive CAN and LIN to things you’ve actually heard of like 4G, WiFi, and Low Energy Bluetooth. Many vehicles today have over a hundred highly customized computers that can funnel data through a gateway and up to the cloud. As with other industries, this data isn’t sent in one unified way. There are a plethora of methods to serialize, compress, and transmit the data to reduce latency, minimize data usage, and increase security. Then once you have the data, you have to figure out what to do with it. Are there patterns that can help you make your product better? How about new revenue generation opportunities? Each question requiring a different visualization technique or machine learning algorithm to solve.


As the IoT market explodes and the data points multiply, we have the opportunity to create an entirely different world from the one we live in today. Unfortunately, diving into IoT can still be scary. There’s an endless amount of communication protocols, unlimited hardware options, different problem sets than those we’ve tackled in mobile and web development, and there’s still, as Jeff Bezos likes to call them, “divinely discontent” customers you are striving to satisfy. Our goal at Auklet is to help entrepreneurs, as well as established companies, accelerate their IoT aspirations, replacing fear with excitement and making it easier than ever to make an awesome IoT product!

Over the Air Updates (OTA)

Most of us take it for granted but updating a web application or mobile app is a well-defined process, and even the smallest company can quickly create a deployment pipeline that utilizes continuous deployment methodologies to roll out an update with every approved commit. To our customers, this happens without their knowledge or at the simple tap of a button on their smartphone. Almost never does a mobile app update result in a bricked phone or a website refresh with a white screen of death showing instead of your e-commerce selection. Seamless over the air updates was something I took for granted when I first started developing at a startup who was focused on digital out of home (DOOH) marketing and was again blown away to find out, was not standard practice at every major automotive manufacturer. Deploying updates to both of these edge devices wasn’t easy and for the most part, didn’t exist.

Early on at the startup, we had devices at multiple restaurants and bars, each of which of course had spotty WiFi and hard to reach TVs and projectors. We had a small embedded device that would plug into the backs of these displays and play through videos and images set by the store owner or third party advertisers. These environments made it extremely difficult to replicate specific issues in our local development environments where we had excellent WiFi, consistently 70-degree temperature, and limited humidity. So during our beta, we started finding a variety of errors. Memory leaks that only appeared after weeks of execution, hardware bricking because of the humidity and heat of the bar, and video jitters caused by poor performing functions and spotty network connectivity only to name a few. All this was happening while we needed to iterate and create new features requested by our beta customers.

As you can guess our devices didn’t have a marketplace or process for deploying updates, so anytime we wanted to do a release we had to go directly to the customer’s location, get up on ladders, look like fools, and either swap out the hardware or flash the existing pieces. I’m all about doing things that don’t scale at a startup, but this was absolutely insane. After enough frustration, I built out a web application that allowed us to distribute updates to our fleet that worked relatively well but took time and energy to maintain that would have been better spent on the actual DOOH application.

After leaving the startup, I went to work for one of the primary North-America based automotive companies. It didn’t take long for my mind to be blown by the fact that the frustration and problems I faced with a handful of devices at my previous job wasn’t being solved at this multi-billion dollar company. They had hundreds of test vehicles at different release levels, spread around the world. Every time a new software release came out, for a single one of the 100 or so modules on the vehicle, an engineer would have to go to the car, flash the software over what’s called an OBDII port, and hope with every fiber of their being that something didn’t go wrong and brick the module or worse, the vehicle. This process was even more frustrating for infotainment and clusters (the things that do navigation and show you how fast you’re going) which had a lot of graphic files and could take up to an hour per vehicle.

Manual Application Updates

Unfortunately, I had the perfect storm occur when I was flashing one of the clusters with a new release and ended up bricking it 45 minutes into the update, about an hour before one of the VPs was set to take the vehicle home for the weekend… This was not only embarrassing but expensive. Even for a billion dollar company a new module whether for development or production isn’t cheap. Luckily we had a spare available and I was able to tear apart the dash, replace the module, and update it. Props to the VP overseeing the program at the time, as he was very gracious about the whole thing.

Most engineers in the IoT space can relate to stories like these because until recently it had been difficult to push updates to edge devices that more than likely had limited network connectivity and very constrained resources. Luckily data plans have become more cost-effective and hardware costs have continued to decline. As we move into the future, it is critical to plan to support OTA on any IoT application you develop. Not only to push feature updates and bug fixes but also to ensure you stay consistent with security patches. Each domain within IoT is becoming more and more competitive by the day, and being able to deploy your latest innovation is essential to making a fantastic product for your customers.

Picking an OTA Solution

There are a growing number of IoT frameworks that support OTA out of the box in some form or another such as AWS Greengrass or There are others that fit more smoothly into projects that don’t want or cannot use one of these frameworks such as Mender which are still usually a much better solution than rolling your own. In a never-ending expansion of options, there are a few things you’ll want to keep in mind as you select the right choice for your team.

happy developer using over the air updates

Rolling Updates

  • Update Sections of Fleet
  • Validate Health of Updated Devices Before Proceeding
  • Schedule When Updates Occur
  • Manage Offline Devices Automatically
  • Status of Update Rollout

If you’ve done web development in the past five to ten years, you’ve probably started to take this fantastic feature for granted. Remember the days where you had to FTP into your Linux server and swap out the HTML, PHP, and JS files manually? Remember when you could SSH into a server and hack away live on production (of course no one has ever done that…). With the proliferation of PaaSes, serverless architecture, and container solutions such as Docker most of us now make a commit and can watch as it’s shipped off to a farm of servers. This farm’s clusters automatically scale up while rolling out the update, verifies everything seems to be healthy, and then determines whether or not to roll back the update or transition the traffic coming in towards the new deployment. These advancements have resulted in one person shops being able to maintain almost 100% uptime with their web applications and scale to relatively large deployments without increasing their IT staff.


The same cannot be said for IoT devices. Up until the last few years, there really wasn’t any off the shelf solution to do over-the-air-updates, much less ones with the ability to orchestrate deployment to thousands of devices that may or may not be online at the time and might disconnect midway through downloading the update. The mass majority of IoT relied on pairing major software updates with hardware refreshes, did updates in store or at dealerships, or had homegrown solutions that enabled them to distribute some level of updates to their devices.

Now you may be thinking, why would it hurt to do an update blast to all the devices at once? The same concepts of uptime on web applications don’t apply to IoT apps, right? You’re correct that unlike web services, rolling out updates over portions of your user base won’t protect other users from downtime with their application. What it does protect you from is if your update suddenly starts bricking devices or causing unexpected behavior on specific revisions of hardware. Rolling updates should enable you to monitor the health of devices that have been updated and set criteria before continuing a deployment.


Version Management

  • Define Version Currently Deployed
  • Revert To a Previous Release
  • A/B Testing Capabilities

Another element of web development many of us have become accustomed to is the ability to manage which version of an application is active on our server clusters. Many frameworks support rudimentary version management such as selecting a release for your entire fleet or specifying a specific release for a given device. More advanced solutions provide you with the ability to dictate which sections of your fleet have a given release and easily enable A/B testing to occur.

I haven’t seen any off the shelf solutions offer it yet, but in the automotive space, we’ve seen some companies with high-end modules do parallel execution A/B testing. This enables them to pump sensor data into the two variants of their applications simultaneously and record how each performs in the real-world situations. This allows companies to push updates to a vehicle, validate the software, and then determine whether or not to switch over to using it as the primary driver or continue using the existing solution while they push out another update.

Environment Variable Management

  • Set Environment Variables Across All Units
  • Notify When Update Has Completed Across All Units

Environment variables are the proven way to store configuration settings and secret credentials that you don’t want exposed to nefarious people. Having the ability to set environment variables is a must for any good IoT framework and fundamental to enabling true OTA capabilities. Without the ability to configure environment variables across your entire fleet you can find yourself in some unfortunate circumstances. Without being able to set variables across your fleet either you end up having to manually remote into each device to configure them, you have to roll your own solution to do so automatically and then maintain that solution, or you have to hardcode the values into your codebase.


Hardcoding the values always seems like the easy way out but please keep in mind that doing so can have the following adverse effects. Anytime you would like to update even one of those variables you have to do a full release of your codebase again. This isn’t the worst thing in the world of web and mobile devices but in IoT any update uses precious data and power. These updates usually place a burden on your customer and will more than likely cause some amount of downtime for them.

Data Management

  • Management of Network Connectivity Loss
  • Automatic Compression of Release
  • Differential Management

In automotive we’ve always had an eye on data usage. Whether it’s the car itself or an after-market add-on that needs to have a cellular plan. Customers usually don’t want to pay for extra data plans and your bottom line improves the less data you have to pay for. That’s why having an OTA solution that does everything it can to try and reduce the overhead associated with pushing out an update is so valuable. This can range from something as simple as compressing your update automatically before shipping it out, utilizing one or multiple network solutions to reduce the cellular data consumption, or determining the differences between your latest release and what’s already on a given device and only sending the pieces that have changed.


Enterprise Solutions

Are you in an industry that has a myriad of compliance requirements? Automotive is notorious for this, and for a good reason. Companies are putting a two-ton metal machine in the hands of the common populace. Depending on your industry this may be important or completely irrelevant. If you haven’t already done so, do your research on the regulations mandated by your industry or internally by your own company. Based on this information ensure that any OTA provider that you choose meets all the requirements or is capable of customizing their solution to meet your needs. If you’re in automotive, I’d recommend taking a look at Airbiquity’s solution :).

Customer Experience

  • How will the OTA update affect your customer?
  • Does the provider have a way for your customer to accept or skip the update?
  • Does the OTA solution require your device to reboot?
  • Does the solution interrupt a customer’s usage if you roll out an update?
  • Can you schedule update windows?
  • How does the solution recover from a failure?

Automotive has some extreme use cases that don’t apply to everyone, but there are two that I love discussing which relate to a customer’s experience.

You’re a new car company and to stay ahead of the curve you’ve made your flagship vehicle capable of doing OTA. You do your due diligence. You have your customers set update windows that you can use to limit what time of day the car will be updated. You add an interface that stops updates from happening unless your customer first approves it. Knowing this foundation is in place you roll out your first update. The notification comes up on one of your users infotainment screen and they accept it. The car then waits until 2 AM rolls around when the update window set and then it starts to update, great! Unfortunately for you this customer has a child that has asthma and it just so happens that tonight she’s having an attack. Your customer straps their child into the vehicle at 2:05 AM as he’s rushing to the hospital. Only to find that the car is updating and won’t be available for another 10 minutes. You best believe you’ve lost that family as a customer forever. Luckily this situation applies to very few areas of IoT and in this particular case, OTA providers for automotive have painstakingly accounted for these exact types of use cases. But I’m sure your industry has similar edge cases that when looked at on mass happen more often than you might expect.

The more applicable situation is your customer is driving down the road and a security update needs to be pushed out ASAP. You can’t require all drivers to stop and pull over to the side of the road. Does the vehicle need to be stopped to receive the update? What happens if the update bricks one of the systems in the vehicle? These types of questions apply to drones, wearables, smart transportation such as bikes, and a host of other applications. Make sure you list out the situations your customers can find themselves in when an update rolls out and ensure that you understand how the solution you choose will manage it on your behalf.


  • Does their OTA solution lock you into a platform?
  • Do they support your target hardware?
  • How has their history been for hardware / operating system version support?

Many IoT platforms are all-in-one solutions that don’t play too nicely with other products. This can be a double-edged sword in that frequently having a single platform allows you to get off the ground quickly but using completely proprietary systems and services tend to lock you in. This will commonly inhibit you from being able to pivot as freely as you’d like or easily change providers if pricing becomes less competitive. Vendor lock-in can be especially hazardous for IoT developers due to the hardware needs of almost every application. There are so many variations to hardware and so many optimizations to be made that over the lifespan of a product companies will often make multiple hardware iterations. Many of which occur in the first 12 months as prototypes are refined, specific pieces of hardware are removed from the bill of materials, and applications are prototyped in a scripting language only then to be converted to another to save on overhead. For these reasons, it is often a better strategy to mix and match providers that focus in on a specific niche and give you exit strategies to either bring things in-house or move to another provider as you grow.

This isn’t to say there aren’t use cases where a full out of the box platform makes sense. Hologram is an excellent example where they provide a white glove solution to not only the hardware, but the data plan, and your cloud solution. These types of solutions are great if you don’t know much about hardware yet and are just trying to get your startup off the ground.

Onward & Upward

Once you’ve integrated a way to deploy updates to your application, then you can start tracking how each new release affects your existing hardware solutions as well as how customers engage with your product. Tracking errors that were missed during QA and unit test cycles becomes mandatory. Even automotive companies with million dollar hardware-in-the-loop (HiL) rigs that run tests for months and test fleets of hundreds of vehicles still miss bugs before deploying updates. That’s why at Auklet we’ve built a monitoring solution specifically for IoT that you can use to track each iteration of your app.

We’d love to hear about how your company did updates before OTA was available or how they’re doing them today! Hit us up on twitter or drop us a note at here.

Additional Reading

As a side note I’d never advocate against still doing your due diligence during validation :). If you’re looking for a more cost effective solution than something like National Instruments or dSpace (excellent companies and solutions but often times too expensive for small and medium size businesses) I’d recommend checking out Tapster who is doing awesome things to make HiL testing more attainable.