A lovely story of firmware, dragons and knights
After decades of service, technology can still offer surprises and challenges. But in the end, we won again 🙂
This is a story of dragons, persistence and cooperation. Oh, and AI, of course.
Into the cave
Recently we were tasked with an apparently simple task: migrating a backend from Ubuntu 16.04 with PHP 7.4 to Ubuntu 24.04 with PHP 8.4. This backend is used by a mobile app and a firmware installed on some appliances.
Easy peasy, right? In our company, that’s something that we already did lots of times. So one of our member started with the usual stuff:
- start a git repo
- add docker to run the backend in its current version, with old requirements
- upgrade versions, check for errors, and adjust the code accordingly
- test, test, test
Meanwhile, I installed and provisioned the new server, migrated all configurations and certificates and started a new test instance of the backend, available on a temporary URL. The app developers produced a test version of the app which connected to this test instance, and everything was fine. We still had to get the ok from the firmware, but we did not anticipate any issue:
- the app was migrated with barely any change in the code
- the server was as similar to the old one as possible
- our test with an HTTP client showed no difference in responses between the old and new server
- the test app was working fine
And yet.
There was no way to make the firmware connect successfully to the new server: it made the first request and then stopped. No error, just nothing. I started to fear we may have missed the hidden dragon in the cave.
The dragon
Any issue with the firmware was really hard to spot and to deal with.
- The firmware is about 10 years old and no one could remember its specific requirements.
- The company which originally created it could not debug it without a physical machine available.
- Updating and distributing a new firmware on all the appliances already used by customers was not an option.
How do you deal with a dragon? With great care, cunning and patience.
We made sure everything was safely stored and started experimenting with web server options. Having the old server and the new server still available was a great boost, as we could experiment without blocking existing customers.
After several blind attempts with no luck we sent an appliance to the firmware company, and they confirmed a very unexpected behaviour: the first request failed because the firmware expected the first response to come in two chunks with specific length.
Let me repeat this: the firmware expected a single http response to come in two chunks, 454 bytes for the first and 64 bytes for the second. Two flushes from the server is fine. A single one is not. How about that for a fragile code, huh?
We started dwelving into any possible Apache hidden configuration or changes between versions, tested with several flush related options, and in the end resorted to building a custom Apache module to customize the flushing behaviour.
The knight
Yes, we had some help by an AI agent: it would have taken much more time to write ourselves a valid custom Apache module to deal with this.
Luckily the firmware company discovered that they did NOT actually expected fixed-lengths chunks but simply different chunks for header and body of the HTTP response. While still a very weird customization, this was much safer and easier than configuring fixed flushes for each response.
So after a few back and forth with the firmware company, we were able to fix the issue. Now the backend runs on an new LAMP server with up-to-date versions, but with a small patch that allows backward compatibility with the firmware.
In the end, this was a very challenging issue, solved with a mixture of safety measusers, hypothesis, custom tools, iterations and persistence. While we mostly do coding, our mission is to solve problems and this was a problem only slightly related to coding.
We love this job!
Foto di Cederic Vandenberghe su Unsplash