Real-Time Embedded User Interfaces Using HTML/CSS/js and Websockets

Modern appliances and electronic devices are becoming increasingly sophisticated. Advanced feature sets require advanced configuration options. Multi level menu systems offer more granular control to the users but at the cost of usability. Unfortunately, most user interfaces are not advanced enough to solve the usability issues and therefore the interface actually becomes the weak link.

Physical interfaces such as button panels or remote controls have many disadvantages. Panel controls require a dedicated display for feedback and physical access to a fixed location. IR remote controls offer only one-way communication with limited feedback and can become difficult to manage for multiple devices.

Many different 3rd party control systems are available to develop fully customizable user interfaces for controlling multiple types of devices. These control systems are essentially software development platforms which provide a very high level of customization and development options. However, the highly technical nature of these platforms makes it cost prohibitive to implement in many situations.

We propose a new type of solution to address these issues and bring simplified management and control to a broader market through embedded control interfaces.

Design Concept

An embedded control interface is a Graphical User Interface designed for a specific appliance, served from an embedded I/O controller connected directly to the appliance. See the illustration below.

 

The embedded I/O controller maintains a proprietary link with the appliance at all times and presents a virtual interface to multiple clients using HTTP. The GUI is built with standard web technologies designed to display within any compatible browser on any computing device. Building and embedding the GUI in this way has two distinct advantages:

  • Universal Access – By using web standards (HTML/CSS/javascript) and ensuring a high level of cross-browser compatibility we allow universal access from any device or operating system, regardless of manufacturer. The software does not have to be precompiled for a specific operating system. This allows us to maximize accessibility with minimal development costs.
  • Ease of Integration – Since the GUI is included with the appliance there is no need for customized development that would be necessary with a 3rd party control system.

Potential Issues

Though this concept solves some problems it does present a few technical hurdles that must be overcome.

How to implement real-time communication between the appliance and embedded I/O controller?

As a manufacturer we have complete control over how our appliance communicates with external systems which makes this a less trivial issue. We are able to broadcast events as they occur. For example, when a user performs an action that changes the state of an appliance, that state change is automatically broadcast out to external devices that may be listening. The embedded I/O controller is serially connected to the appliance and listening for these types of changes. When a response is heard it immediately processes that response to understand what changes have occurred.

A key consideration for this method is that some responses may not be initiated from the GUI but instead from the physical interface. The appliance still needs to broadcast the event regardless of where the change was initiated from and the embedded I/O controller needs to be able to receive and process that response independent of its own requests. This may require a unique communication protocol between appliance and embedded I/O controller where various appliance states can be deciphered from a simple message.

Real-time communication becomes a bit more complicated if you do not have control over the appliance behavior. For example, we have access to the appliance firmware so we can ensure all events are broadcast as they occur. But not every appliance is designed to communicate this way. Some appliances will only respond when queried. Therefore, in order for the embedded I/O controller to maintain a near real-time state model of the appliance it must constantly ask for updates of every possible state. This poses some challenges that could make it unstable to implement.

How to implement real-time communication between the embedded I/O controller and web clients?

Web pages typically rely on user interaction to post and retrieve data. However, a real-time interface cannot rely on user interaction for updates. Additionally, if multiple users can access the same appliance simultaneously then it is important that the information displayed in their interface is always up to date. Otherwise users may unknowingly act on “expired” information.

For example, say we have two users attempting to control an appliance simultaneously. User A is accessing an embedded GUI through a browser from a remote location. User B has physical access to the appliance through a keypad. User B pushes a button that changes the state of the appliance. But User A may not be aware of this change until his GUI refreshes to reflect this new state change. If the GUI does not refresh before User A interacts then problems may occur as User A is acting on “expired” information.

To avoid this we need to make sure the browser client is updated to reflect changes in real-time.

Traditional web interfaces rely on a full page load to update the contents of a page. However, this is highly inefficient as it requests all page content from the server to be resubmitted regardless of whether or not any changes have occurred. This wastes bandwidth and CPU cycles. Handling multiple clients this way could easily overload the limited resources of the embedded I/O controller.

AJAX was developed for this purpose. AJAX allows individual elements to be updated without having to reload an entire page. However, AJAX alone doesn’t solve the problem of getting real-time updates.

In the last few years COMET has become a popular method to handle real-time bidirectional communication between an HTTP server and multiple clients. However, COMET has never been standardized which makes it difficult to implement and work with different browsers. Some of the more popular COMET methods involve long-polling where the client opens an HTTP request and the server keeps it open until an event occurs. The server responds and closes the connection requiring the client to repeat the process. Though this method works, it does have some limitations and scalability issues. Websockets were designed specifically to address the issues with COMET. The basic premise of websockets is that it creates a dedicated TCP socket connection between the server and a client. Updates are sent across the socket connection only when they occur so you can achieve real-time communication with the least amount of resources necessary. Websockets were originally introduced as part of the HTML5 specification but have since been separated into its own specification. As of this writing, Websockets is still being specified by the W3C and the IETF.

Proof Of Concept

With a theoretical concept we set out to develop a proof of concept. For this we decided to create a very simple GUI that could be accessed through a browser to control an appliance.

Appliance

For the appliance we chose our SW-HDMI-4. This is a basic 4 port HDMI switch with an RS232 interface. This interface allows us to send commands and receive updates over RS232.

The protocol is very basic and provides an easily parsible format for machine interpretation. The commands used are:

CommandSyntaxFunction
PWnn=0 for Power OFF
n=1 for Power ON
Turn appliance on/off
PW<CR>Response is PWn<CR> as described aboveQuery power status
COn,m<CR>n=1 for connection output 1.
The SW-HDMI-4 only has a single output
so this value will always be 1.
m=1-4 for the input.
Select the source input to be routed to the output.
COn<CR>Response is COn,m<CR>
n=1 for connection output 1.
m=1-4 for the input.
Query which display is connected to the specified output.
BOn<CR>n=0 for Boost OFF
n=1 for Boost ON
Turn signal Boost on/off
BO<CR>Response is BOn<CR> as described aboveQuery boost status

This SW-HDMI-4 is configured to broadcast all events over RS232. So, for example, if a user pushes the input button on the front panel to change the input source to #3 then the appliance would broadcast “CO1,3” over the serial connection.

embedded I/O controller

For the embedded I/O controller we decided to use a Lantronix Matchport AR. The Matchport features several GPIO lines and an HTTP server running on a custom embedded linux kernel. To get the Matchport fully operational we used the development board connected to the SW-HDMI-4 via RS232 and ethernet side to the LAN.

Software

There were several processes we had to develop for the Matchport to get a functioning prototype.

First we had to establish a serial link between the Matchport and the SW-HDMI-4. This link would be setup automatically on boot up and would stay open as long as both devices were on. Upon initialization the Matchport would query the switch for its current configuration which it would use to create a state model.

The matchport would then begin listening for responses from the SW-HDMI-4. Responses are received and processed and the internal state model is updated to reflect the changes. If any changes occur it would trigger an update to any connected HTTP clients. The communication between server and clients uses a custom websocket connection. Messages are passed back and forth between server and client using JSON.

For the interface we developed a simple point and click GUI as pictured below.

Demonstration

A video demonstration of the working prototype can be seen here:

Notice in the video that there is (nearly) zero latency between synchronization of the GUI and the switch either from actions occurring at the switch or the GUI.

Conclusion

Our working concept demonstrates the ability to provide a real-time web enabled virtual interface for non web enabled appliances. Using AJAX and the latest HTML5 technologies (websockets) we can achieve push notification with efficient resource utilization and near zero latency. As of this writing, websockets is still being developed and is only available in Firefox, Chrome and Safari beta builds but we expect to see it rolled out to stable releases soon, followed by other mobile versions.

With a bit more work we can adapt this system into a modular framework allowing rapid development of advanced virtual user interfaces for other appliances.

9 Responses to Real-Time Embedded User Interfaces Using HTML/CSS/js and Websockets

  1. [...] Real-time junkies! Justin Ireland (@justin_ireland) posts on “Real-Time Embedded User Interfaces, Using HTML/CSS/JS & Web Sockets” [...]

  2. [...] Real-time junkies! Justin Ire­land (@justin_ireland) posts on “Real-Time Embed­ded User Inter­faces, Using HTML/CSS/JS & Web Sockets” [...]

  3. Very cool! Nice work and nice writeup. It’s too bad more common appliances don’t have RS232.

  4. Justin says:

    Yeah I agree. This was initially targeted towards Pro AV equipment which almost always has an RS232 interface. Most consumer appliances only have IR which is unidirectional so this method would not work.

  5. There is no GUI standard or specification for the web.
    I raised this issue with the W3C web accessibility initiative in 2004, and created an RDF schema for GUIs:
    http://www.peepo.co.uk/temp/gui-schema describing such concepts as buttons…
    you may also wish to consider:
    http://www.w3.org/WAI/intro/aria.php

  6. John says:

    Hello,

    Respectfully, I don’t believe this architecture enables real-time GUI control. Websocket communication over a secure network with managed ethernet switches may be deterministic, but as soon as you interact with anything via Windows or any other GPOS (Linux, MAC, etc.), you lose determinism. Contrary to popular belief, Linux is not a real-time OS. Windows RT and Linux RT are “soft” real-time. The existing HARD real-time systems, at this time and to my knowledge, do not have drivers for graphics cards, unfortunately..

    I think a deterministic solution may exist with serial communication between a touch panel running Windows CE because Windows CE is a real-time OS. Otherwise, maybe some genius out there has written a graphics card driver for VXWorks, Phar Lapp, or some other off-the-shelf RTOS.

    Bottom line: if the PC driving the graphics card is not running a RTOS, the interface is not deterministic.

    Another option would be to off-load the image processing to the RTOS with custom image generation algorithms and then use driver software or FPGA algorithms to output these images over a standard line such as HDMI, CameraLink, or D-SUB (for computer monitors or projectors). No matter what it’s a pain in the balls… or has someone else figured this out?

    Best wishes and happy holidays,

    John

    • Justin says:

      Good point John. I guess it depends on what a typical user would consider “real-time”. Based on our testing, the amount of latency between events happening at the appliance and updating the GUI were virtually unnoticeable. While it may not be considered “real-time” by some technical measures, based on human observation it is.

  7. Teis Draiby says:

    I’d love to see the web sockets code for the controller : ). Does it happen to be open source?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comment moderation is enabled. Your comment may take some time to appear.