2018-12-09

Connecting a USB serial console to WRT1900ACS


Goal

A while ago I purchased a Linksys WRT1900ACS-v2 router to replace my aging 15-years firewall (still running kernel 2.4 on an old VIA Eden board!). This device is pretty nice, it's fully supported by Linux mainline, it offers multi-gigabit connectivity with a real CPU (Marvell Armada 385), has lots of RAM (512 MB) and storage (256 MB NAND), and dual-band WiFi. Yes I know, having NAND flash nowadays instead of eMMC is still annoying to deal with, but I won't need to write often, so I certainly can ignore wear leveling since I'll only be loading the rootfs into RAM. This will be plenty of room to run Formilux with nftables.

I don't have much time to devote to its conversion and I figured that if it's still not in production, it's because I don't want to switch to it before I'm 100% confident about the config and it doesn't have an accessible serial console to allow me to recover from my mistakes. Thus I decided to work on exposing a serial console before finishing its configuration.

Serial Connector

Linksys engineers were really cool with this device, you don't need a soldering iron as they already placed the connector inside. You have remove the screws under the rubber pads and strongly pull apart the blue from the black parts to access it.



There are still a few caveats though :
  • the connector is a 2.0mm pitch connector (JST-PH-2.0 to be precise)
  • the on-board marking is reversed compared to the connector's specification, which creates a lot of confusion and can easily fry an adapter since the power pins are at both ends
  • there is something strange with the Tx output, the device refuses to boot if it detects too strong a pull-up (4.7kohm is too strong). I suspect the device tries to pull the line low during boot to detect such a condition in order to stop in debugging mode or something like this.
Thus it is not as trivial as it seems to simply place the connector outside.

After measuring the voltage on the pins and doing a few attempts, I finally came up with this pin-out :


That's fine, I don't need the +3.3V since I intend to connect this to a USB serial adapter which is alimented by the USB bus. So I can use a 4-pin connector which will cover the connector's pins 1 to 4 (motherboard pins 3 to 6). Such 4-pin connectors are easily found with many motherboards which provide a sound connector for the CD/DVD drive. It's the same! Yes it only has 3 wires but that's what we need : GND, Rx, Tx.


Serial adapter

I picked a CH340G-based micro-USB to TTL adapter like this one, which supports working on 3V or 5V (I've set it to 3V) :



Connecting this adapter to the connector revealed the issue with the too strong pull-UP. So I decided to try to attack it various ways. First, I cannot replace the pull-up on the adapter because it's inside the chip itself. In the end, the easiest and most reliable solutions I found were to use either a 2N4403 PNP transistor or a low voltage AO3407 P-channel MOSFET with a much weaker pull-up. These ones would amplify the WRT's Tx pin negative signal to act  stronger on the adapter's Tx input pin. I tried the two following diagrams and finally decided on the MOSFET one since it features an even weaker pull-up :



Since I had to add components to the board, I wouldn't be able to use the existing 6-pin connector, so I desoldered it to make the PCB thinner. Be careful, on mine the Tx PCB pad went off. Fortunately it's didn't serve as a via and is not used, it just made my task a bit more complicated to figure what pin the signal was going to :



Then I noted the soldered the MOSFET and the resistor on the PCB close to the IC :



And connected the extremity of the wires there. Note if you reproduce this design, I left far too long wires, you could shorten them to 15 cm I think.



Drilling a hole

Every device needs to have at least one hole for a console connector :-)

It wasn't easy to find an accessible place for this connector. I tried to place it in front, but this required disassembling the front panel which I failed to. There are screws below the label underneath but despite these the front panel doesn't come off. It looks like there are molten plastic joints at various places holding it firmly in place. In the end I found an empty place above the front right rubber pad. I drilled a large enough hole to let a micro-USB male connector enter, and placed the adapter there on top of dual-sided thick tape against the internal plastic wall. It looks as if it was made for this, there's very little margin there.


I don't want dust nor random metal stuff inadvertently entering the hole, so I wanted to fill it with hot glue. The difficulty with hot glue and USB connectors is that it tends to fill them up so that it's not possible to insert anything anymore there. So I had the idea to insert the USB connector before filling the hole with glue. But I didn't want my USB connector to be glued. Thus I reused the dual-sided tape protection. By definition it's non-adhesive, so I adapted it to the hole's form and placed it around the micro-USB connector. This way I could plug the hole while filling with the glue. The result is perfect and the adapter cannot move anymore.



Reassembly

Reassembling the device is almost as easy as opening it except when you have an excess of cable as I had. You have to find tiny places to push a few centimeters of cable in order for everything to fit. But once reassembled, it looks very clean, and the connector is very easy to access from the front.


And of course it works fine, I can now attach minicom to it at any moment to watch the boot process (yes I know there's no password now, I'll set one at the end of the installation) :



Now I can finish to install it and play with nftables without fear of being locking myself out ;-)

2018-11-11

Replacing 2 AA 1.5V batteries with 2 AA lithium batteries

Context

My digital camera (Canon A2100 IS) uses 2 AA cells as its power source, as can be seen on the photo below from the review site above :
This was one of the reasons why I selected this model, because I didn't want to have to discover that the battery is depleted when I need it, and AA batteries are easy to find. I replaced the batteries long ago with NiMH rechargeable batteries, but the camera is a bit picky on the input voltage and cuts off long before the batteries are depleted.

Failed attempts

NiMH - NiCd

I tried both NiMH and NiCd batteries just in case the discharge curve would be better on these last ones, but none really stands out of the lot. Below 1.2V approximately, the camera complains.

Lithium 1.5V batteries

I found long ago a very interesting type of 1.5V lithium AA batteries made by Kentli. These batteries are in fact made of a smaller 3.7V lithium battery inside, followed by a DC-DC converter to emit a constant 1.5V voltage. They're pretty good for what they are doing. But they are not suitable for this type of device. Indeed, their output voltage regulator suddenly cuts off once the battery's capacity is too low, and the regulated output prevents the camera from indicating it's going to stop working. Not only it's unpleasant to discover that the battery suddenly is dead when you want to take a photo, but it usually fails with the lens out and exposed without protection which is not fun at all. Ideally such batteries should lower the output voltage near the end so that the powered device detects the situation and can gracefully shut down or warn the user. But for torch lamps and laser pointers these ones are perfect.

Single LiFePo4 cell

I was thinking that there's always some margin in devices designed to be powered by batteries, so if two completely charged AA cells can provide about 3.1-3.2V, surely a single LiFePo4 cell (3.2-3.3V approx) would be fine as well. Such batteries are easily found on the net, sold with a spacer made of a simple wire, to replace the second battery without adding any extra voltage.


The problem with LiFePo4 is that the battery capacity is very low, 600-650 mAh on average so in the end I had an even shorter duration than with the NiMH ones.

Single Li-ion cell

I figured that since the LiFePo4 cells send 3.6V to the device once fully charged, there's probably still a bit more margin, so I decided to try a regular 3.7V Li-ion cell (which reaches 4.2V when fully charged), again with the spacer to replace the second battery.

The first difficulty with these batteries is that their advertised capacity is most always fake, so it's very hard to spot valid ones unless you're willing to pay the price. For this size in Li-ion it's reasonable to expect 800 to 900 mAh, not more. Also, at less than $6 per cell it's a low capacity one. But it's not a universal law since it's possible to find expensive ones which are fake as well :-)

Aside the difficulty to select the correct battery, it turns out that the voltage is perfectly suitable for this camera. The camera only warns once the battery is really depleted, and doesn't suffer from the extra voltage. I figured that I'd just have to spot a serious battery and would have an acceptable capacity with the ability to recharge it with a regular Li-ion charger. But isn't it too bad to lose half of the capacity in the spacer ?

Successful solution found

I spent two hours figuring the best type of batteries I'd need to go further and I ended up picking unprotected and flat battery cells. They are slightly shorter than the regular ones because they're designed to support installation of a small protection PCB if needed. I found this, made by Sanyo, which are apparently genuine given the capacity I tested (about 850 mAh each) :


Then I figured that I could install the two in the reverse direction, and connect them in parallel, negative to negative and positive to positive using wires, and isolate one side of each battery so that the resulting block is still two batteries with a single battery voltage. Thus I tried.

First, I prepared the batteries by cutting pieces of the isolation layer in order to solder them together. For this, the batteries need to be installed flipped and aligned. The minus pole should slightly stand out compared to the plus pole, which will receive some solder :


Then using my soldering iron at maximum power, I managed to solder them together without heating the batteries too much :


Finally, the two positive poles were connected together using a piece of wire. A bit of extra solder was installed on top of the wire to make a button which will help make good contact inside the camera :


And that's done! I now have a 3.7V, 1700 mAh battery that presents itself in the same form factor as a set of two AA batteries and which delivers its whole capacity before the camera shuts down. The main difficulty I had was that the space between the two batteries in the camera is very tight and I had to minimize the solder's thickness to let the battery block in. I also had to install some polycarbonate tape on one side to ensure that the metal part closing the battery holder doesn't short-circuit the positive and negative poles.

Unfortunately I didn't figure how to photograph my camera with the trap opened to see the batteries installed, but it's easy to imagine on this block on the photo at the top.

What is nice is that I can still charge it using a regular Li-ion charger supporting 14500 to 18650 batteries. Let's see how long it will last now.

Modding an RC transmitter to support lithium batteries

Context

From time to time with a few friends we play with these RC cars but the problem is always the same : the transmitters require 4 AA-sized batteries. Since we have 4 cars, it means we need to find 16 charged AA batteries before going outside. It often takes much more time to find and test them as the time required to charge the cars themselves! This made me wonder if it would be possible to replace them with a set of USB-rechargeable lithium batteries that are more easily available and easier/faster to charge.

Hardware inspection

The transmitters look like this (outside and battery holder) :


After removing the 7 screws holding the two sides together, the internal board is accessible. Following the wires with a multi-meter shows that the jack plug is connected to the battery through a diode, and that the battery directly connects to this small LDO voltage regulator through the power switch :


The marking "CA33G" indicates an AIC1734 LDO voltage regulator providing 3.3V from up to 12V in. It is said to feature only a 250mV drop at 300mA. I tried with an adjustable power supply and found that the board consumes only 61mA and that the regulator provides 3.3V starting with 3.44V in (144mV drop only). Better, this voltage drop is maintained for lower input voltages and the board seems to work fine down to 2.5V!

Thus I don't need these 4.8-6V input. Using a single lithium battery from 3.5 to 4.2V will be far more than enough. No need for any DC-DC converter either. And given the low power draw, a small one is usable so that I don't have to modify the battery holder.

Solution

I decided to go with some salvaged cellphone batteries. The one photographed here is a BL-5B with a capacity of 890mAh. It includes discharge protection. I simply had to glue a small dirt cheap USB lithium battery charging board to it (you can have 3 of them shipped for less than $1). Then just solder it to the battery connection pads with hanging wires slightly longer than the battery itself. I measured the self-discharge caused by the charging board, it's around one micro-amp so it will need 100 years to discharge the battery. This saves me from having to install switches or connectors.



It is necessary to put some solder tin on the pads before soldering the wires in order to reduce the total heating time and prevent the plastic around from melting. But that's really all. Oh, and of course, it works :-)

I closed everything, installed the battery inside a small plastic bubble protection bag to save it from moving inside, and I can now quickly charge it simply by opening the battery holder and connecting a micro-USB connector to the battery.

Final note

A small note, the charger used above provides 1A by default, some are sold pre-configured to 500mA (just need to change the 1.2k resistor). It's better not to charge small batteries to strongly or they will not last long. Also the charger will stop around 10% of the configured load, which could be reached much earlier than a full charge for smaller batteries. I've already patched two transmitters this way with batteries I had in stock. For next ones I'll probably use 250mAh batteries made for mini-quadcopters, which will provide 4 hours of operation and support being charged in only 20 minutes.

2018-09-22

Gigabit Ethernet Everywhere

What's the purpose ?

Every developer even remotely interested in performance regularly needs multiple physical machines to test the effects of various optimizations. I'm no exception to this, on my desk lie a number of fanless Atom and ARM boards serving as load generators and web servers when I need to validate certain points regarding HAProxy.

The problem is that you don't always have enough network connectivity on the wall. A simple solution consists in buying some external 5-port switches but at some places you then don't have enough power plugs, and having to deal with extra cables and devices at all time for a test once in a while is not always cool.

Additionally when you have to visit customers for some consulting it's commonly problematic not to be able to plug anywhere or directly behind a test machine.

Nowadays there are interesting solutions like this very inexpensive 5 port Gigabit Ethernet switch which I paid less than $13 shipping included. But this one also requires an external power supply.

Very simple solution

One will note that the description mentions a 5V power supply. Thus it can very likely be powered over USB. I bought one and used a USB cable adapter to verify that it worked fine once connected to USB. And indeed, it did.

Thus I opened it, de-soldered the power plug, replaced it with a micro-USB female connector, and that was all. Now I can carry this device in my computer bag every day and if I need to connect it, I can pick whatever micro-USB cable available and connect it to my PC or any other power source.

It would be nice if such low-power switches would ship with micro-USB by default. They would cost even less to produce, and would not need to be shipped with a power adapter.

The modification steps are shown below :

1) open the device by removing the top cover and extract the PCB :

2) desolder the jack plug :


3) prepare a micro-USB connector. This one comes already soldered on a board, I just had to re-cut the PCB a little bit to adapt it to the format of the original jack plug :


4) solder the micro-USB PCB, place a piece of foam on top of it to plug the hole, and put the top cover back. You're done :


That's something very easy to do, which saves you from having to carry an adapter or even a special cable everywhere. I wish the manufacturer will think about it for future versions.

Other thoughts

It's visible on the top right that there's room for an eeprom containing the switch config. The config format is documented in the switch's datasheet. This device is based on a Mediatek MT7530DU. I didn't go into details yet but very commonly this allows to enable VLANs and maybe even port mirroring. For some use cases it can be useful to have a tagged port communicating with each other port via a distinct VLAN. This allows for instance to very easily extend a laptop's interface to 4 ports for routing, firewalling or load-balancing tests. If the device supports port mirroring, it could be even better, supporting being inserted on existing cables and capture the traffic for network troubleshooting purposes.

2018-02-25

USB-based watchdog for ATX power supply

Introduction

Sometimes when developing on low-level stuff like a kernel, you face situations which completely freeze your system. While it's not that much of a problem on a development board, it's more annoying when it happens on a regular PC and you have to move off your chair to power cycle the beast, or worse, wait till you're home when developing remotely.

Usually this doesn't happen because your hardware is supposed to come with a hardware watchdog timer that will reboot the board if hung. But some boards don't have one, or sometimes it doesn't work well or you simply don't have the driver on your experimental kernel.

This article explains how it is possible to implement a watchdog for an ATX power supply, that is controlled over USB. It supports both being controlled by the target device itself (like any regular watchdog timer), or by another machine which will be able to power cycle the tested one.

Background

First, it seems useful to remind how a watchdog timer works. It is a hardware counter that counts down from a configured value, and which triggers the system's reset once the counter reaches zero. The main system periodically refreshes this counter so that as long as the system runs, the counter cannot reach zero (e.g. it sets it to 10 seconds every second). If the system hangs, it stops refreshing the timer and once the period is over, the reset line is triggered and the system reboots.

Second, it happens that some embedded boards do not easily expose certain connectors like the reset pin. At this point it becomes a bit harder to implement a hardware watchdog timer. But there is a solution : ATX power supplies are expected to drive the PWR_OK signal up once the voltage is correct, and drive it down when it's out of specs. More precisely, they connect it to a pull-up resistor, and actively drive it down when the voltage is not within specs. The motherboard is supposed to watch this signal and hold its reset line until this signal is up. This is very convenient because it means we can connect to this signal without even cutting any wire, and drive it low to force the board to reset itself.

So we have our reset input :-)

Locating the PWR_OK signal

This signal is on the grey wire connected to ping 8 of the ATX connector (either 20 or 24 pin) as shown below. In case of doubt, please use your favorite search engine to look for ATX pinout. But be careful, the pinout is often shown from the pin side (front view) and not from the wire side (rear view) :

Controlling the signal

We will need a timer and one solution to communicate with the host and/or with another host. Initially I thought about using a serial port but they are not always available nowadays. Then I thought about using the PS/2 connector and periodically sending commands via the keyboard controller, but some boards do not have such connector. Nowadays almost all boards do have plenty of USB ports however. So it looks like implementing a USB slave device is a good idea, which will make it easy to unplug and connect to another machine if desired.

The difficulty with USB is to find a suitable chip. But not only this is no longer a problem thanks to the V-USB stack but it ends up possibly be the cheapest solution one can think of. This stack is available for very affordable micro-controllers like Atmel (now Microchip)'s ATTiny85. which can be purchased already soldered on a board with all required external components for around $1.50 to $2.00 depending on the board and form factor. Models sold as "Digistump Digispark", "Adafruit Trinket" as well as various so called "ATTINY85 development/programmer board" equipped with a micro-USB connector are all suitable. Two such devices have been successfully used to date :


One point to keep in mind is that the PWR_OK pin can be very sensitive on the motherboard. The ATX specification says that a power supply is required to drain at least 200 µA (i.e. 5V through a 25k resistor), and a motherboard must consider any voltage lower than 2.4V as low. This means that simply connecting a 10k resistor between this signal and the ground may be enough to drive it low with some power supplies. As long as the module is always connected, it's not a problem. But if the module is to be left disconnected, hanging out of the machine, then it will drain the signal through the power lines and will drive it low, preventing the machine from powering up. One pin on the ATTiny85 doesn't cause this : PB5. It's shared with the RESET signal and supports a high voltage, so it doesn't contain any pull-up nor clamp diode. The tests confirmed that indeed, when the module is not connected, only this pin doesn't prevent the board from powering up.

There are 3 solutions to this problem :
  1. use the PB5 pin as the output to control PWR_OK. I'm not happy with doing this as it won't be possible to reprogram the device anymore without providing a high voltage to the chip.
  2. permanently connect the +5V to the device by connecting it to the power supply's +5V line. I'm not fond of this method either, as taking the +5V outside of a machine isn't always safe if there aren't efficient protections (e.g. wires getting accidently pulled off could easily burn if shorted). On some boards this may also require disconnecting the +5V from the micro-USB connector to avoid leaking the current into the USB host.
  3. adding a transistor to only let the current flow when the board is powered. This sounds like the best solution, an only requires one external component.
The boards above all include a LED connected to PB1. We'll use it to indicate when the PWR_OK signal is asserted by the module and to help debugging. It's always convenient to be able to test a driver without having to power down the machine to confirm it works! On such boards, PB3 and PB4 are used for the USB connection. PB5 is shared with the RESET, which only leaves PB0 and PB2 available. Thus we'll use PB0 here to control the PWR_OK signal.

Additionally, in order to protect the device against cheap power supplies which would directly connect PWR_OK to the +5V output, we'll put a series resistor to protect the transistor and the micro-controller against a short-circuit. The micro-controller is supposed to be able to drain and sink 40 mA per output pin. By using 180 ohm, we'll limit the output current to 27 mA in the worst case (just enough to lit a LED), which allows to safely control a wide range of power supplies.

Circuit design

The circuit is pretty simple. A BS170 MOSFET is connected between +5V and PB0, so that the output is only connected when PB0 is low. This MOSFET contains a reverse diode, which allows us to also monitor the output and report it to the USB host. The MOSFET may be replaced with an NPN transistor like a 2N3904 or a BC547, a resistor, and a diode if the voltage monitoring function is desired. Both solutions have been built, tested, and confirmed to work :



The code takes care of disabling the output and the pull-up when the output is not asserted, so that no current even flows through the diode. The finalized boards look like this (before putting them into transparent heat shrink tube) :




Communicating with the chip

The V-USB mentionned above provides USB connectivity. This is fine for HID devices, but these devices require a driver on the host (well, it's not totally exact as they can send keyboard keystrokes but there's no trivial way to send data without a driver or an application). Another solution consists in emulating a modem using the CDC ACM mode. This is exactly what the AVR-CDC code does. It's important however to keep in mind that CDC ACM over a low-speed device is out of spec but general operating systems tend to support it, which is what we're exploiting here.

The AVR-CDC code comes with an example application consisting in a USB-to-RS232 interface realized entirely in software. It's a good starting point for what we need, so the RS232 code was removed and the USB code was modified to instead simply manage a watchdog. The modified code is available here with the pre-compiled firmware in hex format.

The protocol is very simple : the device presents itself as a serial port on the host, and receives some short commands. The following commands are supported :
  • '0'       : disables the watchdog
  • '1'..'8'  : schedules a reset after 2^N-1 seconds (1..255)
  • 'OFF'     : power off
  • 'ON'      : power on
  • 'RST'     : triggers a reset immediately
  • 'L0'/'L1' : turns the LED off/on (to test communication without resetting)
  • '?'       : retrieves current state on 2 bytes. First byte indicates the level of the pin, which is set as an input when not off, allowing to check a power supply's status. '0' indicates it's off (or forced off), '1' indicates it's on (or disconnected). The second byte indicates the remaining amount of seconds before reset.
All unknown characters reset the parser, so CR, LF, spaces etc will not cause any trouble.

All commands which manipulate the timer or the output ('0'..'8', 'ON', 'OFF', 'RST') automatically disable any pending timer. The device takes a great care not to influence the output during boot, and doesn't automatically start. This way it's possible to leave it connected inside a machine and to turn it on by software. It may also be used as a self-reset function by writing 'RST' to the device.

Testing the module

The tests were conducted on an ATX motherboard with an ATX power supply. The module was connected to a netbook PC. The module was forced off and on, then it was verified that the device didn't reset as long as the timer kept being refreshed, and then that stopping the loop (Ctrl-C) was enough to cause the board to restart. The status and the timer were also read before, during and after a reset :


The whole test setup :


The motherboard tested here (Asus P5E3 WS Pro) seems to only sample the PWR_OK signal on the falling edge for a short time, and decides to start anyway after a few seconds. It will prevent from controlling the power from outside. It's not dramatic, it still allows to remotely reset the board, which is the most important.

It is likely that some users may want to enable the timer at boot time. It's easy to do in the code anyway. A change could consist in checking pin PB2 to decide whether or not to automatically start the timer at boot. A variation could consist in replacing the output resistor with a relay connected to +5V or +12V, inserted between the mains and the power supply, or between an external power block's output and the board to control. The system is simple and versatile enough for many variations, possibly not even requiring any code update.

For motherboards featuring an internal USB connector, the Digispark board may be the best, as it includes the USB male connector, so it can directly be plugged into the internal connector without any cable. For internal connections, only the PWR_OK wire is needed.

For remote control, it's preferable to place the module outside of the machine with only the two long wires connected to the ATX power connector. These ones do not represent any particular risk. In case of shortage or connection to the ground, the board will simply be forced off.

Some tests should be conducted to determine whether or not it would make more sense to drive the PS_ON# signal instead. This one is driven low by the board to turn the power supply on. The problem is that some boards wire it to the ground to keep it constantly low. However it would be possible to implement the inverted signal on pin PB2 and decide depending on the motherboard whether to use PWR_OK or PS_ON#.

Downloads and links