Friday, 12 May 2017

RRDTool

RRDTool is a very cool round robin database and graph generation tool. There is a lot of documentation around the web - if you got the time (and patience).

The idea of a RRD  is a fixed number of sample data points in a given database. The size of the database is defined at its creation time. The advantage of such a database is that the space allocated is fixed, and there s no danger of running out of storage space, ever; hence it is ideal for embedded / low resource systems.


Creating a database

rrdtool create eth0.rrd \
	--start now   \
	--step 60	\
	DS:ina:GAUGE:120:U:U  \
	DS:outa:GAUGE:120:U:U  \
	DS:in:DERIVE:120:U:U  \
	DS:out:DERIVE:120:U:U  \
	RRA:LAST:0.5:1:10  RRA:AVERAGE:0.5:5:10 
eth0.rrd is the name of the database (saved filename)
--start  means when do we start populating the database 
--step means sample size in seconds ( enter data every minute  - will get a cron job to do this)
DS  - definition of a dataset: label : type  : heartbeat : lower limit : upper limit
type  can be of :
GAUGE  - save the value vithout any modification  
COUNTER - save the rate (current - last) / sampletime 
DERIVE - same as about with over/underflow adjust
ABSOLUTE - for data which gets reset when read

The key thing to remember is, the data gets modified before saving, for counter,derive types by rrdtool. Counter / Derive is ideal for saving data such as bitrate - as it is computed at ever sampling point, and then saved.

In above database each data entry point will need 4 bits of data (= number of DS definitions)
First two data will be saved as absolute values, next two will be saved as rate values. (in this case it will only start updating database after 2 sets of data has received. We the also define two Round Robin Archives :
RRA:LAST:0.5:1:10    - holds 10 absolute values, updated every minute
RRA:AVERAGE:0.5:5:10 - holds 10 values, averaged every 5 minutes

Instead of creating multiple RRA's  one can have a single RRA , with 1 minute resolution. 
The following will create a foo.rra which hold an year's worth of data in 1 minute resolution:
rrdtool create foo.rrd \
	--start now   \
	--step 60	\
	DS:in:DERIVE:120:U:U  \
	DS:out:DERIVE:120:U:U  \
	RRA:LAST:0.5:1:525600 
and the datebase will only take 8MB of space -Incredibly efficient !                                                                                                                            

Inserting data

As defined in the DS section, we must insert 4 data values at every minute (at every sample point). The command to do so is:
rrdtool update eth0.rrd N:<value>:<value>:<value>:<value>
N neans "now" - inserts the current timestamp. If samples arrive faster than the intended sample time (60 secs) the database will not be updated

Dumping out the contents of RRD

rrdtool dump eth0.rrd  will dump out the contents in xml

Creating pretty graphs ..

coming soon





Sunday, 27 December 2015

Streaming from Banana Pi using gstreamer

Banana Pi M1+ is the upgraded version of the the trusty Banana Pi with a 40 pin header with some very useful interfaces for audio; namely the i2s-0 interface and spdif output pins routed into the  upper end pins.


I coupled my DAB module - which is a i2s master device - hence set Banana Pi as i2s slave (in fex file).  The connection can be tested with alsa record / play utils  ..
  arecord  -D hw:2 -f S16_LE -c 2  -r 48000 | aplay -D hw:3  -
where hw2: is the i2s input alsa device (the DAB input); hw:3 is the usb sound card Once the i2s input is proved working, its time to stream this input to my laptop at 192.168.99.130, udp port 5000. The gstreamer pipeline at Banana Pi end is:
   gst-launch-0.10 alsasrc  device=hw:2 ! 'audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)16, depth=(int)16, rate=(int)48000,channels=(int)2'! udpsink host=192.168.99.130 port=5000
and at my laptop:
 gst-launch udpsrc port=5000 ! 'audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)16, depth=(int)16, rate=(int)48000,channels=(int)2' ! pulsesink
and to play in vlc :
 vlc --demux=rawaud --rawaud-channels=2 --rawaud-samplerate=48000 udp://@:5000
The raw audio datarate for 48k sample rate, S16LE stereo is 1536 Kbits/s. This can easily be handled in a wifi connection, and I had no issues streaming raw over udp. The aggregate data rate will be around with 1.6Mbits/s . If this needs to be carried out in a leaner channel (i.e bluetooth) then opus encoder / decoder is an excellent choice.The opus encoded pipeline is:
 Server : (Banana Pi)
 gst-launch-0.10 alsasrc  device=hw:2 ! 'audio/x-raw-int, endianness=(int)1234, signed=(boolean)true, width=(int)16, depth=(int)16, rate=(int)48000,channels=(int)2' ! opusenc !  udpsink host=192.168.99.130 port=5000
client:
 gst-launch udpsrc port=5000 ! opusdec ! pulsesink
The processor usage for the raw pipeline is about 6~7% and for the opus encoded pipeline is ~25% - excellent !!

Sunday, 4 January 2015

Using Imagemagick to generate a c array for images

Imagemagick is a really versatile tool for manipulating images from the command line. It also is a vary handy tool for generating image bitmaps for embedded systems.
This example is for generating a data array for SSD1306 based OLED displays. These are available  for a few pounds from china / ebay and comes in two forms of interfaces:   i2c or spi interface.  For simplicity I picked up an i2c one.
The data is written as column major. i.e each byte represents 8 vertical pixels. The following image is from the SSD1306 datasheet, describing display ram writing sequence:
click to enlarge
It takes 128x64/8 = 1024 bytes for a full image for these displays. So lets start with a simple 350x174 png image :

now use imagemagick to convert this to an .xbm

  convert -resize 128x64\! 1.png 1.xbm  
The 1.xbm file looks like :
 #define 1_width 128  
 #define 1_height 64  
 static char 1_bits[] = {  
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xC0,   
  0x03, 0x7C, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   
  0x00, 0x00, 0x00, 0x60, 0x87, 0xCF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,   
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x30, 0xFC, 0x81, 0x01, 0x00,   
  0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18,   
 ........ abbreviated ...       
  0x00, 0x00, 0x00, 0x00, 0x00, 0xC0, 0x07, 0x00, 0x00, 0x00, 0x00, 0x00,   
  0x00, 0xC0, 0x7F, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00,   
  0x00, 0x00, 0x00, 0x00, };  
Now we have a c array with 1024 bytes where eight pixels in each byte. as mentioned before, this data is row major, and needs to be transposed to column major array of 1024 bytes. The transpose looks something like :
      __u8 xfdata[1024] = {0x00,};   
      int row, col , offset , pos;  
      pos =0;   
      for (row=0; row <1024; row +=128)  
      {  
           for (col=0; col <128 ; col++)  
           {  
                pos = (0x07 & col);  
                offset = (col >> 3);  
                if ((1<<pos) & data[row+offset+0]) xfdata[row+col] +=1;  
                if ((1<<pos) & data[row +offset+16]) xfdata[row+col] +=2;  
                if ((1<<pos) & data[row +offset +32]) xfdata[row+col] +=4;  
                if ((1<<pos) & data[row +offset +48]) xfdata[row+col] +=8;  
                if ((1<<pos) & data[row +offset + 64]) xfdata[row+col] +=16;  
                if ((1<<pos) & data[row +offset +80]) xfdata[row+col] +=32;  
                if ((1<<pos) & data[row +offset +96]) xfdata[row+col] +=64;  
                if ((1<<pos) & data[row +offset +112]) xfdata[row+col] +=128;  
           }  
      }  

the xfdata[] can now be dumped into SSD1306 ...
note: this display is a dual colour display, where top 16rows are yellow, hence the yellow hats !


Sunday, 14 December 2014

Arduino Due code development without the Java IDE

The arduino Due is quite a nice cortex M3 board to play with, and the whole arduino concept is great;  but I personally dislike IDEs and I do not want to clutter my laptop with Java VM - It would be nice if could develop in my usual vi and makefiles.
As described in this blog,  someone else had the same issue and found a way around it. The only problem is the php script included ,  which refused to execute in my machine. Instead of trying to fix it, it was easy enough to transfer php to a bash script:
Due with usb cable connected to the programming port. Click to enlarge
First download the Arduno IDE for Due. We need the 1.5.8 beta for Due, and extract ii
 wget http://arduino.cc/download.php?f=/arduino-1.5.8-linux64.tgz
 tar -xvf arduino-1.5.8-linux64.tgz -C /tmp/foo/
download and extract atwillys script
 wget http://atwillys.de/content/cc/using-custom-ide-and-system-library-on-arduino-due-sam3x8e/arduino-due-build-tools-setup.tgz  
 tar -xvf arduino-due-build-tools-setup.tgz -C /tmp/foo/  
This will create two directories in /tmp/foo/
 drwxrwxr-x 8 1000 1000 4096 Oct 1 15:57 arduino-1.5.8  
 drwxr-xr-x 3 1000 1000 4096 Sep 10 2013 install  
change to install directory  and remove ._* files  and create a bash script
 cd install  
 find . -name ._* -delete  
 cat > script <<EOF  
 #!/bin/bash  
 if [ $# -eq 0 ]  
 then  
   echo "arduino tool path missing in command line"  
   exit 0  
 else  
 ROOT_DIR=due  
 SCRIPT_RESOURCE_DIR=resources  
 TOOLS_DIR=$ROOT_DIR/tools  
 GCC_EABI_DIR=$TOOLS_DIR/g++_arm_none_eabi  
 BOSSAC=$TOOLS_DIR/bossac  
 HARDW_DIR=$ROOT_DIR/sam  
 DIST_LIB_DIR=$ROOT_DIR/lib  
 DIST_INC_DIR=$ROOT_DIR/include  
 DIST_SRC_DIR=$ROOT_DIR/src  
 SRC_DIR=$1  
 mkdir $ROOT_DIR  
 mkdir $TOOLS_DIR  
 mkdir $HARDW_DIR  
 mkdir $DIST_LIB_DIR  
 mkdir $TOOLS_DIR/g++_arm_none_eabi  
 cp -av $SRC_DIR/hardware/tools/gcc-* $TOOLS_DIR/  
 cp -av $SRC_DIR/hardware/tools/bossac $BOSSAC  
 cp -av $SRC_DIR/hardware/arduino/sam/system/CMSIS $HARDW_DIR/  
 cp -av $SRC_DIR/hardware/arduino/sam/system/libsam $HARDW_DIR/  
 cp -av $SRC_DIR/hardware/arduino/sam/variants/arduino_due_x/linker_scripts $HARDW_DIR/  
 cp -av $SRC_DIR/hardware/arduino/sam/variants/arduino_due_x/libsam_sam3x8e_gcc_rel.a $DIST_LIB_DIR/   
 find $HARDW_DIR -type f -exec chmod 644 {} \;  
 find $HARDW_DIR -type d -exec chmod 755 {} \;  
 find $HARDW_DIR -type f -name *.a -exec cp -av {} $DIST_LIB_DIR/ \;   
 cp -R $SCRIPT_RESOURCE_DIR/include $DIST_INC_DIR/  
 cp -R $SCRIPT_RESOURCE_DIR/make $ROOT_DIR/  
 cp -R $SCRIPT_RESOURCE_DIR/example $ROOT_DIR/  
 sed -i 's/g++_arm_none_eabi/gcc-arm-none-eabi-4.8.3-2014q1/g' $ROOT_DIR/make/make_due_firmware.mk  
 sed -i 's/cu.usbmodemfa141/ttyACM0/g' $ROOT_DIR/make/make_due_firmware.mk  
 sed -i 's/stty -f/stty -F/g' $ROOT_DIR/make/make_due_firmware.mk  
 echo "all done.."  
 fi  
 exit 0  
 EOF  

make the script executable and run it wil the path to the extracted arduino toolchain
 chmod +x script  
 ./script ../arduino-1.5.8  

Once the script is finished, there will be a directory called Due. which is the top directory of the command-line tool chain. cd to examples directory and do a make, with arduino due connected to usb port (/dev/ttyACM0)

Thanks to atwillys.de for all the hard work ..

building the code and programming the Due - click to enlarge

Friday, 12 December 2014

Tiny hackable AR9331 router from china

Atheros AR9331 soc is well established and has been incorporated into many router designs ;  and while searching through the net for nice, inexpensive AR9331 board to play around with,  I  came across this one from china for £25 including postage.

Click to enlarge
Specification :
  • soc : Atheros AR9331 rev 1 MIPS 24kc v7.4
  • 64MB DRAM
  • 8MB SPI NOR Flash
  • One USB2.0 host port
  • One debug port - which also is the +5v power input to the board
  • ethernet - one 100Mbit/s port - usually used as the wan port
  • ethernet - two 100Mbits/s lan ports
  • 802.11 b/g/n wifi with a pretty effective built in printed antenna.
  • i2c and gpio - these are available via the tiny pads on the pcb
flip side showing the io pads - click to enlarge

NOR flash layout: (relative addresses)
 0x000000000000-0x000000020000 : "u-boot"  
 0x000000020000-0x0000007f0000 : "firmware"  
 0x0000007f0000-0x000000800000 : "art"  
BIG FAT WARNING : The board comes pre flashed and make sure never to touch the u-boot sectors, unless you really know what you are doing. The art partition carries the firmware for the wlan - and make sure this 64k sector is backed up. The wlan interface will not come up if this sector is wiped.

Here is the uboot interrupted and dump the help. The factory flashed uboot already has all the sub functions we need; hence no need to mess with it.
click to enlarge
The ethernet ports are not correctly configured in the shipped firmware,. My modified firmware bin file is here.  (md5sum : 43886d43758b69975778bd9722e94973a  openwrt-ar71xx-generic-som9331-squashfs-factory.bin)

absolute addresses (for u-boot):
 0x9f000000-0x9f01ffff : u-boot
 0x9f020000-0x9f7effff : kernel + rootfs 
 0x9f7f0000-0x9f7fffff : art - atheros firmware  

at u-boot prompt, the kernel +rootfs can be erased by:
 erase 0x9f020000 0x9f7effff   
frash erase - click to enlarge
The default u-boot environment setting are as follows :
  • ipaddres   192.168.1.1
  • server ip   192.168.1.2
  • filename f irmware.bin
setup the tftp server, and make sure the server ip is 192.168.1.2. Connect the ethernel cable to one of the lan ports of som9331.  Rename the above file to firmware.bin and:
 tftp   
This will download the firmware.bin file from the tftp server
after the tftp transfer, the image is now in ram @ 0x80800000 
note the filesize in bytes transferred line  (0x7c0000) . Copy this to flash :
 cp.b 0x80800000 0x9f020000 0x7c0000   
finally, copy to flash 
once done, remove the ethernet cabe, and press reset button to boot the shiny new openwrt .. remember to set the root password.



Thursday, 11 December 2014

Now this is what I call a Media Player

I am always on the lookout for cool hardware which can be used as a Linux media player. The last incarnation was based on xbmc, on a Nvidia ION board, with a DVB-S2 PCI satellite card and a 2TB sata disk; and that was almost four years ago. The satellite tuning was based on TV headend and xbmc, with a few kernel tweaks to get the PCI card driver running.  This set-up served me well, but there were few things I liked to change:

  • lowest possible power  - Nvidia ION / Intel atom was not too bad, but nicer to lower the   power consumption to a few tens of watts
  • no moving parts to break - this includes cooling fans platter disks - so that whole system is noise free and reliable
I would have loved very much to turn the Bananpi into a satellite receiver, but the sunxi vdpau drivers are still being worked at, may be some day the folks at Allwinner decide to open up their a/v decoder core. 

This is the receiver I bought ; and arrived in UK in just 3 days. The receiver is based on Broadcom BCM7325, which is at the centre of the board. I was curious as to how the Broadcom achieves micro watt standby power. I have seen how a Broadcom chip struggled to achieve extremely low standby - where we finally resorted to using a secondary  stm32 as the power controller to shut the Broadcom device OFF to achieve micro watt standby power.
This unit uses a PIC 16F866 device (on the front panel board) as the deep standby controller. It also is the the IR receiver. It then communicates with the Broadcom via serial port (on the ribbon cable) at 19200 baud.

click to enlarge
I removed smart card interface (as I will only watch FTA), and added the 40 GB Intel SSD onto the smart card board;  which cleanly fits inside the 22cm x 15cm X 4cm high case. The usb-sata interface board was ripped out of a cheap portable disk case.
On the rear are the usual interfaces, from left to right:

  • F female input for sat coax
  • video out (RCA)
  • audio out L + R (RCA)
  • RS232 debug interface (115200, 8N1) 
  • HDMI out
  • ethernet 100Mbits/s
  • SPDIF out
  • USB host
  • 12v /3A input barrel socket
Just the right amount of interfaces - I am fan of SPDIF - so its a must on any hardware I tinker with. There is another 3 usb host ports, and only one connector populated - Which I have used for the SSD.
The other two ports need 4 impedance matching resistors and a few chokes and capaitors. One cool interface we can is a USB DVB-T tuner. 

The receiver is pre-installed with enigma2 Black Box; and my personal preference is Open-PLi - so first step is to re flash the OpenPLi image.  I am too much in a hurry to to go through a git checkout + OpenEmbedded build, so just using the latest build for vu+ solo.

The zip file contains three files:

 vuplus  
 └── solo  
   ├── boot_cfe_auto.jffs2  
   ├── kernel_cfe_auto.bin  
   └── root_cfe_auto.jffs2  

In order to update the software you need a <2GB usb stick formatted to FAT16. This is a must for most Broadcom devices. Since I couldn't find a 2GB usb, I used a 16GB one with a single 2GB partition with partition id 0x0c formatted to FAT 16.

cfe, is the u-boot-ish bootloader and is quite powerful and flexible, just as u-boot is. Out of the three files unzip, the boot_cfe_auto.jffs2 seems to be some replacement bootloader file which I was not sure I should risk updating. The installed CFE works well, so there was no reason to replace it, all I need is the kernel and rootfs images updated. ( Didnt want to brick the box while updating the bootloader )

Hence the fat-16 formatted disk has the layout as follows:
 vuplus  
 └── solo  
   ├── kernel_cfe_auto.bin  
   └── root_cfe_auto.jffs2  

Plug the RS232<->USB cable, open up screen on /dev/ttyUSB0 115200 and plug the usb stick on the rear USB port and power up the device. The device should not be power down / up for the usb update using the front panel button. knock the power completely off from the wall socket, and plug usb and switch on the wall socket.
The debug interface is very handy at this stage - it nicely spits out the whole flashing process and at the end, prints a message to knock the power off and remove the USB stick and reboot.
Now my shiny new OpenPLi satellite receiver / media player is ready. Here is Servus HD from Austria on Astra 19.2e with modified PLi-HD skin :

Servus HD from Austria - with UI overlayed 
channel UI - click to enlarge

Saturday, 6 December 2014

High Quality USB sound card for Banana Pi

Banana Pi designers have dropped the A20 SPDIF OUT in Banana Pi. So if you need to feed the PCM signal into your amplifier via fiber, this is an excellent module. It is based on Burr Brown PCM2704 device

Click to enlarge
The kernel config needed is already enabled in the config shown here

update : a new Banana Pro is on the way, with  SPDIF output and I2S - check it out here