RSS News Headlines Prompter update.

I made few improvements to my news headlines prompter appliance, so I thought I'd share if anybody cares (or not, have nothing better to do at the moment :-)).
The improvements were mainly done in scripts downloading news and driving the LED matrix message board device, which I modified to run from crontab now. DSL has simplified scheduler, which is made with single threaded perl script. That means if your script is going to run longer that just few seconds or minutes, you need to take into consideration the fact that it will block other scheduled jobs from running. You can also put the scheduled job/script to run in background thus unblocking the MyCron script to parse and run other jobs - just make your script/program aware of other instances that may be already started by cron.
Activation of cron server on DSL is simple. Just added this line in my /opt/bootlocal.sh file:

/usr/bin/perl -w /usr/local/bin/MyCron &

and commented out my previous scripts startup, now they will be activated from crontab file:

#bash ldmtrxmsgbrd! >/dev/null 2>&1 &
#bash getnews! >/dev/null 2>&1

Also, because my DSL does not have DST patch (Busch's legacy), I added time synchronization to the /opt/bootlocal.sh, so the date/time function of the script would be accurate:

sudo /usr/local/bin/gettime.lua nist.expertsmi.com 30

File /opt/crontab now looks like this:

* * * * * * echo Cron timestamp `date` >> /tmp/crontest
* * * * * * /home/dsl/bin/ldmtrxmsgbrd! > /tmp/ldmtrxmsgbrd.trc 2>&1
0,10,20,30,40,50 * * * * * /home/dsl/bin/getnews! > /tmp/getnews.trc 2>&1

The ldmtrxmsgbrd! script (led matrix device driver) is triggered every minute. If no files are present in temporary dedicated directory, it will display date and time on the led matrix device. If there are files, they will be uploaded one by one to the device and script will go to sleep for calculated amount of time after each of them is uploaded (each file is removed from temporary directory after upload). Next instances are simply not going to be triggered by cron server (since it is single-threaded) until the current instance ends its sleep. However ldmtrxmsgbrd! script is capable of detecting already running instance of itself and quit if such is detected, therefore it should work as well with regular (multi-threaded) cron scheduler.
The getnews! script runs every 10 minutes, however if will only download new data when no files are present in dedicated temporary directory. The directory will be empty when ldmtrxmsgbrd! script processes all data files.

Improvements to getnews! script include downloading Atlantic Tropical Weather Outlook RSS from NOAA as well as current conditions and 7-day forecast for my area code. Now the getnews! script looks like this:

#!/bin/bash
# Download RSS feed from BBC - crontab version.
# Parse headlines out to plain text file, split it up
# to 30 lines per file and feed to to ldmtrxmsgbrd! script
# (meaning, store in predefined location /tmp/ldmtrx)

cd ~dsl/bin

mkdir -p /tmp/ldmtrx
cd /tmp
P=`ls /tmp/ldmtrx/ldmtrxnews* 2>/dev/null|head -n1`
if [[ "tt$P" = "tt" ]]
then
        wget -O rss.xml http://en-us.fxfeeds.mozilla.com/en-US/firefox/headlines.xml
        ~dsl/bin/rssparse! rss.xml > headlines_$$.tmp
        cd /tmp/ldmtrx
        split --lines=30 /tmp/headlines_$$.tmp ldmtrxnews
        cd /tmp
        rm headlines_$$.tmp rss.xml

        wget -O rss.xml http://www.nhc.noaa.gov/index-at.xml
        ~dsl/bin/rssparse! rss.xml > headlines_$$.tmp
        cd /tmp/ldmtrx
        split --lines=30 /tmp/headlines_$$.tmp ldmtrxnews2
        cd /tmp
        rm headlines_$$.tmp rss.xml

        wget -O weather.html "http://mobile.weather.gov/port_mp_ns.php?select=3&CityName=Palm%20Harbor&site=TBW&State=FL&warnzone=FLZ050"
        ~dsl/bin/htmlparse! weather.html > headlines_$$.tmp
        cd /tmp/ldmtrx
        split --lines=30 /tmp/headlines_$$.tmp ldmtrxnews3
        cd /tmp
        rm headlines_$$.tmp weather.html

        wget -O weather.html "http://mobile.weather.gov/port_mp_ns.php?select=1&CityName=Palm%20Harbor&site=TBW&State=FL&warnzone=FLZ050"
        ~dsl/bin/htmlparse! weather.html > headlines_$$.tmp
        cd /tmp/ldmtrx
        split --lines=30 /tmp/headlines_$$.tmp ldmtrxnews4
        cd /tmp
        rm headlines_$$.tmp weather.html
fi

Script ldmtrxmsgbrd! that drives led matrix device received improvements as well:

#!/bin/bash
# Crontab version.
# Monitor /tmp/ldmtrx directory for files by ldmtrxnews* pattern.
# If none, send date and time to Led Matrix Scrolling Board and exit.
# If files found, upload texts from the 1st file to the Board in MAUTO mode.
# Sleep for 10 minutes.
# Remove file.

function ShowDate()
{
   DS=`date | awk '{print substr($0,0,16);}'`
   ~dsl/bin/ldmtrxcmd! scrloff ttyS0
   ~dsl/bin/ldmtrxcmd! mext ttyS0
   ~dsl/bin/ldmtrxcmd! stext ttyS0
   ~dsl/bin/ldmtrxcmd! "$DS" ttyS0
}

cd ~dsl/bin
if [[ ! -f /tmp/ldmtrxmsgbrd_sem.tmp ]]
then
        echo "$$" > /tmp/ldmtrxmsgbrd_sem.tmp
        ~dsl/bin/ldmtrxendscript! ttyS0
        if [[ ! -f /tmp/ldmtrxbrd_undetected.tmp ]]
        then
                F=`ls /tmp/ldmtrx/ldmtrxnews* 2>/dev/null|head -n1`
                if [[ -s $F ]]
                then
                        ~dsl/bin/ldmtrxcmd! mauto ttyS0
                        ~dsl/bin/ldmtrxcmd! upl ttyS0

                        ~dsl/bin/loader! $F ttyS0
                        ~dsl/bin/ldmtrxcmd! @EOT ttyS0
                        ~dsl/bin/ldmtrxcmd! scrlon ttyS0
                        ~dsl/bin/ldmtrxcmd! del30 ttyS0
                        LINES=`cat $F|wc -l 2>/dev/null`
                        let SLEEP=30*LINES
                        echo "LINES=$LINES, SLEEP=$SLEEP"
                        rm $F
                        sleep $SLEEP
                else
                        if [[ -f $F ]]
                        then
                                rm $F
                        fi
                        ShowDate
                fi
        else
                echo "Led Matrix Scrolling Message Board device is not present."
                echo "Exiting."
        fi
        rm /tmp/ldmtrxmsgbrd_sem.tmp
else
        echo "Another instance is running."
fi

Note the time for the script to sleep while messages are being displayed/scrolled on led matrix device is now calculated from the number of lines that file actually contains multiplied by 30 seconds instead to be fixed at 900 seconds. There is still space to improvement here, since there may be long lines in the text (especially in weather forecasts) which may need more than 30 seconds to be fully scrolled across. Will be corrected in the next version.

Cosmetic changes to rssparse! as well as new parsing script htmlparse! were done:

Contents of rssparse! script:

#!/bin/bash

cat $1 | awk '

START
{
   pos=1;
   xml=$0
   len=length(xml);
   endp=1
}

{
   while(pos <= len)
   {
      if(substr(xml,pos,7) == "<title>")
      {
         pos=pos+7;
         endp=pos;
         while((substr(xml,endp,8) != "</title>") && (endp < len))
         {
            endp++;
         }
         print "   ",substr(xml,pos,endp-pos)," * ";
         pos=endp+7;
      }
      pos++;
   }
}'

Contents of htmlparse! script:

#!/bin/bash

cat $1 | awk '

START
{
   pos=1;
   xml=$0
   len=length(xml);
   str=""
}

{
   while(pos <= len)
   {
      if((pos <= len - 5) && (substr(xml,pos,6) == "<html>"))
      {
         pos=pos+5;
      }
      else if((pos <= len - 4) && (substr(xml,pos,5) == "<meta"))
      {
         pos=pos+5;
         while(substr(xml,pos,1) != ">")
         {
            pos++;
         } 
      }
      else if((pos <= len - 5) && (substr(xml,pos,6) == "<body>"))
      {
         pos=pos+5;
      }
      else if((pos <= len - 2) && (substr(xml,pos,3) == "<b>"))
      {
         pos=pos+2;
      }
      else if((pos <= len - 3) && (substr(xml,pos,4) == "</b>"))
      {
         pos=pos+3;
      }
      else if((pos <= len - 3) && (substr(xml,pos,4) == "<div"))
      {
         po=pos+4;
         while(substr(xml,pos,1) != ">")
         {
            pos++;
         } 
      }
      else if((pos <= len - 3) && (substr(xml,pos,4) == "<hr>"))
      {
         pos=pos+3;
         if(length(str) > 0)
         {
            str=str " * ";
            print str;
            str=""
         }
      }
      else if((pos <= len - 3) && (substr(xml,pos,4) == "<br>"))
      {
         pos=pos+3;
         if(length(str) > 0)
         {
            str=str " * ";
            print str;
            str=""
         }
      }
      else if((pos <= len - 5) && (substr(xml,pos,6) == "<form>"))
      {
         pos=pos+6;
         while((substr(xml,pos,7) != "</form>") && (pos <= len))
         {
            pos++;
         }
         pos=pos+6;
      }
      else if((pos <= len - 5) && substr(xml,pos,6) == "</div>")
      {
         pos=pos+5;
      }
      else if((pos <= len - 6) && substr(xml,pos,7) == "</body>")
      {
         pos=pos+6;
      }
      else if((pos <= len - 6) && substr(xml,pos,7) == "</html>")
      {
         pos=pos+6;
      }
      else
      {
         str=str substr(xml,pos,1);
      }
      pos++;
   }
}
END
{
}'

When shutting down the system, we need to make sure the cron server does not work before we switch led matrix device into standalone clock mode. Otherwise the lenghty backup process may last long enough for new instances of getnews! and ldmtrxmsgbrd! scripts to be triggered from crontab and alter the device mode. The new /opt/powerdown.sh script now looks like this:

#!/bin/sh
# Put system command to perform upon system shutdown
# Kill cron server
# Set LED Matrix Scrolling Message Board to the clock mode.
bash ~dsl/bin/pwdwn!
# automate system backups
cleanMyDSL.sh
if [ -s /opt/.backup_device -a ! -f /opt/.skip_backup ]; then filetool.sh backup noprompt; fi

Note the ability to bypass the backup step during shutdown was added by the means of manual creation and then detection of /opt/.skip_backup file in powerdown.sh script. That file does not have to be explicitly removed because persistent setup does not include this file, it will be gone after next boot up. Just make sure you have all changes backed up before using this  mechanism by running the tool manually if changes to the system setup/vital scripts were made. It'd be actually a good idea to schedule backup in crontab, only you need to make sure that any backup process that started is finished and it is not duplicated by running another one (create own wrapper script). Bypassing backup process at power down speeds up the shutdown/reboot process greatly in the DSL system configured like mine (boot from flash drive, running in RAM, persistence).

Contents of pwdwn! script:

#!/bin/bash
# Kill cron server.
CPID=`ps|grep MyCron|awk '{print $1;}' 2>/dev/null`
if [[ $CPID -gt 1 ]]
then
        sudo kill $CPID
fi
# Set LED Matrix Scrolling Message Board to the clock mode.
bash ~dsl/bin/inittty! /dev/ttyS0
bash ~dsl/bin/ldmtrxsettime! ttyS0

I did not realize when I started this project that it'd be so much fun. Perfecting the automated system so it can run unattended and deliver its function in reliable and efficient manner is just what is exciting in the job of any engineer.

Thanks for reading.

Marek

PS:

And now traditionally, some pictures:

Pic. 1: Weather news are not great, but not bad either :-)

Pic. 2: It only takes these many processes to run DSL as platform for my news headlines prompter and few extras.

Pic. 3: All runs from RAM.

Pic. 4: Sweet little micro ITX mobo, MSI fuzzy CX700D.

Pic. 5: No news is good news. And I know now that it is late... :-)

Damn Small Linux + Led Matrix Scrolling Message Board = RSS News Headlines Prompter

It's been a long time. Taking a little rest from MKHBC-8-R1 project and in the meantime I purchased Mini ITX motherboard for my electronic projects that may require standalone embedded PC compatible computer controller. I wanted to build quiet, fanless, diskless computer system. I have been a Damn Small Linux distro fan for a long time. So I decided that I'd put together something cool out of that motherboard and DSL. Remember my Led Matrix Scrolling Message Board project? (the first post on this blog). It comes in handy. I imagined I'd like to have a RSS news headlines prompter in my room. What's a better OS to automate such a task than Linux and small solid state computer system that uses little power? For the cost of less than $150 (including parts for my Led Matrix Board, RAM for mobo and flash drive for DSL boot) I achieved my goal. This thing has no case and wires are sticking out, but it is just the way I like it :-).

Let me describe step by step how I built that system.

• I downloaded ISO image of DSL and burned a CD ROM.

• I put 1 GB RAM in the mobo and connected it to the power (it only needs 4-pin ATX power, like the one that goes to HDD).
• I connected keyboard, mouse and monitor and booted up computer to the BIOS.
• Setup boot devices scan sequence to CD ROM, flash, HDD.
• I put DSL CD in external DVD drive connected to mobo via USB and restarted system.
• After DSL boot prompt appeared, I pressed ENTER. System automatically booted to X windows.
• Used DSL wiki page (opened on another computer) to figure out how to install DSL to flash drive.
• In the meantime I realized that DSL does not support mobo's on board ethernet adapter. So, no network (yet).
• Rebooted the system (DSL asked to remove CD so it could boot from flash). System booted from flash, automatically created RAM drive and loaded kernel image and necessary file systems to it. I have now a diskless computer system running entirely from RAM with graphical interface.
• Found old PCI network adapter, compatible with DSL. Shutdown the system, inserted network card in single available PCI slot and booted the system. DSL automatically recognized the network adapter and used DHCP protocol to get IP address from my router. Now I have OS with graphical interface, firefox browser and internet running entirely from RAM, most of it still free for applications.
• I downloaded some extra packages from internet to expand my DSL: GCC compiler, java JDK, samba (file server).
• I read DSL wiki again to figure how to make my extensions stick (persistence, remember, this is OS running from RAM, no permanent storage except USB flash drive).
• I downloaded source code of kernel and follow instructions how to compile it. I stored the code on compact flash drive (not USB stick) attached to IDE flash drive slot that mobo is equipped with.
• I downloaded source code of linux ethernet driver for my mobo from realtec's web page. I follow instructions included with package to compile driver module.
• I figured out how to automatically load network adapter driver module at system startup and make it go up and seek IP via DHCP.  Solution: two lines in /opt/bootlocal.sh script:

cd /home/dsl/src/r1000_v1.04 && sudo make install && sudo depmod -a && sudo modprobe r1000 && cd -

/sbin/pump

I shutdown the system, unplug external PCI ethernet adapter and plug ethernet cable to on-board adapter. Reboot the system...it worked!

So, now all I needed to do was to create set of scripts that would download and display RSS Headlines on my Led Matrix Board.

First, I needed to implement basic communication with Led Matrix Board via serial port. Linux makes it so easy. No need to write programs, all can be done with scripting. The serial protocol can only take text via wire this fast, so to upload commands from text file via serial port I had to create this little script with delay loops for characters and lines made with awk:

dsl@box:~/microcom/scripts/ledmtrx$ cat loader!
cat $1|awk '{for (i=1;i<=length($0);i++){printf substr($0,i,1); for(j=0;j<10000;j++){}; }; print ""; for(j=0;j<100000;j++){};}' > /dev/$2

Script takes 2 arguments: $1 - file name with commands for Led Matrix Board and $2 - serial device name (e.g: ttyS0).

Now, script to send single command to the board:

dsl@box:~/microcom/scripts/ledmtrx$ cat ldmtrxcmd!
printf "$1\n" | loader! - $2

Board can be in script execution mode. It is a default mode after power up. I needed a script that would terminate that mode and enter board into command accepting mode. Script should also be able to recognize if the board is responding (present, ON) or not:

dsl@box:~/microcom/scripts/ledmtrx$ cat ldmtrxendscript!
#!/bin/bash

let "count=20"
rm /tmp/ldmtrxbrd_undetected.tmp

DN=$1
if [[ $DN = "" ]]
then
   DN=ttyS0
fi

IsScriptMode()
{
   ldmtrxcmd! conf $DN
   sleep 1
   M=`cat /tmp/ldmtrxendscript_$DN_$$.tmp|grep OK|tail -n 1 | awk -vFS="|" '{print $3;}'`
   if [[ $M = "4" || ($M != "0" && $M != "1" && $M != "2" && $M != "3" && $M != "4") ]]
   then
      echo "true"
   else
      echo "false"
   fi
}
cat /dev/$DN > /tmp/ldmtrxendscript_$DN_$$.tmp 2>/dev/null &
CPID=`ps|grep cat|grep dev|grep $DN|awk '{print $1}' 2>/dev/null`
while [[ `IsScriptMode` = "true" && $count -gt 0 ]]
do
#   ldmtrxcmd! "\n\n\n" $DN
   ldmtrxcmd! endscript $DN
   let "count=count-1"
done
kill $CPID
rm /tmp/ldmtrxendscript_$DN_$$.tmp
if [[ $count -eq 0 ]]
then
   echo "Unable to disable script mode."
   echo "Possibly Led Matrix Scrolling Message Board device is not present."
   touch /tmp/ldmtrxbrd_undetected.tmp
else
   echo "Script mode disabled."
fi

Script that turns Led Matrix Board into clock ($1 - serial device, e.g: /dev/ttyS0):

dsl@box:~/microcom/scripts/ledmtrx$ cat ldmtrxsettime!
echo "mclk`date +%H%M%S`" > $1

And finally a script that would monitor contents of certain directory in a loop and if files detected, upload their contents to the board, otherwise just display current date and time:

dsl@box:~/microcom/scripts/ledmtrx$ cat ldmtrxmsgbrd!
#!/bin/bash
# Monitor ./tmp subdirectory for files by t* pattern.
# If none, send date and time to Led Matrix Scrolling Board every 15 seconds.
# If files found, upload texts from the file to the Board in MAUTO mode and sleep
# for 10 minutes. Remove file and send the contents of the next file until all
# are processed.

function ShowDate()
{
   DS=`date | awk '{print substr($0,0,16);}'`
   ldmtrxcmd! mext ttyS0
   ldmtrxcmd! stext ttyS0
   ldmtrxcmd! "$DS" ttyS0
}

ldmtrxendscript! ttyS0
if [[ ! -f /tmp/ldmtrxbrd_undetected.tmp ]]
then
        while true
        do
           for F in `ls ./tmp/t* 2>/dev/null`
           do
              if [[ -s $F ]]
              then
                 ldmtrxcmd! mauto ttyS0
                 ldmtrxcmd! upl ttyS0
                 loader! $F ttyS0
                 ldmtrxcmd! @EOT ttyS0
                 rm $F
                 sleep 600
              else
                 rm $F
              fi
           done
           ShowDate
           sleep 15
        done
else
   echo "Led Matrix Scrolling Message Board device is not present."
   echo "Exiting."
fi

All that is missing now is a script that'd download and parse RSS feed:

dsl@box:~/microcom/scripts/ledmtrx$ cat getnews!
#!/bin/bash
# Download RSS feed from BBC every 30 minutes,
# Parse headlines out to plain text file, split it up
# to 30 lines per file and feed to to ldmtrxmsgbrd! script.

let "count=1"

cd ~dsl/microcom/scripts/ledmtrx

while true
do
   wget -O rss.xml http://en-us.fxfeeds.mozilla.com/en-US/firefox/headlines.xml
   rssparse! rss.xml > /tmp/headlines_$count.tmp
   cd ./tmp
   split --lines=30 /tmp/headlines_$count.tmp t
   cd -
   sleep 1800
   rm /tmp/headlines_$count.tmp
   let "count=count+1"
done

dsl@box:~/microcom/scripts/ledmtrx$ cat rssparse!
#!/bin/bash

cat $1 | awk '

START
{
   pos=1;
   xml=$0
   len=length(xml);
   endp=1
}

{
   while(pos <= len)
   {
      if(substr(xml,pos,7) == "<title>")
      {
         pos=pos+7;
         endp=pos;
         while((substr(xml,endp,8) != "</title>") && (endp < len))
         {
            endp++;
         }
         print substr(xml,pos,endp-pos);
         pos=endp+7;
      }
      pos++;
   }
}'

and I was done.

What's left is to add these scripts at system startup so they'd automatically start working after system boots up. Added to /opt/bootlocal.sh:

cd ~dsl/microcom/scripts/ledmtrx

bash ldmtrxmsgbrd! >/tmp/ldmtrxmsgbrd.trc 2>&1 &

bash getnews! >/tmp/getnews.trc 2>&1 &

Rebooted the system and... voila!

Some pictures and screenshots:

Pic.1 - The MSI Fuzzy CX700D mobo. Flash drive with DSL image and extensions in USB port, on-board ethernet used, serial port connects the computer to my Led Matrix Scrolling Message Board. Keyboard, mouse and monitor are not connected as I access the system entirely via network (SSH, samba). I disabled X windows at start up. Ribbon cable is connected but no HDD is in use - just left the cable in the slot since it is a tight fit.

Pic. 2 - No RSS feed available, script ldmtrxmsgbrd! just sends current date and time to the board in 15 seconds intervals.

Pic. 3 - RSS feed arrived, headlines are uploaded to board and displayed in a loop for 10 minutes, then script will scan for the available processed RSS headlines and if not found, will return to displaying date/time.

Pic. 4 - Two x-term ssh sessions opened to the DSL system. On the left I sent command to the led matrix board, on the right is the output from serial port connected to the board.

Pic. 5 - Output from serial port shows that ldmtrxmsgbrd! script sends date and time to the led matrix board.

Pic. 6 - Output from serial port shows script ldmtrxmsgbrd! sending headlines to the led matrix board.

Isn't computing great? I just love this stuff.

Thank you for reading.

Marek Karcz