Node

Streams, Promises and Throttles

Recently in my journey I came across a nice little challenge to read a really large csv, pull data from it and then call a couple of APIs IOT to get some data then output a CSV after processing was complete.  The file I was provided basically contained a long list if IDs which I needed to use to get information from one API then use that information to call Watson.  I hadn’t flexed my muscles with the stream API for a while so it was fun.  I just wanted to share this example in case other are looking for a way to solve a similar problem.

Continue reading…

Putting a newer version of Node.js on LinkIt Smart 7688 Duo

Today I’m going to step you through putting a newer version of Node JS on your LinkIt Smart 7688.  The default version of node available is 0.12.7 which, let’s face it, is completely outdated and essentially useless for to any serious Node.js developers.  Now there are some ongoing challenges with putting a completely up to date version of Node on the MIPS architecture. This is because there are key libraries needed to build Node JS that are not updated on MIPS yet… so since those libraries out out of date we can only get so far.

Setup your linux machine.

I use Ubuntu server 16.04.2 LTS at the time of writing this tutorial.  Get Virtual Box, download an Ubunut 16 .ISO, create a new VM, make sure your disk is around 50GB (5GB is not enough… found this out the hard way), select the ISO you downloaded, install the operating system (be sure to add SSH for convenience).

I alway use SSH for copy paste commands.  To setup you VM so you can ssh to it:

  1. Open the setting for the VM (you can do this while its running)
  2. Go to Network > Adapter 1 > Advanced > Port Forwarding
  3. Forward 127.0.0.1:2200 to 10.0.2.15:22
    • Check your ifconfig to verify your VMs IP address

SSH to it from your host with this:

ssh usename@127.0.0.1 -p 2200

Alternatively you can put the adapter in Bridge mode… then check ifconfig for the VMs IP.

Setup for the build

Following for guide here with modified instruction for things they forgot:

https://docs.labs.mediatek.com/resource/linkit-smart-7688/en/tutorials/firmware-and-bootloader/build-the-firmware-from-source-codes

and with help from the nxhack who is running this repo:

https://github.com/nxhack/openwrt-node-packages/tree/for-15.05

  1. Add the python package (if you didn’t install during Ubuntu OS install)
    sudo add-apt-repository ppa:fkrull/deadsnakes
    sudo apt-get update
  2. Install prerequisite packages to build the firmware:
    sudo apt-get install git g++ libncurses5-dev subversion libssl-dev gawk libxml-parser-perl unzip wget build-essential python2.7
  3. Download the OpenWrt CC source codes:
    git clone git://git.openwrt.org/15.05/openwrt.git
  4. Prepare the default configuration file for feeds:
    cd openwrt   
    cp feeds.conf.default feeds.conf
  5. Add the LinkIt Smart 7688 development board’s and NodeJS to the builds feed:
    sudo nano ./feeds.conf

    Add the following lines to the bottom of the file:
    src-git linkit https://github.com/MediaTek-Labs/linkit-smart-7688-feed.git
    src-git node https://github.com/nxhack/openwrt-node-packages.git;for-15.05

  6. Update the feed information for all available packages to build the firmware:
    ./scripts/feeds update
  7. Change the packages installed as default:
    wget https://gist.githubusercontent.com/nxhack/7ced4d866a59ebc72737589b49a220f8/raw/1bebfe9f6091f55a5856bc4af00da456a4995b09/mtk-linkit.patch
    patch -p1 < mtk-linkit.patch
  8. Fix build error of depend on node.js version:
    wget https://gist.githubusercontent.com/nxhack/4357d51918ba8f3cb5cc00080ad0815c/raw/e96758224ade8cb224523aedf5ea3249a6a59425/MRAA.patch
    patch -p1 < MRAA.patch
  9. Hack for wifi driver so build completes
    Copy kernel objects for support kernel 3.18.45:
    see : MediaTek-Labs/linkit-smart-7688-feed#37

    cp ./feeds/linkit/mtk-sdk-wifi/wifi_binary/mt_wifi.ko_3.18.44 ./feeds/linkit/mtk-sdk-wifi/wifi_binary/mt_wifi.ko_3.18.45
    cp ./feeds/linkit/mtk-sdk-wifi/wifi_binary/mt_wifi.ko_3.18.44_all ./feeds/linkit/mtk-sdk-wifi/wifi_binary/mt_wifi.ko_3.18.45_all
  10. Install all packages:
    ./scripts/feeds install -a
  11. Use node.js custom packages:
    rm ./package/feeds/packages/node
    rm ./package/feeds/packages/node-arduino-firmata
    rm ./package/feeds/packages/node-cylon
    rm ./package/feeds/packages/node-hid
    rm ./package/feeds/packages/node-serialport
    ./scripts/feeds install -a -p node
    
  12. Prepare the kernel configuration:
    make menuconfig
    • Select the following options:
      • Target System: Ralink RT288x/RT3xxx
      • Subtarget: MT7688 based boards
      • Target Profile: LinkIt7688
    • (Optional) GO into Languages > Node.js > Configuration (under the node package) > Select your desired version.
    • (Optional) Enable modules you want (Caution there is a 30MB limit so not all will fit)
      • (Recommended) node-npm (it is a separate module in v6)
    • Save and exit (use the default configuration file without any modification)
  13. !!!BEFORE YOU START!!!
    RUN THIS FROM YOUR VM NOT OVER SSH
    This command will take a while and if your ssh pipe breaks so will your build
    Start the compilation process:

    make

    There are several options you can use with the make command that are helpful.
    – V=99 (gives verbose output during build) or
    – V=1 (shows error, warnings, and notes “less verbose”)
    – &> output.log ( on the end – stores output in a log for later viewing)

A Few Hours Later…

  • After the build process is finished successfully, the resulted firmware file will be under “bin/ramips/openwrt-ramips-mt7688-LinkIt7688-squashfs-sysupgrade.bin”. If its’ not there check the output in your log file or on your screen.  Depending on the hardware resources of the host environment, the build process may take more than 2 hours.
  • You can use this file to update the firmware through Web UI or rename it to lks7688.img to update through a USB drive.

A big thanks to nxhack for making this build possible!

Download images for the version of Node you want:

lks7688-node-v4.img

MD5: d7f724da93a1d916bf777f80516a0f33

SHASUM: f74182c70b937909ad1cba6f97e40b5dd3891962

lks7688-node-v6.img

Includes NPM.

MD5: a17c672f87c4b8fa49a253d5534a9229

SHASUM: 8b19b20d23f1faa94c9dd084ca0271904d1dfa5e

Add a New Page to ReactGo

Adding a new page to your reactGo project can be a lengthy process.  I’m hoping to write a yeoman generator to do this for me in the near future but for now…  Here are the steps.

Steps:

  1. Add a new route to your app >routes.js
    Add Something like: <Route path="routeName" component={page} onEnter={requireAuth} fetchData{[fetchItemData, fetchCartData]} />
    to the Route section.
    Add the necessary supporting imports.
  2. Add a new entry to the navigation.
  3. Create a new page in the pages folder.
    Duplicate existing page, then open.  Update references to new component name.
  4. Add reference to app > pages > index.js
    export { default as ContianerName } from 'pages/ContainerName';
  5. Create a new data fetcher (if necessary)
    Duplicate existing and update path to get data from server and action type to dispatch.
  6. Update the app > fetch-data > index.js with the new reference.
    export { default as fetchMyData } from './fetchMyData';
  7. If a new data set is needed follow these steps: Add a New Dataset to the Store
  8. Create a new Container
    Duplicate an existing component then design and build your page.
  9. After your pages design is complete componentatize what you should.

Questions for yeoman generator:

  • Provide Route name: String
  • Provide page/component name: String
  • Does the page already exist? Boolean
  • If No => Provide Page Description: String
  • Would you like to generate a new CSS Module for the page? Boolean
  • Auth required? Boolean
  • Does it need to fetch any data? Boolean
    • If yes => How many? Number
    • For count:
      • Provide Data Fetcher name: String
      • Does it already exist? Boolean
      • If No => Does dataset exist?

Dockerizing ReactGo

First up you’ll need to be able to successfully run a build for your current ReactGo app.  I assume you have docker installed.

npm run build

Once you’ve got a successful build on your hands go ahead and create your dockerfile.  Example one below:

FROM node:boron

# Create app directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Install app and dependencies
COPY package.json /usr/src/app/
RUN npm install

# Copy build
COPY compiled /usr/src/app/compiled
COPY public /usr/src/app/public

# Set Environment Variables
ENV MONGODB_URI mongodb://mongo/MyAppCollection
ENV NODE_ENV production
ENV PORT 80

EXPOSE 80

CMD ["node", "compiled/server.js"]

You can use a later version of Node if you like I built and tested my on v6.10.

docker build -t <image_name> .

NOTE: DO NOT FORGET THE PERIOD AT THE END (that caused me a serious headache one day…)

While that is building lets launch a mongo container to run our database (we’ll link to it later).  I assume you’ve already pulled mongo from the docker hub: docker pull mongo

docker run -d --name <db_container_name> mongo

Now let test our container locally… I’m going to bind our exposed 80 port to port 80 on localhost to demonstrate how it’s done.  This same principal applies when launching a container out in a production docker environment.

docker run -d -p 127.0.0.1:80:80 -it --link <db_container_name>:mongo --name <app_container> <image_name>

NOTE: notice the MONGODB_URI ENV variable we set in the docker file.  Take special note of underlined part that follows: the mongodb://mongo/MyAppCollection.  This MUST match the link name you give in your run command (e.g. <db_container_name>:mongo).

Check your local host on port 80 and you should see your app.  If not check the logs by using the docker logs <container_ID> command.  Container IDs can be found with docker ps -a command.

After you’ve successfully tested your container you’re ready to push the image to your registry.  You can use docker hub but I’m using Bluemix, since it’s private and I get it for free ;).  After you login to your registry tag your image for release:

Bluemixers need your name space? try bx ic namespace-get

docker tag <image_name> registry.ng.bluemix.net/<name_space>/<app_name>:<version_tag>

The version tag is optional but recommended.  It will default to latest otherwise. Then push it (push it real good)

docker push registry.ng.bluemix.net/<name_space>/<app_name>:<version_tag>

Then wait… forever…

If your on Bluemix you’ll probably want to push up a mongo container as well. (remember to start one up so we can link to it like we did on local host)

For those of you who are not using Bluemix substitute the bx ic commands below for your registry (should be just docker for most of the world)

bx ic cpi mongo registry.ng.bluemix.net/<name_space>/mongo

From there I usually create a dummy container with the Bluemix GUI (website) so I can provision a public IP through Bluemix which I can then reuse when I launch a container with this command.  There is probably a way to provision an IP through CLI but I’m too lazy to look it up.

bx ic run -d -p <my_public_ip>:80:3000 -it --link <db_container_name>:mongo --name <app_container_name> <name_space>/<image_name_or_id>

 

Discoverable JSON based RESTful Services (DJRS)

For any developer who has ever worked with SOAP you know about WSDLs. These standardized files could provide information to an application or and IDE which the developer could then use to easily select an action to do. In the era of declining SOAP usage and the rise of the RESTful APIs it’s time to start thinking about how to make REST even easier to consume than it already is. Easier than easy… yes, easy peasie lemon squeezy.

I won’t lie my first experience with SOAP was a nightmare and even worse I was doing it with PHP and a toolkit that was clearly developed by someone who didn’t want anyone to use php to use their service. So my first experiences were bitter to say the least. But as I worked on other project I foudn other developers using other platforms/frameworks (like .NET) who spoke highly of their experience with SOAP. So that made me ask the question, “What was I missing out on?” The truth was a lot actually. When I saw how easy it was to consume a SOAP service in .NET it seemed to good to be true. But with SOAPs decline in favor of RESTful services the shift went back to something I was ultimately more comfortable with given my experience with SOAP.

So now to the meat and potatoes. What we needs is a good tool that make creating an API easy. BUT WAIT JOE MY COMPANY ALREADY PURCHASED AN ESB AND I HAVE TO USE IT. Well that sucks for you because the big thing that most ESBs I’ve used fail on is providing clear documentation on how to use the service and if it’s JSON based there’s no file generated for a developer IDE to consume. And that’s what we need, an application that allows developers to connect to any data store, provides an easy and intuitive method on configure what should be exposed, what should be auto set, what needs to be validated as well as set up authentication for you and auto-generate documentation and a consumable description of the service that an IDE can use to make connecting to the API easy. We need Deejers (DJRS).

Swagger has already done most of the hard work for this and I think we just need to take it to the next level.

So that’s the goal and I’m starting to build it now and since I’m currently obsessed with Meteor I’m going to use it to build it. Check if anyone has done that yet… NOPE.

Goals:

  • Self Documenting (Swagger)
  • Built in analytics tools
  • User Access Management
  • Easy to Understand UI (For API management)
  • Auto Generated SDKs for languages (Swagger format has consumable client libraries for most languages)
  • Back-end should connect directly to Data-store
    • Should support major data store formats
  • Deployment on on Cloud Foundry will be easy
  • Caching – with query time cache conditions and customization cache times for each end point
  • Support major security formats (Basic, OAuth, JWT ect…)
  • Availability to increase number of running nodes to scale up in high demand situations
  • User can configure the application for any table available:
    • Auto Generate field on condition
    • Designate Required Fields
    • Data Type
    • Schema (Global and per table)

Steps:

  1. Build app that connects and has CRUD access to most common DB formats
  2. Auto-generate basic yaml based on connected datastores
  3. Provide swagger-like endpoint manager to define options
  4. Generate YAML or standard Swagger Config file for Service
  5. … more… but let’s start with that.

JSON Object for 196 Countries with 2 Character Country Code in Name Value Format

I just got done with this… IT TOOK FOREVER! Save your self the pain and copy if you need it.

196 Countries in Country Name and ISO “-2 Code Country Code format. The JSON object is in name value pair format for easy consumption.

Please leave a comment is I misspelled something or if I need to add another country!

Did all the counties from here: http://www.nationsonline.org/oneworld/country_code_list.htm that weren’t in italics.

Continue reading…

The MSNR Stack Step 1

I’ve been letting the idea for the MSNR stack ruminate in my mind for a while now and it was finally time to pull the trigger.  There are an endless array of node frameworks sprouting up but at the end of the day I think its important to do what your comfortable with and for me that’s MVC and SPAs that don’t step it your MVC aka the MSNR stack.

Today I took a very important step toward that goal by setting up my first very own server!  Well it’s not the first time I’ve set up a server or anything but it’s the first time I’ve done it outside of a VM!  I purchased an old HP G5 DL380 recently and decided it was high time to get it up and running.

The first thing I did was mount the sucker:

Then I wired it up, got it some juice (power) and some food (internet) switched it on and it was like the jet of my dreams taking off in my basement.  (The fans are really loud on start up…)

After that I installed Ubuntu 12 LTS since it was certified for my hardware.  Once the install was done it was time to get node installed.  I used a package available here from joyent with great success.

Then I had to make sure it would actually show something if I navigated to it.  By default node does not serve anything like nginx or lamp.  You need to create an application in order for something to display enter the Hello World app:

I’m a fan of nano so I did :

nano ~/hello_world.js

Then I got to work writing my first app (which I stole from the node site).

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(80);

Once it was all ready I did my CTRL+X  Y and Enter to save my file.  Then finally it was time to test!

sudo node ~/hello_world.js

At this point if you did everything right all you should see is a flashing underscore.  You’ll need to use sudo if you are using port 80 like I am.

Finally I wanted to see my master piece in action so I logged on to my domain registrar, created a new subdomain, went to the Advanced DNS editor and redirected the sub-domain to my IP address.  Once that was setteled I went into my At&T 2WIRE router and went into my firewall, selected my server and set it to accept traffic on port 80 (by clicking the Server link and the Web Server selection).

TLDR for all those who skip to the bottom (like myself).

  1. Get a server
  2. Install Ubuntu
  3. Install Node
  4. Create a Node App
  5. Point a Domain to your IP Address
  6. Ensure your Router/Firewall are ready for the traffic.

DONE

http://dev.motoandmustache.com/

Not too bad for an evenings work.

The MSNR Stack

MongoDB + Sails.js + Node.js + React.js = MSNR (mis · ner)

JavaScript frameworks are coming out of the woodwork right now but picking and choosing  which ones to use for your project can be a pain, because there are a myriad of ways to combine these frameworks to build applications.  ToDoMVC covers some of the more popular combinations out right now, however there is one that they don’t cover. The MSNR stack.

Continue reading…