Unreal Expectations

Zoom Busy Light Automation

Michael Leo — Fri, 11 Sep 2020 00:32:42 GMT

Individually addressable LEDs provide a simple way to get status of things around the home without having to pull out phone or looking at a dashboard view. LEDs are visible from 10 or 20 feet away making them easy to see at a glance. I have a couple of Homeseer WD-200+ Z-Wave switches installed with one of them is located in a hallway near the bedroom. As locks change state, the LEDs on the switch turn green (unlocked) or red (locked).

This project came out of wanting to have even more LEDs available on some device that can show status of the house. For example, could I show the status of every window's sensor having an open or closed state. As I searched around for programmable LEDs, the Raspberry Pi came up for various options of programmable LEDs. Looking at Amazon specifically, I came across many busy light products which inspired the idea. Did this idea already exist for the Pi? Of course it did.

The following does not present an exhaustive list of the steps to build what I built, but tries to present enough information and direction to allow someone else to extend and improve.

Working From Home

I have been working from home for the last two years and six of the last eight years. So the work from home part and use of Zoom didn't change much when the California Stay at Home order came down in March 2020. What changed was the home office going from a solitary work environment to a shared environment with four other people. What changed was having conference calls with no interruptions to kids needing help with something at the most inconvient times. Letting members of the family know when I cannot be interrupted makes it easier to manage those interruptions

Hardware for Busy Light

Pi Zero W Starter Kit - Came with everything needed to get up and running with the Pi Zero and 1x8 Blinkt LEDs
pHAT Diffuser - The LEDs can be really bright, this helps with diffusing the lights and makes them look nicer

LED Alternates

Unicorn pHAT - A a 4x8 matrix of LEDs. Requires soldering or GPIO Pogo Pins or GPIO Hammer Header (Solderless)
Unicorn HAT Mini - Does not require soldering

My initial work uses the Starter Kit and Blinkt light, but a newer edition is using the GPIO Hammer Header woth Unicorn pHAT.

Software Architecture

Zoom

We use Zoom at work for remote meetings. This is probably a bit unsurprising since most of the world seemingly started using Zoom. Zoom offers APIs to access their services. They also support Webhooks that will send your service notification messages when events occur. I am interested in hooking into changes to my presence.

Presence Events
The webhook system will send presence events for the following events

When a meeting is manually started or ended
When a meeting on the calendar starts or ends
When status in the Zoom app is manually changed
When the Away timer triggers

Internet Addressibility

A challenge that comes up often with webhooks is that the callback endpoint of the webhook has to be Internet accessible. My Pi Light will live in my home network and so need a path for the notification messages from Zoom to make it into my network and to the Pi.

One option is enabling port forwarding on my router and making the Pi directly accessible from the the Internet. This would also require a Dynamic DNS service so the webhook can find my Internet IP address.
Another option uses a tunneling service such as ngrok.com to provide a public HTTPS endpoint that tunnels to a localhost port. Either I pay for the service to get my own subdomain, or deal with potential of the address changing over time. Ngrok is great for prototyping, but seems problematic for long term use.
Finally, looking at what I already have running at home and Internet hosted, the following items stuck out
- Home Network
  - MQTT Server
  - Homebridge
- Hosted Server
  - Nginx
  - Let's Encrypt Certificate

The following then became the vision for the architecture

MQTT Server

The usage of MQTT Server is a recent addition to my home automation stack. MQTT is a lightweight publish/subscribe protocol built for Internet of Things connectivity. Mosquitto is the server then used for for MQTT brokering. Clients can connect to Mosquitto and publish messages and/or subscribe to receive messages.

Mosquitto supports authentication, authorization and TLS-based communication. In other words, access to the server can be secured.

Mosquitto Bridging

MQTT is built for IoT. Bridging allows one Mosquitto server to connect to another server and publish messages, subscribe to messages or both. This allows software to communicate to a server in their local environment and process messages that branch out to multiple end-points without having to worry about how many other servers and networks might exist behind the curtain.
In this usage scenario, my home network Mosquitto server can connect to the hosted Mosquitto server utilizing an outbound network connection. It can then relay messages from the hosted instance to any clients on the local network.

The home network remains closed to inbound connections.

Nginx Server

Nginx is a web server but will be used for reverse proxy HTTP services and general TCP proxy services. While Nginx may know what HTTP traffic is, it can also proxy the MQTT traffic to the hosted Mosquitto server via the TCP proxy.
Nginx supports TLS termination on HTTP or TCP connections. Integrate with Let's Encrypt support and we have easy certificate generation and renewal.

In this usage scenario, Nginx provides the TLS termination for enabling secure Internet traffic and will proxy HTTP/Webhook traffic and MQTT/Mosquitto traffic to the underlying services on the server.

Homebridge & mqtt-thing Plugin

Homebridge emulates the Homekit API. Plugins to Homebridge can integrate with 3rd party APIs and expose the devices to Homekit. In my initial architecture, Homebridge was run directly on the Pi and this was fine. However, since the Pi Zero only has 512 MB of RAM and running multiple Node applications could strain the memory or swap. I already have an instance of Homebridge running on a Mac Mini, the existing instance could handle the additional plugin with little additional overhead.

MQTT-Thing Plugin

MQTT-Thing is a Homebridge plugin that allows integration of many different accessory types using MQTT. Using the plugin, I can expose an RGB Light to Homekit and when interactions with the light occur in the Home app or Homekit routines, the plugin will publish messages that can be listened for and update the LEDs appropriately.

Custom Node Apps

There are two custom node apps. One runs on the public server, handles the Zoom Webhooks and publishes the presence state to the MQTT server. The other runs on the Pi, handles the MQTT messages from Zoom or Homekit and updates LEDs.

Zoom Webhook & Publishing

The Zoom Webhook code is based on the following samples from Zoom, Zoom Oauth Sample and Zoom Webhook Sample. In my Zoom account configuration, the Webhook Only configuration is not available by default, but OAuth option is available. Therefore, the application had to support OAuth to register access to my Zoom account and then used a Webhook to receive the presence change events. As the presence events come in, the data is reformatted and published to MQTT.

MQTT Listener & Pi Light

The MQTT message listener and Pi LED light control application registers a subscriber to MQTT for two different topics, one for Zoom and one for Homebridge. When a message is received, it is parsed for Zoom or Homebridge commands and the command is then executed.

Steps to Success

1. Acquire Access to Internet Server

Use what you have or use one of the myriad virtual server providers to access a server.

I used Linode for this purpose.

2. Setup the Software on Internet Server

I used docker / docker-compose to install and configure the individual server software for Nginx & Mosquitto

There are many source available for installing and configuring Nginx with Let's Encrypt.

Running Mosquitto within Docker

https://blog.feabhas.com/2020/02/running-the-eclipse-mosquitto-mqtt-broker-in-a-docker-container/

To set up the MQTT reverse proxy and TLS termination for MQTT, see the following:

3. Create the Zoom OAuth/Webhook & Publisher

The node app built off of Zoom's sample application code and then extended it for my purposes. The initial steps get the application up and running to receive the Zoom events via a webhook. The last steps connect it to the Mosquito server to publish the events.

The sample app instruction suggest using ngrok.com, which is fine for development purposes. A later step in the process includes bundling the application into node and proxying through the nginx server.

Step 1. Follow the example Readme at Zoom Oauth Sample to create the OAuth application.

Step 2. Add Webhook Endpoint in the Node Express application following the code at Zoom Webhook Sample.

Step 3. Update the Configurations of the OAuth application in Zoom Marketplace to enable the Event Subscriptions.

Step 3.a In the Feature tab of the OAuth app Configuration, enable Event Subscriptions and add the endpoint URL created in Step 2.

Step 3b. Under the Feature, add the Event Types for User and User Activity so the Webhook will receive the Events when presence changes.

Step 3c. Ensure scopes are set properly. Minimally, the User Information read and write permissions are required.

You may need to reactivate the app with the node application to ensure it has proper scopes and webhook is fully activated. The webhook should be receiving the presence messages.

Step 4. Publish the messages to mosquitto

The mqtt module was added to the node application. I then created a status.js file and Status class that contained the connection and publshing to mqtt.

The events that Zoom sends are not necessarily in order that they occur. Occasionally, an event is received that is older than a previous event received. The code in handlePresenceEvent keeps track of the last event and ensures the latest received event has a dateTime that is newer before the event is published.

const moment = require('moment');
const mqtt=require('mqtt');
const storage = require('node-persist');
let path = require('path');

let appDir = path.dirname(require.main.filename);

let addDebugging = () => {

};

class Status {
    constructor({debug, mqtt:{url, options}}) {
        this.isInit = false;
        this.lastEvent = undefined;

        this.client = mqtt.connect(url,options)

        this.client.on("connect",function() {
            console.log("connected");
        });
        this.client.on("error",function(error){ console.log("Can't connect"+error);});

        if (debug) {
            addDebugging();
        }

    }

    async init() {
        await storage.init({
            dir: `${appDir}/storage`
        });
        this.lastEvent = await storage.getItem('lastEvent');
        if (this.lastEvent && this.lastEvent.dateTime) {
            this.lastEvent.dateTime = moment(this.lastEvent.dateTime);
        } else {
            this.lastEvent = {};
        }
        this.isInit = true;
    }

    async handlePresenceEvent(event) {
        if (!this.isInit) {await this.init()}

        if (event.event !== 'user.presence_status_updated' ||
            !(event.payload && event.payload.object)) {
            return;
        }

        let eventObject = event.payload.object;

        let normal = {
            dateTime: moment(eventObject.date_time),
            email: eventObject.email,
            profileId: eventObject.id,
            presence: eventObject.presence_status
        }

        if (!this.lastEvent.dateTime || this.lastEvent.dateTime.isBefore(normal.dateTime)) {
            this.lastEvent = normal;
            await storage.setItem('lastEvent', this.lastEvent);
            console.log("Updating last presence status");

            if (this.client.connected) {
                let topic = `zoom/${normal.profileId}/presence`;
                this.client.publish(topic, normal.presence);
                console.log("published: ", topic, normal.presence);
            }
        } else {
            console.log("Ignoring presence status");
        }
    }
}

module.exports = Status;

And the Webhook sample method was updated to call the handlePresenceEvent() method instead of dealing with Webinar events.

// Set up a webhook listener for your Webhook Event - in this case we are listening to Presence and other events
app.post('/webhook', bodyParser.raw({ type: 'application/json' }), async (req, res) => {

    let event;

    try {
        event = JSON.parse(req.body);
    } catch (err) {
        res.status(400).send(`Webhook Error: ${err.message}`);
        console.log("Failed to parse body", err);
        return;
    }
    // Check to see if you received the event or not.
    console.log(event)
    if (req.headers.authorization === process.env.verificationToken) {
        res.status(200);

        if (event.event === 'user.presence_status_updated') {
            console.log("Presence Webhook Received.")
            status.handlePresenceEvent(event).catch(reason => {
                console.log("handlePresenceEvent threw exception", reason);
            })
        }

        res.send();

    } else {

        res.status(403).end('Access forbidden');
        console.log("Invalid Post Request.")
    }
});

Step 5. Add node.js app to docker container & proxy the nginx secure port

Public Side Complete

At this point, the left side of the diagram is complete. The Zoom notifications should be received by the the Internet server and published to an MQTT server.

4. Mosquitto on Private Network

I set up a separate instance of Mosquitto in bridge mode on a Mac Mini in my local network. This was done since I already had Mosquitto running and it prevented the need of the Raspberry Pi Zero to connect to the Internet accessible Mosquitto server. However, this step is a bit superfluous and the Node app on the Pi can connect directly to the public Mosquitto server.

http://www.steves-internet-guide.com/mosquitto-bridge-configuration/

mosquitto.conf Bridge Configuration

connection bridge-node
address example.com:8883
remote_username 
remote_password 

topic # in 0 "" ""
bridge_cafile /usr/local/etc/mosquitto/DST_Root_CA_X3.pem

The cafile for DST_Root_CA_X3.pem had to be downloaded and installed on my server. The certificate is a root certificate that Let's Encrypt's intermediate certificate files have been cross-signed against. This ensures that when the Mosquitto server connects over SSL it can verify the certificate it receives from Nginx. (Read)[https://letsencrypt.org/certificates/] on the Let's Encrypt certificates and signing. There are other options for Root certificates.

5. Create MQTT Listener & Pi Light Node App

The overall code base behind the MQTT listener and process to update the light is much larger than what is presented. I am planning on providing a UI to directly update the light from a Web Page, but that is not built yet. Most of the code to handle the webhook events and homebridge events condenses into 4+ files:

App startup
MQTT Listener
Common LED Class
LED Implementation Binding Class

App Startup

// app.js
// Bring in environment secrets through dotenv
require('dotenv/config')
const Listener = require('./lib/listener.js');
const ledImpl = process.env.ledImpl || 'pi-null-led';
const deviceId = process.env.deviceId || 'pi-zero-wh';

let listener = new Listener({
  deviceId,
  ledImpl,
  debug: true,
  mqtt: {
    url: process.env.mqttURL,
    options: {
      username: process.env.mqttUsername,
      password: process.env.mqttPassword
    }
  }
});

MQTT Listener

// lib/listener.js
const mqtt=require('mqtt');

class Listener {
  constructor({deviceId, ledImpl, debug, mqtt:{url, options}}) {
    const {createLedHandler} = require(`./${ledImpl}.js`);

    this.deviceId = deviceId || 'pi-zero-wh';

    this.led = createLedHandler();

    this.lastEvent = undefined;

    this.client = mqtt.connect(url,options);


    this.client.on("connect",function() {
      console.log("connected");

      this.client.subscribe(["zoom/#", `home/busylight/${this.deviceId}/#`], [], (err, granted) => {
        console.log("Subscribe:")
        console.log("  Error: ", err);
        console.log("  Granted: ", JSON.stringify(granted, null, 2));
      });
    }.bind(this));

    this.client.on("error",function(error){
      console.log("Can't connect"+error);
    });

    this.client.on('message', this.handleMessage.bind(this));
    
  }


  async handleMessage(topic, message) {
    console.log("handleMessage: " + JSON.stringify({topic, message: message.toString()}, null, 2));
    if (topic.startsWith("zoom/")) {
      let {rgb: [red, green, blue]} = await this.led.setPresence(message.toString());

      // Send stat message to MQTT Thing to update the state
      const pubTopic = `home/busylight/${this.deviceId}/light/stat/rgb`;
      const pubMessage = `${red},${green},${blue}`;
      
      this.client.publish(pubTopic, pubMessage);

    } else if (topic.startsWith("home/")) {
      this.handleHomeMessage(topic, message.toString());
    }
  }

  async handleHomeMessage(topic, message) {
    //console.log(`topic.indexOf("/cmd/"): ` + (topic.indexOf("/cmd/") > 0));

      if (topic.indexOf("/cmd/") > 0) {
        //console.log(`topic.indexOf("/rgb"): ` + (topic.indexOf("/rgb") > 0));
        if (topic.indexOf("/rgb") > 0) {
          const regex = /^\d\d?\d?,\d\d?\d?,\d\d?\d?$/;
          //console.log(`message.match(regex): ` + message.match(regex));
          if (message.match(regex)) {
            let json = `[${message}]`;
            let rgb = JSON.parse(json);
            //console.log(`message.match(regex): ${json}`);

            this.led.setColor({rgb});
            this.client.publish(topic.replace('cmd', 'stat'), message);
          }

        } else if (topic.indexOf("/power") > 0) {
          if (topic.indexOf("/zoom") > 0) {
            this.led.enablePresenceUpdates(message.toString() !== "true");
            this.client.publish(topic.replace('cmd', 'stat'), message);
          }
          if (topic.indexOf("/light") > 0) {
            this.led.setPower({power: message.toString() === "true"});
            this.client.publish(topic.replace('cmd', 'stat'), message);
          }
        }
      } else if (topic.indexOf("/stat/") > 0) {

      }

  }
}

module.exports = Listener;

Common LED Class

The Led class is a base class that handles most of the common logic of interpreting the message and keeping track of the last state of the light. The child classes then interact with the native node modules to update the light based on the common settings.

// lib/led.js
const storage = require('node-persist');
let path = require('path');

let appDir = path.dirname(require.main.filename);

class Led {
  constructor() {
    this.isInit = false;
    this.disablePresence = false;
  }

  async init() {
    await storage.init({
      dir: `${appDir}/storage`
    });
    this.isInit = true;
  }

  async setPresence(presence) {
    if (!this.isInit) {await this.init()}

    const {enable} = (await storage.getItem('enablePresenceUpdates')) || {enable: true};

    if (!enable) {
      return;
    }

    let color  = {
      rgb: [0, 0, 0],
      clear: false,
      lowlight: true
    }

    if (presence.toLowerCase() === 'away') {

      color.rgb = [255, 255, 0];
    } else if (presence.toLowerCase() === 'available') {
      color.rgb = [0, 255, 0];
    } else if (presence.toLowerCase() === 'do_not_disturb') {
      color.rgb = [255, 0, 0];
    }

    await this.setColor(color);

    return color;

  }

  async setColor(color) {
    if (!this.isInit) {await this.init()}

    console.log("Set color to :" + JSON.stringify(color));

    await storage.setItem('color', color);
  }

  async enablePresenceUpdates(enable) {
    if (!this.isInit) {await this.init()}

    this.disablePresence = !enable;
    //console.log(`enablePresenceUpdates(enable): ${enable}`)
    await storage.setItem('enablePresenceUpdates', {enable});

  }

  async setPower({power}) {
    if (!this.isInit) {await this.init()}

    let color = await storage.getItem('color');
    if (!color) {
      color = {
        rgb: [255, 255, 255],
        clear: false,
        lowlight: true
      };
    }

    if (power) {
      color.clear = false;
    } else {
      color.clear = true;
    }
    await this.setColor(color);

    await storage.setItem('power', {power});
  }

}

module.exports = Led;

Blinkt LED Class

Implementation for the Blink LED

// lib/pi-blinkt-led.js
let Led = require('./led.js');
let Blinkt = require('node-blinkt');

class BlinktLed extends Led {
    constructor() {
        super();

        try {
            this.leds = new Blinkt();

        } catch (err) {
            console.log(err);
            this.leds = {
                setAllPixels: () => {},
                clearAll: () => {}
            }
        }
    }

    async setColor(color) {
        let {rgb: [red, green, blue], clear, lowlight} = color;

        await super.setColor(color)

        if (clear) {
            this.leds.clearAll();
        } else {
            this.leds.setAllPixels(red, green, blue, (typeof lowlight !== 'undefined' && lowlight === true) ? 0.4 : 1.0);
        }
    }

}

module.exports = {
    createLedHandler : () => new BlinktLed(),
};

Unicorn Phat Class

Implementation for the Unicorn Phat module LED

// lib/pi-unicorn-phat-led.js
let Led = require('./led.js');

class UnicornPhatLed extends Led {

    constructor() {
        super();
        this.numberLeds = 32;

        try {
            this.leds = require('rpi-ws281x-native');
            this.leds.init(this.numberLeds, {dmaNum: 10});
            this.leds.setBrightness(25);

        } catch (err) {
            console.log(err);
            this.leds = {
                setAllPixels: () => {},
                clearAll: () => {}
            }
        }
    }

    async setColor(color) {
        let {rgb: [red, green, blue], clear, lowlight} = color;
        let pixel = (green << 16) | (red << 8) | blue;
        console.log(pixel.toString(16));

        let pixels = [];
        for (let i = 0; i < this.numberLeds; i++) {
            pixels.push(pixel);
        }

        await super.setColor(color)

        if (clear) {
            this.leds.reset();
        } else {
            this.leds.setBrightness((typeof lowlight !== 'undefined' && lowlight === true) ? 10 : 100);
            this.leds.render(pixels);
        }
    }

}

module.exports = {
    createLedHandler : () => new UnicornPhatLed(),
};

6. Bonus Homebridge & MQTT-Thing Plugin

I initially set up Homebridge on the Pi Zero itself to keep it separate from the existing instance. However, with so little RAM on the Pi Zero, it took a while to start up and probably started using a bit of swap. So I relented and setup the plugin on the primary Homebridge instance.

Since the Node app is already listening to the MQTT server, it was straight forward to add topics to the MQTT plugin and have the node app process those messages separately from the Zoon topic.

Example MQTT Plugin Config

"accessories": [
        {
            "accessory": "mqttthing",
            "url": "mqtt://example.local:1883",
            "username": "",
            "password": "",
            "mqttOptions": {
                "keepalive": 120
            },
            "logMqtt": true,
            "type": "custom",
            "name": "Busy Light Zero",
            "services": [
                {
                    "type": "lightbulb-RGB",
                    "name": "Busy Light",
                    "topics": {
                        "getOnline": "home/busylight/pi-zero-wh/light/stat/lwt",
                        "getRGB": "home/busylight/pi-zero-wh/light/stat/rgb",
                        "setRGB": "home/busylight/pi-zero-wh/light/cmd/rgb",
                        "getOn": "home/busylight/pi-zero-wh/light/stat/power",
                        "setOn": "home/busylight/pi-zero-wh/light/cmd/power"
                    }
                },
                {
                    "type": "switch",
                    "name": "Zoom Presence Override",
                    "topics": {
                        "getOnline": "home/busylight/pi-zero-wh/zoom/stat/lwt",
                        "getOn": "home/busylight/pi-zero-wh/zoom/stat/power",
                        "setOn": "home/busylight/pi-zero-wh/zoom/cmd/power"
                    }
                }
            ]
        }
    ]

Conclusion

This was a fun project giving me a chance to explore the Raspberry Pi Zero and build on some other technologies I had in place to automate the home.

The amount of custom code was small versus the library reuse and building on the strong work of others.

What's Next

The Zoom webhooks are great for receiving the events when you are connected to Zoom and access the Zoom APIs. This is great as a passive notification system for others to see.

Rather than reactive, I want the light to be proactive and act as reminder to the kids that they have a Zoom meeting scheduled for online class with their teacher. The zoom meeting times are now on a shared calendar. If I can get the Pi to read the calendar and change colors and play a sound when the meetings start, I can more easily ensure the kids are where they should be.To support this, I am looking at integrating Microsoft Office 365 Graph APIs to read the calendar.

Backyard Cafe Lights

Michael Leo — Mon, 15 Jun 2020 15:36:35 GMT

About the Space

Our backyard has a concrete patio. A previous owner added it, ostensibly to install a hot tub, though the hot tub never seemed to have been put in. We have always used the patio for furniture.

The cafe lights were added after a rehabilitation of the backyard. We remodeled a part of the house and during the construction the backyard was trampled upon, used for storage and generally neglected for about 18 months. The backyard had gone from a nice, water wasting, grass filled area to a dilapidated dirt pile. We worked with landscaper to design and clean up the backyard.

We wanted to improve the concrete patio and decided to add some cafe lights to frame the space. After looking at some ideas and options on how to hang the lights in the middle of the space, we used DIY project from HomeDepot's site.

DIY: String Light Poles

Brighten your backyard with these DIY string light poles. Using concrete, lumber and string lights, you can illuminate your yard in style. For more informati...

YouTube

We purchased a set of Enbrighten Cafe Lights to use for the lighting. The string has 24 lights across 48 feet of cabling. The Cafe Lights have a remote control to turn it on/off or change the colors.

Enbrighten Vintage Seasons Color Changing Cafe Lights

Set the perfect mood for every occasion with Enbrighten Seasons LED color changing cafe lights – celebrate holidays, cheer on your favorite teams, or enjoy a...

YouTube

Home Automation

Although the cafe lights have a remote control to turn the lights on and off and change the colors, the lights retain their state when power is removed and reapplied to the lights. If the lights are powered on with the remote and power is removed, it will turn back on when power is applied once again.

Keeping the automation simple, I use a GE Outdoor ZWave Plug. A common plug I have used for the outdoor automations elsewhere. The longest active plug of this brand has been in use for 7 years with most of them being 3 or 4 years old at this point. The only issue I have had with any of these plugs is that the little plug cover eventually wears down from the weather and falls off.

Event Trigger	Event
Scheduled @ 30 minutes before sunset	Turn On
Scheduled @ midnight	Turn Off

The event triggers are just kept to a schedule. Keeping it at 30 minutes before sunset ensures that it is on before darkness sets in or when it is an overcast day. Midnight is an optimum time to turn it off since it is with a 95% confidence that we will not in the backyard after midnight.

HomeKit

While they are automated to turn on and off, there are times when it is necessary to manually change their state. They are exposed to HomeKit in the Backyard room.

What's Next / What's Missing

The Enbrighten Cafe Lights come with a remote control that enables changing the colors of the lights or setting them to one of many preset color rotations. At the moment, they do not provide any out of the box solution to automate or program the color selection. My hardware hacking skills are not up to snuff, so I will have to wait for Enbrighten to develop some programmable remote or another company to offer something similar.

Network Closet & Awesome Home Network

Michael Leo — Mon, 14 Oct 2013 22:09:00 GMT

Network Closet
When I first purchased our new house, I wasn’t aware that a previous owner had updated various outlets in the house and run ethernet cable from the outlets to a closet. We never looked too close at this closet to see that the cables were all running together.

Unfortunately, who ever installed it did all the hard work of running the cables to the closet, but never finished the work of connecting the cables to the patch panel. Although, they did have two cables sliced together to allow port to port connectivity.

Not too long after moving in, I had a professional come in, install some home theatre equipment and fix the network. They fixed the patch panel connecting up the existing network cables. They installed three new runs of cables from the kids’ room, living room, and family room to the closet. They also relocated the Verizon ONT Ethernet cable from the office area to the network closet.

To have power in the closet, I had an electrician install a new outlet in the top of the closet connected to an existing outlet in the bedroom from the other side of the closet wall.

I now have eight hard-wired ports throughout the house. One each in two of the bedrooms, one in office, one in the living room, two in the kitchen, one in the play room, and one in the family room. Seven of the eight ports are connected to an 8-port Gigabit switch with the last port on the switch uplinked to the Verizon FiOS router. The Verizon FiOS router is connected via ethernet cable to the ONT.

Broadband Internet
I work from home using the network pretty much all day. At night the kids like to watch shows on Netflix or iTunes on the AppleTV. Work and play at our house utilizes a lot of bandwidth. After starting to work from home in the last house, I upgraded our Verizon Service from 35Mbps to 150Mbps. This caused a lot of problems at the old house.

We had Verizon FiOS installed as soon as their systems registered it was available for the house, around mid-2007. We therefore had an early ONT box (optical network terminal that converts the fiber into twisted-pair telephone, coax, and ethernet) and the box had to be upgraded to even handle the faster Internet speed. The FiOS router had to be replaced as well since the switch on the router only supported 100Mbps and would be too slow for the broadband. Newer FiOS routers have gigabit switches and so this was replaced. Last, the WAN coax was swapped for ethernet cable. The installer had a lot of trouble switching the ONT boxes and getting our account setup with the new one, including the phone, video, and network. But once it was all setup, the 150 Mbps speed was great. Generally, things download quickly.

When we moved, one of the considerations for new house was having Verizon FiOS available. I just don’t think I could easily go back to the Time Warner cable. Verizon hasn’t yet put in any soft bandwidth caps and I would certainly hit them every now and then. I use Crashplan to back up all of our pictures, music and movies off site and a full backup of these pushes 1 TB when I first started and 1.4 TBs now. The new house also has Verizon FiOS and the move process went relatively smooth getting the phone moved to the new line and the set tops / cablecard recognized for the new account. There wasn’t a good place to install the battery backup, but eventually came up with an OK place to install as it had to be installed indoors out of weather. It was eventually installed behind an access door to the power supply to the jet spa tub.

We continue to have great 150Mbps download from Verizon. It isn’t 1Gpbs that Google is offering in the few places they offer, but it is pretty rare to consider the service disappointing. The only faster option Verizon has at the moment is 300Mpbs, but can’t justify the price on that yet. The 150Mbps easily allows kids to watch their own Netflix Super HD streams while I still do other things on the computer. In three months, I don’t feel we have been restricted by our broadband.

Awesome Local Network
Having the great Internet bandwidth only works with local network not getting clogged by communication of different devices. This is what makes the ethernet cabling so great. I can more easily segment the traffic and not overburden the WiFi network. For example, the home theatre has TiVo Roamio, Samsung Smart TV, and AppleTV that each can use ethernet or WiFi. In addition, the receiver has ethernet for Internet radio, Airplay support or network remote functionality. I have laptop I use most of the day, build server, media server and laptop for Carrie and Kids. There are many more devices, such as Home Automation server, couple more AppleTVs, Time Machine backup server, another Airport Extreme and TiVo Mini. There are the Samsung Smart Washer and Dryer with network access. Finally, there are multiple tablets and phones.

If all these devices were trying to access a single WiFI network, especially during video streaming, it would suck, even though I have dual band wireless-N routers available. By moving most stationary devices to ethernet I lighten the load on the WiFi-only devices and they can make better use of the network as well.

Having the switch in the network closet only makes this better. Devices that need to communicate can generally do it with less interference from other devices. For example, we have TiVo Mini that pairs with the TiVo Roamio. The Mini uses one of tuners on the Roamio to watch live TV. The Mini can change channels by which the Roamio will use a tuner to decode the cable data and then stream the video the Mini which presents on screen. The Roamio sends the network data to the switch which then sends it to the Mini all across ethernet, not affecting WiFi usage. Further, the switch will move the data from the Roamio to the Mini without impacting other devices on other ports. My laptop can browse the Internet which will send data to the switch which is then sent to the uplink to the router, not interfering with the Roamio/Mini data stream. The switch on the Router isn’t used and all traffic on the uplink cable is only data that is going to or coming from the Internet. The Roamio to Mini don’t necessarily have full use of the line, however; there is a second switch with the home theatre equipment. The TiVo, AppleTV, TV, Receiver, and an Airport Extreme are plugged in to a switch with the uplink going go the switch in the network closet. With the second switch also a gigabit switch, the devices aren’t going to create too much interference with the other devices. There is a third switch in the office that has 3 computers, the Time Machine, and the home automation server.

The 1Gbps LAN is great and fast. I copied a 2GB file from old laptop to new one. I was able to get to a near sustained transfer rate of 117 MBps or 936 Mbps. That’s pretty awesome network performance.

With the ethernet and switch, I am able to have two (currently) WiFi access points connected at different points in the house. With simultaneous dual band 80211.N the remaining devices on the network are never too far from an access point. The tablets work almost as good as the other devices for streaming video.

XMPPFramework / iOS / libidn / -all_load failure

Michael Leo — Sun, 04 Dec 2011 01:02:01 GMT

As I integrated the XMPPFramework it requires the libidn library for a couple of methods it uses. The latest version of libidn that the XMPPFramework included as of this writing is 1.15.

Unfortunately, this version seems incompatible with the “Other Linker Flag” -all_load that other libraries seem to want to use. The all_load linker flag appears be a work around for a linker bug that pops up when object files within a static library only contain a category and no classes.

From the source: http://developer.apple.com/mac/library/qa/qa2006/qa1490.html

IMPORTANT: For 64-bit and iPhone OS applications, there is a linker bug that prevents -ObjC from loading objects files from static libraries that contain only categories and no classes. The workaround is to use the -all_load or -force_load flags. -all_load forces the linker to load all object files from every archive it sees, even those without Objective-C code. -force_load is available in Xcode 3.2 and later. It allows finer grain control of archive loading. Each -force_load option must be followed by a path to an archive, and every object file in that archive will be loaded.

Interestingly, most coding conventions will create header and implementation files for categories that are just the categories; hence exposing the defect. Some library developers have come up with a simple workaround so that the force_load or all_load flags are not required. The workaround is a simple macro that is added to the category implementation file that just creates a class so that the bug is not exposed. Don’t know if it originated in Three20, but this is first place I came across it. RestKit also implemented the macro.

Unfortunately, other library developers have not yet implemented this, and still require the all_load flag to be used.

The use of the all_load flag has been causing the XMPPFramework’s use of libidn to fail to link. In part this appears to be the use of older libidn that is included as fat binary, static library. Use of the all_load flag often causes this linker error:

Undefined symbols for architecture i386:
“_libintl_bindtextdomain”, referenced from:
_idna_strerror in libidn.a(strerror-idna.o)
_pr29_strerror in libidn.a(strerror-pr29.o)
_punycode_strerror in libidn.a(strerror-punycode.o)
_stringprep_strerror in libidn.a(strerror-stringprep.o)
_tld_strerror in libidn.a(strerror-tld.o)
“_libintl_dgettext”, referenced from:
_idna_strerror in libidn.a(strerror-idna.o)
_pr29_strerror in libidn.a(strerror-pr29.o)
_punycode_strerror in libidn.a(strerror-punycode.o)
_stringprep_strerror in libidn.a(strerror-stringprep.o)
_tld_strerror in libidn.a(strerror-tld.o)
“_libiconv”, referenced from:
_str_cd_iconv in libidn.a(striconv.o)
_mem_cd_iconv in libidn.a(striconv.o)
“_libiconv_open”, referenced from:
_str_iconv in libidn.a(striconv.o)
“_libiconv_close”, referenced from:
_str_iconv in libidn.a(striconv.o)
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The iconv errors can be fixed because the iOS framework includes libiconv.a. It does not include libintl.a. After downloading version 1.25 of libidn and getting it to install on iOS 5.0, I was able to get rid of errors related to intl.

Here are the commands I used to compile libidn.1.25 and create a fat binary with arm6, arm7, and i386 architecture files.

$ ./configure --host=arm-apple-darwin --disable-shared CC=/Developer/Platforms/iPhoneOS.platform/Developer/usr/llvm-gcc-4.2/bin/llvm-gcc-4.2 CFLAGS="-arch armv7 -fmessage-length=0 -pipe -std=c99 -Wno-trigraphs -fpascal-strings -O0 -Wreturn-type -Wunused-variable -Wunused-value -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.sdk -fvisibility=hidden -gdwarf-2 -mthumb -miphoneos-version-min=4.0 " CPP=/Developer/Platforms/iPhoneOS.platform/Developer/usr/llvm-gcc-4.2/bin/llvm-cpp-4.2 AR=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/ar $ make clean $ make $ cp lib/.libs/libidn.a libidn-arm7.a

$ ./configure --host=arm-apple-darwin --disable-shared CC=/Developer/Platforms/iPhoneOS.platform/Developer/usr/llvm-gcc-4.2/bin/llvm-gcc-4.2 CFLAGS="-arch armv6 -fmessage-length=0 -pipe -std=c99 -Wno-trigraphs -fpascal-strings -O0 -Wreturn-type -Wunused-variable -Wunused-value -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.sdk -fvisibility=hidden -gdwarf-2 -mthumb -miphoneos-version-min=4.0 " CPP=/Developer/Platforms/iPhoneOS.platform/Developer/usr/llvm-gcc-4.2/bin/llvm-cpp-4.2 AR=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/ar
$ make clean
$ make
$ cp lib/.libs/libidn.a libidn-arm6.a

$ ./configure --host=i686-apple-darwin --disable-shared CC=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/llvm-gcc-4.2/bin/llvm-gcc-4.2 CFLAGS="-arch i386 -fmessage-length=0 -pipe -std=c99 -Wno-trigraphs -fpascal-strings -O0 -Wreturn-type -Wunused-variable -Wunused-value -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.0.sdk -fvisibility=hidden -gdwarf-2 -mthumb -miphoneos-version-min=4.0 " CPP=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/llvm-gcc-4.2/bin/llvm-cpp-4.2 AR=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/ar
$ make clean
$ make
$ cp lib/.libs/libidn.a libidn-i386.a

$ lipo -create libidn-i386.a libidn-arm6.a libidn-arm7.a -output libidn.a

After importing the new libidn,a, I still received the linking errors with libiconv. Added that to the library link list and afterwards all worked.

Multiple Language Meta Descriptions / One Page

Michael Leo — Tue, 17 May 2011 15:53:41 GMT

During a post implementation design review today, the design and implementation failed to meet the basic one requirement of the enhancement. The requirement was to ensure the site in question would respond to Katakana characters when searched through google.co.jp. The english title of the site is returned quite high, but when using the Katakana characters it fails.

The site in question is english only and so it mostly makes sense that it fails in google search results for Katakana. So the requirement was how to make the site appear utilizing Katakana.

Technical Solution

The developer went off and designed and implemented a solution. The solution relied upon the Accept-Language header in HTTP request to load a resource file and return a Katana localized version of the title and description if the preferred language is Japanese. Logically, seems to make sense. Change the default language on the browser and get the localized content.

Except there is a hole in the solution you can drive a bus through. When the googlebot crawler indexes sights it doesn’t have different crawlers for google.com versus google.co.jp versus other localized google sites. It has a single googlebot that indexes pages and doesn’t provide a language header. With the change, google would not appear to have interpreted the page in any way different, which is the case because it doesn’t get any different HTML.

From the Google Webmaster Blog we know content negotiation is bad.

@Simon Content negotiation is bad because search engines will only be able to see one of the versions (and might never see the other one). Also, many users (including me) don’t like it when they click on a search result that is not shown in the same language as their query (and the snippet) (actually, I personally don’t just not like it, it makes me quite mad :-)).

SEO Solution

Examining that the technical delivery failed, I backed up from there on the original solution concept.

The solution concept was to just add the Katana characters after the title and as a second description. According the the following information who experimented upon other results found, as expected, that google ignores multiple meta-description tags. It points to other article about partial success in google summaries using more relevant sentence from multi line description.

So the point of this is to try and see how multi languages will work in context of a description. Will update with results.

Update 1

The first launch failed miserably because the theme is not setting or pulling in the meta title or description fields.

Install Your Own URL Shortener

Michael Leo — Fri, 04 Mar 2011 22:26:36 GMT

A while back I acquired domain, zod.im with intent of setting up my own shortened URL service. While there is nothing wrong with the current offerings by bit.ly or TinyURL, I was academically intrigued by the notion of setting this up.

YOURLS

I went with open source package YOURLS. It was pretty easy to setup and has a nice interface. The documentation didn’t provide me everything I needed, but what it didn’t provide was easily discovered. Examples:

I don’t install MySQL databases on a daily, weekly, or monthly basis, so I had to go look that back up since the commands weren’t included directly in the setup documentation.
I front nginx as the web server and chose to look up changes to nginx configuration as the yourls installer provides just content for .htaccess
I use WordTwit for notifications of posts on this blog to send to Twitter. The current version did not have yourls support, but looks like the next Pro version might have more pluggable solution. So made updates to plugin to use this now.

Installation

Pre-Requisites

This assumes the starting point of installation already has nginx installed with fastcgi able to interpret php.
This also assumes MySQL is installed and operational.

MySQL

Create the database

mysql> CREATE DATABASE yourlsdb;

Create the user & Grant

mysql> CREATE USER 'yourlsdbuser'@'localhost' IDENTIFIED BY 'password';

mysql> GRANT ALL PRIVILEGES ON yourlsdb.* TO yourlsdbuser@localhost;

Install YOURLS

Download the code

Check here for the latest version.
$ wget http://code.google.com/p/yourls/downloads/detail?name=yourls-1.5.zip

Unzip the code in directory you want as root

$ sudo mkdir -p /var/www/zod.im/web $ cd /var/www/zod.im/web/ $ unzip /yourls-1.5.zip

Update the configuration

The following is the minimum settings that need to be changed for the private linking to work. Read through the configuration for other changes to consider.

Copy the sample configuration

$ cp includes/config-sample.php user/config.php

Update the database settings

Use vi or whatever your favorite editor is to edit the new file and follow the information in the file.
Change the database settings.
... define( 'YOURLS_DB_USER', 'dbuser' );

/** MySQL database password */
define( 'YOURLS_DB_PASS', 'dbpassword' );

/** The name of the database for YOURLS */
define( 'YOURLS_DB_NAME', 'yourls' );

/** MySQL hostname */
define( 'YOURLS_DB_HOST', 'localhost' );
...

Update the base site URL

/** YOURLS installation URL, no trailing slash */ define( 'YOURLS_SITE', 'http://example.com' );

Update the cookie hash

As the comment shows go here to get a new random hash.
/** A random secret hash used to encrypt cookies. You don't have to remember it, make it long and complicated. Hint: copy from http://yourls.org/cookie **/ define( 'YOURLS_COOKIEKEY', 'qQ4KhL_pu|s@Zm7n#%:b^{A[vhm' );

Update the username and password

The username and passwords are in the configuration file with password in clear text. Since this isn’t secure, I recommend not using a password that works anywhere else.
/** Username(s) and password(s) allowed to access the site */ $yourls_user_passwords = array( 'username' => 'password', 'username2' => 'password2' // You can have one or more 'login'=>'password' lines );

Setup Nginx for Rewrite

Edit the nginx configuration for the domain. The configuration of YOURLS provides an .htaccess file with rewrite rules which is fine for Apache, but not so well with nginx.

server {
server_name www.zod.im www.zodim.com;
rewrite ^(.*) http://zod.im$1 permanent;
}

server {
listen 80;
server_name zod.im;

access_log /var/log/nginx/zod.im.access.log;
error_log /var/log/nginx/zod.im.error.log;

root /var/www/zod.im/web/;
index index.php index.html;

error_page 404 /404.php;

location / {
# If the file exists as a static file serve it directly without
# running all the other rewite tests on it
if (-f $request_filename) {
expires max;
break;
}

if (!-f $request_filename){
set $rule_0 1$rule_0;
}
if (!-d $request_filename){
set $rule_0 2$rule_0;
}
if ($rule_0 = "21"){
rewrite ^/.* /yourls-loader.php last;
}
}
location ~ .php$ {
set $script $uri;
set $path_info "";

if ($uri ~ "^(.+\.php)(/.+)") {
set $script $1;
set $path_info $2;
}

include /etc/nginx/fastcgi_params;
fastcgi_pass 127.0.0.1:5555;
fastcgi_index index.php;

fastcgi_param SCRIPT_FILENAME /var/www/zod.im/web$fastcgi_script_name;
fastcgi_param PATH_INFO $path_info;
fastcgi_intercept_errors on;

root /var/www/zod.im/web;
}

Restart nginx

Go through setup

Open browser and go to the admin page, login and go through initial setup process.

http://example.com/admin/

You get two screens to see. First is initial welcome with button to install. Click this and get the second telling you it has created the database tables.

The second screen may complain about not being able to write the .htaccess file, but since nginx is used, that can easily be ignored.

Conclusion

The process to create your own personal shortening service can take a little as little as 15 minutes with YOURLS. It will probably take longer to figure out your short domain name and acquire it.

Impressions of Sony Dash

Michael Leo — Thu, 17 Feb 2011 09:29:29 GMT

For Christmas I received the Sony Dash as present from my wife. Having worked with computers, networking, and all sorts of technology in general I am both amazed and dismayed by Sony Dash at the same time.

Since this article can live for a long time and the Sony product team does seem to making monthly/bi-monthly updates, some of these comments may become outdated.

Amazed

Portal Integration
The device uses a web application to help install and configure the apps that run on the device. The portal makes it relatively easy to find apps that are available, but if the number grows, it will need to address some things to make it easier to find useful ones.

Unfortunately, a lot of the configuration is via Flash, which makes sense given the apps on the Dash itself are built in Flash. This makes it impossible to use with iPad. If the Dash lives in the bedroom, it is easier to configure it with the iPad rather than laptop that usually stays downstairs. It’ll probably work with Android tablets, but don’t currently have one of those.

Netflix
We don’t have a TV in our bedroom, and while the prospect of watching a movie on 7 inch screen isn’t going to be the bedroom home theater of choice, it is still an interesting device to have Netflix access on. It is able to access my Wireless-N WPA2 network which is good as I would like to phase out the B/G WEP. Anyway, Netflix runs fairly well on it. Netflix is probably the most ubiquitous service I have access to use.

Dismayed

Internet Radio
Given that the Dash does not provide AM/FM radio built-in, this is the best app to use to listen to your local stations. Unfortunately, this looks like one heck of a unfinished app; more to the point it appears to be more proof of concept that indeed Internet Radio can be streamed and is missing the consumer oriented features that make it easy to use. In my world of development, this is would be a manual test harness the backend developer put together to prove the playing of audio streams works.

Then Internet Radio Streamer allows for user to manually add streams that can then be played. By manually adding streams, you have to know the URL of the stream and type it in completely and the content type of stream, m3u or pls. There is no pre-screened list of stations or search functionality. You can only manually add new or edit & delete existing streams. The list of added streams can’t even be reordered or categorized.

OK, so to use the app I need to use my computer to go find stream URLs and add them to the Dash. Not too big a deal since the device does allow adding and configuration of apps on the web site. I can fire up Safari, find the streams and copy/paste to the Dash configuration site. Does this app allow for configuration on the web site? No. So now you have to have computer within eyes view of Dash to enter the stream URL.

I don’t go looking for audio streaming URLs everyday, so didn’t know where to go. My first instinct was to use put in radio stations into search box, “knx 1070 audio stream url”. This had success in getting me to pages that would play the stream online, but made getting the URL difficult. I used Safari Web Inspector to find the URL. One down, many to go.

Eventually, I decided to go a different route. I eventually found some aggregator sites that provided better access to the URL. The main site I used for most stations I eventually added was vTuner. VTuner provides an aggregation list of audio streams and an API to access the list. Their API has been added to many devices, just not the Sony Dash (yet?). It seems the Internet Radio Streamer app can be made vastly more user friendly.

Responsiveness
The thing is slow to respond to inputs. I hit a button on screen and then need to wait several seconds most of the time until I can tell whether or not it recognized the input. I understand the device is intended to be inexpensive, which probably limits what strength processor can be purchased, but wow.

Parting thoughts

While I have explored some apps, I haven’t had an opportunity to fully find the ones that fully interest me. I like the Dash and can’t wait to see what updates come in the future to make it more useful.

ForbesPrint.com and Open Source

Michael Leo — Tue, 22 Jun 2010 21:22:40 GMT

A year ago, my wife, daughter and I visited my wife’s aunt and uncle in Waterford, Michigan area. She signed me up to do a small web site for them to promote the printing business. The printing business has been in the family since 1946.

I decided to use open source tools as much as possible to deliver the site. Long term, I don’t expect to have access to tools I use at work and I actually know I shouldn’t use tools from work to work on external projects. And so, I stayed with open source tools as much as possible.

I set up the website relatively quickly with ruby on rails, but did not launch it for awhile. While I like the ruby on rails approach to rapid development I haven’t used ruby enough to learn the syntax and be productive in it. In the meantime between developing the site and actually launching it, I worked on a project for a client that utilized symfony. This gave me enough opportunity and time to learn php and the symfony framework. So the in-development site was migrated from a ruby on rails to symfony project.

One thing I liked when I set up the rails project was the usage of capistrano to manage the deployment of the application to my slicehost instance. I decided to see how it could work with symfony as well. In the end, I was able to get a good amount of information from the following site. Besides paths and server changes to the configuration presented on the page, I utilize git as version control.

I have found great value in git over subversion. I like the fact that git does not require server software to run. Getting it up and running with just a dedicated user and ssh was a snap. Even better, git doesn’t manage every file with 3 extra files per instance like subversion seems to do. To copy a subversion branch from one server to another takes forever due to all the .svn files. Git can easily be moved around and ability to remove git from a file structure works just as easily.

In the end, I built the very small site, ForbesPrint.com with the following tools:

php for scripting language
symfony for php framework
nginx for web server
fastcgi for php runtime environment
git for source control
gimp for image editing
slicehost for server hosting utilizing Ubuntu virtual machine
SpringSource Tool Set for IDE with Aptana Studio plugin for php project support

Now that the basics are in place, I hope to expand the site to better accommodate their needs and growth the business.

Blogging on the iPad

Michael Leo — Fri, 30 Apr 2010 21:31:13 GMT

My new 3g iPad arrived today and I have spent the evening having a look through the app store for apps that are built for the device. One of the apps I found was WordPress for the iPad. This is my first post on the iPad and the initial impression is that it is easy to do, but it is prone to moving the cursor around a little too easily.

So far the other apps I have tried are Netflix and USAToday. As my previous post mentioned, I imagine the most use of this device will be reading in certain room. I’ll have to wait until later to see how that goes.

On a technology front, my latest project involves the use of SDL Tridion for CMS. I don’t have too much to state about the product as yet. I’ll wait until the developers get going with the implementation before I mention anything about it.

MAMP & PECL/UUID Module Working On Snow Leopard

Michael Leo — Sun, 11 Apr 2010 23:33:05 GMT

I had a previous article that discussed in detail on how to get imagick php plugin working on MAMP. As always, as you get more exposed to technologies and frameworks you find new ways to do things. One thing to keep in mind with MAMP is that 1.8.4 is still 32-bit build. Once it goes 64-bit, I am sure the whole thing will get easier.

In this instance, I was working through adding UUID generation support to my PHP code. While the uniqid provides a function to generate IDs, it is not a GUID specification. There is PECL UUID module that provides “a wrapper around libuuid from the ext2utils project”. After looking around for how to add it to MAMP, I came across the following article that talked about how to enable the uploadprogress module. utilizing the same process I attempted to build the uuid module. Unfortunately, the uuid module failed to compile. This eventually lead to a defect report for the uuid module about Mac OS X and older version of files. Attached was a patch file; patched the files and remade. Installed, reloaded and all was good.

The following are the steps I used for the installation of uuid module.

Download and extract the uuid module
Grabbed the 1.2 version of the uuid module.
macbook:Downloads mleo$ wget http://pecl.php.net/get/uuid-1.0.2.tgz ... mabook:Downloads mleo$ tar -xzf uuid-1.0.2.tgz macbook:Downloads mleo$ cd uuid-1.0.2

Get & Apply Patch
Copy patch code from here. Save the contents into mac.patch file. Apply patch. It did require entry of the path to two test files during the patch process.
macbook:uuid-1.0.2 mleo$ patch < mac.patch patching file config.m4 patching file php_uuid.h can't find file to patch at input line 103 Perhaps you should have used the -p or --strip option? The text leading up to this was: -------------------------- |diff -urp uuid-1.0.2/tests/uuid_mac.phpt uuid- |1.0.2.mine/tests/uuid_mac.phpt |--- tests/uuid_mac.phpt 2008-04-01 08:59:22.000000000 -0700 |+++ tests/uuid_mac.phpt 2008-08-14 10:21:57.000000000 -0700 -------------------------- File to patch: tests/uuid_mac.phpt patching file tests/uuid_mac.phpt can't find file to patch at input line 116 Perhaps you should have used the -p or --strip option? The text leading up to this was: -------------------------- |diff -urp uuid-1.0.2/tests/uuid_time.phpt uuid- |1.0.2.mine/tests/uuid_time.phpt |--- tests/uuid_time.phpt 2008-04-01 08:59:22.000000000 -0700 |+++ tests/uuid_time.phpt 2008-08-14 10:22:50.000000000 -0700 -------------------------- File to patch: tests/uuid_time.phpt patching file tests/uuid_time.phpt patching file uuid.c

Configure and Build the Module
I'll leave the prompts out of this one to make it easier to copy and paste the set of commands, but each line in the below code block is different command.

/Applications/MAMP/bin/php5/bin/phpize MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS="-arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp" CCFLAGS="-arch i386 -arch x86_64 -g -Os -pipe" CXXFLAGS="-arch i386 -arch x86_64 -g -Os -pipe" LDFLAGS="-arch i386 -arch x86_64 -bind_at_load" export CFLAGS CXXFLAGS LDFLAGS CCFLAGS MACOSX_DEPLOYMENT_TARGET ./configure --with-php-config=/Applications/MAMP/bin/php5/bin/php-config make make install

Add module to php.ini
Edit the /Applications/MAMP/conf/php5/php.ini file and add "extension=uuid.so".

Test Installation
Can check info to see if uuid module loads and run the uuid_create() function to see if it returns.

macbook:uuid-1.0.2 mleo$ /Applications/MAMP/bin/php5/bin/php -r 'print phpinfo();' ... uuid UUID extension

Version => 1.0.2 (stable)
Released => 2008-04-01
CVS Revision => $Id: uuid.c,v 1.9 2008/04/01 15:58:52 hholzgra Exp $
Authors => Hartmut Holzgraefe 'hartmut@php.net' (lead)
...

macbook:uuid-1.0.2 mleo$ /Applications/MAMP/bin/php5/bin/php -r 'print uuid_create() . "\n";'
85C43B35-1416-435F-AEFF-3E4693ACEE65
macbook:uuid-1.0.2 mleo$

Most Popular Room for iPad

Michael Leo — Wed, 31 Mar 2010 21:46:11 GMT

My prediction for the iPad is that some survey will come out 6 months after the release that finds people use their iPad more often in the bathroom than any other room in the house. The logic on this is simple. People already have laptops, desktops and fully functional computers available in the living room, office and/or bedroom. The bathroom is really one of last bastions of truly tech free peace and quiet.

Using YAML as Response Format for Services in Spring

Michael Leo — Tue, 02 Mar 2010 11:26:20 GMT

Recently, we were developing an iPhone application where the iPhone developer was insistent on utilizing YAML as the response format for service calls. This seemed odd given the prevalence of JSON as a response format and an already built library for parsing and handling JSON within Objective C that works with the iPhone SDK. The other advantage to JSON is there is another project that has built a JSON view resolver for converting model/domain objects into JSON. The advantage with YAML is that it inherently supports circular references and can resolve them during serialization and deserialization. While domain objects can have circular references in design, I try to keep them minimal and out of the model objects to ensure most serialization techniques work easily.

I couldn’t quickly find any YAML Spring View support so went about working through my own. Using the source of the JSON Spring View project as baseline I was quickly able to build out a baseline view handler. I say baseline because I haven’t fully unit tested the code as yet.

YamlView Class
This makes use of the jyaml library.

public class YamlView extends AbstractView { private static final String DEFAULT_YAML_CONTENT_TYPE = "application/yaml";

public YamlView() {
super();
setContentType(DEFAULT_YAML_CONTENT_TYPE);
}

/**
* Creates a YAML String from the model values.
*/
@SuppressWarnings("unchecked")
protected final String defaultCreateYaml(Map model) {
return Yaml.dump(model);
}

/**
* Creates a Yaml String from the model values.
*/
@SuppressWarnings("unchecked")
protected String createYaml(Map model, HttpServletRequest request, HttpServletResponse response) {
return defaultCreateYaml(model);
}

@SuppressWarnings("unchecked")
@Override
protected void renderMergedOutputModel(Map model, HttpServletRequest request, HttpServletResponse response)
throws Exception {

response.setContentType(getContentType());
writeYaml(model, request, response);
}

@SuppressWarnings("unchecked")
protected void writeYaml(Map model, HttpServletRequest request, HttpServletResponse response) throws Exception {

String yaml = createYaml(model, request, response);
response.getWriter().write(yaml);

}
}

Configuration: applicationContext.xml
In one of the Spring configuration files where the web presentation layer is set.
... ...
The order is in place because this configuration file has multiple ViewResolvers. The xmlViews.xml then has the configuration:

And finally, the usage within a controller:

public class ExampleController extends AbstractCommandController { ... @Override protected ModelAndView handle(HttpServletRequest request, HttpServletResponse response, Object command, BindException bindException) throws Exception {

...
String viewType = "jsonView";
if (request.getRequestURI().endsWith(".yaml")) {
viewType = "yamlView";
}

return new ModelAndView(viewType, model);
}
...
}

It really is amazing the little amount of custom code required these days to get great results.

Gadgets Galore - The iPad would just be a small incemental add

Michael Leo — Fri, 29 Jan 2010 00:59:16 GMT

The Apple iPad was announced yesterday and I do believe the hype was so insane leading up to the announcement that Steve Jobs could have introduced a semi-transparent 1/4 inch laser screen that ran for a month on full charge and people would still have made fun of the name. I kind of think the first generation of the iPad should just have been called the iPod Touch XL as it more like the Touch and less like the iPhone.

The day after led to all the news and I came across an interesting AP article that stated

… the average household owns about 24 electronic gadgets …

and this got me thinking about how many we own in our household of 3 people. I came up with the following list of over 40 items that are used on regular or semi-regular basis and are likely plugged-in sucking power.

The List

I’ll list the items in the home theater and summarize the others.

Home Theater: 10 Gadgets

Sony Receiver & Bose Surround Sound Speakers (2001)
Sony Plasma TV –Living Room (2003)
Sony DVD player (2004)
TiVo HD* (2007)
Sony PS3* (2006)
Apple AppleTV* (2007)
Actiontec Cable/Moca Modem* (2007)
Linksys Simultaneous Dual Band Router* (2009)
Western Digital NAS* (2008)
Logitech Universal Remote (2008)

Arguably, the receiver and speakers are 2 items, but the one is really not useful without the other in this case.

Audio Players: 4 Gadgets
We currently have 3 working iPods (Video, Mini, Shuffle) not counting the iPhone. The Mini and Shuffle aren’t used much anymore. The Video is hooked up to the screen in the car to provide hours of Handy Manny to my daughter on long car trips. We also have a Delphi Portable XM Satellite receiver that my wife uses at work.

Cameras: 5 Gadgets
We have 5MB and 8MB point and shoot digital cameras. I received a Canon 20D DSLR for Christmas in 2005 and added an eyeFi Wireless SD card (with CF adapter) a couple years later. We added an HDD Digital Camcorder before my daughter was born.

Kitchen: 4 Gadgets
In the kitchen is set top box hooked up to TV with a Mac Mini connected to network via a Netgear MOCA bridge.

Bedrooms: 7 Gadgets
We have 2 bedrooms but have kept to keep one of them free of television, and at some point will probably remove the television from the second one as well. But currently across the two bedrooms we have Desktop computer & printer, set top box, TV, and Slingbox. To have the computer and slingbox connected to the network I use Linksys Dual Band Ethernet Wireless Bridge and Slink Ethernet over Power bridge.

Other Entertainment: 4 Gadgets
I purchased an Amazon Kindle for my wife this past Christmas. The Kindle is amazingly better in functionality than the Sony eReader purchased 2 years ago. The eReader isn’t used anymore and isn’t in this list. The other items include 2 Apple Airport Expresses purchased about 3 years apart. One was for the home theater long before the AppleTV was added and the other was for bedroom and printer. Finally, we have a Brookstone wireless speaker that is more often hooked up to one of the Airports now.

Portable Game Stations: 2 Gadgets
A Sony PSP and Nintendo DS.

Phones: 3 Gadgets
We still have regular landlines and 2 smart phones, an Apple iPhone 3G and Samsung Blackjack II.

Work Related: 4 Gadgets
My wife and I each have a laptop provided by our employers and so maybe these not count towards the list. I also have a Linksys Travel Router and the all important iGo power adapter with 8 or so different tips; something has to be able to provide power to all the devices.

Cars: 2+ Gadgets
Add to this our two cars that both have built-in GPS and one that has a DVD player with back seat screen and XM satellite radio.

So based on this number, adding an iPad to the household is only a 2% increase in the number of gadgetry we have to deal with.

Connected Devices

Going through that list of gadgets and counting the number of items that can connect to the local network or Internet via Ethernet, MOCA, or 802.11 came to 19 devices. Having so many devices is part of why I have tried to upgrade and replace network devices more than anything else. The Ethernet over Powerline worked for the Slingbox in one bedroom but not the desktop in another room. The desktop got the Dual Band Ethernet Bridge and this performs pretty well. The Mac Mini was also using wireless and wanted to see if I could do better. I have Verizon FIOS and they use MOCA for the set top boxes to connect to the network and receive programming guide data as well as Video On Demand. I started looking at MOCA bridges and added a Netgear device for the Mac Mini to connect to the router. So I have pretty much tried every major consumer based physical medium for local area networks.

Replacing Gadgets

Most of the items in this list were purchased over the course of 10 years so it’s not like we went hog wild in a short period of time. I have generally been lucky with electronics and have had them run as long as needed. The only exception seems to be wireless routers, Logitech remotes, and of course mobile phones. Over the course of the 10 years I had a wired router, 1 wireless B, 3 wireless G, and 2 wireless N routers. The switch from B to G to N to N (simultaneous dual band) was more about speed increases than failure. The Logitech remotes on the other hand… I think I had 3 that hit the wood floor and failed. The mobile phone changes consisted of 9 phones over 12 years. 4 changes were due to service provider changes. 3 were due to failure or loss. Only 2 were explicitly for something new. Switching to AT&T to get the iPhone was for something new.

Unplugged, but Serviceable

I won’t get into the unplugged devices, mostly it is older gaming systems.

Wrapping It Up

So, do I think I will get an iPad. Maybe, but not soon after it is released. I really think it is no more than extra large iPod Touch in this first generation presented. However, it has potential to open floodgates of creativity in further development of apps. Additionally, Apple has yet to release features for iPhone OS 4.0 and I imagine it will provide more interesting features for the next iPhone and updates to the iPad.

One Line Command - Print Directory Tree

Michael Leo — Tue, 26 Jan 2010 16:56:45 GMT

While this blog is new, I want to ensure I keep content going lest it goes abandoned and six months from now I wonder what happened. Most of my original inspiration on putting content here is based on useful work I do. Not having anything interesting to share from current work, I’ll share a segment of things I find interesting and useful from time to time.

So this post is kicking off my category of “One Line Commands”. Over the years I have found myself finding, grepping, cutting and PERLing through directories and files to find some useful information to solve some issue. This first example is a PERL script that recurses through directory tree printing and adds indent based on the current level. Output can be sent to a file and imported into Excel. It is setup to ignore subversion files/directories because at the time I wanted to get a full directory tree from a checked out repository.

perl -e 'use File::Find; sub pl {my ($sc) = @_; print "\t" while ($sc-- > 0); } sub wanted { $f=$_; $fn=$File::Find::name; return if $fn =~ /.svn/; my $sc=($fn =~tr/\///); if (-d $fn) {pl($sc); print "$f/\n";} else {pl($sc); print "$f"; pl(20-$sc); print "$fn\n";}}; find(\&wanted, ("."));'

Copy and pastes into one command line entry.

MAMP & Imagick on Snow Leopard

Michael Leo — Tue, 05 Jan 2010 16:51:20 GMT

Most web searches for adding the imagick extension to MAMP suggest using MacPorts to install ImageMagick and then use this compiled version as the library to running ‘pecl install imagick’.

As I was in the process of setting up a clean Mac Book Pro, I tried going through this process. I quickly ran into some problems getting the imagick extension running under MAMP. This is a tail of that adventure.

I started with the latest version of MAMP 1.8.4 and MacPorts 1.8.2 installed.
Then installed ImageMagick

$ sudo port install ImageMagick

This went off without a problem. Could then easily run the ‘convert’ program to see if is working. Next step felt like to install imagick extension.

$ cd /Applications/MAMP/bin/php5/bin

And just make sure the right version of PECL is executed via ‘./’ else may get Mac OS X installed or even MacPorts installed version.

$ ./pecl install imagick

The process doesn’t stop with error, but immediately it has gripes about missing header files.

downloading imagick-2.3.0.tgz ... Starting to download imagick-2.3.0.tgz (86,976 bytes) .....................done: 86,976 bytes 12 source files, building WARNING: php_bin /Applications/MAMP/bin/php5/bin/php appears to have a suffix 5/bin/php, but config variable php_suffix does not match running: phpize

grep: /Applications/ MAMP/bin/php5/include/php/main/php.h: No such file or directory

grep: /Applications/ MAMP/bin/php5/include/php/Zend/zend_modules.h: No such file or directory

grep: /Applications/ MAMP/bin/php5/include/php/Zend/zend_extensions.h: No such file or directory

Configuring for:
PHP Api Version:
Zend Module Api No:
Zend Extension Api No:
configure.in:158: warning: AC_CACHE_VAL(lt_prog_compiler_static_works, ...): suspicious cache-id, must contain _cv_ to be cached
../../lib/autoconf/general.m4:1998: AC_CACHE_VAL is expanded from...
...
aclocal.m4:4641: _LT_AC_TAGCONFIG is expanded from...
Please provide the prefix of Imagemagick installation [autodetect] :
$

The first problem encountered was simply getting ‘pecl’ to compile the imagick extension as the MAMP distribution does not include the php headers where pecl is expecting them.

There is probably a way to get pecl to pick them up properly, but in the interest of time, one can just create a link from

$ cd /Applications/MAMP/bin/php5

$ ln -s /Applications/MAMP/Library/include include

And then re-run ‘pecl’

$ ./pecl install imagick ... Please provide the prefix of Imagemagick installation [autodetect] : /opt/local building in /var/tmp/pear-build-mleo/imagick-2.3.0 running: /private/tmp/pear/temp/imagick/configure --with-imagick=/opt/local ...

and provide ‘/opt/local’ when prompted for prefix of Imagemagick installation. Everthing seemed to go well. Finally, modified /Applications/MAMP/conf/php5/php.ini to add

extenstion=imagick.so

And modified /Applications/MAMP/Library/bin/envvars to add

DYLD_LIBRARY_PATH="/Applications/MAMP/Library/lib:/opt/local/lib:$DYLD_LIBRARY_PATH"

Finally ran:

$ /Applications/MAMP/bin/php5/bin/php -i | less

to see if it was enabled. Unfortunately, the imagick module was not listed. After a little investigation, in the php error log was the following entry:

PHP Warning: PHP Startup: Unable to load dynamic library '/Applications/MAMP/bin/php5/lib/php/extensions/no-debug-non-zts-20060613/imagick.so' - (null) in Unknown on line 0

After some various searching and other, checking what I did wrong, I finally considered the files:

$ file /Applications/MAMP/bin/php5/bin/php /Applications/MAMP/bin/php5/bin/php: Mach-O universal binary with 2 architectures /Applications/MAMP/bin/php5/bin/php (for architecture ppc): Mach-O executable ppc /Applications/MAMP/bin/php5/bin/php (for architecture i386): Mach-O executable i386

$ file /Applications/MAMP/bin/php5/lib/php/extensions/no-debug-non-zts-20060613/imagick.so /Applications/MAMP/bin/php5/lib/php/extensions/no-debug-non-zts-20060613/imagick.so (for architecture x86_64): Mach-O 64-bit x86_64

So discovered that MAMP is a Universal binary, just not an x86_64 universal binary. So, first, make sure to rebuild imagick.so with i386 binary support and just to be safe, add those settings for the php header files as well.

So, back to the php distribution and run configure setting some flags, although this may not be needed, it can’t hurt.
$ MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS="-arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp" CCFLAGS="-arch i386 -arch x86_64 -g -Os -pipe" CXXFLAGS="-arch i386 -arch x86_64 -g -Os -pipe" LDFLAGS="-arch i386 -arch x86_64 -bind_at_load" ./configure --enable-shared

loading cache ./config.cache checking for Cygwin environment... (cached) no ...

Copy the headers back to MAMP location. And re-run pecl to build universal binary.

$ ./pecl uninstall imagick

$ MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS="-arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp" CCFLAGS="-arch i386 -arch x86_64 -g -Os -pipe" CXXFLAGS="-arch i386 -arch x86_64 -g -Os -pipe" LDFLAGS="-arch i386 -arch x86_64 -bind_at_load" ./pecl install imagick

Run php printinfo and
$ /Applications/MAMP/bin/php5/bin/php -i | less

dyld: NSLinkModule() error dyld: Library not loaded: /opt/local/lib/libMagickWand.2.dylib Referenced from: /Applications/MAMP/bin/php5/lib/php/extensions/no-debug-non-zts-20060613/imagick.so Reason: no suitable image found. Did find: /opt/local/lib/libMagickWand.2.dylib: mach-o, but wrong architecture

And discover that MacPorts isn’t creating universal binaries.
$ file /opt/local/lib/libMagickCore.2.dylib /opt/local/lib/libMagickCore.2.dylib: Mach-O 64-bit dynamically linked shared library x86_64

Again, head to the favorite search engine and look at how MacPorts can work with Universal binaries. After some criteria changes came across this simple set of details.

MacPorts provides a variant +universal for all installs that use configure script to initialize build process that will build Universal binaries. The specific architecture has some defaults based on the OS and current hardware.

A bit of pain to add +universal to every port install command, so make +universal, uh, universal it can be added to the /opt/local/etc/macports/variants.conf file as last line:

# To specify global variants to use for all port builds, # customize this file to list variant settings you want. # # Be sure to uncomment/define the variants_conf setting # in the system wide ${prefix}/etc/macports/macports.conf # file or in your personal ~/.macports/macports.conf to # point to this file to enable this feature. # # Any variants specified here that are not supported by # a port will just be ignored. Multiple variants can be # specified per line, or one per line is also allowed. # # Example: # +ipv6 +no_x11 +universal

And to force a rebuild of anything you have installed

sudo port upgrade --force installed

Finally, we try php again:

$ /Applications/MAMP/bin/php5/bin/php -i | less

... imagick

imagick module => enabled
imagick module version => 2.3.0
imagick classes => Imagick, ImagickDraw, ImagickPixel, ImagickPixelIterator
ImageMagick version => ImageMagick 6.5.8-0 2010-01-05 Q16 http://www.imagemagick.org
ImageMagick copyright => Copyright (C) 1999-2009 ImageMagick Studio LLC
ImageMagick release date => 2010-01-05
ImageMagick Number of supported formats: => 194
ImageMagick Supported formats => 3FR, A, AI, ART, ARW, AVI, AVS, B, BGR, BMP, BMP2, BMP3, BRF, BRG, C, CAL, CALS, CAPTION, CIN, CIP, CLIP, CMYK, CMYKA, CR2, CRW, CUR, CUT, DCM, DCR, DCX, DDS, DFONT, DNG, DOT, DPX, EPDF, EPI, EPS, EPS2, EPS3, EPSF, EPSI, EPT, EPT2, EPT3, ERF, FAX, FITS, FRACTAL, FTS, G, G3, GBR, GIF, GIF87, GRADIENT, GRAY, GRB, GROUP4, HALD, HISTOGRAM, HRZ, HTM, HTML, ICB, ICO, ICON, INFO, INLINE, IPL, ISOBRL, JNG, JPEG, JPG, K, K25, KDC, LABEL, M, M2V, M4V, MAP, MAT, MATTE, MIFF, MNG, MONO, MOV, MP4, MPC, MPEG, MPG, MRW, MSL, MSVG, MTV, MVG, NEF, NULL, O, ORF, OTB, OTF, PAL, PALM, PAM, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF, PDFA, PEF, PFA, PFB, PFM, PGM, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG24, PNG32, PNG8, PNM, PPM, PREVIEW, PS, PS2, PS3, PSD, PTIF, PWP, R, RADIAL-GRADIENT, RAF, RAS, RBG, RGB, RGBA, RGBO, RLA, RLE, SCR, SCT, SFW, SGI, SHTML, SR2, SRF, STEGANO, SUN, SVG, SVGZ, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TTC, TTF, TXT, UBRL, UIL, UYVY, VDA, VICAR, VID, VIFF, VST, WBMP, WMV, WPG, X, X3F, XBM, XC, XCF, XPM, XPS, XV, XWD, Y, YCbCr, YCbCrA, YUV

Directive => Local Value => Master Value
imagick.locale_fix => 0 => 0
imagick.progress_monitor => 0 => 0

...

Somewhere during this adventure I definitely considered dumping MAMP and using apache, php and mysql from MacPorts but MAMP does generally provide a simple package installation and a GUI for quickly starting and stopping the services.