AddedBytes.com Blog (formerly ILoveJackDaniels.com)

Blogging in Business

dave@addedbytes.com — Fri, 14 Nov 2008 10:15:44 GMT

When I started at Active Parity, one of the first things I wanted to do was help the company to get a blog started. They're a great way to show off your knowledge and communicate with clients. My first post over on the new blog is up, and it's on exactly that subject: Blogging in Business - specifically, why businesses should blog and what the potential downsides are. The first of many posts over there, I'm sure!

Personal Development: To Do

dave@addedbytes.com — Thu, 21 Aug 2008 12:34:54 GMT

On Hacker News, ambition posted a to-do list inspired by / taken from this excellent bit of advice from Chris Wanstrath. Which got me thinking about what I want to work on and with in my spare time.

I've been meaning to organise my side-projects better. Like everyone else, I have lots of ideas and little time to make anything of them. I have a folder packed with projects at 95% completion, sitting there unloved because I got distracted, or found something better to use.

The problem with that is that taking projects to 95% is ultimately demotivating. It breeds guilt, and that's not helpful. And a project at 95% doesn't pay you back for the time you put in to it. You eventually need to release something if you don't want to end up looking back and seeing missed opportunities and wasted time.

In addition to a collection of projects on the go and ideas, there are technical skills I want to develop. I'm learning Python, and Linux server administration. I'm interested in looking into Objective-C and Cocoa. jQuery is great but I need more time with it. My "Dave! Play with PostgreSQL!" post-it is faded it's been on my wall for so long. And I need to stay sharp with the languages and technologies I use day-to-day.

Some fat needs to be trimmed.

I need to leave time for new things, too. Stuff I've not heard of yet. I'm always going to be distracted by shiny new technology. I think that's a good thing. But I want time to experiment and to tinker. If I earmark all my time for projects, I'm not going to suddenly lose interest in web technologies and tools. No, I'd start cutting into time I've promised to other things. Voila - the guilt's back and the schedule's shot. Back to square one.

So I've spent some time thinking about what I really want to get out of the time I spend on personal projects, and come up with a to-do list. I expect this to change over time, and while I don't expect for a second that you, the reader, will have the same goals or that this list itself will be useful to you, I hope if you're in a similar position it helps you to get a handle on things and get back to spending your time doing what you enjoy.

Keep on blogging!
Keep on making cheat sheets!
Move AddedBytes (set up server).
Thin out project folder and pick 2 to work on until finished.
Write a web service.
Write SVN Statistics app in Python (learn Python).
Rewrite site management VB app in Python (learn Python).
Learn Objective-C and Cocoa by writing a Useful Small Mac App (decide on what app!).
Learn a new PHP framework.
Get involved in an open source project.
Update and release more code from AddedBytes.com under open source license.

That should keep me going for a while. Next, I need to flesh out some of those ideas and work out how much time I can put into them.

Readability Code Open Sourced

dave@addedbytes.com — Fri, 01 Aug 2008 12:01:00 GMT

In July 2004 I wrote some code to calculate the readability of text using the most common algorithms available (Flesch-Kincaid and Gunning-Fog). The code hasn't aged well, and had many flaws, especially when it came to the subject of syllable counting.

Syllable counting is a tricky prospect. Consider the following sentence, for example: "I moped about, hopeful that my moped would be back on the highway soon". Sound innocuous? There's a pair of homographs in there (two words, spelled the same, that sound different) - and these have different syllable counts depending on which of the two words you mean. Words can be almost identical, with the same order and number of consonants and vowels (and it's that order you generally use to calculate syllable numbers) - "sired" has one syllable, while "sided" has two. Throw in prefixes, suffixes, plurals and compound words and you've got yourself a challenge.

Syllable counting is a minefield, with a small set of rules and a massive set of exceptions to handle.

That said, I've spent some time working through a set of test data and have come up with a small set of rules to take on the task. It helped tremendously having the work of Greg Fast (creator of Perl module Lingua::EN::Syllables) handy for reference, and setting up a decent set of unit tests allowed me to experiement with different rules until I found a set that works. So far. I expect to find more and more exceptions as time goes on, and hopefully the rules can be expanded to account for them.

It wasn't just the syllable counting that was bad. The code was inefficient, disorganised, and incapable of handling anything unpredictable (every extra space counted as an extra word, for example). There were lines in there that didn't make any sense. And I hadn't documented anything, so couldn't tell you why I'd added them in the first place. Oh to be that young and inexperienced again ...

So, as with the other releases in the last few weeks, I went back and rewrote the code properly. The new and improved version has been released as a Google Code project by the name of PHP Text Statistics. It's released (as with the other projects I've set free recently) under a New BSD License.

TextStatistics.php

It consists (so far) of a single class that will tell you various things about the text you feed it:

String length
Letter count
Syllable count
Sentence count
Average words per sentence
Average syllables per word

It will also calculate the readability of the text you enter according to the 6 known algorithms (links go to Wikipedia):

TextStatistics.php4

There is also a PHP4 compatible version of the code. At the time of writing, it returns the correct scores for test data, though given PHP4's decline and the rise of PHP5, this version may not remain as current as the previous file.

tests/

Next thing to be aware of is the unit tests included in the project. There's no easy way to check your calculations are correct, unless you have a set of verified numbers to compare them against. So, I put together (so far) three files with a variety of different tests for the code. These tests should be run with PHPUnit and at the time of writing they all pass (which means there's not enough of them yet).

tests/TextStatisticsTest.php

The basic unit test class lists a large selection of words and compares their calculated syllable count with their actual syllable count (worked out the old fashioned way). It includes a variety of tests to ensure sentence counting and word counting both work as intended. It also includes a small selection of sentences, for which readability scores have been calculated by hand, and checks that the class returns the correct scores for these items.

tests/TextStatisticsKiplingIf.php

Rudyard Kipling's If is, aside from a brilliant piece of inspiring poetry, one long sentence comprised of lots and lots of short words (take a look - impressive how few of the words are multi-syllabic). This file contains a selection of tests to run against If. It checks all of the words of the poem have their syllable count correctly calculated, and that all of the readability scores are correctly calculated, by matching the calculated scores against hand-calculated numbers.

tests/TextStatisticsMelvilleMobyDick.php

Herman Melville's Moby Dick is up next (well, the first paragraph is - I'm not prepared to count, by hand, the number of syllables in the entire book). Like If, it is (I believe) in the public domain, so can be used for this sort of purpose without complications. It's also a brilliant read. This file contains a selection of tests to run against the first paragraph of Moby Dick. It checks all of the words of the passage have their syllable count correctly calculated, and that all of the readability scores are correctly calculated, by matching the calculated scores against hand-calculated numbers.

Get Involved!

This project can benefit from the involvement of people in many ways. Initially, the most helpful thing anyone can do is find words whose syllable count is not correctly calculated by the script and add a new test for that word. There are going to be a lot out there (especially compound words, like "shoreline", and odd words that are not pronounced according to normal rules, like "simile").

The class could be expanded to give more information about text - like letter frequencies, word and phrase freqencies (useful for SEO) and unique word count, among other things. I've made a start on making the code multi-byte character set safe, but there's lots more to do there too.

The really brave could add more test text, too. Paragraphs of (public domain) text provide an excellent way to check the tool is working as it should. I'd suggest using either the Kipling or Melville file as a template to work from, and prepare for a boring few hours. You get a great feeling of satisfaction at the end, though, when the whole thing is done!

There's a discussion group for ... well, discussion. Suggestions, comments and feedback all welcome. If you would like to get involved in this project, start there (or email me), or grab a copy of the code from the SVN repository on Google Code.

Cheat Sheet Requests Updated

dave@addedbytes.com — Mon, 28 Jul 2008 19:18:14 GMT

The Cheat Sheet Requests system has so far been more successful than I had hoped it could be, so I'm chuffed, and very grateful to everyone who has contributed so far.

Three problems have arisen, however. The first is the sheer volume of requests. There are simply too many to give people a reasonable selection to choose from, so the voting becomes skewed in favour of the items already at the top of the list.

The second problem is abuse, which has of course been rife. Lots of people requesting all sorts of dark and depraved cheat sheets. Amusing, but as requests go live instantly, not ideal.

Finally, there are only so many hours in the day, and I'm never going to be able to fulfil every cheat sheet request (though I am working on something to help people make their own).

Something that was unexpected but entirely positive was (is) the volume of requests and votes for non-techy cheat sheets. A "Girls" cheat sheet is in high demand, though I suspect it would take far more than a A4 sheet to list anything useful about women. And that's assuming any of the content could be agreed on by anyone. "French", "Guitar", "Chess" and "Leadership" are all great ideas, and I'm definitely up for doing more non-geek ones. That said, I suspect "Squid" and "Cockroaches" are not entirely serious requests.

So, I've changed the system. Now, you can still request anything you like, but only short-listed ideas will be displayed for voting. These are the ones I've picked out of the requests list that are very likely, depending on votes, to become cheat sheets. The top of the pile, so to speak. At the moment, the list is 15 requests, drawn from the most popular and the most interesting to date.

Those that aren't on the short list won't be forgotten or ignored. I'm going to start organising the requests into categories, and create a new section for requests and votes for those who want to see, and add to, the full list of requests.

For those of you not part of the Google Group, the new Python and Subversion cheat sheets are both in the preview stage there, so if you want to have a say about what goes on them, or just see what they look like before they're properly released, take a look at the Added Bytes Cheat Sheets Google Group.

Modem Emulator Open Sourced

dave@addedbytes.com — Fri, 18 Jul 2008 03:09:00 GMT

In July 2004 I released a modem emulator (a.k.a. a throughput throttling proxy). It was created to help give designers a sense of how their sites function for people with slower connections.

I've had to take it offline a number of times due to the volume of traffic and the various ways it was being used (turns out it was a highly effective way to bypass workplace web filters).

Not only that, the code was badly out of date (code soup, not an object in sight, no real validation ... the shame) and badly needed an update.

It's been sitting there, half-working and half-not, and begging for an update for almost exactly 4 years. Ultimately, the choice was to update it or kill it permanently.

So, I spent some quality time rewriting the whole thing, pretty much from the ground up, and now with pleasure announce that it has been turned into an open source project (yes, another one) and the code is now available from Google Code under a New BSD License.

With any luck, this will allow more people to make this tool part of their workflow.