It's not art, it's design

I recently read Eric Ries' Lean Startup, it's excellent, a must read for anyone doing any sort of product development, not only software.

The nutshell version that I wish every developer & designer should would take to heart:

What you're building is not art, it's a functional workflow. Your design is an experiment to make that interaction the most satisfying user experience possible. You are not your user.

New beginnings, new ideas

What an amazing time to be a developer. It seems like 10 year old hypothetical situations have suddenly exploded into reality. Node.js, github, NoSQL, Metro, WinRT, ASP.NET OSS, it's like a whirlwind. As someone who's always specialized in not specializing it's almost overwhelming to try keep up.

What I do know is this:

The web won. 

So long Silverlight, Java and Flash. JavaScript wins (as frightening as that sounds) I think we owe "alert" a bit of respect, time to learn how to do it properly, even on the server.

Javascript, HTML & css are now the lingua franca of everything, not just the net. If you think Apps are better, you miss the point, you can write apps in HTML and Javascript thanks to PhoneGap and Metro. (I can't come up with a decent acronym for it, JHaCss? CJaH? CEH? - techincally it's ECMAScript).

Even databases, I don't think it's a case of "Why NoSQL?" anymore. In most use cases I have to ask "Why SQL?", and do I really need to host my own? (Quick plug for Cloudant and MongoLab ;) ) 

Javascript still sucks.

As my Mum always used to say, winning isn't everything. And my the rules of  portability don't change. Write once, debug everywhere holds more than ever. If you're fluent in something like C#, javaScript is incredibly aggravating. 

To the point,

I'm starting a new learning app, it's a Metro C# music sequencer I've dubbed "Seque Beat" for a bit of fun.

https://github.com/doobiwan/SequeBeat

Nothing really there at the moment, I want to explore portable assemblies, TDD, all the good stuff, feel free to join in.

Yay for Metro and Windows 8! Boo for lazy haters .... ;)

Why do all my posts start as comment posts.....


Look, consider me a bit harsh if you will, but if a developer considers themselves just a "Silverlight Developer" they deserve to feel pain. No one technology can solve every problem, the beauty of ".NET", or lets rather say "Managed" applications if you will, is not write once run everywhere, that was debunked a good 10 years ago after Java failed miserably at it. What C#/XAML/Visual Studio/LINQ give you is a set of very powerful, specialized tools you take with you and use as necessary.

I do windows I do web, I do services, I do some Silverlight, and I'd be happy to tackle Mac, Android or iOS with Mono if the opportunity ever presented itself.  What ports is the language and the patterns, the rest is too difficult to abstract over such diverse platforms. I learnt that first from Miguel de Icaza (who weighs in fantastically here http://tirania.org/blog/archive/2011/Sep-15.html) and secondly from  painful experience.

And that is exactly the issue with Metro.

Under the hood Silverlight is the closest cousin to WinRT there is, they're both subset CLRs based on native implementations rather than pure CLR due to constraints - Silverlight needs to be tiny to be downloadable, WinRT needs to be as performant as possible. Silverlight is designed for browser downloadable RAD desktop apps. WinRT is designed for App Store sourced, "run on any processor" Touch apps.

I won't pretend to be fluent in architect, but I guess you could call them domain specific language implementations.

So now what?
If you really want reuse what you should really be looking at is WCF and/or REST. Not necessarily even server side, run a windows services and make your Metro, Silverlight, HTML, WPF and, well frankly anything, just a thin UI implementation.

Just remember, every time you over engineer, a Project Manager kills a kitten.

On Frameworks

This was a comment in response to a post by Jeremy Miller. It got a bit wordy so I figured I might as well keep it ;) 


I love the idea of choice, I love that there are multiple Frameworks, and I would love them all to be uber successful.

But... It comes down to portability, long term support,  company culture, and having a common toolbox. 

I've been at this game for over a decade, and the biggest problem I  encounter over and over again (besides beancounters ;) ) is legacy frameworks. By that I mean things that in their day were considered by someone as the tool de jour, but have fallen out of favour and/or and maintenance and now hold products back from forward migration, or keep the people maintaining those apps from progressing their skills. Tools that effectively require developers to become specialist of themselves.

It could be anything  - COM components, CMS frameworks, web Frameworks, DataSets, XML islands, Microsoft Entlib ... ;)  etc etc

Developer "A" implements "Excellent Architecture Foo" and alls good, because he understands and buys into it. He leaves, developers "B" comes along that has a different architecture perspective and it doesn't gel. Something like a culture change from "Software Factories" to YAGNI. Everything turns into a hack, and next thing you have is spaghetti (usually with meatballs). 

The way I (try to) avoid that is by very careful consideration. Choosing between Fubu, OpenRasta, MonoRail and ASP.NET, is not just about their relative merits, but their long term impact on my products and my teams, and our commitment to maintaining those products. It means every new hire has to be upskilled, and in many cases it's not the easy stuff you need skills in, like building pages, it's the hard stuff, like debugging and performance which only comes with experience.

That's why developers migrate towards the "most popular" frameworks, it's self preservation. In Java it's Spring, in Ruby it's Rails, in .NET, well, it's "Microsoft sourced" because it's the usually the largest support team. It's not always the case. nHibernate, JSON.Net, MVVM Light, log4net, DotNetZip come to mind as OSS tools that have managed to build enough momentum, and stay portable and relevant. 

Introducing “Sydney Trains & Ferries”

So my first app submission, it’s a great feeling.

There’s a little bit more the ST&F than a phone app with a bit of data so I thought it’d be a good idea to break it all down. ST&F is probably more Azure than it is Windows phone. Not expecting to make much on it also presents some interesting dilemmas and really challenged me to be razor sharp about what’s really important when it comes to cloud architecture. In other words – cost.

The WP7 app

The app is relatively straight forward. While uniqueness can be good, sticking to conventions allows users to focus on achieving their goals rather than wowing at chrome.

I need to give a big shout out to the developer community, for tools and guidance:

• http://sterling.codeplex.com/
• http://mvvmlight.codeplex.com/
• http://qdfeed.codeplex.com/
• http://c4fdevkit.codeplex.com/

The source data

ST&F gets its data from http://www.131500.com.au. Once unzipped, it’s a 1.5-2GB XML dataset comprising about 70 different transport services in and around Sydney. The schema itself is quite challenging to deal with. My first attempt at transforming a complete dataset took around 3 hours to run on my desktop PC, which is far from ideal. So as they say … to the cloud!

The data upload routine has now been farmed out to windows azure worker roles, the transformation now takes around 15 minutes to run using 4 worker roles, spinning up the instances often takes more time than the run itself (hopefully an issue that will be addressed by the azure compute team one day). So for about a dollar I save myself a whole bunch on frustration. It still has it’s issues – some documents are co-dependant, so there’s some workflow involved that’s invoked manually. In the near future this will be mitigated, but that’s a subject for another post.

Oh, did I mention I despise XML namespaces? They are everything that’s wrong with the world …. well almost.

The eternal problem of (current) cloud services

I’ll give you 2 guesses …. oh you got it on the first – cost.

While excelling in being able to handle the cost of burst loads like data transformations effectively, what cloud compute sucks at is low volume. Minimum cost of an SLA compliant web solution is US$180. Eeep. Sure you can use a stand alone extra small instance, but you’re still looking at $40+, and for less that that I can use old fashion web hosting, where’s the fun in that?

Time to rethink what you thought about ‘web services’.

App Data

What I realised while building the app is that while on the surface one thinks of “querying” service information, in fact, most lookups are unique key based requests like “Get Service Times For Stop Location”. As my previous posts spoil – I started from a typical “Table/Entity” mindset and tried this in Azure table store with frustrating results. It’s a bit challenging querying Table Store from WP7, which meant I’d most likely have to run a web role, which was a cost I was keen to avoid.

Taking a step back I realized that being able to identify each request is actually unique – things like “Chatswood Station” or “Bankstown Line” means each request can be represented in REST, so, blobs to the rescue! The data processing done in azure now transforms hierarchical semi relational data into thousands of static blobs. Sure there’s a ton of duplicated data, but blob store is cheap as chips per GB and I don’t need to run a web role. I have to watch out for the number of requests, but that’s mitigated by caching in the Sterling database.

In Summary

I had a lot of fun, but this is just the beginning. First the phone app needs to pass cert, then there are still a number of perspectives and views I need to present. The azure processing pipeline is also undergoing some re-engineering, so stay posted.

Here we go …

Sydney Trains & Ferries (beta) has just been submitted to the Windows Phone 7 AppHub. Here’s hoping everything goes smoothly.

As they say “Necessity is the mother of invention”, and in this case I needed an app for my own daily commute, hence Sydney Trains & Ferries was born. Admittedly the app is far from ‘complete’, the intention is to evolve through user input, so hop onto the feedback link and get screaming at me

The first think I have to ask is:

Do you want better Rail and Ferry info first or should I focus on getting bus info into the app?

Other than that, I’ll put together a few technical posts on what’s going on under the hood. Sydney Trains and Ferries is running on Windows Azure and presented some interesting problems, some of which I’m still busy solving.

Stay tuned.

The future of cloud computing

 

I like to think I get cloud computing. I like to think I get it pretty well from a philosophical perspective as well as a commercial one. With the mass commercialization of cloud computing underway, the message seems to have been distilled down to “rent your pre-existing configurations on the web, oh and we’ll come up with a scalable store”. Rackspace, Amazon and Microsoft all do exactly that  and they’re selling the vision short.

These ‘first generation’ Cloud services remind me a lot of the anecdote of the first cars. They looked like motorized carriages because that was the prevailing point of reference, no one took the time to re-evaluate the actual problem the carriage solved, removed horse specific requirements and built the car around the engine, not the other way around. I think that’s where cloud computing is currently “Carriage cars”.

There are those that get it. I think Heroku gets it, and subsequently AppHarbor. Funnily enough I think GoDaddy gets it.

Current large scale cloud hosts maintain a billing granularity of one “machine”. They infer that an application requires a machine to run, and subsequently 2 machines for failover. Under certain circumstances that may be true, but for most applications, especially new startups, activity could largely be measured in single digit percentages cpu time, basically putting 2 whole VMs to waste.

In other words, VMs are not a unit of work.

Even these services offering of PaaS fall short of what I consider ideal. They’re basically automated or programmatic ways of provisioning services that would otherwise require IT staff. In other words, it doesn’t abstract away the problem it just turns an IT problem into a programming one.

In the other corner we have Heroku and AppHarbor, and certainly a few I don’t know. Abstract away the hardware completely, there’s no waiting 15 minutes to “spin up a new server”. It works because apps don’t really need VMs. VMs are an easy out for the providers, they’re not good solutions for developers.

Funnily enough that brings me to GoDaddy of all companies. I recently saw they’re offering “4GH” hosting. Basically it’s auto scaling clustered web hosting services. They offer it at entry level prices closer to traditional web sites and my immediate reaction was “exactly”. With IIS isolation and throttling I don’t need a whole VM to run an app. I don’t need two servers for redundancy, the point of clusters is to distribute load. Take a few hundred applications and span them across a dozen servers, that’ll give 90% of users what they want at a fraction of what “cloud” hosting currently costs.

Goodbye tables, hello blobs

Okay, so after working with azure table storage for two weeks I can finally say I get it. Or more appropriately, I get what it’s not. It’s not, “table” storage …

If you think of a table of data as a series of rows of related entities which you can group and query you can be easily misguided into thinking you get azure table store. In reality Azure table store is more like “Azure Indexes” because in order to make them useful you have to contort you data for each view you wish.

For example, if you want to "group by” you’re better off creating a new table by pivoting your data on the group term. If it’s  particular record you want, you could search on partitionkey and rowkey, BUT, you could just as easily make that a single row partition, or … a blob.

That’s the situation in my use case. I thought because I am reading semi relational data a table would be most appropriate. But I’ve come to the conclusion that actually, I’m better of sticking JSON objects in blob store.

Oh, Table Storage ...

I'm having a ton of fun with Azure, but the one "version 1.x" feature that bugs me is the immaturity of table storage.

The fundamental ideas behind table storage are great - hi speed, low cost, non relational data. The issues I have are with the details of the implementation.

First off there's the supported structures. Why to we have to use flat objects? Why limit things to, basically,  a property dictionary, when  the semantics support graph serialization? Maybe constraints are good, but I find this limits the transparency of working with Table storage.

My next issue, while it's all well and good using 'similar' semantics to other technologies like Data Services, LINQ and Entity Framework, it kinda defeats the DRY principle in that it's not the same. Why was table storage not provided as a provider for Entity framework, as well as an abstraction over Data Services?

Lastly that brings me to Dev storage. Look, having local easy to configure tools is great, but there are known issues that just hurt the experience Oliver Jones covers some details here: http://deeperdesign.wordpress.com/2010/03/10/azure-table-storage-what-a-pain-in-the-ass/

My thought is, why mess around "faking" table storage over express anyway, when the whole point is to avoid SQL Server?

I figure the semantic and dev fabric issues could actually be solved quite easily, together - release a 'production' ready local version of table storage. This would help unified architecture and the dev experience.

Windows Azure Table Storage stupid mistakes

Don't program when you're tired ...

I've been working through the Azure labs and as usual I don't stick exactly to the script (how're you supposed to learn anything if you don't challenge the boundaries? ;) ) So after 2 late hours I couldn't figure out why:
"storageAccount.CreateCloudTableClient().CreateTableIfNotExist("ThingOnes")
"
Created tables, but the more common
"CloudTableClient.CreateTablesFromModel(typeof(ThingContext),

                                                    storageAccount.TableEndpoint.AbsoluteUri,

                                                    storageAccount.Credentials);"

didn't.

I just noticed now that in my "ThingContext" here:
"
        public IQueryable<ThingOne> ThingOneEntities

        {
            get
            {
                return this.CreateQuery<ThingOne>("ThingOnes");
            }

        }
"
the property name of the entity set was different to the EntitySetName I declared.

Setting them the same made everything happy again.

In other words ... magic strings are evil!