Introducing “Sydney Trains & Ferries”

So my first app submission, it’s a great feeling.

There’s a little bit more the ST&F than a phone app with a bit of data so I thought it’d be a good idea to break it all down. ST&F is probably more Azure than it is Windows phone. Not expecting to make much on it also presents some interesting dilemmas and really challenged me to be razor sharp about what’s really important when it comes to cloud architecture. In other words – cost.

The WP7 app

The app is relatively straight forward. While uniqueness can be good, sticking to conventions allows users to focus on achieving their goals rather than wowing at chrome.

I need to give a big shout out to the developer community, for tools and guidance:

• http://sterling.codeplex.com/
• http://mvvmlight.codeplex.com/
• http://qdfeed.codeplex.com/
• http://c4fdevkit.codeplex.com/

The source data

ST&F gets its data from http://www.131500.com.au. Once unzipped, it’s a 1.5-2GB XML dataset comprising about 70 different transport services in and around Sydney. The schema itself is quite challenging to deal with. My first attempt at transforming a complete dataset took around 3 hours to run on my desktop PC, which is far from ideal. So as they say … to the cloud!

The data upload routine has now been farmed out to windows azure worker roles, the transformation now takes around 15 minutes to run using 4 worker roles, spinning up the instances often takes more time than the run itself (hopefully an issue that will be addressed by the azure compute team one day). So for about a dollar I save myself a whole bunch on frustration. It still has it’s issues – some documents are co-dependant, so there’s some workflow involved that’s invoked manually. In the near future this will be mitigated, but that’s a subject for another post.

Oh, did I mention I despise XML namespaces? They are everything that’s wrong with the world …. well almost.

The eternal problem of (current) cloud services

I’ll give you 2 guesses …. oh you got it on the first – cost.

While excelling in being able to handle the cost of burst loads like data transformations effectively, what cloud compute sucks at is low volume. Minimum cost of an SLA compliant web solution is US$180. Eeep. Sure you can use a stand alone extra small instance, but you’re still looking at $40+, and for less that that I can use old fashion web hosting, where’s the fun in that?

Time to rethink what you thought about ‘web services’.

App Data

What I realised while building the app is that while on the surface one thinks of “querying” service information, in fact, most lookups are unique key based requests like “Get Service Times For Stop Location”. As my previous posts spoil – I started from a typical “Table/Entity” mindset and tried this in Azure table store with frustrating results. It’s a bit challenging querying Table Store from WP7, which meant I’d most likely have to run a web role, which was a cost I was keen to avoid.

Taking a step back I realized that being able to identify each request is actually unique – things like “Chatswood Station” or “Bankstown Line” means each request can be represented in REST, so, blobs to the rescue! The data processing done in azure now transforms hierarchical semi relational data into thousands of static blobs. Sure there’s a ton of duplicated data, but blob store is cheap as chips per GB and I don’t need to run a web role. I have to watch out for the number of requests, but that’s mitigated by caching in the Sterling database.

In Summary

I had a lot of fun, but this is just the beginning. First the phone app needs to pass cert, then there are still a number of perspectives and views I need to present. The azure processing pipeline is also undergoing some re-engineering, so stay posted.

Here we go …

Sydney Trains & Ferries (beta) has just been submitted to the Windows Phone 7 AppHub. Here’s hoping everything goes smoothly.

As they say “Necessity is the mother of invention”, and in this case I needed an app for my own daily commute, hence Sydney Trains & Ferries was born. Admittedly the app is far from ‘complete’, the intention is to evolve through user input, so hop onto the feedback link and get screaming at me

The first think I have to ask is:

Do you want better Rail and Ferry info first or should I focus on getting bus info into the app?

Other than that, I’ll put together a few technical posts on what’s going on under the hood. Sydney Trains and Ferries is running on Windows Azure and presented some interesting problems, some of which I’m still busy solving.

Stay tuned.

The future of cloud computing

 

I like to think I get cloud computing. I like to think I get it pretty well from a philosophical perspective as well as a commercial one. With the mass commercialization of cloud computing underway, the message seems to have been distilled down to “rent your pre-existing configurations on the web, oh and we’ll come up with a scalable store”. Rackspace, Amazon and Microsoft all do exactly that  and they’re selling the vision short.

These ‘first generation’ Cloud services remind me a lot of the anecdote of the first cars. They looked like motorized carriages because that was the prevailing point of reference, no one took the time to re-evaluate the actual problem the carriage solved, removed horse specific requirements and built the car around the engine, not the other way around. I think that’s where cloud computing is currently “Carriage cars”.

There are those that get it. I think Heroku gets it, and subsequently AppHarbor. Funnily enough I think GoDaddy gets it.

Current large scale cloud hosts maintain a billing granularity of one “machine”. They infer that an application requires a machine to run, and subsequently 2 machines for failover. Under certain circumstances that may be true, but for most applications, especially new startups, activity could largely be measured in single digit percentages cpu time, basically putting 2 whole VMs to waste.

In other words, VMs are not a unit of work.

Even these services offering of PaaS fall short of what I consider ideal. They’re basically automated or programmatic ways of provisioning services that would otherwise require IT staff. In other words, it doesn’t abstract away the problem it just turns an IT problem into a programming one.

In the other corner we have Heroku and AppHarbor, and certainly a few I don’t know. Abstract away the hardware completely, there’s no waiting 15 minutes to “spin up a new server”. It works because apps don’t really need VMs. VMs are an easy out for the providers, they’re not good solutions for developers.

Funnily enough that brings me to GoDaddy of all companies. I recently saw they’re offering “4GH” hosting. Basically it’s auto scaling clustered web hosting services. They offer it at entry level prices closer to traditional web sites and my immediate reaction was “exactly”. With IIS isolation and throttling I don’t need a whole VM to run an app. I don’t need two servers for redundancy, the point of clusters is to distribute load. Take a few hundred applications and span them across a dozen servers, that’ll give 90% of users what they want at a fraction of what “cloud” hosting currently costs.

Goodbye tables, hello blobs

Okay, so after working with azure table storage for two weeks I can finally say I get it. Or more appropriately, I get what it’s not. It’s not, “table” storage …

If you think of a table of data as a series of rows of related entities which you can group and query you can be easily misguided into thinking you get azure table store. In reality Azure table store is more like “Azure Indexes” because in order to make them useful you have to contort you data for each view you wish.

For example, if you want to "group by” you’re better off creating a new table by pivoting your data on the group term. If it’s  particular record you want, you could search on partitionkey and rowkey, BUT, you could just as easily make that a single row partition, or … a blob.

That’s the situation in my use case. I thought because I am reading semi relational data a table would be most appropriate. But I’ve come to the conclusion that actually, I’m better of sticking JSON objects in blob store.

Oh, Table Storage ...

I'm having a ton of fun with Azure, but the one "version 1.x" feature that bugs me is the immaturity of table storage.

The fundamental ideas behind table storage are great - hi speed, low cost, non relational data. The issues I have are with the details of the implementation.

First off there's the supported structures. Why to we have to use flat objects? Why limit things to, basically,  a property dictionary, when  the semantics support graph serialization? Maybe constraints are good, but I find this limits the transparency of working with Table storage.

My next issue, while it's all well and good using 'similar' semantics to other technologies like Data Services, LINQ and Entity Framework, it kinda defeats the DRY principle in that it's not the same. Why was table storage not provided as a provider for Entity framework, as well as an abstraction over Data Services?

Lastly that brings me to Dev storage. Look, having local easy to configure tools is great, but there are known issues that just hurt the experience Oliver Jones covers some details here: http://deeperdesign.wordpress.com/2010/03/10/azure-table-storage-what-a-pain-in-the-ass/

My thought is, why mess around "faking" table storage over express anyway, when the whole point is to avoid SQL Server?

I figure the semantic and dev fabric issues could actually be solved quite easily, together - release a 'production' ready local version of table storage. This would help unified architecture and the dev experience.

Windows Azure Table Storage stupid mistakes

Don't program when you're tired ...

I've been working through the Azure labs and as usual I don't stick exactly to the script (how're you supposed to learn anything if you don't challenge the boundaries? ;) ) So after 2 late hours I couldn't figure out why:
"storageAccount.CreateCloudTableClient().CreateTableIfNotExist("ThingOnes")
"
Created tables, but the more common
"CloudTableClient.CreateTablesFromModel(typeof(ThingContext),

                                                    storageAccount.TableEndpoint.AbsoluteUri,

                                                    storageAccount.Credentials);"

didn't.

I just noticed now that in my "ThingContext" here:
"
        public IQueryable<ThingOne> ThingOneEntities

        {
            get
            {
                return this.CreateQuery<ThingOne>("ThingOnes");
            }

        }
"
the property name of the entity set was different to the EntitySetName I declared.

Setting them the same made everything happy again.

In other words ... magic strings are evil!

Windows phone 7 impressions : 2 weeks on

So it's 2 weeks on since I finally got my grubby mits on my Omnia 7, and overall I'm pretty happy with it. It's solid without being cumbersome and the UI is really slick.
The AMOLED screen is a marvel of crispness.

Having said that the phone, as any v1 product, has issues. The most jarring being without doubt the combination of tombstoning and capacitive face buttons. Being unceremoniously dumped out of "the harvest " and having to wait a minute again to resuming is something that would prevent me recommending the phone.

Overall though, I'm pretty excited about what should be in the pipeline for wp7 when I see top class apps like ninemsn and bye bye brains.

That and the fact I managed to type this entire post on my phone without issue is a testimony to both the keyboard and the browser.

Oh and the Omnia 7 camera is pretty good, I just don't get the reasoning of having  the default video format unplayable on the Xbox 360.

From a dev perspective the tools are really excellent, with a bit commitment I should have something presentable soon.

Silverlight ... HTML5 ... *sigh*

*sigh*

Steve Ballmer and Bob Muglias comments on HTML5 & Silverlight are nothing new. Well, except it being by the CEO, the relevant VP and at PDC. I guess that can look worrying if you want to it to.

I've actually found the fallout quite positive. In truth their comments were completely expected, if Microsoft was trying to "take over teh world" by installing silverlight on every device I'd say they were foolish and naive. It failed for Java, it failed for Flash, why would Silverlight be any different? Silverlight fulfills a different role to html, that explaination was the only bit missing from the keynote.

That's what's been great about the aftermath: The role and future of Silverlight has never been better expressed in so many ways and in such detail. If these perspectives had been made clear during or before PDC it would have been a none issue. Here's some of my favourites:

Bob Muglia's response
http://team.silverlight.net/announcement/pdc-and-silverlight/

Great detailed post from Mike Taulty
http://mtaulty.com/CommunityServer/blogs/mike_taultys_blog/archive/2010/11/01/silverlight-versus-html5-really.aspx

MVPs Jeremy Likeness & Laurent Bugnion also weighed in nicely 'from the coal face' on the subject:
http://csharperimage.jeremylikness.com/2010/10/so-whats-fuss-about-silverlight.html
http://csharperimage.jeremylikness.com/2010/11/silverlight-and-html5-microsofts.html
http://blog.galasoft.ch/archive/2010/10/30/my-position-on-the-silverlight-debate.aspx
Rather go look at the portable library project (http://player.microsoftpdc.com/Session/638f610a-ea51-4aef-9657-e3fe425ae745/469.163) to go see exactly what makes Silverlight so awesome.

Think of Silverlight as more “Thin Client” than “Rich Web”

There's a lot of justify your platform going on at the moment. I see Mike Taulty has weighed in on the subject.

The thing is that the 'HTML5 lobby group' aka Apple and Google, have a lot to gain by denigrating Silverlight and Flash. iOS doesn't support it and Google's core strength has been by commoditizing the web with its thousands of JavaScript programmers. HTML5 is all well and good, but in the rush to save the universe people are getting a bit blind. Add these two behemoths to the voices of the thousands of "script kiddies" that consider a couple of lines of copy and pasted PHP and JavaScript programming, and you'd think we're reached software Nirvana.

The truth is HTML is still HTML and Flash and Silverlight are not just about streaming video.

Even using a better browser like Chrome or Firefox, how many times have you been frustrated waiting for a page to load? Have you gotten irritated, or worse – had to apologize to a customer, while trying to find what you were doing in a mass of tabs or accidently closed the wrong tab and lost their state? What about sitting waiting for something to happen while JavaScript has a meltdown and there's no way of knowing? That's just from a user perspective. From a developers view, JavaScript is a complete PitA, and trying to do LoB or Verticals levels of complexity is a killer, never mind the testing time.

Then there's the whole question of running disconnected. Page storage is all well and good but what happens when *shudder* you have to move to another page? What about device integration?

The truth is Silverlight (and Flash) presents real value to application developers but is a bit difficult to quantify if you haven't felt the pain of trying to develop large HTML apps.

But that's just me.

Dear Microsoft, you were SPOT on. Not.

I guess it's the same old tail. Microsoft, a good idea, squandered. Right idea, wrong era.

...Picked up and owned by Apple.
The iWatch Is Real: It&#8217;s the New iPod Nano [PIC]

SPOT was Microsoft's attempt at smart watches, announced back in 2003, and after 5-6 years of spinning it's wheels it was canned. Barely 2 years later and Apple are back with a real how-to.

Music, Phones, touch PC's, Watches etc etc. Apple gets it right first time and executes in a focused manner. There's a lesson to be learned there.....