Dan Hilton

After a longer than planned hiatus, here is the second part of my building enterprise web applications blog post.

Standing on the shoulders of Apps

One of the aspects of django that I really liked when I first came to it was the ability to plug in apps that gave you discretepieces of functionality. Need your users to have an avatar? there is an app for that. You then have the ability to override the templates,wrap the views and even subclass the models to customise the app's functionality without having to branch the code of the app. This gives yougreat flexibility when trying to build new applications. However, as with any selection of open source code, there are some stars and some dead-ends. I've tried to list the apps that I've recently used on a large project to give you a guide:

  django-auth-ldap
   If you need to authenticate against an ldap server or an Active directory tree, then this app (availble on Pypi) offers a great, easy to
   configure way to allow your existing internal  userbase to login using your chosen SSO solution.

 django-sentry
   When your dealing with a range of integration points and apps, you need help assessing and dealing with errors. Sentry is
   a great app that can aggregate and alert you to issues with your deployed project. You could also have this functionality
   with something like Splunk but there is always value in showing live issues to developers as well as systems teams.

 django-json-rpc
   If you need to create ajax interfaces for rich interfaces, django-json-rpc is one of the nicest solutions for creating hooks
   for frontend developers to connect to. It works by having a decorator which you use to wrap a view function, automatically
   (via a registration function in your urls.py) exposing those functions via a json endpoint and gives you an autogenerated
   documented console to try out the functions with. Great for exposing functionality quickly.
 
 django-haystack
   Fast becoming the defacto django search app, haystack is that easy and good that it deserves it's reputation. If only installing
   and administering Solr on RHEL 5.5 was as easy. In all seriousness, if you need the faceting and power that Solr brings, read up
   on installing and administering it on your choosen deployment os. If you're going large scale you can have a dedicated Solr node /
   cluster as it talks via xml over http.

 django-staticfiles
   With all these great apps, how do you manage the various static files that come with them? Need seven copies of jQuery? Unlikely.
   django-staticfiles (which has now been added to django trunk it's that good), goes through your installed apps and copies all the
   media files to one single location which you can then serve all media from.

 django-mailer
   Email is still the most likely way of alerting a user to an event (even though jabber, sms, twitter are faster). The problem with
   sending email in django is that if you send an email in a view function, you either have to handle the error there and then or fail
   silently, logging the error. With django-mailer, you can queue the mail, using the current EmailBackend and then handle resending failed
   mail asynchronously.

 Non-django python libraries

 suds
   If you have to integrate with a .net webservice (a SOAP service to the rest of the world), suds is brilliant. It abstracts the painful
   SOAP xml, giving you a python object to deal with. It's a great moment when you can integrate a .net webservice into your project in
   less time than drinking a cup of coffee. We used suds to create a django utility library exposing certain functionality behind a simple
   api. Be aware of security here though. If your organisation has deployed Kerberos correctly, you can for view functions, take the krb5
   token and pass it backwards. If not, you're going to have to write a decent delegation framework, which allows your app to perform
   operations on the user's behalf. A great idea that we didn't use was to use oauth (as used by Twitter, Facebook, Google, etc) and have
   users allow your application to perform certain actions on their behalf. There are .net libraries for oauth 2.0 as well so you would
   have to have a bridge but worth looking at.

 pip
   Pip is a great package management system for python, which when combined with virtualenv is a winner for repeatable installations. The
   particularly useful aspect is the ability to freeze the current packages in site-packages creating a requirements file which you can
   pass to pip to create a completely identical setup. It's not as fully featured as buildout but no where as complex as buildbot.

Since January of this year I've been engaged in building what I guess many would call an enterprise web application in Django. I use the term enterprise in the sense that the project I've been tasked with building is large in nature of importance to the organisation and also in parts quite complex. It's been a very useful and educational experience, one which I would gladly repeat in Django. I'm going to break this down into three posts I think, the first being about some of our toolchain and possible improvements on it, second on django apps we used and some measure of how good they are and finally some advice on how to run similar projects and some pitfalls we experienced.

Toolchain

We developed on linux desktop machines in a variety of emacs, vim & nano. I know a bit hardcore, but we did make use of various plugins to help us, including the excellent vimmate plugins. Each developer ran a local instance of the django app, using the default server and sqlite3 for speed. We used virtualenv and pip requirement files to manage any external dependencies and used fabric to automate sections of the app deployment. We also worked with the internal systems team to move all of the Linux depenencies to Puppet as well, which will eventually give us a completely automatic build. We're not there yet as we still need to get postgresql database creation online and stable within our fabric / puppet coupling.

We made use of existing subversion hosting internally, we used tagging for releases, keeping trunk always in a runnable, non-broken state. We have a suite of tests for the django apps we wrote, written both as unit tests and functional tests where appropriate. We haven't got any in the browser tests using say selenium, but I'll cover that in the final post.

We used postgresql which for our domain is great, we don't do anything particularly hard with it (like GIS, real-time updates, etc) so we haven't had to tune it nor look at connection pooling for example. I've broken out our fabric script to be able to be able to setup our database on any host so that as the system grows the sysadmin team can migrate to a dedicated db server.

Aside from commenting code, we've used sphinx to create both enduser documentation and internal developer guides as we wanted to document our api and some other aspects of the system in a way that will be easy to pick up. For end-user documentation we took a demand led approach starting with cheatsheets and quickstart  guides and fleshing out where questions come up.

Issues

Developing on Linux when you can't customise your working environment is a pain. A lesson I should of learnt early and acted on, was to lock down our target deployment environment (currently Fedora 12, and then RHEL 6 when it comes out of beta) so we could have set up a continous integration system such as Hudson. We have a test suite that is regularly run but could do with being automated. The ability and the need to be able to build a new instance of the system to test say a sandbox system integration cropped up repeatedly, slowing us down as we have to request a new VM to be built. What I'd do in future is maybe get a machine built using something like Ubuntu private cloud, that we could have had control over so as to remove a dependency on sysadmin support.

From a project management point of view we used basecamp with our UX designers, Trac with internal developers & RT with IT support desk. I'd say that basecamp worked well, but some way of digitally commenting on wireframes would have been really handy. They were produced in Axure which did a great job of automating documentation creation for the front-end developers.

Lessons learnt

1. Staging environments that match production are a very good thing.
2. Take the pain early, it'll save you over the development cycle.
3. It's amazing what you can achieve just using vim

Next time: django apps - conventions, finding them & measuring quality.

Legacy is a very loaded word in digital; any project or piece of code, branding, visual design can be described as legacy even though it may still be fit for purpose and do what was originally asked of it. What is and isn't legacy is purely a point of view, it can be that everyone holds the same point of view but it is still a qualitative measure of a website or system. There is no discrete definition of what is and isn't legacy. A billing system written in Bash maybe in the minds of most a legacy system or a banking system written in COBOL, but then some may say that a php site built only 12 months ago is legacy because it doesn't meet the requirements of today.

It's most loaded use is by salespeople and people selling a particular new technology. Phrases such as "integrates with legacy systems", "increase your ROI on legacy systems", all ring alarm bells in my mind as there is an instant assumption that legacy equals crap. That is not always the case and in many respects you must understand the full business requirements of system to understand it. There are however related issues which can be indicative of an issue with a system, code rot, technical debt and the associated passive rot and active rot. These all describe the degrading of a program or piece of code that through environmental changes or "a death of a thousand cuts", leads to a reduced functionality and a movement away from it's core use. 

In an ideal world all projects - no matter how small or big - would come with extensive documentation (project aims, use cases, wireframes, planing docs) however we don't live or work in an ideal world. We live in a commercially driven reality, whether you're billing by the day, responsible for a large business change project or just trying to get something done in an evening. It is about achieving balance between your goals within your constraints. And this by it's very nature does lead in to the creation of legacy projects; they are epoch specific, a creature and a composition of their time. Imagine trying to build a website that has to last 50 years - how would you start it? how could you build it? 

Legacy I feel is not a bad word, it's the emotional connotations we place upon systems we label with it that are our problems, our own internal beliefs that new is better (which in many cases it is!) without understanding the mental blocks that puts on us as programmers or designers as we work with the system / project. 

In a nutshell: be kind to the decisions of those who have gone before you; who knows what your code will look like in the future.

A recurring theme in many of the offices / companies I've been involved in has been the quality and atmosphere of the offices that I've worked in.

Some people (and companies) don't pay too much attention to them, opting for the cheapest / simplest choice at time of need, which I believe is a classic false economy. When working with online projects, you're dealing with a wide range of skill sets and problems; some of which are creative fluffy ones and some that are hard crunchy ones. These both require different types of people and different types of environments. Some things are a given though and for the benefit of anyone building a creative online office space I give you a brief list of key performance indicators to get right:

1. Beverages - Coffee must be strong and / or of good quality. Tea must range from PG / Tetleys to 'crazy' tea - Lapsing / Green tea etc. You need a water cooler. Do not get a constant heating urn - your staff need that 5 - 10 minutes break to defocus and gain some perspective on what it is they're struggling with.
2. Ergonomics - Don't scrimp on chairs nor monitors, your staff can't work for you without their eyes and backs. Provide a decent set-up process as part of their induction. Ergonomics and health & safety is not a joke - it's the law and it shows that you do care about them as individuals more than just the economic value they provide to the organisation.
3. Noise - A noisy office can be a great creative powerhouse of fantastic ideas; don't let it become a common room of banality. Let people wear earphones, be tolerant of random conversations and not afraid to poke their nose into a conversation if they have something of value to add.
4. Art - Is a very dangerous area. If you impose it on people, they will resent it if it is seen to be expensive; if you decline it you run the risk of having no soul. Best option? Let people decide what they want on their walls, have an art office team to go and seek ideas and maybe even commission themselves. But no nudity OK?
5. Whiteboards - Big, lots and in places where people can gather and stare at them. Whether it's notices, new data models or just drawing stick people they help people communicate ideas and work through them with others.
6. Dresscode - nothing ripped and nothing with swear words, otherwise your free to go I say. You may find that the types of people you employ will impose upon themselves a herd dress code - I heard of a local agency that has had to start having a cowboy day as so many employees turn up in check shirts.
7. Perks - Be different, but useful & relevant: free snacks are great but not if they are full of fat and sugar. Encourage cyclescheme, gym membership, cultural engagement and family time - flextime is a great one for this. Give random acts of kindness, that way the perk is always noticed. Different professions have weird perk cultures; sales is the obvious outlier with incentives and performance related pay at the heart of the remuneration package.

In short, as others have said, "Talent makes capital dance."  If you want to retain talent, you better make sure they can dance. For more inspiration:

This ain't no disco | blog about agency studios

It's been a while (note: years...) since I last tried to blog / publish my thoughts anywhere. However, there have been a number of things that have itched me to actually blog about or at least write about them over the last couple of months. So, over the next couple of time periods I will periodically post something of interest to me, probably digital, probably related to something technical or political, who knows. 

It's an adventure.