Practical people. Practical problems. Practical advice.

Posts in category Architecture

Test-Driven Development

May17
2012 Leave a Comment Written by ecomware

If it’s wrong, I don’t want to be right

We all have vices. Software engineers typically have more of them than the average guy. (I don’t really know that for sure, but it seems true intuitively, and it’s fun to say.) Some vices, as we all know, are more or less socially acceptable, especially if we make adequate attempts to conceal them. My ex-girlfriend’s father would sneak to the garage to smoke cigarettes. I didn’t realize he did that until someone told me during a family get-together many years after I met him. And I don’t care that you’re a 37-year-old Lego aficionado, as long as you keep that secret locked away in a closet like the rest of us self-respecting gentlemen. And, let’s face it, YouPorn.com (98th most popular site in the US) isn’t more popular than Catholic.com (6,198th most popular site in the US) for any reason other than the obvious one: you people are messed up.

I’m investigating Test-Driven Development 1, and the phrase deserves capital letters because I’m currently reading a book about test-driven JavaScript development, which only happens in places where Test-Driven Development–the big, corporate kind–is celebrated. To understand how I feel about this, let me share a previous experience. I was once asked how I might solve a problem for a large development team who had achieved 98% code coverage. They had had reasonable success, maintained average (not enviable) quality, but were beginning to miss deployment deadlines as marketing had gained control and began to push for features. The summary of their problem, neatly delivered, beckoned for the reply, “Why, you should push back on Marketing, and [insert whatever software engineering people say here].” Of course, I said, “I recommend you stop wasting time on tests, welcome a trivial decrease in quality, and get back to spending 37 of your 39-hour work weeks writing actual code”. Imagine that, I proposed writing actual code to deliver actual features.

Today, I ran this: js-test-driver – Remote javascript console – Google Project Hosting . And, you know what? It felt good. It felt really, really good. I immediately heard the siren song luring me to the island of non-stop, self-executing, regression-, integration-, just-make-it-up-as-we-go-because-it-feels-so-good-and-reassuring-testing. That went on for about 25 minutes until I looked at my watch and realized I had accomplished nothing. It’s really easy to get a temporary feeling of pleasure and productivity from that activity though, and I had to stop and remind myself how idiotic it sounds to suggest that it’s too hard to just write code, so we should write some code to help us write code.

While I would never make the case that no code should ever be tested 2, I do think the software engineering culture is at the far (and unhealthy) end of the continuum. Like smoking, or legos, or your while-the-wife-is-out-shopping Internet activity, it would be better for all concerned if you enjoyed this sort of thing more privately.

Footnotes

1 By investigating, I don’t mean to say I’m learning it for the first time. Having written code as an application-tier developer at a big solution company and as a UI developer at a big digital agency, I maintain two paradigms, one of which has required me to do various-phase-unit-testing-type activities within certain contexts. I mean to say I’m investigating it the way you might research any political party that opposes yours before, say, a big election.

2 I blame Kent Beck. He could’ve done more to prevent this nonsense from happening by issuing some sort of warning, like the text that appears before a JackAss video.

Framework Myths

May09
2012 Leave a Comment Written by ecomware

The software industry is overrun with truisms and platitudes. Do not repeat yourself, we’re told over and over. (Unless you’re repeating an engineering cliche, of course.) Keep it simple, stupid. Composition over inheritance. Abstract the UI through MVC. And so on.

Most developers know, intuitively, that no framework, architecture, or process is correct for all situations. There are no silver bullets, after all. (Again, a truism.) And most developers know that you cannot design too far into the future, that you shouldn’t refactor too early, and that the 80/20 rule always applies.

But I want to know how it’s possible that so many dramatically different styles of software development can exist within teams full of people all saying the same things? Is it possible, to borrow another cliche, that some of you are doing it wrong?

Several times I have watched small groups of people in meetings repeat exactly the same set of details to each other and derive from those details diametrically opposed conclusions. They see the same set of facts, agree that a similar problem exists, and then come to differing conclusions about how to effect a solution.

I believe some portion of the time, the parties disagree fundamentally about the problem due to differences in deeply held values and beliefs, even though on the surface they agree to a practical definition of the problem.

But I believe in some portion of the time the main difference lies in the beliefs about, and expectations of, the consequences of certain behaviors either contributing to the problem or proposed as solutions.

For example, it should be easy to see how disagreement might arise between two people who hold different values, say between one who’s highest value is wealth and another who’s highest value is happiness. Those two people can coexist, but it would almost always require an understanding of the other’s driving value, and some effort to compromise.

But what about the differences that arise between two people who value wealth, perhaps regarding which behaviors create wealth?

Reasonable people might make the case that perhaps both proposed behaviors might create wealth. Granted. But I propose that what happens much more frequently in that situation is that one of the people is just wrong.

And so it is in software. Some of your ideas about components, generalization, simplicity, silver bullets, and MVC are just wrong. They don’t contribute to pliable software. They don’t contribute to software quality. And they don’t contribute to your company’s profitability.

Posted in Babble

Presentation Layer Architecture

Apr18
2012 Leave a Comment Written by Digital Marketing

My specialization is the class of websites optimized for large retail (B2C) clients A. There is usually an e-commerce component and a market research component, but there is always a branding component. My specialization is not building web applications; I do not build web pages like Gmail or Maps 1. I have built sites using several programming languages (with and without template-driven systems) including perl, php, and java. And I have specialized in both application-tier programming (using those preceding languages) and full-time front-end development (using strictly CSS, HTML and JavaScript) at different times. Finally, I have worked with many teams with workflows optimized toward different influential, functional groups. In some cases the business (product or project management) professionals were most influential 2. In other cases either software engineers or front-end developers had influence. In one or two cultures, QA leaders were the most influential group.

There were varying levels of Agile, BDUF and Waterfall employed at each. Some teams did weekly sprint planning and daily scrums, others were full-on Waterfall with minimal testing. And while it is often very difficult to isolate a quality (read: attribute (n.) or property) of an architecture and attribute (v.) its manifestation to one specific cause, it is fairly easy to intuit those qualities (read: attributes (n.)) that have repeatedly manifested in each of my teams. First, the production-environment quality (read: bugs, downtime, etc) has been mostly consistent: tolerable, but not warrantable. And second, the most influential, functional group is always the retail client. Their needs–sales, research, and branding, as stated above– produce a constant need to innovate and stand apart, and almost always instigate frequent change and configuration of their web assets. No digital marketing professional wants their site to be either stagnant or anything like their competitors. 3 (Even though they do all ask for a site like Apple.) This necessity to change often and expediently is a primary constraint on our development systems–the systems I’ve specialized in–which all of my teams have felt. Some of my teams have explicitly identified that constraint and optimized for it, and others haven’t. It is this attribute of my teams that I feel has led to the most significant differences between them.

In teams driven by front-end developers, it was not uncommon to see a new JS library introduced in every release 4. Or to see simple forms using microscopic MVC patterns and unnecessary asynchronous requests. Teams with talented (if stereotypical) software engineers are attracted to portlets, velocity, wicket, filters, modularization, generalization, and objects and components. QA-driven teams believe they need more stories, need to be involved earlier in the process, need a higher rate of coverage during regression testing, or need to centralize communication. Business teams just want more features. Put this over here, that over there, and add a video overlay. And they need those things yesterday.

These push-pull relationships determine how we spend our time each day. If we recognize that we need to iterate (just generally, not in the Agile sense) to properly serve the customer, and that quality levels will remain roughly static no matter which system of development we employ, we can develop an architecture that truly suits our purpose and meets the primary needs of each of the functional units–especially the customer.

Bibliography
[BEAUTIFUL]
Beautiful Architecture, Edited by Diomidis Spinellis and Georgios Gousios. Copyright 2009 O’Reilly Media. 978-0-596-51798-4

[SCALABLE]
Scalable Internet Architectures by Theo Schlossnagle. Copyright 2006 by Sams, 978-0-672-32699-8

[SOFTWARE]
Software Systems Architecture: Working With Stakeholders Using Viewpoints and Perspectives, 2nd Edition by Nick Rozanski, Eóin Woods. Copyright 2012 Addison-Wesley Professional, 978-0-321-71833-4

[QUALITY]
The Economics of Software Quality by Capers Jones, Olivier Bonsignour. Copyright 2012 Addison-Wesley Professional, 978-0-13-258220-9

Footnotes
A I’ve participated in the development of websites for most of the American auto manufacturers, some of their suppliers, and many other tangentially-related businesses.

1 http://maps.google.com was built by software engineers. Its architecture was influenced by its relatively static purpose: it shows maps. I have been told that the browser-executed assets, including HTML, CSS and JS were output by a program written strictly for the purpose using the C programming language. I differentiate web applications from other web sites using a simple criterium: Do end-users perceive the full user experience and purpose of their interaction (the general context) to be very similar to desktop software? Gmail and maps are applications.

2 Honestly, all teams are business driven, but the amount of push back from software engineering resources to business initiatives and product development differs from organization to organization. Business-driven teams are those that experience minimal push back from the software engineering resources.

3 Quality tends to co-exist with highly static software. I suspect the extreme rate of change is what limits quality levels in organizations who service this kind of client. So the question is: as developers, shall we directly optimize our process toward a quality target or a pliability target?

4 If the library wasn’t completely new, as in upgrading from mootools.js to jQuery (or EXT JS to YUI), the version was upgraded. Front-end developers cannot stand any implementation pattern for more than a week.

Posted in Business & Marketing - Tagged , , , , , , , , , , , ,

ARCHITECTURAL STYLES: JSON vs XML

Feb07
2012 Leave a Comment Written by ecomware

Overview

Let’s start simple. JSON is an acronym. It stands for JavaScript Object Notation. JSON was better known once as object literal syntax. That’s what it is, but you won’t find many recruiters using that phrase in their job descriptions. (Memorize none of this, by the way.) If you know ECMAScript well, you know JSON.

XML is extensible markup language. It’s often described as the semantically rich Tupperware ® for data because its syntax is nested like Matryoshka dolls and allows for informational attributes that “seal in” meaning.

Both have schemas.

JSON

Object literal syntax is easy to parse and work with in host contexts like browsers. Browser add-ons, like Firebug and other developer toolbars, make it possible to edit JavaScript and execute it. And since (valid) object literal syntax is just JavaScript, it’s convenient for front-end developers.

AJAX, which used to be an acronym for asynchronous JavaScript and XML and has since been abridged to mean asynchronous JavaScript requests for any type of data, often required JavaScript developers to write code to convert from one format (XML) to another (JavaScript data types) as part of the web page. By converting the structure of the asynchronous response entity bodies to object literal syntax instead of XML, they saved themselves* a lot of work. Many developers will claim it also saves processing power on the client, which is sometimes true, but this property was identified retroactively. (In other words, this was a way to cut corners before it was sold as a way to improve run-time performance.)

It’s also important to note that JSON’s popularity is partly due to the fact that recruiters and development managers write their job descriptions with input from the geeks in their IT departments, which induces any applicants to bone up on the trends the teams are following. It’s reinforcing.

A few years ago, a reason to use XML over JSON would have been the ability to reuse XML between components of enterprise-level systems without much conversion by the connecting elements. The prevalence of architectures using literal syntax for conveyance gave way to the development of libraries that could structure and evaluate JSON within other contexts such as Java and PHP, so this isn’t strictly true anymore.

Development teams have followed this trend all the way to the end of the spectrum: JSON is now often used as a canonical form of data storage. (This reminds me of the days when people actually put images and PDF files in their MYSQL databases. You can, but it doesn’t mean you should.)

XML

It’s very difficult to create reusable patterns of logic if the data doesn’t have delimiters. And, in fact, it’s often necessary to communicate many things about data in addition to where it starts and stops. Sizes, names, contexts, spacial relationships, languages, etc., XML’s flexibility and support for metadata cannot be beat. It really is ideal as a container, if not a medium. By contrast, JavaScript object notation is perfect for delineating data types. It’s a subset of a language syntax designed to build code structures. It is not ideal for capturing metadata. When JSON is used as a markup language, it’s really performing two jobs. (It’s OK to use a screwdriver to pound tacks when you have to, but most know that a hammer outdoes a screwdriver in the jobs it was really intended for.) XML is optimized for housing data, and for those with data-centric perspectives, JSON will quickly show its limits.

XML sticks to the principle of generality, which provides the benefit of configurability. You can keep your data in an XML container with all of the labels you need and still serve representations of it as they’re needed. For example, XML can be morphed into different XML in an effort to change the semantics of the data, or into completely different data structures (like JSON). And transformations are relatively simple to perform using XSL.

XML does have two advantages over JSON in the browser context. XML has a dedicated mime type. JSON has two. (JSON is both JavaScript and text/JSON, which is weird. This is a tell that we’re using it for two jobs.) But the XML mime type doesn’t cause those weird errors when requested asynchronously. This is the first advantage. The second, and much bigger advantage, however, is that browsers support the pairing of XML with CSS and XSL. XML may be turned into a conventional HTML-structured DOM on the client. It can also be styled using CSS directly. These qualities cannot be overstated. The ability to cache DOM templates as discrete XSL resources on the client side is extremely valuable in some cases where latency is an issue.

My Opinion

From a system perspective, I’m almost always in favor of storing data as XML. Where data is necessary for reuse, especially outside of our own systems, I support XML. In situations where it doesn’t really matter, I quit and go work somewhere else.

* They saved themselves a lot of work. But the middle tier (application layer) developers that are responsible for structuring those response bodies usually do the conversion via some other mechanism such as XSL, Java, or a to-JSON library.

Posted in Babble - Tagged , , , , , , , , , ,

SEO and Site Speed

Nov23
2011 Leave a Comment Written by ecomware

At the time of this writing, Notes is listed as one of the top 2 million sites globally according to www.alexa.com. You can view the site information here: http://www.alexa.com/siteinfo/ecomware.com.

Two months ago, Notes was in the top 7M. Three months before that, it was in the top 4M. That’s a lot of change in six months.

One of the factors that causes ecomware.com (a micro blog with barely 50,000 readers annually) to rank so high is that it’s linked to from Liferay.com, and regarded as a good source for simple Liferay theme information. Link neighborhoods are important, and having embedded a link to myself in Liferay.com certainly helps ecomware’s visibility.

One of the factors that causes ecomware.com to rank so poorly, especially when it dropped into the 7M range, is caching. During the summer, I was experimenting with document freshness, and set the cache expiration date on the home page (the blog index page) to 14 days into the future. This, I assume, caused the major sites to believe the content was stale.

To speed up ecomware.com, I followed many of the well known steps for reducing latency. I consolidated and condensed the .js and .css that were part of the theme, and in cases where the pre-published source files are small, I simply embed them in the markup as part of the publishing action. This means I have just one include of each type. I also eliminate the head requests for these resources by using an expires header. I layer the real-time assets using AJAX. Subsequent page loads will cache at approximately 85-95%, and without head requests. I also use a publishing mechanism in front of Word Press to serve static pages so that I’m not incurring a real-time processing cost on each request. (Big frameworks have this already, but it’s simple enough to implement on a small scale.) I also leverage XML and XSLT on the client side. In cases where the presentation is almost identical across many assets, I cache a single XSLT file and serve literal XML data to the client. (The transformation creates a normal HTML DOM on the client side.)

The end result is that I can serve this site from a shared host/DB with reasonable performance (especially on subsequent loads) and still reap the benefits of good search ranking.

Tagged , , , , , , , , , , , , , , , ,

CSS URLs: The effect of relative and absolute pathing

Oct01
2011 Written by ecomware

Most people who work with CSS understand the difference between relative and absolute paths, but there is one situation that doesn’t immediately come to mind that can be very useful.

Let’s say we have a document hosted at:

http://www.a.biz/index.html

Let’s say that document contains a link tag (or an import directive) to an external CSS file:

<link href="stylesheet.css" rel="stylesheet" type="text/css" />

Inside this style sheet, we have the CSS selector:

body {
    background: url("styles/bg.png") repeat-x scroll 0 0 #FFFFFF;
}

First, the browser computes the base URL (http://www.a.biz) of the document, and appends the path to the style sheet to it. The style sheet will be downloaded from:

http://www.a.biz/stylesheet.css

Second, the browser will attempt to download the image. Because the image is relative, a base URL must be calculated. In most cases, the base URL is the same as the document itself, and so it is in this case. The base URL (http://www.a.biz) is prepended to the relative path of the image, and the resource is downloaded. In this case, the image will come from:

http://www.a.biz/styles/bg.png

Many developers assume the base URL is calculated the same way for both resources (style sheet and image), but it isn’t. The base URL for the relative image path is found by using the absolute URL of the style sheet, just as the base URL for the relative style sheet path is found by using the absolute path of the document (index.html).

This means that if we change our link tag in http://www.a.biz/index.html to what follows, the relative path of the image will change.

<link href="http://www.b.biz" rel="stylesheet" type="text/css" />

The URL of the style sheet is no longer relative to the document. The base URL of the style sheet is now http://www.b.biz, which means the computed absolute value of the image resource is:

http://www.b.biz/styles/bg.png

This can be very useful in cases where similarly-branded web content is distributed across hosts. It can be used in development effectively, too. Using this technique, it’s possible to develop a single set of resources (style sheets, images, etc) and reuse them. This enables sharding of the UI resources, and is especially effective in cases where real-time content cannot be cached behind a CDN by removing the resource requests to another architecture.

Posted in Code, CSS Examples - Tagged , , , , , , , , , , , ,

GoDaddy’s php.ini and .htaccess for parsing .xml files

Aug08
2011 Written by ecomware

The title says it all, doesn’t it? I’ve read plenty of web articles on this, and they’re all variants of the same thing. In a nutshell, you need to create a local php.ini file and a .htaccess file. The specific file names and their respective entries will vary slightly depending on what version of PHP your hosting account is using. (There are more than 2; Some accounts use both 4 and 5.)

My hosting account runs PHP5, primarily. My .htaccess file has the entry below. Note that it doesn’t have the -cgi (as in: AddHandler x-httpd-php5-cgi) suffix.
AddHandler x-httpd-php5 .xml

My php.ini, php4.ini, and php5.ini files (yes, I have all 3) include this single line:
short_open_tag = “0″

Now files with the .xml extension will be parsed by the PHP interpreter. You’ll need to remember to manually set the content-type header of your files if they’re not HTML. In my case, I had to set the content type header to text/xml.

I had a lot of files to add the line to, plus I wanted to gzip them, so the following command came in handy: (It should all be on one line.)

find *.pdf.xml -type f -exec sed -i '1i <?php  /*Some PHP Stuff*/ ?>' {} \;
Posted in Code, PHP Examples, Shell, XML and XSLT Examples - Tagged , , , , , , , , , ,

Load jQuery below, use it above

Aug08
2011 Leave a Comment Written by ecomware

If you want to build a quick website, you’re going to aggregate, deflate* and cache your JS files. This means you’ll make one request for JavaScript, and hopefully you’ll do it near the bottom of your document: Your script request will be one of the final elements in your body tag.

Your page will be fast, but this prevents you from writing any script that uses the yet-to-be-defined jQuery variable earlier in the page, which is a common need in template systems like blogs, WP, etc.

Here is a way to keep your pages quick, and still use your jQuery wherever you want: It’s called the pre element. Instead of writing your inline script inside of script tags, write it in pre tags with a special class that you designate as executable. Example:

<pre class=”jQuery_executable” style=”display:none;”>
/* Do some jQuery stuff here */
jQuery(function() {
/* Do some stuff */
});
</pre>

Add a short plug-in to your aggregated JS that will do the following:

jQuery('.jQuery_executable').each(function() {
    eval(jQuery(this).toString());
});

This has two benefits:
1. You’re able to write inline code even though jQuery hasn’t yet been defined.
2. You’re DEFERRING the execution of your inline script until the end of the page. (Unlike using jQuery(function(){}), your JS isn’t even parsed.)

This has at least one drawback:

jQuery('body').append(jQuery(unescape(escape("<div>Elements</div>"))));
Posted in Code, JavaScript Examples - Tagged , , , , , , ,

Cleaning up WordPress and the Buffet Framework

Jul29
2011 Written by ecomware

Notes isn’t much of a blog. It’s just a collection of notes, ideas and solutions that I want to remember and reuse. I don’t spend a lot of time working on its design or architecture.

I did spend a few minutes looking at its performance recently. Firefox reported that WordPress and the Buffet Framework were making nearly a dozen calls for JS and CSS files. The scripts were all loading from the head. The page weight was over 150KB, which is a lot considering it’s almost all text, and it wasn’t being cached. Nothing was being gzipped. Finally, I estimated there was a lot of unnecessary processing going on for each request, including database access – an expensive action in our hosting service.

Fixing almost all of the above took little more than 2 hours. The scripts went to the bottom. The requests were consolidated. The page is served with a cache (expires) header. And the largest files are gzipped. A large amount of unnecessary processing was removed from the primary template files (header.php, footer.php, page.php, etc).

In total, this reduced the initial page weight by 50%, and subsequent page loads are much faster (even when navigating to interior pages that represent a primary content cache miss).

How much wasted time and energy loss is a result of badly configured WordPress-type sites? With millions and millions of them out there, it may be a lot. Novice users benefit from the “ease of configuration and use” that such sites provide, but the big advantage they offer to the publisher is a huge drawback to the consumers of the content (hopefully a large audience).

Posted in Babble

UTF-8

Jul11
2011 Written by ecomware

I have a small web project that processes form data on the server. I struggled with some encoding issues until I defaulted the document to UTF-8 encoding, and set STDIN and STDOUT to utf8 on the CGI script. (The Apache configuration is not specifying an encoding, so the meta tag works.).

Document Meta:
<meta charset="utf-8" />

Perl CGI
binmode(STDIN, ":utf8");
binmode(STDOUT, ":utf8");

Posted in Code - Tagged , , , , , , , , , ,
← Older Entries