Testing designs on non-functional prototypes

I find myself grappling with the odd project management and user feedback issue as I trundle through the big project that takes up the bulk of my working day at present (and likely will through to next summer). In particular one issue has been eating at me for a while. I find it challenging to do really useful user testing on click-through HTML prototypes when they’re not yet functional enough to just hand off to an end user for something like task completion analysis. Any thoughts readers may have would be helpful.

A little background

The project I’m dealing with is a major redesign of the course and program catalogue on our public web site. The phase I am presently working on is only focused on blending what have previously been completely separate data sources which publish separate program marketing ‘sitelets’ with official institute program pages into one ‘micro-site’ for each program we offer. The part-time course catalogue and search functionality are later phases.

The process to this point

In getting to the point where we’ve got a visual design and web-based prototype, I followed a pretty standard development approach:

  • examined existing usability analysis on our catalogue for key issues
  • conducted primary stakeholder research to uncover additional content management and presentation issues
  • developed a comprehensive design and functional requirements document
  • developed rudimentary IA and tested it with stakeholders via online AJAX card sort exercise
  • produced application storyboard/userflow and wireframes (with interaction notes) of all key presentation and content management interfaces
  • our team produced photoshop composites and HTML prototypes to demonstrate rudimentary end-user features
  • multiple presentations and demos have been conducted for broad, management and school-based stakeholder groups to solicit feedback, including current summer students
  • the feedback will be amalgamated in the next couple weeks into a design ‘lock-down’ so that we can code the application

My problem

While satisfying all stakeholders or blending data might seem like the obvious issue, it’s not. While the coding process will create a fully functional beta application which we can use for user testing, I have desperately wanted to test the prototype, but found it difficult. The best solution I could come up with was to present the demo to the group of students and then have them answer a short paper-based survey to solicit their opinion, particularly since so much of it is not functional that any user testing would have created more questions than answers.

The survey methodology is not nearly as valuable in my mind as observing them interacting with it, because my concern is that certain visual or interaction design issues which might be caught while someone is using it, won’t become apparent until we’re beta testing the application, after it’s too late for design changes.

I’d love to hear thoughts from anyone who’s had to conduct user tests on prototypes.