I thought Experiment 2 was a great example of the pain of trying to fit your users' feelings into an "ideal" model.
When we started LALife.com, it was with the idea of making a real estate site with lots of great statistics and quantitative data. We gave grades for how safe things were, how good the schools were, multiple heatmaps, assessor data, census data, nearby amenities -- a buffet of data. And sure, people who also fit in the mold of trivia collectors thought this was amazing.
But the more we talked to people, the more we found out that they really didn't use this data, even when they said they appreciated it. We had really overshot the market -- users almost never delved into the statistics, yet took our grades as absolute authority. What people wanted first of all was just insight about whether it was a "nice place" or not, a maddeningly vague concept.
Ultimately, we trashed countless tables and statistics and scaled it back to one number. Yes, one 0-99 number that shows you, well, how "nice" a place is to live. We were so afraid of generalizing things like this because everyone is different and has different priorities and so on. But trying to make things custom for everyone is a losing game, and it turned out "is it nice?" is something everyone already knew intuitively.
So insanely, we went from having 50 extra data points on a home for sale to having one "superscore". But a funny thing happened, which is it became much more successful in the site's actual mission, to help people understand a home's neighborhood without having to visit in person. And now we're providing an insight that is compact enough that we can put it everywhere and people can digest it instantly.
So although we would ultimately like to give people a more data-conscious mindset, the tool and the user need to agree on that commitment. So we're accommodating the user's actual mental model while we work towards expanding it.
PS: We have our scoring model working nationwide, but as you have seen from the Zillow thread, it's hard to get nationwide home listings. We're working on getting homes for rent on http://www.padrank.com/ so you can sign up there if you want to see how it works throughout the US.
This phenomenon is actually quite common. In retail, it's called the customer funnel: http://en.wikipedia.org/wiki/Purchase_funnel. The idea is you want to capture as much awareness with something really simple, usually it's a brand name, e.g. Apple. When customers express more interest, by coming in Apple store/website, they are presented with more design/technical details. Finally, only when they show purchase intent, they are shown price/configuration/delivery options.
When we started LALife.com, it was with the idea of making a real estate site with lots of great statistics and quantitative data. We gave grades for how safe things were, how good the schools were, multiple heatmaps, assessor data, census data, nearby amenities -- a buffet of data. And sure, people who also fit in the mold of trivia collectors thought this was amazing.
But the more we talked to people, the more we found out that they really didn't use this data, even when they said they appreciated it. We had really overshot the market -- users almost never delved into the statistics, yet took our grades as absolute authority. What people wanted first of all was just insight about whether it was a "nice place" or not, a maddeningly vague concept.
Ultimately, we trashed countless tables and statistics and scaled it back to one number. Yes, one 0-99 number that shows you, well, how "nice" a place is to live. We were so afraid of generalizing things like this because everyone is different and has different priorities and so on. But trying to make things custom for everyone is a losing game, and it turned out "is it nice?" is something everyone already knew intuitively.
So insanely, we went from having 50 extra data points on a home for sale to having one "superscore". But a funny thing happened, which is it became much more successful in the site's actual mission, to help people understand a home's neighborhood without having to visit in person. And now we're providing an insight that is compact enough that we can put it everywhere and people can digest it instantly.
So although we would ultimately like to give people a more data-conscious mindset, the tool and the user need to agree on that commitment. So we're accommodating the user's actual mental model while we work towards expanding it.
PS: We have our scoring model working nationwide, but as you have seen from the Zillow thread, it's hard to get nationwide home listings. We're working on getting homes for rent on http://www.padrank.com/ so you can sign up there if you want to see how it works throughout the US.