Russell Beattie’s Weblog

It's amusing to see a bunch of people in the tech community having trouble reconciling their love for the iPhone vs. how closed and proprietary it is. It's a real conundrum... The iPhone 3G is the best mobile phone there is, bar none, in both functionality and usability. It's also relatively inexpensive, widely marketed, and easy to get down at your local mall so tons of your friends and family have one as well. And there's nothing else like it on the market now, or in the forseeable future.

However, from a technology perspective, it's about as closed as they come. And this has caused some real consternation in the tech community. They're trying to work through the issue it seems, but haven't really quite gotten it straight in their minds yet.

First there's Gina Trapani writing about how you should avoid the iPhone, despite the fact that she doesn't actually follow this advice. She lays out the very real problems that the iPhone has from a Free Software and DRM perspective, but in the end simply covers up the Apple logo in a "minor rebellion" and keeps using it.

Then there's Tim Bray, writing about his Mobility Blues, because Java development on most phones sucks, Android is nonexistent, and developing for the iPhone despite how nice it is, is akin to being a sharecropper. He doesn't actually address development for Blackberry, Symbian or Windows mobile devices because really, the entire post is summed up as, "God, the iPhone rocks, I wish it were open, or at least had a decent JDK from Sun." :-)

Then there's Tim O'Reilly - who's been a huge iPhone booster despite the fact that his company was essentially built supporting open source - wondering about "devices and services that people love so much that they even love to hate them." I don't think anyone loves to hate the iPhone (except competitors obviously), it's that *they hate that they love it*. There's a difference.

One of the amusing things in O'Reilly's post was at the end where he wrote that Jeff Weiner asked him to "write something that explains why the iPhone is such a paradigm-shifting device," which he agreed he should do. It's sort of another attempt to reconcile the love/hate relationship with the device: "I have to love it! It's 10x better than anything else before it!" But is that true? I don't think so.

I really don't see any paradigm shift beyond the normal Apple integration and marketing magic. Anyone who's used smartphones for the past few years knows that the iPhone doesn't do anything that another comparable device like a Nokia N95 does - it just integrated them better. There are some innovations like the multi-touch screen, but in general the iPhone succeeds because the whole ends up being more than the sum of it's parts. It's not like it's 10x better than anything else, however. It all depends on your perspective and what you use your phone for. Ask a heavy Blackberry user if they prefer an iPhone's email client, for example, or a Nokia user chatting for free using integrated VoIP if they'd like to give it up, or ask a heavy Danger HipTop users if they want to give up their 24/7 IM connections and I doubt you'd get many takers. That said, ask any Motorola RAZR user what they prefer and there's no contest, so it really just depends on where you're coming from.

I've loved the iPhone since the moment I started using it (check the archives, baby), and have only gained more respect for it as time goes on. But as a developer and business person I've always been wary of it, and maybe a little annoyed that so much attention was lavished on it to the detriment of other more open platforms. I take the long view, however. The iPhone will never gain a monopoly like Windows - there's just too much competition in the market. Competitors from all sides - from the Intel backed MID devices to the Nokia backed mobile devices to the Linux backed open devices - will eventually catch up in terms of features and functionality. So therefore it's just a matter of time before there are more open alternatives that don't require any sort of sacrifice to use. Just like I use a Sony Vaio laptop with Ubuntu on it instead of a MacBook, with little to no loss of features and a much expanded universe of possibilities, eventually we'll see the same in the mobile market. Until then, I happily have an iPhone in my pocket and not worry about it.

That doesn't mean I'm content - every time I pull it out of my pocket I almost let out an audible sigh. My biggest regret so far as a professional in the mobile space is that I had nothing to do with that device, actually. I wish I had something to do with it because it's so nice, and has brought smartphones to the masses in a way that no one else could. It would have been nice to have been part of that effort.

I could actually see what Jobs was going to do years ago. Here's what I wrote back in 2004:

Steve Jobs has a mobile phone. I'm not sure which mobile phone it is, but he's definltely got one. And he hates it. He curses at it every day. He hates it like he hated the original IBM PC. He hates how hard it is to add contacts and make calls and he cringes at the web experience and the Java games, if he's even bothered to try them. He holds it in his hand during long trips and admires some things about it, but knows *he could do it better.* He knows that if Apple decided to make a mobile phone, it would be the most intuitive and elegant mobile phone in the world. And he wants that phone.

And he got it. But that's the thing: It's Steve Job's phone, and you shouldn't forget it. You can use it, you can love it, you can praise it, buy apps for it and let your kids play with it. You can do everything you want, but just remember, it's not yours and never will be.

If you're in technology, this is the thing to understand and once you accept this, it's pretty easy to enjoy how nice the iPhone is. Like leasing a BMW or something. It's not yours, but it doesn't make driving it any less enjoyable.

:-)

-Russ

So I got the new iPhone 3G, and have upgraded my old one to the 2.0 firmware, and have been pretty impressed with some of the innovative applications that are available online. I was thinking about this to myself while keeping my 6yo son occupied today at a restaurant with a free Pong-clone that I looked up and downloaded while we were sitting there. It's simple and fun, though he discovered soon enough that tilting the phone towards your opponent makes the ball go that much faster, and soon I was afraid we were going to snap it in half trying to tilt in the opposite sides direction.

That said, there's a lot of apps that are just plain missing from the app store - most of them which have been available on much less capable mobile phones for years now. Considering some of the crap that's in the catalog, I wonder if there's just a vacuum of Mac-toting Objective C developers, or if Apple is actively keeping these apps out of the AppStore?

Some stuff that's missing is just plain geeky, I admit. The first things I want to see are an IRC client, Jabber client, an SSH terminal and a Remote Desktop or VNC client [Update: There is a VNC client already! :-)]. Yes, I'm a geek - but so are the people who usually write these sorts of apps, which makes me think there's not really a dearth of these types of apps, but that Apple is only allowing "commercial" types of programs in the App store.

Beyond this stuff, I'd love to see a real VoIP client (Skype, or SIP) - all the advanced Nokia phones have this stuff integrated into them already - so it can't be carrier pressure or anything unless they're being overly cautious. I'd also love to have a camcorder app, or at least a decent camera app that doesn't lack features like the integrated one does (timer, digital zoom, etc.). And what about an Opera or Mozilla browser, maybe with integrated Flash? There doesn't seem to be any real technical reason for them not to be there.

Those are sort of "nice to have's", but an example of an important and popular app that's missing is MobiTV. Live streaming TV has been a staple on American mobile phones since I got my Nokia 6620 EDGE phone years ago. Not being able to have the option to watch live TV on HSUPA 3G networks is a almost a crime. I can't imagine that MobiTV ignored this platform outright, can you? And YouTube and Pandora already do video and/or streaming. Something else must be going on.

Thinking about it I realized the AppStore doesn't have any apps with a subscription model. They're all apps that are buy-once, use forever. Until I realized this, I had simply mentally compared the Apple AppStore to the Qualcomm Brew deck that's been on CDMA phones for years, but now I realize that in fact, it's actually much less capable. (Sorry Apple fans, they are neither the innovator here, nor apparently the most full-featured either.)

The comparison is actually quite accurate in general though. I still think it's shitty that I have to go through the iTunes interface to install apps at all - this is just like how Brew phones work as well. Why can't I just load up an app from any URL? Apps should be thought of as music, and just like I can "sideload" my iPhone with MP3s from my CDs, I should be able to load up apps from independent sites as well. This is exactly like the Qualcomm model. But you know, I remember being at one of the first MobileMondays in San Francisco years ago and Rocket Mobile was presenting, and Marc Canter sitting in the back having an absolute coronary hearing about all the limitations of the Brew app platform and being absolutely aghast that any developer would want to play that game, regardless of the revenue model. It seems however, that he's among the few, and Apple especially seems to able get away with this crud without so-much as a negative peep from the blogging or developer community. Hey, if you don't like it, don't develop for it, right? Sure, that's fine, I honestly just can't believe there's so many developers who decided to accept it.

The sad part about it is most of the stuff I'm missing were available on my Nokia 6600 in 2003. So if that's the case, why do I use an iPhone? Because it's the best damn mobile phone on the freakin' planet, with the biggest screen and nicest UI - that doesn't mean I need to eat Apple's shit and like it though.

All that said, I do have to say that there's apps for the iPhone that I *haven't* seen on any of the other smartphones out there even with years head start, so there's definitely a thriving and innovative community supporting the iPhone, and for that I'm happy and look forward to seeing what comes next. But just imagine how much *more* innovation we would see if these developers were actually using an open platform instead?

-Russ

I decided to whip up an experiment called Roomatic to test out some ideas I had to give Twitter some more forum-like community functionality like I've been posting about lately. The end result ended up being a way of using Twitter like how you would use IRC, with the channels based on #hashtags that are included in messages and presented in a standard chatroom interface.

At first I was aiming towards making a sort of Twitter-based clone of FriendFeed's comment functionality (which I may still try to do) so you'll see that Roomatic's rooms don't just have to be hashtags, but also "hashurls" (I just made that up, can you tell?). In other words, you can pre-pend a URL with a hashmark like this: #http://tinyurl.com/3c9ljq, and it becomes a room based on that URL. (Try it to see what I'm talking about).

That was the base of the functionality I wanted to use to enable comments, but then as I started messing with the Twitter and Summize APIs, I realized I could make it a bit more "live" using the JSON API. It ended up working out really well and enabled me to make a service out-of-the-box so that I didn't have to have users add Roomatic as a friend to use it (like you do with hashtags.org), nor did I have to store any user data. The Twitter POST just gets passed through to the server, and the rest comes directly from the JSON query. The result is the same sort of functionality I was looking for, but with minimal resources needed on my server.

I'm *so* glad that Twitter bought Summize, as even though they encouraged people building stuff on their API, I was worried about hammering at their servers too much. Each page will poll the JSON Search API once every 3 seconds for updates - and even though you can set it so that it only asks for updates since a specific tweet ID (which I did), it still seemed like a lot to do to some third party. Twitter, however, is supposed to handle it. :-)

This isn't much different than the functionality that Summize/Twitter Search already has, truth be told, but I wanted to see how it would "feel" with a slightly tweaked UI and updates that are shown immediately. I'm not particularly adept at Javascript or JQuery, so it's a bit crufty at the moment, but it gets the general idea across. (If you know ways of improving the way the Javascript works, *please* feel free to ping me).

Anyways, because only the original room requests and posts go through the server and the rest is done via JSON, it's actually quite light to host, so I thought it would be fun to throw the site out there, even in its crufty state. Try it out and tell me what you think.

Oh, and I'll be in #roomatic if you want to chat.

:-)

-Russ

Update: Tweaked the site a bit so that #hashtags aren't the default, just the room word itself. Hashtags can still be used to narrow the results with a %23 added to the room name.

Ahh, so it's that time again, when I post about needing to find a job - only this time I really mean it. :-) I've spent the past couple months decompressing after the failure of Mowser, and now looking at my bank account, it's time to get serious about moving on. Happily I can say that I'm rested, semi-tanned, and completely bored out of my mind, so finding a nice place to do some cool stuff sounds very appealing to me at the moment.

If you have a job that you think I might be good for, please email me at russ@russellbeattie.com, or call me at +1 415 606 5345.

My resume is here. The question you might be asking after seeing it is what exactly is it that I do? Well, that's a good question. I'm not a heads-down programmer any more, nor do I have real experience in product or project management, and though I did start my own company, it died... So that leaves me with hand waving, I guess. Do you need a good hand-waver (consultant, strategist, analyst)? If so, I'm your man!

Actually, I swore that if I failed at Mowser, I was going to shift gears and do something else like write for a living, or maybe be a full-time analyst (I did apply to Forrester last week, but they haven't responded yet). That said, the salary you can make in a technology company is still better than what you get as a writer, so it'd be sort of silly to just jump ship if I can still get a decent gig at one of the web companies or startups around. It's going to be a tough haul, though... I've had some phone interviews already with people just looking for low-level PHP contractors and they didn't go well, and since I don't think anyone in the mobile-web business wants to talk to me any more (for obvious reasons, as I think they're doomed) I really have limited options. If you know of something, definitely send it my way!

Thanks!

-Russ

The news broke last night that Twitter is going to buy Summize, which I think is fantastic because it's such a great service which really adds a ton of value to Twitter. Om, in his jetlagged state, wrote about why it's important here: the idea is that analysis of conversations is good for contextual advertising, etc. "We monitor collective attitudes being expressed right now on the web".

That's great that Summize thinks of itself that way, as I've personally thought that's been the best part of the service - and something I've used daily for the past couple weeks. The "Trending Topics" link section on their front page is really, really cool - much moreso than the search stuff. (I wish in fact there was a feed of just those links so you could see topics as they bubble up to the list. I'd also love to see them expanded on as well - a bigger top 100 list per day for example.)

The reason I think they're so cool is that you get to take the pulse of a thousand conversations from a million people or so and find out what they're interested in at any given moment. This is pretty great for someone like myself that tends not to be aware of popular issues outside my little tech bubble I live in. For example, lately I've learned that the Bachelorette was coming to the end and seemed to be something lots of people were interested in. (I had zero idea the show even existed). Also, I've seen movie reviews pretty consistently, and news topics as they pop up. Right now, because of the population of Twitter is left-leaning alpha-geeks and power users, the topics tend to be on that side of things with "Obama" being a semi-permanent member of the list, as well as navel-gazing type links - "Oooh, boy! Another AIR client for Twitter!" - and technology stuff like this morning's Drobo 2.0 announcement. But as the user base expands, the topics should get much more broad, and it'll be interesting to see what they are.

It's like those year-end zeitgeist reports that Google and Yahoo! do where they list the most important search terms of the year, but live and constantly updated. And more importantly, I don't just see that people are searching "about" something, but I get to see their opinion on it as well. Do they like the thing they're mentioning, or do they hate it? That's a key differentiator from a "popular search terms" list we've seen up til now.

I think if Twitter doesn't end up buying Summize, or even if they do that they should re-focus a lot of their effort in this area as this stuff is valuable to a ton of other services out there. (This was no doubt Summize's original business plan...). I just have that gut feeling of its inherent value, and it seems huge to me. Imagine a "topic engine" that did nothing but scour the latest blog posts, tweets, forum posts, etc. and created live topic lists for tons of areas? Like Nielson on steroids. It seems to me it'd be a really valuable position to be in: "Why is that important? Oh, Summize said it was."

All that said, it seems like something that Google or Yahoo! could do in a heartbeat since they're already spidering so much content out there. I wonder if we'll see something from them sometime soon now that it's been pointed out?

-Russ

Just trying out the new Adobe Flash 10 for Linux support for v4l2... yay!

-Russ

Even though I signed up for it months ago, it took me until just last night and today to figure out WTF was going on at FriendFeed. I'd see raves about it pop up on TechMeme, I'd go try it, not have any clue what the use of it was, and then forget about it again. It seemed like chaos and noise to me and no better than my current news reader.

So last night I went through and started cleaning everything up - I got rid of all my feeds first, and just added in my blog, Twitter and Disqus. Then I deleted all my friends, and went through and added a few back in by hand. That seemed to clean things up quite a bit, and this morning I confirmed FriendFeed was much more useful.

This is really the fault of FriendFeed's signup design because of its emphasis on adding your Facebook friends. When you sign up to the the service, it asks if you want to import from Facebook - and then automatically adds a bunch of those users who are already signed up if it finds matches, auto subscribing you to their set of feeds and vice versa. That just added tons and tons of feeds right away. Combine that with the "feature" of seeing 'friend of friends' feeds and the fact that you can sign up to the service like I did, add in a bunch of links, and then never return to it again, is just a recipe for noise.

So now that I cleaned everything up, I'm grokking what's going on quite a bit more, and there's a few things that are going on that are interesting - and not just another mashup of feeds. There are (at least) three distinct parts to the system that I finally have figured out.

The first part of the service should almost be called "Friend's Feeds" (possessive and plural) as the idea is that you and your contacts can add in all the feeds from their various services they use regularly and they will appear on your home page as an "activity stream". Each user's profile page is also sort of a summary of their online activity as well which if you think about it is incredibly useful. Rather than having to create a feed which splices in all the various services, FriendFeed does it for you and creates a nice web-viewable page for people you know (private or public, your choice) and also creates a "feed of feeds" as well. Though I've had readers of this blog complain when I spliced in various other feeds like my del.icio.us links or my Twitter posts, it's still a great reflection of what I want to share online - so it's very cool.

Side note: One has to wonder if FeedBurner - who had a limited feed splicing in their service from years ago - will get into the act and offer a more complete service that competes on this level? I already use FeedBurner to offload the robot activity from my blog's feed, it's tempting to also offer my FriendFeed spliced feed as well. Reversing this, I wonder if FriendFeed will ever provide the stats of FeedBurner or the ability to map domains?

The second part of FriendFeed is the direct posting/sharing feature. You can share links and tweet-like posts directly on FriendFeed, bypassing the other systems you may use. This is sort of brilliant. Once users start using FriendFeed and get others to subscribe to it, they'll star to wonder, "Why post this link on del.icio.us or use Twitter to post a random thought, when I can just share it straight to FriendFeed instead?" Though at first, you might not have enough friends who know about your FriendFeed stream, so you might be afraid of posting something that no one else sees - as more of your friends start using it (or are aware of it), there's a tipping point where it becomes

Finally, the last part is the comments/forum features which have been layered on top of all the items that have been posted to the service. Each item - whether it's a Flickr photo that's been imported via your Flickr feed, or a post you wrote directly in FriendFeed using the "Share Something" button - is the basis of a public conversation where other FriendFeed users can add their thoughts by adding a comment. Additionally, you can choose to share an item or a feed directly into a FriendFeed Room, where the comments on those items are seen by everyone in that room, whether they subscribe to the original item's feed or not.

Had I understood how much FriendFeed acts like a forum last night, I would have added to my mongo-post about the subject. The fact that I discovered after I had already posted about how compelling forums are, and how FriendFeed is gaining traction quickly only goes to reinforce the point of my post I think.

There's actually a huge similarity in how 4Chan works, and how FriendFeed works which unless you're obsessed with the subject you may not notice. When a user comments on an item in FriendFeed, it "bumps" the feed back to the top of your activity stream. This is very similar to how many forums - especially the anonymous ones - work as well. It serves to help continually push the topic and encourage users to respond to each other, as the discussion keeps coming to the top until it loses steam.

So FriendFeed does some really interesting things, and some more that I haven't paid much attention to yet (the Digg-like "joe liked this" rankings for example), but honestly, it's a total *mess*. It's sort of the anti-Twitter in terms of usability and focus. It's so completely geared towards the power user and uber-geeek its not funny - though with a name like FriendFeed, it's not surprising, most people still have no idea what the hell a 'feed' is. But even for the most jaded Web 2.0 super user, the service's multitude of entry points (feeds, sharing, comments) makes it so easy to devolve from a useful service into pure noise, as I personally can attest to.

FriendFeed needs a serious cleanup, IMHO. Some problems:

* Their tab metaphor is all screwed up - some things get new tabs, others are in sub-tabs/links. Tabs pop up and then disappear. It's just horrible.

* Your account settings are accessed by a link up in the corner, and has some functionality you'd expect in the friend settings tab, which is hidden off on the right.

* The Facebook integration is just a bad way to start off (like I said above) - I don't subscribe to everyone in my Facebook's feeds for a reason.

* Showing me friend-of-friends by default is just dumb, and the setting to turn it off is really hard to find and should be in a global settings page.

* There's no way to filter which feeds of my friends I see. Just because someone's added in 30 different sources, doesn't mean I need to see them all. Why is it all or nothing?

* There's no list of Rooms... which I think is weird and probably just an oversight for now.

* Profile pages have everything except the most basic info! It needs a blurb and a "main link" as well. Some people have added me, and it's hard to figure out which of their many feeds is best to learn about who they are.

* Is FriendFeed mainly an activity stream, a microblog system, or a forum? They don't have to cut features, but they need to choose a focus.

Okay, so that's all my thoughts for now. Feel free to add russellbeattie to your FriendFeed friends or subscribe to my feed-of-feeds, which only has a few services right now, but will probably expand as I use the service more.

-Russ

Think about the last time you were trying to find an answer for something online - where did you eventually find the solution? It may depend on what you were looking for, but for me the answer is invariably contained in a web forum somewhere online. Blogs will many times have information about what I'm looking for, but almost always there's a forum thread or two out there that has as much or more info about whatever it is I'm looking for. Forums are as old as the Internet itself (newsnet, bulletin boards, AOL boards, etc.), and yet they're still fascinating to me.

They're so useful, varied and popular and yet they're also so broken in many ways. I really don't think enough effort has been dedicated to "figuring out" why they work, and what can be done to improve on the standard formula. Forums have a pretty common structure - you have a main page where you see "categories" of things to leave messages about, then within those category pages are "threads" or "discussions" which show the first message in the the thread, and the number of replies to it. Clicking on the thread leads to a list of messages about that subject.

This tried and true model serves some of the biggest websites out there. They're not big as say, Yahoo! or AOL, but they collectively have millions of members who produce millions of posts every day. The amount of knowledge and effort going into those sites is incredible - but they seem almost completely under the radar of entrepreneurs or investors because they are so common.

Yet look at the graph I included above from Compete.com - there's so much interest in Twitter and yet it only just recently passed GaiaOnline in terms of users, and hasn't caught up yet with the image forum DeviantArt. Sure, Twitter's growth is fantastic and viral, but look at those other sites, chugging along with millions of members and posts, and there's dozens more just like those out there.

This is what I've been obsessing about lately. I've actually written about forums before, but I'm looking at them with fresh eyes and noticing how interesting they are, and yet how little they've been modernized. I have the sensation there's a huge opportunity here waiting to be taken advantage of.

If you step up a few levels, it's easy to think of *everything* as a forum of one sort or another, yeah? Social Networking contain forums, profile pages with "wall" posts are like personal forums, Yahoo! and Google groups are forums, blogs with comments are forums, company feedback pages are forums, etc. etc. But I'm thinking of the topic-based, shared-experience forums rather than these smaller discussion pages.

I think that may be what attracts me to forums so much. I always go back to my experience with Facebook most of my interactions using the system are one-on-one. I approve new friends, I get a few messages, I leave and read Wall posts - everything is done in the context of one person talking to another, though in the view of the public. I've gone hunting around for interesting forums or groups in Facebook with real content and there doesn't seem to be much there. It's a person-centric social structure, not an object-centered one like most of the forums. You go to the massive IGN forums to talk about games, GaiaOnline to talk about Anime and you go to DeviantArt to share interesting images. These are things I can understand and relate to - it's the difference between going to a party, and going to a meetup or event. I rarely do either, but I much prefer meetings with a purpose, just like I much prefer forums.

The genres of forums are interesting as well to me. I've been fascinated with Japan's 2ch "image board" forum for years now, and recently explored the English-language equivalent at 4chan.org. Both are anonymous forums (which I initially wrote about years ago), neither requiring users to log in, with most posts being written by "Anonymous". Both are also incredibly influential - 2ch much moreso in Japan, but 4Chan is the source of many of the Internet memes we've seen over the past couple years (Rickrolling, etc.). According to Wikipedia, 4Chan's /b/ "random" topic has wracked up an incredible 70MM posts in its 4 years.

Well, most of that is probably the absolute worst trash you can imagine. Even if you've seen it all, hanging out on 4chan for a bit will scar your retinas and challenge your faith in the good of humankind. That said it's the ultimate expression of free speech, and hey, also pretty amusing at times. :-) Putting aside moral and ethical questions for a moment (the racism, sexism and homophobia on /b/ is pretty disconcerting...) I'm just thinking of the numbers. The volume of posts is such that you don't actually have to ever use the "next page" links, simply refresh the topic pages, and posts will bump up over and over again. There's obviously something about this type of forum which is incredibly compelling.

Well, all forums actually. People post and post and post to them. Think 4Chan's number of posts is huge? According to Big-Boards, GaiaOnline has over 1.3 BILLION posts! They're obviously the exception (and are such a force, they're launching their own MMRPG based on their forums, which is crazy), but IGN has 181MM posts in its site, Nexopia has 160MM.

This is what excites me, because it seems for all their size and popularity, most of the popular boards use off the shelf software to do most of their work. And maybe that's fine since the compelling part of the forums ostensibly aren't the features of the site itself, but the topic it's covering. Companies are already taking advantage of the fact that there's little need to innovate - Ning, for example, is essentially just a forum site. Browsing the most popular or featured "networks", their basic forum component is always the core element - with the other features like extensive profiles and friends being secondary. Lefora is focusing just on the forum itself, not even bothering with the social stuff. This makes sense, since it seems that once you've created a forum around a topic, traffic and posts seem to follow regardless of additional features.

But what's to say that by changing some of what's considered standard in forums, there wouldn't be something that's even more compelling?

A few years ago I was doing some thought experiments around this topic when tagging was all the rage and ended up with what I called a "hyper forum" (I wrote about it here.). The idea was to add tags to each post in a normal forum, so you could then follow posts in various dimensions - by date, by thread and by tags. It didn't work out very well as it was incredibly confusing, but it was an interesting experiment.

Going in the opposite direction of adding to forums, how about simplifying instead. For example, is there really a need for both topics and threads? Is there a better way for non-registered users to discover topics (say by word frequency or page-view popularity)? Is there a better way for users to keep track of posts and replies? Essentially what I'm thinking of is a microblog version of a forum - simplified and streamlined.

One of the things I'm obsessed with is "user friction", which is why I love anonymous forums so much. For example, there's zero friction involved in participating in a 4Chan thread. You don't have to register, and the upload field is part of the main form. You just write what you want, add a file and you've instantly added content to the forum. But the site just looks like hell and takes ages to understand what's going on.

The problem however, with all forums especially ones you can post anonymously to is spam and/or illegal activies and content. Years ago I remember wondering why Yahoo! didn't do more to emphasize their message boards or chat rooms. Those are the ones that used to be part of Yahoo! Messenger or linked at the bottom of news posts. Randy Farmer - who's been doing virtual community stuff longer than I've been using a computer mouse regularly - sat me down and explained how the more anonymous a community is, the more work it takes to maintain and the less value it has to users and ultimately to advertisers. The message boards were filled to the brim with spam and the chat rooms were filled with nothing but crude behavior and high probability of "grooming". Since human moderation was expensive, and ultimately impossible, Yahoo! eventually got rid of that stuff as it was just too taxing and/or dangerous to maintain.

Again, this goes back to why Twitter is so interesting. The whitelist system cleans up so much of that stuff it's incredible (as I wrote about here), thus also making it quite valuable. Sure, you might get requests to be be friends with 100 bots a day, but if you don't add them, you don't see their crap in your messages. The question is, how can you duplicate this inherent value in a forum?

I've got some ideas, which is why I was excited to play with the Laconi.ca codebase a bit a couple days ago at my catchall domain Foozik. The first thing I did was bump up the maximum length of a post to 280 characters, and add in automatic embedding of image and YouTube links. It's not a forum, but it's at least a bit more in line with some of my thoughts for features that I think would be interesting to have. I'll most likely start in on the code again from scratch though, as I want it to be unique, but it's good to test out ideas.

I envision this sort of combination of Tumblr, Twitter and 4Chan in my head - where it's super easy to start posting various content types, and easy to keep track of a thread and replies, but with much less baggage of a traditional forum with their tables and links and weird avatars and sigs cluttering the interface. The questions is how to keep the quality up and spam out, and that I'm not sure about just yet... but I feel like I'm close.

Maybe just a couple more years, and I'll have figured it out. :-)

-Russ

I was just about to embark on a post yesterday about my latest obsession which is web-based forums (actually, it's a return of an old obsession) when identi.ca launched with their open source PHP-based Twitter clone, so I just had to try it out. I threw it up on foozik.com if you want to see. It took me a while to get the dependencies working, but it seems pretty cool.

It's a great effort, looks good, and promoted in all the right ways. Evan (the guy behind identi.ca and the laconi.ca code base) did a great job creating a nice little project with some cool features like OpenID, Jabber support and the beginnings of a federation system.

Looking at the code, however, it's doomed.

The core architecture just isn't made to scale, and a day after it launched identi.ca already seems to be paying the price, even after adding a bunch more servers. Here's the the problem in a few lines of code:

$notice = DB_DataObject::factory('notice');

# XXX: chokety and bad

$notice->whereAdd('EXISTS (SELECT subscribed from subscription where subscriber = '.$profile->id.' and subscribed = notice.profile_id)', 'OR');

$notice->whereAdd('profile_id = ' . $profile->id, 'OR');

$notice->orderBy('created DESC');

Even the comments express this is "chokety and bad". Ignoring the use of the PEAR::DB data object stuff (that's adding abstractions on top of your database that you can't afford to have) this code shows that the design of the system is fundamentally flawed. The core problem is the query itself - it's expensive as hell: "Get all the notices (messages) where I am subscribed to the publisher." Oh, man. As the database grows, the indexes will have to get huge, and as there's more subscribers and more subscriptions between subscribers, it's going to be impossible for that query to keep up.

The lesson from Twitter is that microblogs aren't Content Management Systems at all, but are instead Messaging systems, and have to be architected as such. SMTP or EDI are our models here, not publishing or blogs.

Here's how a microblog system has to work to scale: All the messages created by users have to go into a Queue when they're created, and an external process then has to go through one by one and figure out which messages go into which subscriber's message list. As the system grows and more messages are created, the messages may arrive in your "inbox" slower, but they will still arrive. This type of system can be easily broken up into dedicated servers and multiple processes can handle different parts of the read/write process, and the individual user message lists can be more easily cached - as once a page is created that contains messages, it doesn't change.

If you don't set it up like this? Well, you're seeing what happens at Twitter and identi.ca now. As the number of users scale up, and the number of messages increase, the load quickly overwhelms any relational database until it'll become impossible to keep up. Structuring microblog systems essentially like self-contained web-mail is the key. Just think about it - Hotmail survived and scaled after it launched in the 1990s when 512MB of RAM was considered a lot and 10GB hard drives were "big". It's not about *power* or throwing more/bigger hardware at the problem, it's about architecture.

Another example: Lots of web forums out there get millions of new posts a day by tons of users (GaiaOnline.com had 5MM posts last week, for example), but they scale just fine because the format of forums have been designed to scale. These sites work because they simplify queries, facilitate caching, and not guaranteeing instant updates. Simplification and caching are to me the fundamental aspects of web scaling. This is the way it's been for a decade now... Want to survive a Slashdotting, for example? Get your DB out of there and export your pages as .html static files. These same principles can be applied even to a microblog/messaging system as well.

Once this is widely accepted (and I'm sure there are many that would argue with me), the thing that will separate these types of services won't be whether they stay up (ala Twitter), but how fast your subscription messages are updated. Some services might be smaller or offer more features but not update as quickly whereas others will pride themselves on being as close to real-time as possible. The key is that it's all about messaging, not publishing. (Oh, and this also facilitates federation as well, but that's another topic).

This said, all is not lost for the Laconi.ca code. The good thing is that it's open source, and not only that, but using the GNU Affero license, which means any changes made on anyone's server needs to be released back as well. Hopefully now that there's a code base to start from, some others (like myself) who are too lazy to start from scratch can get in there and start tweaking and shaping the code into something that will fundamentally scale better.

I envision lots of microblog services out there, actually. Even though services do have a network effect going for them (the reason Twitter has survived for so long), the idea of smaller microblog services is very appealing. One comment on Identi.ca yesterday spelled out it's usefulness perfectly - a person wanted a version she could use in her classroom as a way for students to ask questions. Very cool - yes you could use a live chat room or a simple forum or e-group, but bringing the subscription model into it would add a very interesting dynamic that I think better reflects how people really interact. That's just one example, but I think there's even more out there.

Just my thoughts for now. I'm going to write more about web forums in a bit.

-Russ