1. REST == HTTP GET
2. HTTP+POX == HTTP GET & POST
3. SOAP == HTTP POST, with interop issues

When I’m talking to people about web services, and I hear them use various terms, this is what I feel they’re talking about most of the time.

REST

REST almost always indicates HTTP GET with a query string. The results are probably XML, but lately they could also be JSON.

No other HTTP verbs are used (especially PUT and DELETE), and it’s highly likely that all requests are routed through the same URI.

For example:

http://api.example.org/rest?action=search&query=trachtenberg

HTTP+POX

Nobody uses this term. However, there are some web services that, while not “true REST,” use more than HTTP GET. I place them in this category.

These likely exist because the service allows you to submit a large quantity of data that is not human readable, so it does not make sense to place as part of a query string. Blobs of HTML and pictures come to mind.

For example, the eBay “XML API”. You can submit items over eBay web services. Since the item description is pure HTML, it exceeded the maximum query string length our web server would accept at the time.

Thus, the eBay XML API was invented: you HTTP POST an XML document, we return an XML document.

Again, like REST, there’s no concept of resources. Everything goes through the same URI. The action is indicated either as part of the query string or in POST body. (Or both, as in the eBay case.)

SOAP

SOAP is identical to HTTP+POX, except that you’re required to use XML Namespaces and XML Schema.

You are also going to have interop issues when trying to generate the SOAP envelope to send or parse the SOAP envelope that’s returned.

XML Namespaces and XML Schema are minor headaches, but people are willing to deal with them. People understand that, in theory, namespaces are good, even if XML Namespaces are a little funky.

XML Schema can be confusing. However, it does allow you to validate the response, which most people find to be somewhat useful. The fact that XML Schema may not be the best way to describe XML data is a different issue.

If SOAP merely had these two issues, people would work through it. The real problem with SOAP is that the specification is so confusing, people can’t build interoperable clients and servers. This drives people mad.

People know how to generate arbitrary XML (it’s just text after all) and send a HTTP POST request. They also know how to parse XML, particularly when they know ahead of time what they should expect. (That’s why PHP 5’s SimpleXML extension is a great web services client.)

What they cannot do is decipher a WSDL file to determine what crazy combination of XML the server is expecting (especially when the server is not following the standard). And, assuming they can get through that, they cannot figure out how to repeatedly hammer their SOAP client into producing the magical combination of XML necessary to placate the SOAP server.

That’s the difference between HTTP+POX and SOAP.

It’s not the complexity. Well, it’s not complexity in terms of needing to build up a large document. It’s complexity in terms of trying to understand the hundreds of pages of XML specifications to generate the request and not having good examples of working XML documents to crib off of.

As to whether the message name should go in the URI? Nobody cares.

Looking at the Table of Contents, I’m particularly interested in “Appendix A: HTTP status codes and when to use each one.” (Yes, I am serious.)

When I designed “Dude, Where’s My Used Car,” my mashup between eBay Motors and Google Maps, I churned through a number of different ways to signal failure. My biggest issue is that there’s so many places my request can fail, I’m not sure how to “correctly” represent that back inside my mashup.

For example, in response to a user-generated action (such as filling out a form and clicking submit), my mashup initiates a HTTP GET AJAX request (well, AJAJ because I’m using JSON instead of XML) which hits my web server. I then use PHP to parse the request, generate a corresponding SOAP request to eBay’s web service gateway, wait for result, pick it apart and convert the bits I want to JSON. This data is then passed back to the browser, where I (well, Dojo, actually) evals it.

When everything works, everything works.

But what happens when the user enters an invalid ZIP Code? eBay will return an error, and I’d like to proxy this back to the application, so I can toss up a message.

Should I return 200 OK and pass back the message? That’s one option, but I’d prefer not to need to check the data structure to see if I got the normal data I expected or some nasty error instead.

Besides, Dojo lets me specify a separate error handler callback function. It seems a cleaner design to handle all the errors there. But then I can’t return 200. What to use instead? 400 Bad Request?

Then there’s cases where the error is not the user’s fault, but either my fault or eBay’s fault. What should I do if eBay returns a SOAP envelope my PHP script can’t parse? 502 Bad Gateway? Or if the eBay server returns a SOAP Fault? The same 500 Internal Server Error I got back from eBay?

Should I not let my web server’s interaction with eBay bleed back into browser?

Here’s another example: when eBay’s server is too busy, it returns Error Code 10007. I can retry (and hopefully fix the problem) inside of my PHP script without even needing to message back to the browser. But if I did ultimately give up retrying, should I pass back 503 Service Unavailable? Am I on the hook for translating between SOAP and REST in my server-side proxy PHP script?

Maybe I can use 4xx codes for problems which are “user generated” in some form. These are things which require the user to take an action to remedy. And for things which are broken but otherwise outside of the control of the user (such as my SOAP client being unable to understand eBay’s SOAP response) I can use 5xx.

I actually like this idea, but I’m wondering if there’s a “standard” way to indicate errors and failures in a AJAX application. I can’t find one, but I’m not very familiar with what’s out there.

PS: My vote is to leave off the “With Ruby” part of the title, and go with plain “REST Web Services”.

For example, at this year’s ETech, I broke through with a killer presentation titled: “eBay Web Services: A Marketplace Platform for Fun and Profit.”

Knowing a good thing when he sees it, Patrick from Google tries to respond at this month’s ZendCon with: “Scrub (Ajax), Wash (SOAP) and REST: use Google Checkout and AdWords APIs with PHP for fun and profit.”

Mixed metaphors and the ugly imagery of Patrick in the shower aside, I am not put off in even a small way by his outright thievery. For I am already sixteen steps ahead.

I am proud to introduce, coming on October 31th, for the very first time, my latest and greatest talk: “eBay Web services: Great platform or greatest platform?”

Truly Lincolnish.

PS: You are only allowed to read this if you have a sense of humor. This is meant to be funny. I don’t think Patrick is going around stealing all my talk titles, or even just my good talk titles. Besides, I am stealing my most recent one from Stephen Cobert, so I can’t really claim ownership to that now can I?

SOAP is obviously the big one. I use the PHP 5 ext/soap extension, which is great, but there’s actually another PHP SOAP extension that might be even better. No, it’s not PEAR::SOAP or NuSOAP; it’s axis2.

If you haven’t heard of axis2, you’re not alone. That’s because it’s a pecl extension that’s still in beta, so there aren’t a lot of people using it yet. However, it’s a PHP version of the Apache Axis 2.0 SOAP stack.

Unlike ext/soap, which is recreating SOAP one feature at a time, Axis 2.0 already supports a good portion of the WS-* specifications. So all that needs to happen is for someone to write the hooks between PHP and Axis 2.0, which is far easier than actually writing the features themselves.

I met a number of people from WSO2, the company that’s writing Axis 2.0, and they’re quite excited about the extension. However, they did mention it’s still in beta, so I haven’t actually spent any time using it yet.

The other PHP XML extension I’ve been hearing a lot about is SDO. SDO is an attempt to provide a standard data interface regardless of the backend datasource. So, for example, you can interact with XML data in the exact same manner as information pulled from your database.

At OSCON and ApacheCon, I’ve run into a couple of people from IBM who have been doing the heavy lifting on this extension, and we’ve had a number of interesting chats with them about eBay Web services and SDO, but I hadn’t had any free time at work to install the extension.

Therefore, when I had a little free time today between the end of my session and lunch, I sat down and reproduced a short code example that I had wrote using ext/soap with SDO instead.

Here’s the key portion of the original code:

// Print Titles and Mileage
if (isset($response->SearchResultItemArray)) {
    foreach($response->SearchResultItemArray as $item) {
        printf("%s\n\t%d miles\n", $item, $item->ItemSpecific['Mileage']);
    }
}

This iterates through a search result for eBay Motors listings and prints out the title and the mileage for each individual item.

Normally the code would be far more complex, but through a series of ext/soap class mappings I wrote that implement the IteratorAggregate and ArrayAccess interfaces, along with the __toString() magic method, I’ve managed to abstract away a number of the complexities.

Best I can tell, SDO doesn’t give me quite the same level of control, but it does implement a number of these features for me out-of-the-box.

Here’s my rewrite using SDO:

foreach ($root->SearchResultItemArray->SearchResultItem as $item) {
    $title = $item->Item->Title;
    $mileage = $item->ItemSpecific["NameValueList[Name='Mileage']"]->Value[0];
    printf("%s\n\t%d miles\n", $title, $mileage);
}

It’s not quite as brief, but I do get this nice XPath-like filtering that lets me pull out the the value of the car’s Mileage in one line. Pretty handy.

I had to implement the ArrayAccess interface to get this to work under ext/soap, which included this method:

public function offsetGet($name) {
	if (! is_array($this->NameValueList)) {
		$this->NameValueList = array($this->NameValueList);
	}

	foreach ($this->NameValueList as $NameValueList) {
		if ($NameValueList->Name == $name) {
			return $NameValueList->Value;
		}
	}

	return null;
}

Not the hardest thing in the world to write, but this is just one of the custom class maps that will arise in our data schema, and SDO takes care of them all automatically.

Still, right now I think I prefer my ability the greater control I have over the interface with ext/soap. Our SOAP schema isn’t that pretty since there are lots of list, array, and hash wrappers. Through classmaps and interfaces I can turn these into native-looking PHP arrays and hashes.

I may be introducing a leaky abstraction, but I think this is better than exposing a NameValueListArrayType for people to wrangle with.

Like ext/soap, SDO requires you to define your data using an XML Schema. While we actually publish a stand alone XML Schema file, SDO will also happily parse a WSDL file, too, which is nice.

However, it will not directly consume a SOAP message because the WSDL doesn’t include any mention of the SOAP envelope wrapper. I needed to rip out the contents of the SOAP body into a separate XML document in order to get SDO to parse my data. Oddly, this corresponds perfectly with our “XML API,” so I could use that to retrieve properly formatted data that I can pass directly to SDO.

Where SDO really falls down for me is performance. Admittedly, eBay is a pathological case, but our WSDL file is 2.94 Megs in size. When I feed that to SDO, it takes 8.5 seconds to process the XML Schema data. Yikes!

I don’t mind a one-time start up hit, but it doesn’t appear that SDO can cache a parsed version of the schema. In contrast, ext/soap has both an on-disk and in-memory WSDL cache.

This means it takes SDO about 8.7 seconds to process the schema, load in data, and print out the information — and all with locally stored files.

In contrast, ext/soap takes as little as 0.55 seconds to pull in a cached version of the WSDL from the disk, go out over the network to query eBay via SOAP, wait for eBay’s SOAP server to run a database query on its end and send back a SOAP response, parse the SOAP envelope, and print out the exact same data. If I was using the in-memory cache, I think it’d be even faster, and I’m sure the big bottleneck here is talking to eBay.

Now, I’ve spent all of 60 minutes playing with SDO, so it’s quite possible that I’m missing some obvious configuration flag. If not, I hope IBM can do something to help speed up the performance, since while I could probably write a script to break apart our schema to create individual files that contain all the possible types on a per-call basis, I am hoping I don’t need to.

In the meantime, I’m going to continue exploring SDO to see what else I can do with it because I’ve sure I’ve only grazed its surface.

[Update: I cannot get the axis2 extension to compile, despite trying a number of different versions of Axis2c and both Linux and Mac OS X. I may try again tomorrow.]

As usual, I had a great time at the show. It was fun to see all my old friends and make new ones.

After poking around Feedster and Technorati, I’m glad to see other people are equally excited by these ideas:

The best demo was one that showed how you can integrate eBay with Microsoft Outlook, giving you a single place to manage your auctions. One can envision many similar uses that aggregate task-based communications into your email software.

Trapper Markelz

[Joe Belfiore] demo’d a cool add in that eBay created for monitoring auctions. Although not an avid eBayer this is something I’ve wanted for a while.

eBay had a talk about building their live.com search gadget – source speaks louder than words – it’s available on www.microsoftgadgets.com.

Jon Gallant

Later today I’m hosting a round table, so if you’re still at MIX, come on by the eBay table and I’ll buy you lunch.

Other links:

• Jon Gallaway
• VB-tech weblog

Yesterday, was the big Bill G keynote and 1-1 chat with Tim O’Reilly. During Bill’s opening remarks, he gave the eBay Web service a nice plug by saying “eBay is an extreme example where half the product listings are done in a programmable way.” Technically, it’s 47% of eBay.com listings, but what’s 3% among friends?

Later on, Dean Hachamovitch, king of IE 7, showed off eBay’s new support for viewing search results via RSS directly within the browser. Even better, we’ve integrated support for Microsoft’s Simple List Extensions to RSS, so you can sort and filter eBay items by category, format, price, etc. I think it’s a great way of using RSS outside of news syndication.

After lunch, I was on a panel titled “Web 2.0: Show Me The Money,” with Tim O’Reilly, Jeremy Zawodny, Michael Arrington, and Royal Farros. At first, I was worried we couldn’t fill up the entire hour and fifteen minutes, but we actually ran three minutes late and could have kept going. I don’t know if that was a good or bad thing, but a number of people have come up to me after the panel to say they enjoyed it, so I’m going to assume we were at least entertaining, if not actually informative.

Here’s the round-up from the blogsphere:

• Jeremy Zawodny
• Joe Duck
• Luc Van Braekel
• Randy Holloway

I hustled from my panel to Christin Boyd’s Office 2007 talk, where she demoed (in grand style) an eBay and Outlook integration, where you can pull in the items your watching and bidding on from eBay directly into Outlook. They appear directly inside a folder that you can sort, label, etc. Even better, they appear on your calendar, so you get a reminder 15 minutes before the auction closes. She even overwrote the “Reply” button on the ribbon turning it into a “Bid on eBay” button. Quite cool!

This morning, Joe Belfiore demoed this in front of the entire MIX 06 crowd during his morning keynote as an example of Office integration with third party sites using Web services.

Right now, I’m taking a short break before lunch, and then I’m off to hear Alan Lewis demo his eBay Live.Com Gadget. He’s learned all sorts of practical information about combining widgets and gadgets with Web services, and he’s going to share best practices with the attendees.

I got a chance to play around with the gadget over the past week, and it’s quite nice. Kudos to Alan, Rob, and Tim, for the design, programming, and UI. They really took this from idea to concept to actual code all by themselves. In particular, they added this nifty feature where the gadget will intelligently expand and truncate the search results depending on the width of your screen. Very impressive.

If you’ve made it this far, I’ll share my one Vegas celebrity almost sighting. Yes, a real “appears in the National Enquirer” celebrity, not a tech “has an a-list blog” celebrity. While we were at dinner last night, Britney Spears rolled into the restaurant. Unfortunately, no thanks can be shared with “It’s 30 seconds too late, but now’s when I’m going to mention this” Arturo, who didn’t alert the people at the table with our backs to the entrance. I admit to shamlessly trying to “go to the bathroom,” but she was hidden away in a private room, and the bathrooms are in the casino, so that line didn’t work so well.

It’s quite nice and takes advantage of a not-well-announced new rate limiting feature that you should expect to learn more about later this month. :) No particular reason for the suspense, except that I’ve been too busy to give it the proper attention it needs.

If you’re the type of person that runs Google Desktop and uses eBay, you should pick it up.

[S]earch engines like Google get their revenues from advertising because people use these search engines, but they don’t share these advertising revenues with the end users who help them get the revenue.

Later on, Josh hits on my number one long-time issue with making money from search:

[Google should] be concerned about the lack of network effects, but also by the related fact that the switching costs both as a searcher and as an advertiser are so low.

I use Google now because it’s the best. But I used to use AltaVista. And before AV, I used Lycos. If (when?) someone else comes along, I’m going to switch.

Google has no inherent structural advantage over other companies in regards to search. They are no lock-in costs or network effects. They’re just better at it. But companies such as Microsoft and Yahoo! and eBay can also hire engineers who can write search engines. They can even hire away Google engineers.

If eBay sellers will go through the hassle of selling their items through multiple channels, you better believe advertisers will go where they get the best bang for their buck, too. As long as Google continues to aggregate demand and provide the “best” clicks, they’ll continue to capture massive value.

However, when other companies catch up, either by providing a better service or by paying me to use their search engine, Google will need to cut into their AdSense margins to remain competitive. This hurts Google and all search engines on the supply side.

In related news, Tim Bray talks today about “The Future Search Market” He describes an application that has a Web search window and:

When someone types in “Britney Spears” or “Mayan Eschatology”, I send the query off three different search engines who pay me a small retainer for the privilege of getting them…. You could imagine an alternative setup in which you send the search terms to the engines and all they come back with is their per-click bid price, and then you only send the actual search to the winner.

It’s certainly true that companies such as Google and Yahoo! pay to be the search provider for popular applications, such as Firefox and Safari. They also pay to be the search provider for popular Web applications, such as AOL and the Washington Post. However, this doesn’t need to apply to big companies.

Another way Google currently pays for traffic is through their AdWords program. Tim’s idea applies just as well (if not better) to any page on the Web that runs contextual ads — which are essentially embedded search results intelligently served-up based on a fancy back-end algorithm. This algorithm guesses what a visitor would have typed into the search box (if only one had existed).

Right now, Google (presumably) has a nice margin between what they charge advertisers (via AdSense) and payout (via AdWords). This makes them a tidy profit.

But Yahoo! and Microsoft have similar products. As a content provider, you’d switch from AdWords if you made more money from the Yahoo! Publisher Network. But right now, your choice of contextual advertisements is all or nothing: you’re either with Yahoo! or Google or Microsoft. You have a few choices, but you’re forced to pick one and stick with them until you switch.

This is quite coarse and inefficient. You’re leaving money on the table because it’s not really a question of whether you make more money overall from one company or another. What you should really care about is whether you can make more money for them (and thus from them) for this specific visitor at this specific instant in time.

Tim’s idea becomes:

[A]n alternative setup in which you send the search terms page URL to the engines and all they come back with is their per-click bid price, and then you only send the actual search to serve contextal ads of the winner.

If Microsoft and Yahoo! want to make a big dent in Google’s AdWords business, they should provide a pricing Web service for contextual advertisers. By providing transparency in the market, they’d commoditize away Google’s demand side, too.

If you were a company that made money from multiple channels and you were facing a large scary competitor that made 99% of its money from advertising, wouldn’t you do your best to erode as much profit margin from advertising as you could?

For this talk, I decided to demonstrate Web services using a number of real Web services, so you can get a flavor for how people are actually implementing Web 2.0. Specifically:

• del.icio.us (bought by Yahoo! earlier this week)
• flickr (bought by Yahoo! earlier this year)
• eBay (almost bought by Yahoo!, almost bought Yahoo!)

And, of course, there’s the obligatory Google Maps reference because, well, it’s a rule or something. I think. I can’t remember.

This was my first presentation done in Keynote. It’s certainly easier to make non-ugly slides in Keynote than in PowerPoint, which almost seems to lead you down the path of ugly slides. You think Microsoft could invest in a few good new templates instead of still using the ones their intern programmers designed in 1992.

If you’re going to be at ApacheCon, let me know. I am arriving Monday afternon and leaving Wednesday night, and staying at the conference hotel. I will have a rental car, so if you know a good place to eat or drink or visit that’s off the beaten path, we can help each other.