Web Services

How eBay Uses Metadata to Enhance Its Web Services

Alan Lewis has a great article up on XML.com on embedding meta-data inside WSDL files.

Since we rev our Web services API every two weeks, we run into versioning problems that aren’t well covered by existing practices. For example, we define a whole set of complexTypes, but those types can morph overtime and we want to maintain backwards compatibility whenever possible. Or, a piece of data may be mandatory in one call, but optional in another.

Seeking a standards-based solution, our documentation team turned to the appInfo element defined as part of XML Schema. It’s quite a nice idea, and also allows us to auto-generate reference documentation from the WSDL file itself. We’d love for other companies who encounter a similar problem to take a similar approach, so that we can pool resources.

If you’re interested, check out Alan’s piece.

ning! double ning!

A very good friend of mine has been working at 24 Hour Laundry for many months now. Unfortunately, I was unable to convince him to share any information about his top secret project. Well, he did say it was “quite interesting,” but that didn’t get me very far.

Therefore, I was quite excited to see Russ blog about the launch of ning.com. Now I could finally see what all the excitement was about. Well, I must agree with David. It certainly is quite interesting.

To quote from the faq: “Ning is a free online service… for people to build and run social applications.” Think of it as a mashup between Jot and, well, something else. Jot is Joe Kraus’s application wiki platform, and something else is the union of flickr + del.icio.us + friendster + Google Maps.

I wanted to blog about it immediately; however, I really didn’t have anything interesting to say. Not that it’s stopped me before, but still. As a rule, I feel that in order to really understand something, you need to get your hands dirty using it. And in this case, I couldn’t just explore the example sites, I actually had to figure out how to build an application on the ning platform.

Fortunately, I know PHP 5 and so does ning.

I did the simplest thing I could possibly do that was still of value to me. I cloned the sample restaurant rating application; extended it to add two fields: airport code and location; and created the “Airport Restaurant Review” guide.

At first, I ran into a little bit of a problem understanding that I had to use the “my” namespace. Data went in fine, but it didn’t come out. Exception city, sweetheart. Sigh.

I must admit, I hate frameworks because I never have any idea how to go about debugging them.

However, I left the application alone for a day and it managed to debug itself. Well, not really, but after I came back it only took me five minutes to give in and actually read the docs to guess the proper fix. All better now.

Speaking of developer docs, I can tell David has really been cranking away on them. There’s *a lot* of material there: tutorials, API references, and full-featured sample applications.

This Airport Restaurant Review guide is actually something I’ve wanted to write for a over a year now, and even managed to convince someone to do for me, but we never launched it. Now it’s up and running in less than an hour of programming time.

Pretty cool.

My next task is to refactor the application to create a separate airport class instead merely adding properties to an AirportRestaurant class. Unfortunately, I’m not quite sure how to migrate the underlying data in the store, so I’m kinda stuck. The key to ning is the ever-evolving nature of the applications, so I’m sure I’m missing something here.

But while the code is necessary part of a social application, it’s not sufficient. You need users and data and a community of people to care about what’s happening inside your application. That’s what makes it social.

In other words, a photo sharing application with 200 photos isn’t flickr, even if it looks the same and has feature parity. In fact, your photo has less value outside of flickr than in, even though it’s the same photo with the same set of tags.

So I guess one of the questions on my mind is “is there room for more than one flickr on the Web?”

Some types of sites lend themselves to a “the rich get richer” design: marketplaces (eBay), payments (PayPal), communications (Skype), IM (AIM, Y! Messenger, ICQ).

Others do not: search (Y!, Google, MSN, Ask), e-mail (Y!, Gmail, Hotmail, ISPs), content (Y!, Google News, MSNBC, blogs, newspapers, etc.)

Do social networks fall into the first category? Or are people perfectly happy forming their own little social cliques? Dunno.

I suspect we’ll end up with something similar to IM. There are multiple good-sized networks that have a clear advantage due to their size, but not necessarily a “winner take all” outcome, such as what Microsoft’s done with Windows and Office.

One of the advantages of ning is a shared data store, so people can remix data from other applications into their own. On the face of things, this looks like an excellent way to aggregate data across multiple micro applications into a unified global database of socially created content — exactly what a social network needs to become powerful.

However, you permission to other people’s data is read-only, so I can’t actually propagate my Airport Restaurants reviews back into the main set of core Restaurant reviews.

This leads to data fragmentation, which I think undermines the benefit of a social network.

It’d be great if I could request write access to someone’s data store. If a AirportRestaurant isa Restaurant (or aggregates a Restaurant), then everything should map correctly.

This certainly complicates matters — if you think refactoring was hard before, how do you coordinate it now? — but adds a great deal of value from my point of view.

I’m pretty sure ning will continue to evolve, and I’m quite interested to see where this quite interesting platform goes in the next few months.

The business case for mashups

I’m speaking at Web 2.0 today on Mash-ups 2.0: Where’s the Business Model?. It should be fun.

BTW, Web 2.0 is jam packed with people. Sessions are SRO. This is the second conference I went to in the past month (Microsoft PDC being the other) that sold out. I’m hoping this is a sign that the tech industry is on an upswing.

PS: I promise I will blog about something other than when I’m speaking real soon. I have some thoughts surrounding Web services and mashups that I’m going to try out in the panel. Nothing brilliant, but some thoughts I had from working on a little application. If I don’t get shouted down, I’m going to put them out there.

Jon Udell on SOA

Jon Udell wrote a nice piece in Infoworld on SOA (aka Web services), where he talks about the SOAP vs. REST camps and also investigates some of the new Web Services standards (WS-*) that Microsoft and IBM have been pushing. (BTW, I consider eBay’s “XML API” to be more similar to REST than SOAP.)

It’s a good business-level overview of what’s going on in Web services, and includes a few business examples of people using WS-* in the wild. (i.e. actual deployed programs, not made up “proof of concepts.”)

Additionally, eBay Web services gets a small mention at the top of page 2:

Today’s most visible exemplars of WS-Lite — Amazon and eBay — use Web services in a point-to-point way. In that mode there’s not much difference between SOAP/WSDL APIs and REST APIs, so it’s not surprising that developers who work with these platforms overwhelmingly prefer the REST flavor.

Jon blogged about the article, where he provides some background on the piece. His post include a link to his excellent Bill Gates interview from last week’s Microsoft PDC.

Zend/PHP Conference Coming Soon!

I just checked my calendar and was shocked to discover the Zend/PHP Conference is just around the corner — October 18-21.

I’ve been part of a team of people from the PHP community working for months with Zend (including Zeev and Andi) to help put together a great business-focused PHP conference. I’m really excited for this show.

There are lots of great PHP conferences, but this is the only one emphasizing PHP inside large and medium sized business. There’s still great technical content (Wez on PDO, Chris Shiflett on Security, Andrei and Andi on Unicode, George on Building Scalable PHP Applications). However, I’ve already heard those talks before. :)

What I’m going to attend are all the new talks. Such as, John Allspaw from Yahoo! on their LAMP hardware layout, John Roberts from SugarCRM on building a commercial business on top of an open source application, and Jeff Winner from Propel on Managing Development and Operations Together.

Of course, I’m also going to the Empowering eBay Research with PHP and Web Services session. I’m doing the intro, and then quickly getting out of the way for Andrew. He founded Terapeak. Terapeak is providing some amazing data analysis tools that crunch the gadzillion mega-bytes of active and closed listing data to help eBay sellers (among others) determine the state of eBay and the economy. There’s no better way to see buying and selling trends that to look at our data, and Terapeak is all based on PHP.

Come to the show. When I’m not in a session, I’ll be at the eBay/PayPal booth in the exhibit hall, and I’d love to chat.

Adam’s Fall Conference Schedule

Now that’s I’m back from FOO Camp, I’m checking my calendar to see what conference events I have coming up. For those of you keeping track (hi mom!), here they are:

  • Microsoft PDC: September 13-16 in LA. I haven’t managed to wrangle a chance to speak yet. :)
  • Zend/PHP Conference: October 18-21 in Silicon Valley. Speaking about eBay Web services. Final talk TBD.
  • ApacheCon US 2005: December 10-14 in San Diego. Speaking on “Consuming Web Services Using PHP 5”.

I may also go to Web 2.0 and the 4D Summit, but I have yet to finalize those events.

The ApacheCon talk should be quite cool, as I will be showing off nifty real-world web services examples. For example, sucking del.icio.us RSS bookmarks into a WordPress blog, or mashing up eBay Web services search results with the Google mapping API. Here’s the full abstract:

As we move into the world of Web 2.0, PHP developers must increasing include Web services in their toolkit of skills. This session covers how to implement REST and SOAP clients using the latest PHP 5 extensions, such as ext/soap, SimpleXML, and xsl.

This is not an academic talk discussing theory and specifications. Examples show applications of popular Web services, including del.icio.us, eBay, and Google Maps.

Don’t be left behind. Come to this session and learn how to integrate Web services into your code.

I have most of the code already written in various places, so there’s only the problem of creating the slides. Once I know more about my other talks, I will pimp them here, so stay tuned for all the details.

How ext/soap converts SOAP data structures to PHP

I haven’t seen any documentation on how ext/soap converts SOAP results to PHP data structures, so I thought I would make a quick blog entry. If I get time, I will try to make this all pretty, confirm I am actually correct, and add it to the official docs.

Normally, the conversion is simple:

Adam

Becomes an attribute of “Person” and a value of “Adam” under a stdClass instance.

stdClass Object
(
[Person] => Adam
)

Attributes are also converted to properties:

stdClass Object
(
[gender] => male
)

When there is both text and attributes, the text is mapped to a property name of _ (underscore).

stdClass Object
(
[Person] => stdClass Object
(
[_] => Adam
[gender] => male
)

)

Of all the bits that might trip you up, this underscore mapping bit is it, as I don’t know where this came from. :)

However, it’s only done when there is straight-up text. If there is a nested XML element, then this doesn’t happen, as in:

Adam

This becomes:

stdClass Object
(
[Person] => stdClass Object
(
[Name] => Adam
[gender] => male
)

)

That’s all for now. The OSCON WiFi connection is super slow.

eBay SOAP using PHP 5 ext/soap and AJAX

I’ve been playing around with a nifty skunkworks project using our SOAP API, the ext/soap extension in PHP 5, and the fad du jour of AJAX.

Progress has been moving a long at a nice clip, as long as I can find time to work on it. One of my main goals is to more extensively test the compatibility between ext/soap and eBay’s SOAP interface. With the release of PHP 5.0.4, you can finally make SOAP API calls to our servers. I’d done enough to confirm this, but I’d done minimal testing to verify if this applies to all 100+ API calls. Now I’m trying to throw some real-world tests at the extension.

The bad news is that I’ve found a bug. The good news (for PHP) is that the bug is on eBay’s end. :)

The eBay SOAP API is almost 18 months old, but historically many of our developers have used our legacy XML API, which is based on a different codebase. That older codebase has extensive burn in and testing. The newer SOAP codebase is still well-tested (we have over 10 dedicated API QA engineers in addition to the many millions of API calls made by developers), but there are some dark little corners that are not as well-lit.

It turns out I stumbled down one early this morning involving our Motors site. Apparently, according to our SOAP API, the car I’m searching for has 154242 miles and, by an odd twist of fate, was also released in the year 154242. Heh.

We’re in the process of a giant migration, where all the legacy XML API developers move over to share the new SOAP API codebase. (You don’t need to move over to SOAP as part of this transition, but the data schema is changing to one that’s more well-thought out and coherant.) I suspect as we close in on the cutoff date of next June, we’ll run into a few more of these buglets. This is why we’re encouraging people to start porting sooner rather than later.

Unfortunately, when you expose so much data in so many different ways off of a dynamic database driven site, you’re never going to be able to find everything ahead of time. Fortunately, with more eyeballs, all bugs are shallow, and I know our API team will roll out the fixes in no time.

On the AJAX side, JavaScript is a surprisingly cool language. Its always gotten a bad rap, but I believe that’s more due to browser incompatibility issues than fundamental core design problems. For example, in this project I’m using a function closure within a function closure. How zany is that?

Much magic indeed.