Forum » Development Discussion

Request for feedback on the next version of the Scrobble API

 
  • Request for feedback on the next version of the Scrobble API

    It's been over a year since we asked developers what they would like to see in a new version of the Scrobble API (which you may know as the "submissions protocol").

    We have taken all of that feedback on board, as well as the many e-mails that have been sent to us directly where developers have raised
    certain issues they have with the current API and given us ideas for new features. We are proud to finally present a first draft version of what this new
    API might look like and how it might work.

    We want to stress that this is in no way final or complete. We are releasing it as a "request for comment" from you, our developer community, as you have the most experience implementing our APIs and will be the ones writing new applications or updating your existing ones to use the new API.

    So, without further ado, here is a list of proposed changes and new features for the new version of the scrobble API:

    • As hinted at above, the somewhat confusing "submissions protocol" will be renamed to the "Scrobble API" as it's all about scrobbles.
    • The Scrobble API will become part of the Last.fm Web Services API under a new "Scrobble" package. We hope that this simplifies development by having *one* unified authentication, request and response mechanism for all our Web Services. We also hope that this will lead to applications which currently just scrobble to doing more with the rest of our API and vice versa with the end result being cooler apps with more features. This does mean the API will change significantly but we think the more logical method and parameter names make things much more intuitive. This also means the Scrobble API will send XML or JSON responses just like the rest of our webservices so you can do parsing and conversion of responses in a number of different ways. We will keep these small and simple so you can still treat the responses like a string and grep for "failed" or "ok" values without needing an XML parser if you so choose.
    • Moving to the web services API means that there will no longer be the requirement for a separate client id. Instead you will need to create a Last.fm web services account and can then start scrobbling right away, without the need for the manual "can I please have a client id" e-mail step. We will expand the information linked to a web services account to allow developers to specify their scrobble application name, associated web address and, wait for it, their own custom scrobbler icons which will be displayed next to the "Now listening using..." message on a user's profile. This was our second most requested feature so we're happy to be able to add this and do away with the generic grey Last.fm scrobbler icon that a lot of you have had to put up with.
    • By associating all scrobbles with a web services API key we can do improved tracking on the backend and in future we hope to use this to provide you with information like how many scrobbles we received using your application, create charts of the most popular scrobblers, compare musical tastes of people using different scrobblers and so on.
    • We will be adding the most requested feature from developers.... more detailed error messages! We will return more information in a response indicating why a scrobble failed and we will also inform you if a scrobble was ignored (instead of returning "OK" and saying nothing more like we currently do). This should ease debugging of errors significantly.
    • We will add corrections information to the response where relevant so you can inform the users of your applications to fix their tags (or you can just ignore this information).
    • We will remove loving and banning of tracks from the Scrobble API so that developers only have to call the webservice methods Track.love and Track.ban instead of having to perform both calls as is the case now. In a similar vein skipping tracks will be moved from the Scrobble API to a web services Track.skip method.
    • For those of you who are using our radio streaming API, we will also be changing the XSPF playlist so that it contains a "stream id" for each song. This should then be sent through to the Scrobble API and Track.love, Track.ban and Track.skip calls related to that track. This will enable us to associate all these different pieces of information with a specific track recommendation that we made which in turn will help us to make even better recommendations in future.
    • We will be adding the (optional) ability to send the album artist in cases where this is not be the same as the track artist.
    • We will make the track duration parameter optional as not all scrobble applications have access to this information.
    • Sending single scrobbles will be a different API call to sending batched scrobbles. Developers who want to write a simple scrobbler quickly can just go for the single scrobble method without having to deal with the (slightly) more complicated batched scrobble call (which is still the preferred method for sending bulked scrobbles in cases where they have been cached etc.)

    That's quite a list of changes but of course there were also a lot of great ideas which haven't made the cut but with this new version we are reserving the ability to add new features to the API. We will do this by adding extra parameters or methods to the API but these will always be optional and will be done in such a way that backward compatability with existing clients is guaranteed. We'd also like to point out that we won't be dropping support for the 1.2 protocol anytime soon, so your existing applications will work for a number of years to come without requiring any changes.

    After a round of feedback from you (which will hopefully not take longer than a month or two) we plan to release a beta version of this API for those of you keen on trying it out and giving us more feedback and helping us find bugs. We will be actively following the discussion of this thread and will regularly update this forum to let you know how we are progressing. Below is the full proposed API documentation in all it's technical detail, so grab a cup of tea,
    have a good read and let us know what you think.

    Scrobble API Documentation (draft)

    Response Handling
    HTTP Status Codes
    All scrobble methods will return one of 3 http status codes:
    • 200 OK. This is returned by successful requests. It is also returned when a request is filtered and rejected by the server, in which case resubmitting the request will not help.
    • 400 BAD_REQUEST. The is an error in the request. This include missing parameters and bad authorisation. The client SHOULD NOT repeat the request without modifications.
    • 503 SERVICE_UNAVAILABLE. This is a hard failure on the server side, the client should try again later. If three hard failures events occur consecutively, the client should re-authenticate.

    Caching
    It is recommended that the client hold submissions in a local queue if the submission process fails with a hard failure, since the server connectivity may be variable (either because of network outage, or server failure). This cache should be retained over client restarts, allowing the user to close the client and restart later without losing their scrobbled tracks. The cached tracks must be submitted in chronological order according to the time when they were listened to and before any new tracks are submitted.

    Scrobble.nowPlaying
    Params
    • artist (required): The artist name.
    • track (required): The track name.
    • album (optional): The album name.
    • albumArtist (optional): The album artist, if this differs from the track artist.
    • duration (optional): The length of the track in seconds.
    • mbid (optional): The MusicBrainz Track ID.
    • sk (required): A session key generated by authenticating a user via the authentication protocol.
    • api_key (required): A Last.fm API key.
    • api_sig (required): A Last.fm method signature. See authentication for more information.

    Auth
    This service requires authentication. Please see our authentication how-to. This is a write service and must be accessed with an HTTP POST request. All parameters should be sent in the POST body, including the 'method' parameter. See rest requests for more information.
    Errors
    • 2 : Invalid service - This service does not exist
    • 3 : Invalid Method - No method with that name in this package
    • 4 : Authentication Failed - You do not have permissions to access the service
    • 6 : Invalid parameters - Your request is missing a required parameter
    • 9 : Invalid session key - Please re-authenticate
    • 10 : Invalid API key - You must be granted a valid key by last.fm
    • 11 : Service Offline - This service is temporarily offline. Try again later.
    • 13 : Invalid method signature supplied

    • These error codes are subject to change.
    • 113 : Artist name did not pass filters.
    • 114 : Track name did not pass filters.
    • 115 : Album name did not pass filters.

    Note: Errors indicating a request did not pass filters, will return a 200 OK. Do not resubmit the request.
    Example success response
    <lfm status="ok">
    <artist>The Very Best</artist>
    <album>Warm Heart of Africa</album>
    <track>Julia</track>
    </lfm>
    Example failed response
    <lfm status="failed">
    <error code="6">Missing required parameter artist</error>
    </lfm>
    A note about corrections
    Responses can contain meta data corrections. If Last.fm has a correction for the submitted artist, album or track, the nowPlaying submission will still be recorded under the submitted names, but the response will contain the corrected names. Eg: if a user submits artist=Bjork, album=Post track=Isobel, the response will be:
    <lfm status="ok">
    <artist>Björk</artist>
    <album>Post</album>
    <track>Isobel</track>
    </lfm>

    A client application could optionally suggest to correct the id3 tags in a users library. As corrections are not perfect, we would suggest that the user be asked if they want the correction to be applied, rather then doing automatic corrections. The scrobble will still be recorded under the submitted names.

    Scrobble.submit
    Params
    • artist (required): The artist name.
    • track (required): The track name.
    • album (optional): The album name.
    • albumArtist (optional): The album artist, if this differs from the track artist.
    • timestamp (required): The time the track started playing, in UNIX timestamp format (integer number of seconds since 00:00:00, January 1st 1970 UTC). This must be in the UTC time zone.
    • duration (optional): The length of the track in seconds.
    • mbid (optional): The MusicBrainz Track ID.
    • streamid (optional): The stream id for this track received from the radio.getPlaylist service.
    • streamauth (optional): The 5-digit Last.fm recommendation key.
    • sk (required): A session key generated by authenticating a user via the authentication protocol.
    • api_key (required): A Last.fm API key.
    • api_sig (required): A Last.fm method signature. See authentication for more information.

    Auth
    This service requires authentication. Please see our authentication how-to.
    This is a write service and must be accessed with an HTTP POST request. All parameters should be sent in the POST body, including the 'method' parameter. See rest requests for more information.
    Errors
    • 2 : Invalid service -This service does not exist
    • 3 : Invalid Method - No method with that name in this package
    • 4 : Authentication Failed - You do not have permissions to access the service
    • 6 : Invalid parameters - Your request is missing a required parameter
    • 9 : Invalid session key - Please re-authenticate
    • 10 : Invalid API key - You must be granted a valid key by last.fm
    • 11 : Service Offline - This service is temporarily offline. Try again later.
    • 13 : Invalid method signature supplied

    • These error codes are subject to change.
    • 113 : Artist name did not pass filters.
    • 114 : Track name did not pass filters.
    • 115 : Album name did not pass filters.
    • 116 : Timestamp did not pass filters.
    • 117 : Duplicate timestamp received.
    • 118 : Max daily scrobbles exceeded.
    • 119 : Stream auth was invalid.

    Note: Errors indicating a request did not pass filters, will return a 200 OK. Do not resubmit the request.
    Example success response
    <lfm status="ok">
    <artist>The Very Best</artist>
    <album>Warm Heart of Africa</album>
    <track>Julia</track>
    </lfm>
    Example failed response
    <lfm status="failed">
    <error code="6">Missing required parameter artist</error>
    </lfm>
    A note about corrections
    Responses can contain meta data corrections. If Last.fm has a correction for the submitted artist, album or track, the scrobble will still be recorded under the submitted names, but the response will contain the corrected names. Eg: if a user submits artist=Bjork, album=Post, track=Isobel, the response will be:
    <lfm status="ok">
    <artist>Björk</artist>
    <album>Post</album>
    <track>Isobel</track>
    </lfm>

    A client application could optionally suggest to correct the id3 tags in a users library. As corrections are not perfect, we would suggest that the user be asked if they want the correction to be applied, rather then doing automatic corrections. The scrobble will still be recorded under the submitted names.

    Scrobble.submitBatch
    Submit a number of scrobbles in one POST request. This should be used when a number of scrobbles have been cached. Multiple scrobbles may be submitted by indexing each scrobble's parameters with an increasing integer from 0 up to a maximum of 49. The tracks must be submitted in chronological order according to when they were listened to (i.e. the track identified by timestamp[0] must have been played before the track identified by timestamp[1]).
    Params
    These parameters must be specified for each scrobble, where i is an integer indexing the scrobble.
    • artist (required): The artist name.
    • track (required): The track name.
    • album (optional): The album name.
    • albumArtist (optional): The album artist, if this differs from the track artist.
    • timestamp (required): The time the track started playing, in UNIX timestamp format (integer number of seconds since 00:00:00, January 1st 1970 UTC). This must be in the UTC time zone.
    • duration (optional): The length of the track in seconds.
    • mbid (optional): The MusicBrainz Track ID.
    • streamid (optional): The stream id for this track received from the radio.getPlaylist service.
    • streamauth (optional): The 5-digit Last.fm recommendation key.
      These parameters need only be specified once.
    • sk (required): A session key generated by authenticating a user via the authentication protocol.
    • api_key (required): A Last.fm API key.
    • api_sig (required): A Last.fm method signature. See authentication for more information.

    Auth
    This service requires authentication. Please see our authentication how-to. This is a write service and must be accessed with an HTTP POST request. All parameters should be sent in the POST body, including the 'method' parameter. See rest requests for more information.
    Errors
    • 2 : Invalid service -This service does not exist
    • 3 : Invalid Method - No method with that name in this package
    • 4 : Authentication Failed - You do not have permissions to access the service
    • 6 : Invalid parameters - Your request is missing a required parameter
    • 9 : Invalid session key - Please re-authenticate
    • 10 : Invalid API key - You must be granted a valid key by last.fm
    • 11 : Service Offline - This service is temporarily offline. Try again later.
    • 13 : Invalid method signature supplied

    Example success response
    The submit batch method does not return information for each scrobble, the response provides a summary of how many scrobbles were accepted and how many scrobbles were ignored. If scrobbles were ignored due to filtering, the batch should not be resubmitted.
    <lfm status="ok">
    <scrobbled>15</scrobbled>
    <ignored>0</ignored>
    </lfm>
    Example failed response
    If any of the submitted scrobbles result in BAD_REQUEST the whole batch is rejected.
    <lfm status="failed">
    <error code="6">Missing required parameter artist</error>
    </lfm>

    bleh
    Edited by roserpens on 18 Mar 2010, 16:58
    • Russ said...
    • Alumni
    • 18 Mar 2010, 16:35
    That's all top-notch.

    I would possibly suggest you implement the batch submit as an XML post as opposed to a form-style request. Then you could get rid of those ugly array-style indexes. Thus:


    <submission>
    <scrobble>
    <artist>Artist 1</artist>
    <album>Album 1</album>
    <track>Track 1</track>
    </scrobble>
    <scrobble>
    <artist>Artist 2</artist>
    <album>Album 2</album>
    <track>Track 2</track>
    </scrobble>
    </submission>


    Which raises the question: how different is this from the bootstrap API, and could you merge it entirely?

    (If you do this it would be worth adding a strong recommendation to gzip the request.)

  • the question why Scrobble and not just Track floats through my mind but then i might grab a cup of tea first.

    (Hi, Russ :)

    • dougma said...
    • Alumni
    • 19 Mar 2010, 05:44

    Re: Request for feedback on the next version of the Scrobble API

    Nice work!

    roserpens said:
    Caching
    The cached tracks must be submitted in chronological order according to the time when they were listened to and before any new tracks are submitted.


    What happens if I can't do this? Like, I might have multiple devices in different states of connectedness... so my scrobbles _are_ going to be submitted out of order. But I don't want to lose them!

    • [Deleted user] said...
    • User
    • 19 Mar 2010, 14:57

    Scrobbling on iphone Spotify app

    Is it possible? Will it be made possible in the future?

    • julians said...
    • Alumni
    • 19 Mar 2010, 15:08

    Geo-tagging scrobbles

    Have you thought about the possibility of geo-tagging scrobbles in some way?

  • @HodgeStar - interesting thought, we'll discuss that as an option. A few other decisions still need to be made regarding how this will all integrate into the rest
    of the API which will influence this. So that would affect the package but the rest of the docs above would stay the same.
    @Russ - those array style indeces aren't great I agree, but the XML option would be totally different to all other calls so we've gone for consistency, however bringing
    bootstraps into consideration might change things. Will have think, thanks for the input.
    @dougma - The current API requires scrobbles in order so this isn't something that has changed. We had quite a few discussions about this and decided to leave it as
    it was but maybe we will rethink it before we release this.
    @kjblatherwick - it most certainly is possible but you will have to ask the kind people at Spotify to add that to their app. Let them know!
    @julians - that was considered but for now we're not entirely sure what we would do with geo tags and, due to the volume of data, we would rather not gather anything
    unless we've got a solid reason for it. Having said that, with this version of the API we're reserving the right to add extra, optional fields in the future so this
    is something we could easily add should we decide to.

    • Russ said...
    • Alumni
    • 19 Mar 2010, 16:14
    As I recall, we were going to remove the scrobble-in-order requirement completely. There's no technical reason for it to exist any more, and it just complicates historical scrobbling. I think there are better ways of doing abuse prevention.

    • mxcl said...
    • Alumni
    • 19 Mar 2010, 16:51
    Being able to send scrobbles individually and batched is inspired. Good call.

    Adding corrections information is great. Great plan.

    Track.skip seems a bad choice of namespace relative to Radio.skip. Although I admit that ban only applies to radio, so it is a continuation of an existing bad choice… :P However ban is the in the same category as love. While skip would feel more natural under the Radio namespace IMO.

    I agree with Russ regarding the batch submit using XML. The code to iterate the numbers and form encode correctly was always fiddly to write. And XML is more readable than form requests. But I also agree that continuity is important. Balancing act. If it was my choice I'd side with the more human readable data that is also easier to write.

    I agree with Doug that you will get caches submitted after other scrobbles. Real Last.fm users scrobble at home, work and on their mobiles. It is inevitable that caches will be submitted out of chronological order.

    Overall looks great, good work. Can't wait to try it out! I'll race you, whoever releases their next product first wins?

  • Differentiating between media types

    I'd love to see some podcast integration from last.fm and it seems that this would need to start by identifying "tracks" as they come in as podcast episodes rather than music tracks. If you were correctly identifying the source (and feed!?) of the track when scrobbling from iTunes or elsewhere then you'd be able to track independant podcast episode/feed popularity etc.

  • Haven't read through, but I wanted to throw out a couple of quick notes. I'd make a few changes to accommodate future requirements change should the need arise.

    First, eliminate hard-coded "mbid" and replace it with key/val pair "extidtype" and "extid." Extidtype can be a numeric identifying external id source e.g. 1 = MusicBrainz; 2 = Discogs; 3 = Echonest, etc. For now, obviously, the scrobbler will only support MusicBrainz type. Yet, if (when) change occurs, the API to support such a change is already in place.

    Second, add in support for extended data via inclusion of "extdatatype" and "extdata" key/val attribs. Doing so allows inclusion of additional features over time. For instance, you might decide to enable something akin to "scrobble to tag" allowing auto-tagging during scrobble. Point is, structure is already in place to add in additional key/val data/functionality, and there's a structural fall-back mechanism in case extended data suddenly becomes a necessity.

    Again, haven't read through, but "something" akin to a client/session key appears necessary to support multiple client scrobble during failure recovery. Cached chron order requirement without accommodating multiple client scrobble appears problematic.

  • Re: chron order requirement

    I think that refers only to the tracks in one cache. That makes sense and there's no problem implementing it. We already can submit older scrobbles from a different cache.

    • Ganymed said...
    • User
    • 19 Mar 2010, 19:32
    Very nice, can't wait to get my hands on this.

    However, I'd love to see the submissions protocol supporting scrobbles with the same timestamp, e.g., scrobbles that stem from an iPod playcount import where only the last played timestamp is available along with the playcount. From what I see in the proposed API, I'd get an error 117 in such cases.

    Ideally, the site would detect those scrobbles and list them in a special way in the profile (e.g., <track> * <playcount> (<last played>)) -- but this is only a raw idea.

    What's important is, that there'll be a supported and official way to submit those scrobbles and get them counted (which sadly is currently only possible by the official client which uses an undocumented part of the current protocol).

  • re. scrobbling with the same timestamp:

    What about tracking the device where the track was scrobbled? This should make the decision rather easy, or?
    This could also be made anonymous (e.g. numbers for different devices) when submitting the data to last.fm server.

    believe in music to survive
  • Will it be possible to scrobble a streaming radio which is not from last.fm, like a shputcast radio??

    • mxcl said...
    • Alumni
    • 21 Mar 2010, 04:01
    karlos_rocks said:
    Will it be possible to scrobble a streaming radio which is not from last.fm, like a shputcast radio??
    This is already possible if you set the right source. R iirc.

    Edited by mxcl on 21 Mar 2010, 04:02
    • mxcl said...
    • Alumni
    • 21 Mar 2010, 04:02

    Re: Scrobbling on iphone Spotify app

    kjblatherwick said:
    Is it possible? Will it be made possible in the future?
    It's up to Spotify, isn't it?

    • toc-rox said...
    • User
    • 21 Mar 2010, 18:31
    Timestamp: Accept additional date/time formats like ISO 8601 and not only Unix epoch time.

    • tburny said...
    • Forum Moderator
    • 21 Mar 2010, 19:59
    This is very great news!
    But I partly disagree on the XML-submit thing. Althoug XML is really nice, you could leave the old model if doing so, as like others mentioned before this would not be very "api style". Plus there is already XML-Rpc for the current API which should be suitable enough(although there's a bit of overhead) and would not break the request model/compatibility to the other api calls.
    I as a developer would really love it if there would be no more extra extra exceptionally stuff for scrobbling api :)

    Combo.fm: Combine your favourite radio stations! | My Blog | scala-lastfmapi | Cache2k - A high performance Java in-memory cache
    P.S.: Do not click here
    throw new PokemonException(); //Gotta catch 'em all
    My forum post reflects my personal opinion :)
    • mxcl said...
    • Alumni
    • 21 Mar 2010, 20:23
    toc-rox said:
    Timestamp: Accept additional date/time formats like ISO 8601 and not only Unix epoch time.
    meh. Converting times to epoch is easy. I don't see why Last.fm have to waste their efforts, especially considering how many other formats there are.

  • Is the provided timestamp (play start time) guaranteed to match up to the "date uts" attrib returned by the user.getRecentTracks call?

    Probably already the case (forgive my ignorance of the existing imp), just looking for confirmation.

    • ampjuke said...
    • User
    • 22 Mar 2010, 11:24
    Great news! Hope the beta is ready soon.

  • I'm against Russ regarding the multi-track scrobbing. I prefer the array indexes. Please remember that not all scrobble clients are written in WWW friendly languages. For example I have a C client. Generating XML would be a real PITA. https://sourceforge.net/projects/liblastfm/

    Regarding the authentication - As long as there is a provision to write scrobble middlewear (like my library), the fewer logins the better, the fewer accounts needed, the better.

    • segin said...
    • User
    • 22 Mar 2010, 18:30
    toc-rox said:
    Timestamp: Accept additional date/time formats like ISO 8601 and not only Unix epoch time.


    @toc-rox
    Do you even know why Unix epoch time is used? Most computer systems and software internally uses Unix epoch time as a simple integer counter. Using simple printf/scanf or atoi/itoa conversion back and forth is far more simpler than using a rather crappy standard such as ISO 8601.

    When dealing with data not intended for human consumption, use a data format that is easiest for computers themselves to handle.

    @roserpens
    Also, of good use, would be an example scrobbler, and an example of an XML exchange between client and server. I never used the webservices API, instead only using the scrobbler API.

    • adammw said...
    • User
    • 23 Mar 2010, 06:00
    For NowPlaying method, how about also including a startTime (optional) parameter so you can work out how much of the song has been listed to thus far?

Anonymous users may not post messages. Please log in or create an account to post in the forums.