The best solution so far...

 
    • Phage said...
    • User
    • 24 Jan 2010, 17:34

    The best solution so far...

    Hi All,

    This problem has been bothering me for some time now! I listen to a range of artists from grindcore(songs rarely longer than a few mins) to progressive music where songs can be 20+mins long and so my charts are becoming rather distorted over time :(

    Last.fm has the ability to fix this, in their database, tracks store their duration in milliseconds. Although I've found a few slight issues with this. Last.fm doesn't store some of the track lengths, my script will create a list of these, 800ish of my tracks don't have a length on last.fm these are then looked up in the Musicbrainz database. Also more worryingly is that some track have the wrong length, Imperivm by Ictus is an example of this I've found, last.fm says the track is 14 mins long when it's actually 39mins long, I don't know how to fix this yet...

    Importantly my script will fix a problem I see with alot of these other solutions
    that have been appearing, mine doesn't use any kind of averaging for the track
    length, it uses the real lengths to come to a (hopefully) more accurate answer.
    Averages don't cut it for me :/

    Also my script will do this calculation for all your plays ever, not just the last 12
    months, important when you have 5 years of scrobbles!

    Things that my method will not overcome:
    * Last.fm don't scrobble tracks below 30secs (a problem for grindcore)
    * Last.fm stores no data on how long a track was listened to for

    To work on in the future:
    * Produce charts that look like last.fm's / allow easy embedding into a last.fm
    profile
    * Store the downloaded dataset in a local database.
    * Make it more user friendly!
    * Add in some filtering so script can do last 3/6/12 months etc

    It's rather work in progress at the moment, probably not very user friendly to get
    working either, but if you'd like to use it this is what you need to do....

    You need to get an API account from last.fm here: http://www.last.fm/api/account
    - The reason for this is because the calls needed for this script require an "API
    key", this is because they pull back alot of data and last.fm don't like people who arn't making them money pulling too much information from their database. And they apparently require you to cache all data locally for a week, I'm not doing this yet, so don't run this script more than once a week unless you want to potentially get locked out!

    relevant snippet from the legal stuff:
    "4.4

    You will implement suitable caching in accordance with the HTTP headers sent with web service responses. You will not make more than 5 requests per originating IP address per second, averaged over a 5 minute period, without prior written consent. You agree to cache similar artist and any chart data (top tracks, top artists, top albums) for a minimum of one week."

    1. Apply for a last.fm API account
    - At this point I should probably tell you to read the Terms Of Service at here
    (http://www.last.fm/api/tos) as by using the last.fm web service you are bound by them.

    2. Download the script here: http://www.mediafire.com/?113iide5cm9d0oj

    3. Install ruby from the one click installer:
    http://rubyforge.org/projects/rubyinstaller/

    4. Put your API key into settings.rb, the line beginning
    "api_key="

    5. Run the script from the command line, move to the directory the script is in then run something like this... c:\ruby19\bin\ruby.exe lastfm.rb "Phage" (or your
    username)

    6. It should take a few minutes to run, once finished results should appear in
    results.txt in the same directory as the script, any tracks with a length of 0 in the
    last.fm database will be output to errors.txt

    7. If you get an error, post it here, obviously errors boiling down to particular
    machine configurations are unlikely to be fixed :( Obviously I can't spent a lot of time fixing errors on this but I'll give it a shot.

    8. Tracks that couldn't be found on Musicbrainz are output into the mb_errors.txt
    file, it's probably worth checking your tags against what's in the MB database, or if the track isn't in it, think about submitting it to MB. (currently you need to empty
    this file each time the script runs)

    The script is probably a little rough around the edges at the moment, and written mainly for my own benefit, but i thought it was worth sharing, constructive criticism is very welcome! There isn't much error handling at the moment, but it ran for long enough to get all 232 pages (50 tracks per page) of all the plays in my library.

    if you improve something, please let me know :)

    Phage

    Edit - put the files back up on mediafire, also made some small changes. Have also started work on getting the results put into a DB.

    Edited by Phage on 12 Dec 2010, 17:48
    • Phage said...
    • User
    • 31 Jan 2010, 18:12
    A new version can be downloaded here:
    http://www.sendspace.com/file/8syvv4

    Improvements
    * Now correctly escaping special characters and accents in track, artists and album names when building URI to request information from last.fm and Musicbrainz

    * Output files are now have the date appended to the front

    * Some improved error handling

  • iScrobbler has a feature called Local Charts that keeps track of time length of all scrobbled tracks.

    My example is that Godspeed You! Black Emperor is #33 on my tracks played list, but #1 in time by almost a full day...

    • Phage said...
    • User
    • 23 May 2010, 20:04
    if I installed and used that now, would it calculate these local charts for all the songs I've listened to in the past?

  • i don't know honestly...

    i bought a new computer about a year ago and it only has the songs/tracks i've listened to since then. but that could be because i lost all of my play counts when I switched over to my new computer. It may take your play count information and start it up from there...

    sorry i've got no real answer for you there...

    • preater said...
    • User
    • 31 May 2010, 11:09
    I wanted to try this out but it looks like it has now disappeared from sendspace.com. Could you upload it somewhere else? Pastebin is nice for scripts / code.

    Thanks.

    • Phage said...
    • User
    • 12 Jun 2010, 20:07
    I uploaded it again here:
    http://www.sendspace.com/file/2prol2

    I could whack it on pastebin but it's split over a few files and i didn't see how I could do that (with a quick look) on pastebin. I had the zip from last time around so i just put it up on sendspace!

    • Voffe said...
    • User
    • 25 Nov 2010, 16:53
    Any chance to reaupload the script somewhere as it has disappeared from sendspace again?

    • Phage said...
    • User
    • 12 Dec 2010, 16:00
    The files should be up on mediafire for at least a while now!

    • Voffe said...
    • User
    • 12 Dec 2010, 16:50
    Hi.
    Thanks for reposting the script. However, as I tried to run it, I got an error message right in the beginning...

    I tried to run it on my own system with Windows XP and it gave an error:
    <internal:lib/rubygems/custom_require>:29:in `require': no such file to load --
    lfmdb (LoadError)
    from <internal:lib/rubygems/custom_require>:29:in `require'
    from lastfm.rb:5:in `<main>'

    After that I tried it on a unix shell I have access to, to see if the problem was with my Ruby installation or something, but it produced another error:

    ./lfmdb.rb:1:in `require': ./settings.rb:3: syntax error, unexpected tIDENTIFIER , expecting $end (SyntaxError)
    from ./lfmdb.rb:1
    from lastfm.rb:5:in `require'
    from lastfm.rb:5

    Am I doing something wrong? I applied for an API account which last.fm said would be active right away, pasted the API keys to settings.rb and run "ruby lastfm.rb "Voffe"".

    • Phage said...
    • User
    • 12 Dec 2010, 17:47
    I think something has changed recently in Ruby (don't think I've kept very upto date with it) and now the local directory isn't included in the include path. I have now fixed this and reuploaded the file.

    • Voffe said...
    • User
    • 15 Dec 2010, 16:26
    Now I managed to get the script running but still have some problems. On my Windows system it seems to run like a charm for the first four pages of tracks I've been listening for most, but then gives a segmentation fault. Maybe it's a bug in ruby(?) or something. I'll try later with a different version of Ruby or something...

    Here's the error message anyhow, if it helps anything:

    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/source.rb:116: [BUG] Segmentation
    fault
    ruby 1.9.2p0 (2010-08-18) [i386-mingw32]

    -- control frame ----------
    c:0015 p:0014 s:0100 b:0100 l:000099 d:000099 METHOD F:/Program Files/Ruby192/li
    b/ruby/1.9.1/rexml/source.rb:116
    c:0014 p:0010 s:0097 b:0097 l:000096 d:000096 METHOD F:/Program Files/Ruby192/li
    b/ruby/1.9.1/rexml/source.rb:228
    c:0013 p:0013 s:0094 b:0094 l:000093 d:000093 METHOD F:/Program Files/Ruby192/li
    b/ruby/1.9.1/rexml/parsers/baseparser.rb:159
    c:0012 p:0042 s:0091 b:0091 l:000090 d:000090 METHOD F:/Program Files/Ruby192/li
    b/ruby/1.9.1/rexml/parsers/baseparser.rb:199
    c:0011 p:0038 s:0060 b:0060 l:000059 d:000059 METHOD F:/Program Files/Ruby192/li
    b/ruby/1.9.1/rexml/parsers/treeparser.rb:22
    c:0010 p:0028 s:0048 b:0048 l:000047 d:000047 METHOD F:/Program Files/Ruby192/li
    b/ruby/1.9.1/rexml/document.rb:230
    c:0009 p:0105 s:0044 b:0044 l:000043 d:000043 METHOD F:/Program Files/Ruby192/li
    b/ruby/1.9.1/rexml/document.rb:43
    c:0008 p:---- s:0039 b:0039 l:000038 d:000038 FINISH
    c:0007 p:---- s:0037 b:0037 l:000036 d:000036 CFUNC :new
    c:0006 p:0108 s:0033 b:0033 l:00185c d:000032 BLOCK lastfm.rb:123
    c:0005 p:---- s:0029 b:0029 l:000028 d:000028 FINISH
    c:0004 p:---- s:0027 b:0027 l:000026 d:000026 CFUNC :times
    c:0003 p:0759 s:0024 b:0024 l:00185c d:0016e4 EVAL lastfm.rb:105
    c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
    c:0001 p:0000 s:0002 b:0002 l:00185c d:00185c TOP
    ---------------------------
    -- Ruby level backtrace information ----------------------------------------
    lastfm.rb:105:in `<main>'
    lastfm.rb:105:in `times'
    lastfm.rb:123:in `block in <main>'
    lastfm.rb:123:in `new'
    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/document.rb:43:in `initialize'
    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/document.rb:230:in `build'
    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:22:in `parse
    '
    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:199:in `pull
    '
    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:159:in `empt
    y?'
    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/source.rb:228:in `empty?'
    F:/Program Files/Ruby192/lib/ruby/1.9.1/rexml/source.rb:116:in `empty?'

    [NOTE]
    You may have encountered a bug in the Ruby interpreter or extension libraries.
    Bug reports are welcome.
    For details: http://www.ruby-lang.org/bugreport.html


    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.


    There also seems to be a different type of problem with the script on the *nix system I have access to.
    It seems like the script only locates the song titles, not the artists or albums of the tracks.
    For example, when it tries to calculate the time I've been listening to the song Moya byt Godspeed You! Black Emperor, this is what it shows on the *nix system:
    calling url: http://ws.audioscrobbler.com/2.0/?method=library.gettracks&api_key=<censored>&user=Voffe&page=1
    [[MB]] Looking for: arist - | track - Moya | album -
    Moya

    Obviously it can't find the correct song length from last.fm by just the name of the song and then tries to connect MB just for another failure.
    I tried to save the script files both in ANSI and in UTF-8 but that didn''t seem to be the problem. The Ruby installation on the *nix system is version 1.8.7.

    • Voffe said...
    • User
    • 15 Dec 2010, 23:01
    It seems like the script only works with Ruby 1.9.1,

  • Could you upload it again, please?

    • Phage said...
    • User
    • 23 Jun 2011, 07:21
    sorry I've not managed to make time for looking at any of this stuff of late, I'm gonna try and get back into it soon, or if I find I don't have time, release what I've done so far in some sort of open source way.

    I see the way this ultimatly working is populating a local database with all your track play information and allowing you to edit stuff like the track length (because sometimes it's wrong in the last fm/musicbrainz database) and number of plays etc and then generating a chart from this thats in a form you can put on your profile.

  • This is very interesting, although I think it would be much better in website form, like normalisr. I have the same problem with normalisr. Will look into it some time soon.

    • nospr said...
    • User
    • 28 Jul 2011, 23:09
    when you reupload it please let us know!

    Vision will blind. Severance ties. Median am I. True are all lies.
  • None of the links work. Too bad.

    • [Deleted user] said...
    • User
    • 16 Jan 2012, 17:17
    Can you re-upload the tool please? All your links are now dead :(

Anonymous users may not post messages. Please log in or create an account to post in the forums.