Game Information Scraping (MobySharp v1.0.0.0)

[Rain]

The Premise

Recently I had the need to obtain tons of data (as I am creating my own video gaming frontend (no, it's not for new systems; it is for old computers to play DOS games on.)) but there seemed to be a lack of tools out there for us software developers to use. So, my solution was to just simply make my own damned scraper that obtains detailed information from MobyGames. So, here I am after a day and a half of creating this scraper solution and am sharing it with the community in the hopes that I may save some other developer out there the headache and torture of doing this themselves.

The only functionality that I left out of the scraper is the obtaining of images (as most of MobyGames images are sub-par at best anyhow) and all I care about is the detailed data and don't want to waste any of their bandwidth obtaining said images.

The Dynamic Link Library

Using the library is fairly simple and a straight forward process. What I usually do is declare a global List<MobyGamesSearchResult> that is populated later on in the code. The reason that I declare a global is so that I can easily obtain the name of the game, and the MobyGames.com URL for the application URL (later on in the code examples it will become clearer what I mean.)

Setting up your methods

using MobySharp;
using MobySharp.Methods;

public partial class Form1 : Form
{
       List<MobyGamesSearchResult> _gameList = new List<MobyGamesSearchResult>();
.....

Now that you have declared all of your variables that you will be using throughout the application we can safely populate our results list by calling a search function.

_gameList = new MobyScraper().SearchMoby(searchString);

if (_gameList != null)
foreach (var results in _gameList)
{
    listBox1.Items.Add(results.Name);
}

Where searchString is equal to the name of the game you are looking for (ie DOOM II). You will notice that I also trap a null search result in the code above; as there is currently no method implemented of converting a null search (ie no games found.)

So, now that we have our _gameList populated and have listed all of the search names into a nice tidy listbox we can continue our code as seen whenever the user changes the listbox1 index.

           foreach (var result in _gameList)
           {
                   if (result.Name == (string)listBox1.SelectedItem)
                   {
                       var entry = new MobyScraper().GameInformation(result.Url);


                       labelName.Text = (string)listBox1.SelectedItem;
                       labelPublisher.Text = entry.Publisher;
                       labelReleaseDate.Text = entry.ReleaseDate;


                       labelDeveloper.Text = entry.Developer;
                       labelPlatform.Text = entry.Platform;
                       labelGenre.Text = entry.Genre;
                       labelPerspective.Text = entry.Perspective;


                       richTextBox1.Text = entry.Description;
                       richTextBox2.Text = entry.AlternateTitles;


                       return;
                   }
           }

So, as you can see by the code above we first get a search list of all available titles; and an URL is assigned to each title. After we have made a game selection we cross reference the title name against the selected game and if a match is found we extract the URL and then create a new GameInformation object which will contain all of the data on said game.

The Example Application Showing Data

Anyhow, I hope some of my fellow third party developers get some use out of this; I have attached a sample application as well as the stand alone DLL file in this post.

Thanks guys!

MobySharp.zip

Example Application.zip

[jackhammersalm]

Emb would be silly to pass this up

[brolly]

Now I can probably make a good use of this (even though I prefer gamefaqs every day over moby), nice work Rain.

Any chance you can add also a platform filter instead of filtering by name only?

[emb]

Emb would be silly to pass this up

Pfft... I wouldn't even think of using a software API written by rain... (cough.. Dat API.... Cough)

Ok... I can't lie... I love everything this stellar man outputs. I may nab this and integrate it into an upcoming version of the list generator. Thanks rain!

[Iggy]

Rain...you are a programming BEAST!!! Thanks for all the AWESOME programs you have released to the community!!!

[BadBoyBill]

Nice, would it be possible to have a tab for gamefaqs and moby so we could have a choice, this would have been mad helpful about 5 years ago when doing initial db's, but I think there is still a lot of use for it. Theres actually a small side project im working on right now that I actually could use this

[brolly]

moby isn't really that useful for doing HS dbs, gamefaqs on the other hand is very useful for that job and is the resource I use as it has a few features that we need on the db creation process that simply don't exist in moby.

Moby is good for additional info though as well as artwork.

[Rain]

I looked into adding support for GameFaq's but the return data is not as well laid out as MobyGames and therefore scraping said site for information would be more complex and increase the overall risk of error. I could look into doing that, but probably not within the next few weeks. Bleasby has got me doing other things at the moment lol. I will post example data here of the information the current API can retreive.

[Rain]

moby isn't really that useful for doing HS dbs, gamefaqs on the other hand is very useful for that job and is the resource I use as it has a few features that we need on the db creation process that simply don't exist in moby.

Moby is good for additional info though as well as artwork.

What data are you looking to retrieve that cannot be found via MobyGames.com; just curious and I'll see if I can find a way to compensate.

[brolly]

The 2 major things:

Moby doesn't link the same game released on different regions together so there's no way to know, well actually it does sort of using alternate titles, but the way they do it is just messy and confusing (and missing info many times) whereas gamefaqs does this in a nice clean table.

Content Ratings, these are non-existant in Moby afaik.

Then the genre system on Moby just sucks, gamefaqs has the best genre system I've seen by far.

Good things on Moby are the artwork (which isn't scrapped anyway), separation of Windows/DOS games for example and showing all systems a game was released for which I can see as being something interesting to be used even from within HS (BBB approval pending of course lol). Also descriptions/synopsys are better and much more detailed in Moby.

So both have their pros and cons, but for database creation process gamefaqs is way better and pretty much what I use except for the odd systems for which Moby isn't really that useful anyway.

[brolly]

Looks good, about gamefaqs don't burn yourself with it I have already scrapped most of what I need from it with a scrapper a buddy wrote (apart from ratings), but I believe some of the info there could be used in the future in HyperList for example.

Keep it going with bleasby I bet he can keep you busy for a while lol

[djvj]

lol good luck with bleasby. He's one of the main reasons we are still in beta

[bleasby]

The way you guys say it is like I am the evil twin of you guys. lol

I am not that bad!!! I just like adding more and more stuff...

Rain, I will write to you a private message about it. I am already very happy with the things that you implemented so far, and the speed that you were able to do them, but there is always room for improving so we need to determine a feasible limit anyways (that is me trying to be more restrictive with my desires because the brolly and djvj reprimand, if you let me I will go back to my normal behavior of never wanting to stop adding things).