Announcement: unofficial HS XML Game Database Format for No-Intro based Systems

[RodentVienna]

Motivation:

For almost 3 years, I am hunting for the ultimate experience to create personalized gamelists to use in HS.

For MAME, there are a lot of tools such as RomLister from Jeff, and a whole bunch of additional cuatom datafiles, which provide additional Info, such as e.g. "Game Genre", "Controls used" and "Number of Players".

Then in another axis, you have HS specific tools, the BEST BY FAR being Don's Toolset by fellow member "emb". KUDOS

Another part of the situation is, what HS officials proclaim as "standard" through downloadable XML Databases from HyperList.

Those Databases, from the perspective of the openminded user are limited in many ways, which I'd like to point out right now:

• limitation in which games are allowed to being added to official standard ("official" for No-Intro is "USA Releases" only, plus a few handselected pearls)
  --- This limitation is attempted to be greatly lifted by a new project: The-HyperSpin-Project
  --- KUDOS
  
• limitation in which theoretical additional information official HS XML DBs provide, such as "# players", "control device used", "# buttons" (not all people have an 8 button arcade control panel, aye?)
  
• BUT the biggest limitation in itself, is the capability for the user to define his ultimate personal gamelist, end to end (i'll explain more)
  

with current tools availabe, you cannot start from No-Intro DAT, and expect a result of a) being complete with all information that exists in official HS DB's, and having all non-USA games included, and more specific maybe incorporate No-Intro's Parent-Clone 1G1R functionality (define priority order of regions, and only keep 1 game per region, no clones)

Imagine you can create a gamelist for Super Nintendo", which covers all games released worldwide, but narrowed down to favorize one release region per game.

Then you want to create a "Genre.XML" covering only games that can be played simultaneously for showing off with friends, while your arcade machine only has 6 buttons per player...

Of course, for showing off to friends, you only want to have games in your list, where the IGN rating was above 7.5 AND the average Gamefaqs ratings were above 7.0 ...

I suspect most of you don't even know on all those details, and are happy with how things are on the "official" side per today. No offense though to you users, or the HS staff/supporters.

While I don't feel offended by the situation, I feel limited. I know I am not alone. You are not alone, respectively.

Now I realize, the true limit is in myself, and I cannot expect others to come up with new ideas and visions, while leaning back and wait. Only took me 3 years, to get my ass up and learn XML/XSL for Hyperspin porposes only. Ouch

End 2 End Personal Gamelist Creation - What is it?

The approach is simple:

1) take any no intro dat file from www-no-intro.org

2) enrich the data contained with additional data, such as: genre, rating, IGN-score, #players, control devices, #buttons, year, manufacturer, subgenre, etc.

3) filter the created merged data within your own desire

4) output gamelist to (but not limited to in theory) HS DB XML "unofficial DB format"

if you like it more advanced; the concept as a formula:

1) [No-Intro.dat] + [official HS XML DB] + [custom XML DB with custom information] = [unofficial HSXML DB]

2) [unofficial HSXML DB] + {Tool that can rework "full DB's" into "personalized/filtered DB'S} such as emb's "Don's Listgenerator" = [personalized gamelist]

What we have / don't have

HAVE:

• - open access to No-Intro DAT files (thanks No-Intro Team)
  
• - open access to official HS XML DB files (thanks HS staff; thanks brolly, nogarth and all the other people involved!)
  
• - absolutely brilliant 3rd party tools, such as *drumroll* emb's "Don's Toolset", that allow for simplified workflow for filtering personalized lists
  
• - FULL CONTROL (parse, lookup, rewrite) over any Data contained in XML format, meaning we can build any relational databases out of multiple datasources into one
  
• --- achieved through XSL stylesheet language and XPath queries. No Voodoo involved
  
• - a preliminary XML data structure for additional info mentioned
  

DON'T HAVE:

• - actual additional data
  
• - actual 3rd party tool to handle the new available data for filtering
  
• --- extending emb's "Don's Listgenerator" would be the simplest approach to this, but who m I to dictate the incorporatation of such changes to his toolset
  

What I am looking for

I cannot collect and validate all the "additional data" myself.

It is by far to much for me as a family father of 6 children, a full-time job, and the desire to maintain "my ultimately personalized Arcade Machine"

I'm on the edge so far with developing this workflow approach.

A good and bad example to mention here is the "controls.dat" project for MAME.

It was a great crowdsourcing project while it lasted, but failed ultimately due to the approach their management was driving in regards to validating entries.

I am looking for people who can provide such data in structured form.

I can provide the structure and backend for you to do so, by e.g. a public google spreadsheet file.

You could do your own public google spreadsheets and share.

Then I look for people who

a) know reliable datasources for game information, such as IGN, Gamefaqs, etc

know how to build a data scraper for such datasources, to potentially automate the data collection process to a certain extent

c) can validate entries in such databases from time to time

d) are more professional with XML/XSL than I myself, which can improve on my work here

e) can build HS 3rd party apps / or write XML/XSL DIFF funtionality to incorporate certain aspects in the maintenance process for this workflow (updating gamelists, merging no-intro standard attributes, simplifying, identifying, reporting, etc)

Meantime

I am going to watch this thread closely for any feedback you guys may have, and specifically if you can come up with further extensions to the set of data.

I hope to being able to gather people together for this vast project.

So far I have started to work with the "Super Nintendo Entertainment System" DAT from No-Intro, and I get feasible output.

But there are other Systems covered by No-Intro; each with their own control devices, and potential other sets of information.

The scope of this project will stay, but extend to "unified applicability" to other No-Intro based systems.

Meanwhile I will focus more on extending the coverage for all other No-Intro Systems, which will be enough for me to take given my personal situation briefly summarized above.

FOOD

It's dangerous outside - Take this!

Sample output, from real data:

<?xml version="1.0" encoding="UTF-8"?>
<menu>
  <header>
     <listname>Super Nintendo Entertainment System</listname>
     <lastlistupdate>2013-09-15+02:00</lastlistupdate>
     <listversion>20130907-201003</listversion>
     <exporterversion>RodentVienna Workflow</exporterversion>
  </header>
  <game name="2020 Super Baseball (USA)">
     <description>2020 Super Baseball (USA)</description>
     <cloneof>NOT FOUND</cloneof>
     <crc>TO RECALCULATE</crc>
     <manufacturer>SNK</manufacturer>
     <year>1993</year>
     <genre>Sports/Baseball</genre>
     <rating>Other - NR (Not Rated)</rating>
     <enabled>Yes</enabled>
     <c_region_1g1r>USA</c_region_1g1r>
     <c_region>USA</c_region>
     <c_category>Standard Release</c_category>
     <c_nplayers/>
     <c_controls_device>Super Scope</c_controls_device>
     <c_controls_buttons>6</c_controls_buttons>
     <c_clonepartof>2020 Super Baseball (USA)</c_clonepartof>
     <c_custom1/>
     <c_custom2/>
     <c_custom3/>
  </game>
  <game name="Batman Forever (USA)">
     <description>Batman Forever (USA)</description>
     <cloneof>Batman Forever (Europe)</cloneof>
     <crc>TO RECALCULATE</crc>
     <manufacturer>Acclaim</manufacturer>
     <year>1995</year>
     <genre>Fighter</genre>
     <rating>Other - NR (Not Rated)</rating>
     <enabled>Yes</enabled>
     <c_region_1g1r>USA</c_region_1g1r>
     <c_region>USA</c_region>
     <c_category>Standard Release</c_category>
     <c_nplayers/>
     <c_controls_device>Gamepad</c_controls_device>
     <c_controls_buttons>8</c_controls_buttons>
     <c_clonepartof>Batman Forever (Europe)</c_clonepartof>
     <c_custom1/>
     <c_custom2/>
     <c_custom3/>
  </game>
  <game name="Mario Paint (Japan, USA)">
     <description>Mario Paint (Japan, USA)</description>
     <cloneof>Mario Paint (Europe)</cloneof>
     <crc>TO RECALCULATE</crc>
     <manufacturer>Nintendo</manufacturer>
     <year>1992</year>
     <genre>Miscellaneous</genre>
     <rating>Other - NR (Not Rated)</rating>
     <enabled>Yes</enabled>
     <c_region_1g1r>JPN</c_region_1g1r>
     <c_region>Japan, USA</c_region>
     <c_category>Standard Release</c_category>
     <c_nplayers/>
     <c_controls_device>Mouse</c_controls_device>
     <c_controls_buttons>2</c_controls_buttons>
     <c_clonepartof>Mario Paint (Europe)</c_clonepartof>
     <c_custom1/>
     <c_custom2/>
     <c_custom3/>
  </game>
</menu>

Downloads:

Proof-of-concept-related files involved

http://www.kitchingers.net/dump/snes.zip

[djvj]

Pretty much all this has been discussed countless times over and we even have one or 2 threads entirely about adding more info to the xmls. Our limitation has always been hyperlist. Once HyperList is updated to support the new fields we presented long ago, many of which you have above, those fields can start to be supported and cataloged properly. Did you notice the data folders included with RL? Much of the data we already have, and we have tools to grab a lot of it automatically. Once the new HList is live we can update the current dbs and start going down this road. Until then, just sit tight. There is a lot planned.

[RodentVienna]

I will look at the RL data folder, thanks.

Is your blueprint for data enrichment in a state you can share with me? Dont wanna sit tight meantime. Learning xml and xsl is a great pleasure.

I am working also on a web based solution, that combines in blatant terms hyperlaunch and dons listgenerator. It will be fully based on xml/xslt/mysql with only the db queries executed in php.

Im still learning on my path to become an amateur hobby developer and so far it is exciting.

Sent from my LG-P880 using Tapatalk 4

[Sir Cyrano]

Pretty much all this has been discussed countless times over and we even have one or 2 threads entirely about adding more info to the xmls. Our limitation has always been hyperlist. Once HyperList is updated to support the new fields we presented long ago, many of which you have above, those fields can start to be supported and cataloged properly. Did you notice the data folders included with RL? Much of the data we already have, and we have tools to grab a lot of it automatically. Once the new HList is live we can update the current dbs and start going down this road. Until then, just sit tight. There is a lot planned.

Djdv you know when HyperList will be Update ??

[djvj]

Rodent, we don't exactly have a structure blueprint laid out like you like to do in your posts. There is one thread public on this forum somewhere that has some info we proposed to BBB to add support into HList. That was awhile back now and those fields will be implemented once the new HList is accessible to us. Not really worried about the fields we want to add as the new HList we will be able to dynamically add a new field to all dbs in a couple clicks, something not possible with the current HList or db it is stored on. Rest assured, we are well aware of the fields that are desired/needed in the xmls so that more features can use it down the road.

Cyrano, not for awhile. There might be a beta soon for us to play with and comment on, but something live may not be around for awhile as there is a lot going into it.

I will put this out there now, if I start seeing "when will xxx be out" I will start deleting those posts. Those types of posts only annoy the dev(s) working on it. Please sit tight and be patient. There is so much to work on that we don't need HList for.

[RodentVienna]

not the type of "when will xxx be out"- question, but rather feeling lucky to free up some time... so:

anywhere i could add my support with this?

... still seeing this end to end from a users perspective ...

thx/A