Translating DAT files into XML?

[ninja2bceen]

amazing work dude! I just used the snes... my test included looking at 3 names that didn't have multiple alternate names and I checked 3 with multiple alternative names. Both worked great. Note that some of these games don't have a lot of info in them meaning it simply just displays the name of the game and the copyright while other games have a whole lot of fun stuff included. Amazing work, when does the beta version come out? Or maybe you can slap each system ini file on this thread? I'm going to nab the Atari one now.

[ninja2bceen]

one question.. in one of the files there is a settings ini... what names should we be using for the atari5200?

[settings]

MAME=info

Nintendo Entertainment System=nes

Super Nintendo Entertainment System=snes

Atari 5200=a5200 --->is this right?

[dragon57]

a5200 is what is used in the history.dat file, so I would assume that is correct. The name used for the Atari 2600 is a2600 in the history.dat, the Atari 7800 is a7800.

[dragon57]

I have about six more complete as of tonight. I have my method down pretty good, so I think I'll just keep cranking them out while I continue to have some spare time after work. I'll try to figure out the best way to make the files available for everyone.

[ninja2bceen]

Got any other spiffiness? Is this something nontechnical folk can do? How are u doing this :-)

[dragon57]

I have 19 .ini files complete at the moment.

My workflow isn't all that difficult. Like I say. I do stuff like this all the time at my day job with various apps, utilities and software tools. I purposely didn't use anything that most users would not have access to.

I first wrote a command line utility to parse the system names, descriptions and game/rom names from the history.dat file. This data was written out as a delimited file that I imported into Microsoft Excel. I am using two different free Excel plugins to assist me. One allows me to import the Hyperspin database xml files and the other is the important one. It does the fuzzy matching that is required between the imported data from the history.dat and the Hyperspin xml files (I then have to manually review the output and correct any incorrect guesses from the fuzzy matching). This is a high level flow, and you can't just do this once and be done with it, but it gives you an idea how I am doing it.

[ninja2bceen]

I have 19 .ini files complete at the moment.

My workflow isn't all that difficult. Like I say. I do stuff like this all the time at my day job with various apps, utilities and software tools. I purposely didn't use anything that most users would not have access to.

I first wrote a command line utility to parse the system names, descriptions and game/rom names from the history.dat file. This data was written out as a delimited file that I imported into Microsoft Excel. I am using two different free Excel plugins to assist me. One allows me to import the Hyperspin database xml files and the other is the important one. It does the fuzzy matching that is required between the imported data from the history.dat and the Hyperspin xml files (I then have to manually review the output and correct any incorrect guesses from the fuzzy matching). This is a high level flow, and you can't just do this once and be done with it, but it gives you an idea how I am doing it.

That's sweet! Looking fwd to ur other posts. 19 inis, yes please.

[Dogway]

Thanks a lot for the help although I think I'm a bit late to the party. By now you already have seen the great issue here (sorry to put you down ninja2bceen) that I warned before here and here.

By default:

[super Mario World (USA)]
Alternate_Rom_Name=smwu

and smwu:

USA release. Game developed in Japan. For more information about the game itself, please see the original Japanese release entry; "Super Mario World - Super Mario Bros. 4 [Model SHVC-MW]".

missing all this:

Super Mario World - Super Mario Bros. 4 (c) 1990 Nintendo Company, Limited.

Tackle a whole new world with Mario & Yoshi!

Mario's off on his biggest adventure ever, and this time, he's brought along a friend. Yoshi the dinosaur teams up with Mario to battle Koopa, who has kidnapped Princess Peach once again. Guide Mario and Yoshi through nine peril-filled worlds to the final showdown in Koopa's castle.

Use Mario's new powers and Yoshi's voracious monster-gobbling appetite as you explore 96 levels filled with dangerous new monsters and traps. Climb mountains and cross rivers, and descend into subterranean depths. Destroy the seven Koopa castles and find keys to gain entrance to hidden levels. Discover more warps and thrilling bonus world than ever before!

Mario's back and this time, he's better than ever!

- TECHNICAL -

Cartridge ID: SHVC-MW
Cartridge size: 4Mb.

- TRIVIA -

Super Mario World was released on November 21, 190 in Japan for 8000 Yen.

As per Nintendo tradition, an episode of the Mario series was a launch title for a Nintendo game system (the Super Famicom in this case) and became an immense commercial success. The pack-in launch title for the Super Famicom. The game has sold over 20 million copies worldwide.

Shigeru Miyamoto had wanted Mario to have a dinosaur companion ever since Super Mario Bros., but the dinosaur couldn't fit into the limitations of the Famicom.

Ranked as the 8th best game for a Nintendo console in Nintendo Power's Top 200 Games list. Ranked 7th in Official Nintendo Magazine's 100 greatest Nintendo games of all time. Won Nintendo Power's Game of the Year award in 1991. Named the fifth greatest game of all time in the final issue of Nintendo Power.

Export releases:
[uS] "Super Mario World [Model SNS-MW-USA]"
[uS] "Super Mario World [Model SNS-MW-USA-1]"
[EU] "Super Mario World [Model SNSP-MW-NOE]"
[EU] "Super Mario World [Model SNSP-MW-NOE-1]"
[FR] "Super Mario World [Model SNSP-MW-FAH]"
[FR] "Super Mario World [Model SNSP-MW-FAH-1]"

- SERIES -

1. Mario Bros. [No. TMA1-UP] (1983) 
2. Super Mario Bros. (1985, Famicom) 
3. Super Mario Bros. 2 (1986, Famicom) 
4. Super Mario Bros. 2 (1988, NES): remake of Yumekojo Doki Doki Panic (1987, Famicom) 
5. Super Mario Bros. 3 [Model HVC-UM] (1988, Famicom)
6. Super Mario Land (1989, Game Boy) 
7. Super Mario World - Super Mario Bros. 4 [Model SHVC-MW] (1991, Super Famicom) 
8. Super Mario Land 2 - 6 Golden Coins (1992,Game Boy) 
9. Super Mario Land 3 - Wario Land (1993, Game Boy) 
10. Yoshi's Island - Super Mario World 2 (1995, Super Famicom) 
11. Super Mario 64 (1997, Nintendo 64) 
12. Super Mario Sunshine (2002, Gamecube) 
13. Yoshi's Island DS (2006, DS) 
14. New Super Mario Bros. (2006, DS) 
15. Super Mario Galaxy (2007, Wii) 
16. New Super Mario Bros. Wii (2009, Wii) 
17. Super Mario Galaxy 2 (2010, Wii) 
18. Super Mario 3D Land (2011, 3DS) 
19. New Super Mario Bros. 2 (2012, DS) 
20. New Super Mario Bros. U (2012, Wii U)
21. New Super Luigi U (2013, Wii U)
22. Super Mario 3D World (2013, Wii U)

- PORTS -

* Consoles :
Nintendo GBA [JP] (2001) "Super Mario Advance 2: Super Mario World + Mario Brothers [Model AGB-AA2J-JPN]"

Now as you might probably know, smwu has almost no data, and if this is super mario world just wonder about the rest of games.

I had the idea though to swap the content when messages like these are found. You would look for ]" on the description section, from there you take everything from Model and ]" that gives you the cartridge model you would use to search the game with full data. The model is in the technical section after "Cartridge ID:", if you have your code nicely worked out for sections this might not be very hard. For me personally is too much, a bit over my head I don't feel too comfortable with regex, scripting nor programming either. (edit: not all of them -715 in total- adhere to this scheme, some only show game names)

For now I am doing (by hand) a series database of many popular IPs that I might share as well, example:

Series Ace Attorney:
(2001) Phoenix Wright: Ace Attorney (GBA)
(NDS, iOS Remake)
(2002) Phoenix Wright: Ace Attorney: Justice for All (GBA)
(NDS, iOS Remake)
(2004) Phoenix Wright: Ace Attorney: Trials and Tribulations (GBA)
(NDS, iOS Remake)
(2007) Apollo Justice: Ace Attorney (NDS)
(2013) Phoenix Wright: Ace Attorney - Dual Destinies (3DS, iOS)

Investigation series:
(2009) Ace Attorney Investigations Miles Edgeworth (NDS)
(2011) Gyakuten Kenji 2 (Sequel to Miles Edgeworth) (NDS)

Great Turnabout subseries:
(2015) Great Turnabout Trial: The Adventure of Ryuunosuke Naruhodou (3DS)

Spinoffs:
(2012) Professor Layton vs. Phoenix Wright: Ace Attorney (3DS)

Chronology:
(2015) Great Turnabout Trial: The Adventure of Ryuunosuke Naruhodou (3DS)
(2001) Phoenix Wright: Ace Attorney (GBA)
(2002) Phoenix Wright: Ace Attorney: Justice for All (GBA)
(2004) Phoenix Wright: Ace Attorney: Trials and Tribulations (GBA)
(2012) Professor Layton vs. Phoenix Wright: Ace Attorney (3DS)
(2009) Ace Attorney Investigations Miles Edgeworth (NDS)
(2011) Gyakuten Kenji 2 (Sequel to Miles Edgeworth) (NDS)
(2007) Apollo Justice: Ace Attorney (NDS)
(2013) Phoenix Wright: Ace Attorney - Dual Destinies (3DS, iOS)

[dragon57]

Dogway, you are exactly right. Many of the alternate roms that an automated system might pick will be the wrong ones. I am hoping that the ini files I create will at least be stepping stones for others to build upon.

[Ninja2bseen]

Agreed

Thanks dogway. I think the amazing work should continue. The conversion is correct but the .dat file needs changed/updated. Maybe we need to go to the site datomatic to get no intro dat files? The problem is with the dat file. Maybe there is something to what u are saying though... Get the conversion done, then do a find of u and replace with j but wait that probably wouldn't work. Some games like I was saying has all the data but like u said some has minimum.

http://datomatic.no-intro.org/?page=download

[Dogway]

No problem, I don't have all systems nor full romsets, so I would probably only need to edit like 200 entries as much. Your work is a great time saver in any case.

Anyways I had to edit the inis since I include PAL and NTSC-J games too

[dragon57]

Am I correct in assuming I don't need to go to the effort of creating an ini file for Mame? Aren't the game names in the Hyperspin xml file the same as the game names in the history.dat file?

[Dogway]

Yes, you don't have. Arcade games have fixed filenames.

all ending with info don't need remaps:

[settings]
PCB MAME=info
PCB CPS=info
PCB Namco=info
PCB Namco 246=info
PCB Namco 256=info
PCB Sammy Atomiswave=info
PCB Sega Hikaru=info
PCB Gaelco=info
PCB Sega Model 2=info
PCB Sega Model 3=info
PCB Sega Naomi=info
PCB Sega Naomi 2=info
PCB Sega Naomi 2 GD-ROM=info
PCB Sega Naomi GD-ROM=info
PCB Sega Triforce=info
PCB Sega Triforce 309=info
PCB Taito Type X=info
PCB ZiNc=info
SNK Neo Geo=info
Nintendo Entertainment System=nes
Nintendo Famicom Disk System=famicom_flop
Super Nintendo Entertainment System=snes
Nintendo 64=n64
Nintendo Game Boy=gameboy
Nintendo Game Boy Advance=gba
Nintendo Game Boy Color=gbcolor
Sega Game Gear=gamegear
Sega Master System=sms
Sega Genesis=megadriv
Sega CD=megacd,megacdj
Sega Saturn=saturn,sat_cart

[dragon57]

OK, folks, here is a link to an archive containing all the ini files I have created. Please try them out and let me know if you have issues with any of them. Also let me know if I have missed converting any standard Hyperspin xml file that also has a complement in the history.dat file.-

Edit: url deleted, see below for new link.

[djvj]

dragon, just poking around and I see some errors:

Atari 2600 ini:

[2005 Minigame Multicart (USA) (Unl)]

Alternate_Rom_Name=

[3-D Tic-Tac-Toe (USA)]

Alternate_Rom_Name=3dtictac,3dtictace

Games that are blank should have no entry and not exist in the ini

There are 2 rom names in many of the alt rom names.

[ninja2bceen]

thanks, can we get a matching system names.ini... sort of a continuation of dogways list to support each of the ini's created?

[dragon57]

dragon, just poking around and I see some errors:

Atari 2600 ini:

[2005 Minigame Multicart (USA) (Unl)]

Alternate_Rom_Name=

[3-D Tic-Tac-Toe (USA)]

Alternate_Rom_Name=3dtictac,3dtictace

Games that are blank should have no entry and not exist in the ini

There are 2 rom names in many of the alt rom names.

Great, thanks for the feekback. I can easily filter out the blank records.

I asked about the records with the multiple rom names a few posts back and were told they worked. Was that just a random happen chance that it worked? Some of the roms have many more than two alternate names. What is the correct format for more than one alternate name, separate lines with each rom name having its own Alternate_Rom_Name=?

[dragon57]

thanks, can we get a matching system names.ini... sort of a continuation of dogways list to support each of the ini's created?

Good idea. I'll add that to the next version of the archive after I hear back from djvj.

[djvj]

First, HL will never use comma as a delimiter because a comma is a valid character in a filename. How would you parse a filename with multiples if it had a comma in it and comma was your delimiter? Rare but its possible.

Second, all the info you need is on the HL google site. https://sites.google.com/site/hyperlaunch2/alter

[puckman]

Alternate_Rom_Name_1=pacmanbl

Alternate_Rom_Name_2=pacmanbla

Alternate_Rom_Name_3=pacheart

Alternate_Rom_Name_4=pacman

Alternate_Rom_Name_5=pacmod

[dragon57]

First, HL will never use comma as a delimiter because a comma is a valid character in a filename. How would you parse a filename with multiples if it had a comma in it and comma was your delimiter? Rare but its possible.

Second, all the info you need is on the HL google site. https://sites.google.com/site/hyperlaunch2/alter

Thank you sir for answering a basic question. I am still coming up to speed on Hyperspin in general and Hyperlaunch specifically. The link is very helpful and it all makes perfect sense.

[dragon57]

Here is an updated archive with the changes pointed out and requested. Give it a go and keep pointing out anything that needs updating.

Edit: url deleted, see below for new link.

[SIMPLYAUSTIN]

I believe we are all saying the same thing :-) Austin will probably have to convert the ini to an xml but yep all is exciting

I doubt this would be no problem. So in theory, I could match up all current media artwork with the MESS shortnames for all the systems it supports?

[RodentVienna]

Retired like EMB.

EMB did the opposite of retiring, but just is not on HS anymore:

www.retrofe.com

[Dogway]

Here is an updated archive with the changes pointed out and requested. Give it a go and keep pointing out anything that needs updating.

As I said before NeoGeo should be info. Neogeo entries (AES) are very limited and even so you missed many games (ie androrun is in the history.dat but not in your neogeo.ini)

You also forgot Nintendo Famicom Disk System, although some guys will tell you "screw with that" because most (all?) games are japanese.

[bleasby]

I asked about the records with the multiple rom names a few posts back and were told they worked. Was that just a random happen chance that it worked? Some of the roms have many more than two alternate names. What is the correct format for more than one alternate name, separate lines with each rom name having its own Alternate_Rom_Name=?

Just to clarify why it seamed to work for some cases.

Hyperlaunch uses this regex expresion to find the game on the history.dat file.

"i)" . "\$\s*" . historyDatSystemName . "\s*=\s*.*\b" . romNameToSearch . "\b\s*

Basically what it does is to search for a line that starts with the $ symbol, followed by the system code, followed by the equal sign and finally, that has the game name defined on the map ini files that you created.

For example, it searches for: $info=3dtictac

However on history dat it is possible to define multiple games that would share the same info on the file. This is done by using the comma separation.

For example: $info=3dtictac,3dtictace

If you make a ini file like the example that you did:

[3-D Tic-Tac-Toe (USA)]
Alternate_Rom_Name=3dtictac,3dtictace 

The regex search will still find that line because it will think that the game name is actually the entere string ("3dtictac,3dtictace").

This is not desirable for a few reasons, for example, if any additional game is added on that same line on an updated version of the history.dat file, as for example on here:

$info=3dtictac,NEW3dtictacDUMP, 3dtictace

The game will not be found anymore. While using the propper one game per ini key setup will keep everything working even after history.dat file updates.

There is another more important reason why to follow the naming conventions set on the rom mapping structure, that djvj pointed out. The history dat support on HyperLaunch was designed to be perfectly compatible with the rom mapping ini files.

Once the community is able to build propper rom mapping files for release groups like the MESS software lists, NoINtro, Tosec,... you would need to just copy the MESS software list map ini files to the history.dat folder to have a perfect match between games and history info. The oposite can be done also, use the history dat inis on the rom mapping feature, to allow you to use roms whithout having to renaming them or to choose what is your preferable language for play games automatically. This will only work if you follow the naming conventions set on the link provided by djvj.