XBMC ROM utilities

Wintermute0110

XBMC ROM utilities 0.1


Table of Contents

Introduction
xru-console
Configuration file
Understanding ROM filters
No-Intro One ROM-One Game (1R1G) DATs
Updating/Copying filtered ROMs
Art Work
xru-mame
Configuration file
Creating the MAME ROM database
Understanding MAME filters
Updating/Copying MAME ROMs
NFO files
Artwork files
xru-launcher-AL
Configuration
Checking launchers
Examples
Simple filtering of a console ROM collection
Splitting MAME ROMs into several launchers for each driver
Changelog
Version 0.1

Introduction

XBMC ROM utilities (XRU) is a set of Python scripts that allows you to filter your ROM collections to remove unwanted ROMs from XBMC's launcher plugins like Advanced Launcher, ROM Collection Browser, or any other front-end. If the ROM collections are updated or a new version of the emulators installed (which may require some new ROMs to be updated) XRU is able to synchronise the ROM directories of your HTPC quickly and effectively.

XRU includes three programs

  • xru-console filters a ROM collection with No-Intro names and copies the filtered list to the destination directory of your choice. Multiple ROM collections, each with a different filter, can be configured. Multiple filters can be configured to the same ROM collection, for example to split the ROMs into USA, Europe and Japan regions. Also, if artwork is available, it may also be synchronised.

  • xru-mame takes the MAME-generated XML game database and merges it with Catver.ini. Then, a set of filters can be defined (for example, to remove mechanical games or select games that use a lightgun) and filtered ROMs are copied to different destination directories. It also creates NFO files with game information for launchers and copies local artwork if available so offline scrapers can be used.

  • xru-launcher-AL parses XBMC's Advanced Launcher configuration file and scans your ROM launchers for missing ROMs. It prints a report with the launchers you have to update.

xru-console

xru-console requires a mandatory command to specify the main action to do. Several commands also require a filterName. Optionally, some command actions can be modified with options.

$ xru-console.py [-options] command [filterName]

The following commands are supported.

  • usage

    Prints a summary of the commands and options.

  • list

    Lists every ROM set system defined in the configuration file and some basic information. Use --verbose to get more information.

  • list-nointro <filterName>

    Lists every ROM defined in the No-Intro DAT file.

  • check-nointro <filterName>

    Scans the source directory and reads the No-Intro XML data file. Checks if you have all the ROMs defined in the No-INTRO DAT and reports the number of missing and unknown ROMs.

  • list-tags <filterName>

    Scans the source directory and reports the total number of ROM files, all the tags found, and the number of ROMs that have each tag. It also display ROMs without any tag.

  • check-filter <filterName>

    Applies ROM filters defined in the configuration file and prints a list of the scored ROMs. used.

  • copy <filterName>

    Applies the ROM filters defined in the configuration file and copies the contents of the source directory into the destination directory. This always overwrites ROMs in the destination directory and may take a long time for huge ROM collections and network-mounted destination directories.

  • update <filterName>

    Like copy, but only copies ROMs if file sizes are different. This may save a lot of time.

  • check-artwork <filterName>

    Reads the ROMs in destDir, checks if you have the corresponding artwork files, and prints a report.

  • copy-artwork <filterName>

    Reads the ROMs in destDir and tries to copy the matching artwork to the destination directory.

  • update-artwork <filterName>

    Like copy-artwork, but only copies images if file sizes are different. This may save a lot of time.

The following options are supported.

  • -h, --help

    Prints a short command reference

  • -v, --verbose

    Prints more information about what's going on. Can be used several times to get even more information.

  • -l, --log

    Saves program output in xru-console-log.txt.

  • --logto <logName>

    Saves program output in the file you specify.

  • --dryRun

    don't modify the destination directory at all, just print the operations to be done.

  • --cleanROMs

    Deletes ROMs in the destination directory not present in the filtered ROM list (unknown ROMs).

  • --cleanNFO

    Deletes redundant NFO files in destination directory (unknown NFO files).

  • --cleanArtWork

    Deletes thumbs and fanart in the artwork destination directories not present in the artwork list (unknown Artwork).

Configuration file

xru-console requires a configuration file, named xru-console-config.xml, where you store the directories and filters for every ROM set you want to filter. The configuration filter is an XML file. A typical configuration filter looks like this.

<!-- Example configuration file for XBMC ROM utilities -->
<ROMcollections>

<!-- Nintendo ROMS -->
<collection name="SNES NoIntro" shortname="snes">
  <NoIntroDat>/home/xbmc/roms/DATs/SNES Parent-Clone (20140608-202038).dat</NoIntroDat>
  <ROMsSource>/home/xbmc/roms/roms-nintendo-snes/</ROMsSource>
  <ROMsDest  >/home/xbmc/NUC-remote/ROMs/roms-nintendo-snes/</ROMsDest>

  <ThumbsSource>/home/xbmc/roms/ArtWork/nintendo-snes-boxes/</ThumbsSource>
  <FanartSource>/home/xbmc/roms/ArtWork/nintendo-snes-snaps/</FanartSource>
  <ThumbsDest  >/home/xbmc/NUC-remote/ROMs/ArtWork/nintendo-snes-thumbs/</ThumbsDest>
  <FanartDest  >/home/xbmc/NUC-remote/ROMs/ArtWork/nintendo-snes-fanart/</FanartDest>

  <filterUpTags>Europe, World, Rev 1</filterUpTags>
  <filterDownTags>Japan, Beta, Proto, France</filterDownTags>
  <includeTags>Europe, USA</includeTags>
  <excludeTags>Japan</excludeTags>
</collection>

<!-- Sega ROMS -->
<collection name="Sega Mega Drive NoIntro" shortname="genesis">
  <ROMsSource>/home/xbmc/roms/roms-sega-genesis/</ROMsSource>
  <ROMsDest  >/home/xbmc/NUC-remote/ROMs/roms-sega-genesis/</ROMsDest>

  <filterUpTags>Europe, World, Rev 1</filterUpTags>
  <filterDownTags>Japan, Beta, Proto</filterDownTags>
  <includeTags>Europe, USA</includeTags>
  <excludeTags>Japan</excludeTags>
</collection>
</ROMcollections> 

Every ROM collection filter requires a <collection> tag. The attribute shortname defines the name of this collection to be used with the commands that require a <filterName>. name attribute defines a longer name to better describe this ROM collection filter and it is useful if you setup different filters for the same ROM set. <ROMsSource> defines the source directory where you have all the ROMs in your collection. <ROMsDest> defines the destination directory where you filtered ROMs will be copied. Both tags are mandatory for every ROM collection filter.

Understanding ROM filters

To use xru-console, your ROM collection needs to have the No-Intro naming conventions. Only ROMs in .zip are currently supported.

Tip

If you have a ROM collection with different naming (TOSEC, Goodtools, or completely random names) then you can rename your ROMs with a ROM manager like CLRMamePro or ROM Vault.

For example, a (very abridged) No-Intro collection for the Super Nintendo may look like this.

Super Mario All-Stars and Super Mario World (Europe).zip
Super Mario All-Stars + Super Mario World (USA)
Super Mario World (Europe) (Rev 1).zip
Super Mario World (Europe).zip
Super Mario World (Japan) (En) (Arcade) [b].zip
Super Mario World (USA).zip
Super Mario - Yossy Island (Japan) (Rev 1).zip
Super Mario - Yossy Island (Japan) (Rev 2).zip
Super Mario - Yossy Island (Japan).zip

Every ROM in the collection has a base-name and then one or more tags. Every single tag is inside parenthesis and the most common tags are the region where the ROM belongs: Europe, USA and/or Japan. Some ROMs may have complex tags, separated with commas, as in the following example.

Super Solitaire (Europe) (En,Fr,De,Es,It) (Proto).zip
Super Solitaire (USA) (En,Fr,De,Es,It).zip

xru-console is able to remove all the tags from the ROM name and complex tags are also separated into simpler tags.

For every game in the ROM collection there are usually several ROMs, mainly released in different regions (Europe, USA, or Japan). In your XBMC launcher ideally you only want one ROM for every game, corresponding to the ROMs in your favourite region (or regions). Also, there may be several released version of your ROM and you just want to play the best available one.

xru-console makes two passes when filtering the ROMs in the source directory. Firstly, tags are removed from the ROM name, and all ROMs that have a common name after tag removal are grouped together. This is roughly like grouping ROMs into a parent/clone set. Note that the order of the ROMs in every set is arbitrary. For example, the SNES ROMs in the example above will be grouped like this.

<pcloneList> "Super Mario All-Stars and Super Mario World"
  Super Mario All-Stars and Super Mario World (Europe).zip | (Europe)

<pcloneList> "Super Mario All-Stars + Super Mario World"
  Super Mario All-Stars + Super Mario World (USA).zip      | (USA)

<pcloneList> "Super Mario World"
  Super Mario World (Europe) (Rev 1).zip                   | (Europe), (Rev 1)
  Super Mario World (Europe).zip                           | (Europe)
  Super Mario World (Japan) (En) (Arcade) [b].zip          | (Japan), (En), (Arcade)
  Super Mario World (USA).zip                              | (USA)

<pcloneList> "Super Mario - Yossy Island"
  Super Mario - Yossy Island (Japan) (Rev 1).zip           | (Japan), (Rev 1)
  Super Mario - Yossy Island (Japan) (Rev 2).zip           | (Japan), (Rev 2)
  Super Mario - Yossy Island (Japan).zip                   | (Japan)

The first pass of the filtering process uses the configured labels <filterUpTags> and <filterDownTags>. For every tag you write inside <filterUpTags>, separated with commas, every ROM on every parent/clone set will get some points. For example, if you write this.

<filterUpTags>Europe, World, Rev 1</filterUpTags>

Then ROMs having the Europe tag will get 3 points, ROMs having the World tag will get 2 points, and ROMs having the Rev 1 tag will get one point. If a ROM has several wanted tags then it will get more points. Note that the tags you want more should be placed first.

The <filterDownTags> filters works in a similar way, but instead of adding points to the score it subtracts points. For example, the following filter.

<filterDownTags>Japan, Beta, Proto, France</filterDownTags>

Will subtract -4 points for ROMs having the tag Japan, -3 points for the ROMs having the tag Beta, and so on. In the case of this filter, you specify ROMs you don not want first. Following with the example, if the filters are applied to our ROM list, then the scores will be.

<pcloneList> "Super Mario All-Stars and Super Mario World"
  3 | Super Mario All-Stars and Super Mario World (Europe).zip | (Europe)

<pcloneList> "Super Mario All-Stars and Super Mario World"
  0 | Super Mario All-Stars + Super Mario World (USA).zip      | (USA)

<pcloneList> "Super Mario World"
  4 | Super Mario World (Europe) (Rev 1).zip                   | (Europe), (Rev 1)
  3 | Super Mario World (Europe).zip                           | (Europe)
  0 | Super Mario World (USA).zip                              | (USA)
 -4 | Super Mario World (Japan) (En) (Arcade) [b].zip          | (Japan), (En), (Arcade)
 
<pcloneList> "Super Mario - Yossy Island"
 -2 | Super Mario - Yossy Island (Japan) (Rev 1).zip           | (Japan), (Rev 1)
 -3 | Super Mario - Yossy Island (Japan) (Rev 2).zip           | (Japan), (Rev 2)
 -3 | Super Mario - Yossy Island (Japan).zip                   | (Japan)

After scoring the ROMs, the first pass of the filtering takes the ROM with highest score on every set. If more than one ROM have the same highest score, then the first ROM in the list is selected. Following with the example, after the first filter the ROM collection will look like this.

<pcloneList> "Super Mario All-Stars and Super Mario World"
  3 | Super Mario All-Stars and Super Mario World (Europe).zip | (Europe)

<pcloneList> "Super Mario All-Stars and Super Mario World"
  0 | Super Mario All-Stars + Super Mario World (USA).zip      | (USA)

<pcloneList> "Super Mario World"
  4 | Super Mario World (Europe) (Rev 1).zip                   | (Europe), (Rev 1)
 
<pcloneList> "Super Mario - Yossy Island"
 -2 | Super Mario - Yossy Island (Japan) (Rev 1).zip           | (Japan), (Rev 1)

Out of a total of 9 ROMs, so far only 4 will be copied into the destination directory.

The second part of the filtering process allows you to remove unwanted ROMs with specific tags. Users in Europe or USA normally do not want Japanese ROMs at all. To remove ROMs belonging to the Japanese region configure.

<includeTags>Europe, USA</includeTags>
<excludeTags>Japan</excludeTags>

This filter will exclude all ROMs with the tag Japan, but only if the ROM has not the tags Europe or USA. This is to prevent filtering ROMs with tags like (Japan, USA) or (Japan, Europe). After the second pass of the filtering process, the filtering is over and the list of ROMs to be copied to the destination directory will look like this.

<pcloneList> "Super Mario All-Stars and Super Mario World"
  3 | Super Mario All-Stars and Super Mario World (Europe).zip | (Europe)

<pcloneList> "Super Mario All-Stars and Super Mario World"
  0 | Super Mario All-Stars + Super Mario World (USA).zip      | (USA)

<pcloneList> "Super Mario World"
  4 | Super Mario World (Europe) (Rev 1).zip                   | (Europe), (Rev 1)

The list of tags for every ROM collection can be displayed with the list-tags command, which also prints how many ROMs in the collection have that tag. This command is very handy to know which tags to put in the configuration file for that ROM collection. Note that if a ROM has more than one tag it will be counted twice. For example:

$ ./xru-console.py list-tags snes
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Listing tags]
Filter name = snes
[Tag histogram]
...
   104  Fr
   120  En
   173  Rev 1
   191  BS
   220  Beta
   626  Europe
   968  USA
  1925  Japan

Before you copy/update your filtered ROM collection, you may want to check what will happen and the check-filter command does precisely this. For example:

$ ./xru-console.py check-filter snes
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Check-filter ROM]
Filter name = snes
Using No-Intro parent/clone DAT
Parsing No-Intro XML DAT
Parsing No-Intro XML file ./Nintendo - SNES Parent-Clone (20140608-202038).dat...  done
Total number of games = 3366
Number of parents = 1828
Number of clones = 1538
[Filtering ROMs]
...
<ROM set> Super Mario All-Stars + Super Mario World
   0 [IH] Super Mario All-Stars + Super Mario World (USA).zip
<ROM set> Super Mario All-Stars and Super Mario World
   7 [IH] Super Mario All-Stars and Super Mario World (Europe).zip
...
<ROM set> Super Mario Kart
   7 [IH] Super Mario Kart (Europe).zip
   0 [IH] Super Mario Kart (USA).zip
  -3 [EH] Super Mario Kart (Japan).zip
<ROM set> Super Mario RPG
  -3 [EH] Super Mario RPG (Japan).zip
<ROM set> Super Mario RPG - Legend of the Seven Stars
   0 [IH] Super Mario RPG - Legend of the Seven Stars (USA).zip
...

Have a look at the end of next section for a quick explanation of the flags and numbers in this listing.

No-Intro One ROM-One Game (1R1G) DATs

The first pass of the filtering process groups different versions of the same game together. However, based only on the ROM names this process is far from perfect and will produce wrong results for some games. For example:

<pcloneList> "Super Mario All-Stars and Super Mario World"
  Super Mario All-Stars and Super Mario World (Europe).zip | (Europe)

<pcloneList> "Super Mario All-Stars + Super Mario World"
  Super Mario All-Stars + Super Mario World (USA).zip      | (USA)

Will be grouped into different parent/clone sets, however it is clear that they are just the European and USA version of the same game. Actually, this example (however real) is not very common. Most of the games belonging to the same set having different name are the Japanese versions of the game.

For many users, the basic filtering will be fine. However, for picky users this problem may be solved using the No-Intro One ROM-One Game (1G1R) DATs. A typical No-Intro DAT looks like this.

game (
  name "Sonic The Hedgehog (Japan, Korea)"
  description "Sonic The Hedgehog (Japan, Korea)"
  rom ( name "Sonic The Hedgehog (Japan, Korea).md" size 524288 crc AFE05EEE )
)

game (
  name "Sonic The Hedgehog (USA, Europe)"
  description "Sonic The Hedgehog (USA, Europe)"
  rom ( name "Sonic The Hedgehog (USA, Europe).md" size 524288 crc F9394E97 flags verified )
)

However, from DAT-o-MATIC a No-Intro DAT in XML format having proper parent/clone lists are available. This is also referred to as a 1G1R list. To download a 1G1R XML DAT for your favourite system, go to DAT-o-MATIC, click on the P/Clone XML link, select your console system in the drop-box on the top, and finally click on the Download button. This is how a 1G1R XML DAT looks like.

<game name="Sonic The Hedgehog (USA, Europe)">
  <description>Sonic The Hedgehog (USA, Europe)</description>
  <release name="Sonic The Hedgehog" region="EUR"/>
  <release name="Sonic The Hedgehog" region="USA"/>
  <rom name="Sonic The Hedgehog (USA, Europe).md" size="524288" crc="F9394E97" status="verified"/>
</game>
<game name="Sonic The Hedgehog (Japan, Korea)" cloneof="Sonic The Hedgehog (USA, Europe)">
  <description>Sonic The Hedgehog (Japan, Korea)</description>
  <release name="Sonic The Hedgehog" region="JPN"/>
  <release name="Sonic The Hedgehog" region="KOR"/>
  <rom name="Sonic The Hedgehog (Japan, Korea).md" size="524288" crc="AFE05EEE"/>
</game>

If you pay attention, there is a cloneof attribute on every clone ROM in the collection. This allows xru-console to make a perfect parent/clone list before applying filters, and this will produce a better filtered list.

In order to use a No-Intro 1G1R XML DAT with xru-console, you need to configure the following option for every collection filter where you want to use it. Make sure you use the correct XML DAT for the collection!

<collection>
...
  <NoIntroDat>./Sega - Genesis Parent-Clone (20140601).dat</NoIntroDat>
...
</collection>

In No-Intro 1G1R mode, xru-console makes the main parent/clone list using the XML DAT instead of scanning you source directory. Then, the two pass filtering is applied. Finally, your source directory will be scanned and for every ROM set in the parent/clone list, the ROM with highest score will be copied into the destination directory. If you do not have that ROM, then the second ROM in the scored ROM set will be tried, and so on.

In No-Intro 1G1R mode, two additional commands are available. list-nointro <romSetName> lists all the games in the No-Intro 1G1R XML DAT. For example:

$ ./xru-console.py list-nointro genesis
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Listing No-Intro XML DAT]
Filter name = genesis
Parsing No-Intro XML DAT
Parsing No-Intro merged XML file ./Sega - Genesis Parent-Clone (20140601).dat...  done
...
<game> Sonic The Hedgehog (USA, Europe)
<game> Sonic The Hedgehog (Japan, Korea)
<game> Sonic The Hedgehog 2 (World) (Rev A)
<game> Sonic The Hedgehog 2 (World)
<game> Sonic The Hedgehog 2 (World) (Beta)
<game> Sonic The Hedgehog 3 (Europe)
<game> Sonic The Hedgehog 3 (Japan, Korea)
<game> Sonic The Hedgehog 3 (USA)
...
<game> Zoop (USA)
Number of games = 1681

check-nointro <romSetName> will scan you source directory and will report how many ROMs you have, how many ROMs you don't have (missing ROMs), and how many ROMs you have not listed in the DAT (unknown ROMs).

$ ./xru-console.py check-nointro genesis-local-nointro
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Checking ROMs with No-Intro XML DAT]
Filter name = genesis-local-nointro
Parsing No-Intro XML DAT
Parsing ./DAT-No-Intro/Sega - Genesis Parent-Clone (20140601-111936).dat...  done
[Scanning ROMs in sourceDir]
[Report]
Games in DAT = 1681
Have ROMs    = 1681
Missing ROMs = 0
Unknown ROMs = 47

If you specify the option -v, xru-console will print the missing and unknown ROMs. -vv will also print the ROMs you have.

Note that xru-console only looks at the file name of the ROM! It does not check the ROM checksum like more advanced ROM Managers, for example CRLMamePro or ROM Vault.

Before you copy/update your filtered ROM collection, you may want to check what will happen and the check-filter command does precisely that. For example:

$ ./xru-console.py check-filter snes
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Check-filter ROM]
Filter name = snes
Using No-Intro parent/clone DAT
Parsing No-Intro XML DAT
Parsing No-Intro XML file ./Nintendo - SNES Parent-Clone (20140608-202038).dat...  done
Total number of games = 3366
Number of parents = 1828
Number of clones = 1538
[Filtering ROMs]
...
<ROM set> Super Mario All-Stars and Super Mario World
   7 [IH] Super Mario All-Stars and Super Mario World (Europe).zip
   0 [IH] Super Mario All-Stars + Super Mario World (USA).zip
...
<ROM set> Super Mario RPG - Legend of the Seven Stars
   0 [IH] Super Mario RPG - Legend of the Seven Stars (USA).zip
  -3 [EH] Super Mario RPG (Japan).zip
<ROM set> Super Mario World
  11 [IH] Super Mario World (Europe) (Rev 1).zip
   7 [IH] Super Mario World (Europe).zip
   0 [IH] Super Mario World (USA).zip
  -3 [EH] Super Mario World (Japan) (En) (Arcade) [b].zip
  -3 [EH] Super Mario World - Super Mario Bros. 4 (Japan).zip
...|  ||
   |  |+-- Have flag
   |  +--- Included flag
   +------ Score

In this ROM listing, you can see the parent/clone sets, the score of each ROM, if the ROM is included or excluded, and if the ROM exists in the source directory or not. Note that if you use a No-Intro XML DAT you may have missing ROMs. However, if you do not use the DAT then you will always have all the ROMs in this list.

Updating/Copying filtered ROMs

Once you have setup your configuration file and checked the filter output (note that you can configure several filters for the same ROM source directory, for example, to place Europe/USA games and Japanese game into different destination directories) then it is time to copy your filtered ROMs from the source directory into the destination directory.

The copy <romSetName> command will filter your ROMs and then copy, one by one, every filtered ROM into the destination directory.

$ ./xru-console.py copy genesis
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Copy/Update ROMs]
Filter name = genesis-local
Using directory listing
[Reading ROMs in source dir]
Found 1728 ROMs
[Filtering ROMs]
[Scanning sourceDir for ROMs to be copied]
[Creating list of ROMs to be copied/updated]
[Copying ROMs into destDir]
  0% <Copied> 10 Super Jogos (Brazil).zip
  0% <Copied> 16 Zhang Ma Jiang (China) (Unl).zip
  0% <Copied> 3 Ninjas Kick Back (USA).zip
...
 99% <Copied> [BIOS] Sega CD 2 (USA) (v2.00W).zip
 99% <Copied> [BIOS] Sega TMSS (World).zip
 99% <Copied> [BIOS] X'Eye (USA) (v2.00).zip
[Report]
Copied 962 ROMs

The update <romSetName> command operates in a similar way to copy. However, rather than copying all the ROMs, each ROMs in sourceDir is compared with the same named ROM in destDir, and the file will be copied only if the destination ROM does not exist or the file sizes are different. This is very useful when you destination directory is network-mounted.

$ ./xru-console.py update genesis --cleanROMs
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Copy/Update ROMs]
Filter name = genesis
Using No-Intro parent/clone DAT
Parsing No-Intro XML DAT
Parsing No-Intro XML file ./Sega - Genesis Parent-Clone (20140601-111936).dat...  done
Total number of games = 1681
Number of parents = 1010
Number of clones = 671
[Filtering ROMs]
[Scanning sourceDir for ROMs to be copied]
[Creating list of ROMs to be copied/updated]
[Updating ROMs into destDir]
  0% <Copied > Action 52 (USA) (Alt 1) (Unl).zip
 31% <Copied > Gain Ground (World) (Alt 1).zip
 32% <Copied > Ghostbusters (World).zip
 35% <Copied > HardBall III (USA) (Unl).zip
...
 98% <Copied > Zany Golf (USA, Europe).zip
 99% <Copied > [BIOS] LaserActive (USA) (v1.02).zip
[Report]
Copied ROMs     47
Updated ROMs   821
[Cleaning ROMs in ROMsDest]
<Deleted> ATP Tour Championship Tennis (USA).zip
<Deleted> Action 52 (USA) (Unl).zip
...
<Deleted> [BIOS] Sega CD 2 (USA) (v2.00W).zip
Deleted 122 redundant ROMs

By default, only the newly copied files are displayed during the update process. If you want to check also the updated files, then use the option -v in the command line.

With both copy and update commands, you can use the flags --dryRun, --cleanROMs and --cleanNFO.

--dryRun will not perform any operation in your files. Use this flag if you want to test what would happened with your ROMs, and if you are happy with the results execute xru-console without --dryRun option. Note that if you use --dryRun, --cleanROMs and --cleanNFO will show you inaccurate outcomes, since the destination directory is not modified.

--cleanROMs will delete any unknown ROMs in the destination directory. After copying/update the filtered ROMs, the destination directory will be scanned and any ROM found not in the list of filtered ROMs will be deleted. Use this with caution! The list of deleted ROMs in destDir is always printed.

Both Advanced Launcher and ROM Collection Browser create NFO files having the scrapped ROM information, like year of the release, genre, etc. Use the --cleanNFO to remove any unknown NFO files, in a similar way to the --cleanROMs flag. The list of deleted NFO files in destDir is always printed.

Art Work

Both Advanced Launcher and ROM Collection Browser have good scrapers that will download most of the artwork for your ROMs. However, the scraping process may be very slow for huge ROM collections (for example, a collection having 2,000 ROMs may take up to 10 hours to scrap). This problem may be solved if you have locally available collections of artwork. xru-console can copy the matching artwork for your filtered collection in the directories you specify, and then you can choose the offline scraper that simply reads those files from the disk instead of downloading them from the Internet.

xru-console currently uses two types of artwork files, named thumbs and fanart, which correspond with the two artwork categories used by Advanced Launcher. Roughly speaking, thumbs are images that will be displayed for every game in a prominent position (this is XBMC skin and listing-mode dependant) and fanart will appear as a background. You can think of the thumbs as the game box and the fanart as the in-game screenshot.

Important

Artwork must have .png extension (lowercase).

Full collections of No-Intro ROMs exist out there. However, it is very difficult to find a complete artwork collection that have boxes/screenshots/titles. Partial artwork collections are available, although. To overcome this problem, xru-console will try to replace any missing piece of artwork with another one belonging to the same parent/clone set of the filtered ROM, in the case you have a partial artwork set which does not belong to the region of the ROMs you are filtering. We will see how this works later with an example.

If you have local Artwork available and want to copy it into destination folders (synchronised with your filtered ROM list), then configure the following into your ROM collection

<collection>
...
  <ThumbsSource>/home/xbmc/roms/ArtWork/nintendo-snes-boxes/</ThumbsSource>
  <FanartSource>/home/xbmc/roms/ArtWork/nintendo-snes-snaps/</FanartSource>
  <ThumbsDest  >/home/xbmc/NUC/ROMs/ArtWork/nintendo-snes-thumbs/</ThumbsDest>
  <FanartDest  >/home/xbmc/NUC/ROMs/ArtWork/nintendo-snes-fanart/</FanartDest>
...
</collection>

You may check you configuration with the check-artwork <romSetName> command. For example:

$ ./xru-console.py check-artwork genesis
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Check-ArtWork]
Filter name = genesis
Using No-Intro parent/clone DAT
Parsing No-Intro XML DAT
Parsing No-Intro XML file ./Sega - Genesis Parent-Clone (20140601-111936).dat...  done
Total number of games = 1681
Number of parents = 1010
Number of clones = 671
[Optimising ArtWork file list]
[Updating ArtWork]
...
<<  ROM  >> Sonic The Hedgehog (USA, Europe).zip
 Original   Sonic The Hedgehog (USA, Europe)
 Have T     Sonic The Hedgehog (USA, Europe).png
 Have F     Sonic The Hedgehog (USA, Europe).png
...
<<  ROM  >> Speedball 2 (Europe).zip
 Replaced   Speedball 2 - Brutal Deluxe (USA)
 Have T     Speedball 2 - Brutal Deluxe (USA).png
 Have F     Speedball 2 - Brutal Deluxe (USA).png
...
<<  ROM  >> [BIOS] X'Eye (USA) (v2.00).zip
 Not found
[Report]
Number of ROMs in destDir  = 868
Number of ArtWork found    = 722
Number of original ArtWork = 179
Number of replaced ArtWork = 543
Number of have Thumbs    = 722
Number of missing Thumbs = 0
Number of have Fanart    = 707
Number of missing Fanart = 15

Note that for the game Speedball 2 no artwork was found for the filtered game in destDir. However, a replacement belonging to another ROM in the parent/clone set was found. Some other ROMs, particularly BIOSes, won't have any artwork and will produce a miss.

When you are happy with the results, you can copy/update your Artwork with the commands copy-artwork <romSetName> and update-artwork <romSetName>. You can use the --cleanArtWork flag if you want to delete unknown Artwork into the thumbs and fanart destination directories. For example:

$ ./xru-console.py update-artwork genesis --cleanArtWork
XBMC ROM utilities - Console ROMs version 0.1.0
[Parsing config file]
[Updating/copying ArtWork]
Filter name = genesis
Using No-Intro parent/clone DAT
Parsing No-Intro XML DAT
Parsing No-Intro XML file ./Sega - Genesis Parent-Clone (20140601-111936).dat...  done
Total number of games = 1681
Number of parents = 1010
Number of clones = 671
[Optimising ArtWork file list]
[Updating ArtWork]
  1% <Missing Fanart> Adventurous Boy - Mao Xian Xiao Zi (China) (Unl)
  5% <Missing Fanart> Awesome Possum Kicks Dr Machino's Butt! (USA) (Beta)
...
 96% <Missing Fanart> World Series Baseball 98 (USA)
[Report]
Copied thumbs      0
Updated thumbs   722
Missing thumbs     0
Copied fanart      0
Updated fanart   707
Missing fanart    15
[Cleaning ArtWork]
Deleted 0 redundant thumbs
Deleted 0 redundant fanart

xru-mame

xru-mame is able to filter a MAME ROM collection. This tool is somewhat more sophisticated than xru-console because MAME keeps a full database with a lot of information about the arcade game it runs. This database can be used with a lot of filtering options to fully customize the list of games in your front-end.

xru-mame requires a command that defines the main action to do. Also, many commands require the name of the filter to use.

$ xru-mame.py [options] <command> [filterName]

The following commands are supported.

  • usage

    prints a summary of the commands and options.

  • reduce-XML

    Takes MAME XML as input, picks the useful information, and writes an stripped XML with only meaningful information. The reason for doing this is because MAME XML file is huge and takes a long time to process it. After reducing it, all subsequent processing should be much quicker.

  • merge

    Takes MAME XML (reduced) info file and Catver.ini and makes an output XML file with all the necessary information for proper game filtering.

  • list-merged

    List every ROM set system defined in the merged MAME XML information file. Use --verbose to get more information.

  • list-categories

    Reads Catver.ini and makes a histogram of the categories (prints all available categories and tells how many ROMs every category has).

  • list-drivers

    Reads merged XML database and prints a histogram of the drivers (how many games use each driver).

  • list-controls

    Reads merged XML database and prints a histogram of the game controls: buttons, players and input devices.

  • list-years

    Reads merged XML database and prints a histogram of the game release year (how many games were released on each year).

  • check-filter <filterName>

    Applies filters and checks you source directory for have and missing ROMs.

  • copy <filterName>

    Applies ROM filters defined in the configuration file and copies the contents of sourceDir into destDir. This overwrites ROMs in destDir.

  • update <filterName>

    Like copy, but only copies files if file size is different (this saves a lot of time, particularly if sourceDir and/or destDir are on a network-mounted filesystem).

  • check-artwork <filterName>

    Reads the ROMs in destDir, checks if you have the corresponding artwork files, and prints a report.

  • copy-artwork <filterName>

    Reads the ROMs in destDir and tries to copy the matching artwork to the destination directory.

  • update-artwork <filterName>

    Like copy-artwork, but only copies images if file sizes are different. This may save a lot of time.

The following options are supported.

  • -h, --help

    Print short command reference

  • -v, --verbose

    Print more information about what's going on

  • -l, --log

    Save program output in xru-mame-log.txt.

  • --logto [logName]

    Save program output in the file you specify.

  • --dryRun

    Don't modify destDir at all, just print the operations to be done.

  • --cleanROMs

    Deletes ROMs in destDir not present in the filtered ROM list.

  • --generateNFO

    Generates NFO files with game information for the launchers.

  • --cleanNFO

    Deletes unknown NFO files in the destination directory.

  • --cleanArtWork

    Deletes unknown ArtWork in destination directories.

Configuration file

xru-mame requires a configuration file, named xru-mame-config.xml, where you setup the ROM directories and filters. A typical configuration filter looks like this.

<!-- Example configuration file for XBMC ROM utilities -->
<MAMEConfig>
<General>
  <MAME_XML      >./DAT-mame/mame-0154.xml</MAME_XML>
  <MAME_XML_redux>./DAT-mame/mame-0154-reduced.xml</MAME_XML_redux>
  <Catver        >./DAT-mame/Catver.ini</Catver>
  <MergedInfo    >./DAT-mame/mame-0154-merged.xml</MergedInfo>
</General>

<MAMEFilter name="main">
  <ROMsSource>/home/xbmc/roms/roms-mame/</ROMsSource>
  <ROMsDest  >/home/xbmc/NUC-remote/ROMs/roms-mame-main/</ROMsDest>

  <ThumbsSource>/home/xbmc/roms/ArtWork/nintendo-snes-boxes/</ThumbsSource>
  <FanartSource>/home/xbmc/roms/ArtWork/nintendo-snes-snaps/</FanartSource>
  <ThumbsDest  >/home/xbmc/NUC-remote/ROMs/ArtWork/nintendo-snes-thumbs/</ThumbsDest>
  <FanartDest  >/home/xbmc/NUC-remote/ROMs/ArtWork/nintendo-snes-fanart/</FanartDest>

  <MainFilter>NoClones, NoSamples, NoMechanical, NoBIOS, NoNonworking</MainFilter>
  <Driver>not cps1</Driver>
  <Categories>not Mature and not Casino and not PinMAME and not Fruit_Machines</Categories>
  <Controls>not Mahjong and not Gambling and not Hanafuda</Controls>
  <Buttons>buttons == 1 or buttons == 2</Buttons>
  <Players>players == 1 or players == 2</Players>
  <Years>year >= 1990 and year &lt; 2000</Years>
  <!-- <YearsOpts>YearExpansion</YearsOpts> -->
</MAMEFilter>

<MAMEFilter name="cps1">
  <ROMsSource>/home/xbmc/roms/roms-mame/</ROMsSource>
  <ROMsDest  >/home/xbmc/NUC-remote/ROMs/roms-mame-cps1/</ROMsDest>

  <MainFilter>NoClones, NoSamples, NoMechanical</MainFilter>
  <Driver>cps1</Driver>
</MAMEFilter>
</MAMEConfig>

Creating the MAME ROM database

The first step before games can be filtered is to create a MAME game database that xru-mame can use. First, check what is your MAME version with the following command.

$ mame -?
M.A.M.E. v0.154 (Jul 23 2014) - Multiple Arcade Machine Emulator
Copyright Nicola Salmoria and the MAME team

MAME is an emulator: it reproduces, more or less faithfully, the behaviour of
...

In this example MAME has the version number 0.154. This number will be used in the configuration files so when you get a new version of MAME you won't mess up game databases.

Next, create the MAME XML game database. Execute the following command to place the XML database into mame-0154.xml:

$ mame -listxml > mame-0154.xml

Now, create a basic xru-mame-config.xml configuration file. Note that in this example all the MAME related game databases will be stored in the directory DAT-mame placed where you have the xru-mame.py script for convenience.

<MAMEConfig>
<General>
  <MAME_XML      >./DAT-mame/mame-0154.xml</MAME_XML>
  <MAME_XML_redux>./DAT-mame/mame-0154-reduced.xml</MAME_XML_redux>
  <Catver        >./DAT-mame/Catver.ini</Catver>
  <MergedInfo    >./DAT-mame/mame-0154-merged.xml</MergedInfo>
</General>
</MAMEConfig>

MAME XML game database is huge and has much more information than needed for game filtering. Next, the game database will be "reduced", that is, non-relevant information will be stripped off and a shorted game database will be created (named mame-0154-reduced.xml in the above configuration file example). Execute:

$ ./xru-mame.py reduce-XML
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Reducing MAME XML game database]
Reading MAME XML game database...
NOTE: this will take a looong time...
Parsing MAME XML file ./DAT-mame/mame-0154.xml...   done
Reducing MAME XML database...
Building reduced output XML file...
Writing reduced XML file ./DAT-mame/mame-0154-reduced.xml

Please be patient! This reduction process will take several minutes and about 2 GB of RAM will be used. However, all subsequent commands will be much faster. The reduction will shrink mame-0154.xml to mame-0154-reduced.xml from about 150 MB to about 15 MB.

Next, download the file Catver.ini from Progetto EMMA (make sure you get the English version) and place it in the DAT-mame directory. The following command will merge mame-0154-reduced.xml with Catver.ini to create mame-0154-merged.xml and then your are done. Execute:

XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Building merged MAME filter database]
[Parsing Catver.ini]
[Parsing (reduced) MAME XML file]
Parsing MAME XML file ./DAT-mame/mame-0154-reduced.xml...   done
[Merging MAME XML and categories]
[WARNING] Category not found for game 18wheelu
...
[WARNING] Category not found for game r4700le
[Writing output file]
Output file ./DAT-mame/mame-0154-merged.xml

If you get some some warnings about non-found categories for some games just ignore them. You can now list the games in your merged XML database.

$ ./xru-mame.py list-merged
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Short listing of reduced MAME XML]
Parsing merged MAME XML file './DAT-mame/mame-0154-reduced-merged.xml'...  done
005
|-- driver = segag80r.c
|-- sampleof = 005
|-- description = 005
|-- year = 1981
|-- manufacturer = Sega
|-- driver status = imperfect
+-- category = Maze
...
keycus_c430
|-- driver = src/mame/machine/ns11prot.c
|-- isdevice = yes
|-- description = KEYCUS C430
+-- category = System
[Report]
Number of games = 30551
Number of clones = 20513
Number of games with samples = 573
Number of devices = 907

You are now ready to start filtering your MAME ROMs.

Understanding MAME filters

In order to understand how the MAME filters work, let's try a basic filter named main. Edit your xru-mame-config.xml and add the following.

<MAMEConfig>
<General>
  <MAME_XML      >./DAT-mame/mame-0154.xml</MAME_XML>
  <MAME_XML_redux>./DAT-mame/mame-0154-reduced.xml</MAME_XML_redux>
  <Catver        >./DAT-mame/Catver.ini</Catver>
  <MergedInfo    >./DAT-mame/mame-0154-merged.xml</MergedInfo>
</General>

<MAMEFilter name="main">
  <ROMsSource>/home/wintermute/mame-roms/</ROMsSource>
  <ROMsDest  >/home/NUC-remote/ROMs/roms-mame-main/</ROMsDest>

  <MainFilter>NoClones, NoSamples, NoMechanical, NoBIOS, NoNonworking</MainFilter>
  <Driver>not cps1 and not cps2 and not cps3 and not neogeo_noslot</Driver>
  <Categories>not Casino and not PinMAME and not Fruit Machines</Categories>
  <Controls>Joy or Lightgun and not(Mahjong or Hanafuda)</Controls>
  <Buttons>buttons == 1</Buttons>
  <Players>players == 1</Players>
  <Years>year == 1990 or year == 1991</Years>
</MAMEFilter>
</MAMEConfig>

Every MAME filter requires a <MAMEFilter> tag and there the name attribute defines the name you will used to invoke other command for this filter. Every filter require the <ROMsSource> and <ROMsDest> tags, which define the source directory when you have all your ROMs and the destination directory where filtered ROMs will be copied.

The rest of the tags inside MAMEFilter allows to define filters to discard the games you don not want and keeps the ones you want.

Main filter

The first filter is configured with the tag <MainFilter>. Here you can write the following filtering options (separated by commas if more than one).

  • NoClones will filter all the cloned games. Clones are derived games, maybe in a different language other than English or bootleg games. Most users will choose this option.

  • NoSamples will filter all the games that use samples.

  • NoMechanical will remove all the mechanical games. Most of them are slot or pinball machines and do not work anyway.

  • NoBIOS will remove all BIOS ROMs. BIOS are non-playable ROMs necessary for other ROMs to work. Required BIOS are automatically added as dependencies (that is, if a game you want needs a specific BIOS or device they will be automatically included regardless of your filters), so using this option is always safe and recommended.

  • NoNonworking will remove all games that do not work or have serious emulation problems or emulation is preliminary.

Note that games belonging to the devices type are removed automatically. Those are not really games but emulated devices and are non-playable. If a ROM you want requires some device or BIOS these will be automatically added as dependencies so removing devices (and BIOS) is a safe operation.

Tip

If a ROM you want requires a device or BIOS in order to work they will be added automatically to the filtered list as dependencies regardless of your filter configuration. This guarantees that you will be able to run all your ROMs. CHDs are not supported yet as dependencies (you will need to copy CHDs manually to your destination directory).

Driver filter

After the main filter comes the driver filter, which is defined inside the tag <Driver>. By default all drivers will be included. If you specify a driver, then only games belonging to that driver will be included. The not operator in front of a driver name excludes games belonging to that driver. You can also specify multiple drivers separated by and and or operators and use parenthesis to change the operator precedence. For example:

(a) <Driver>cps1</Driver>
(b) <Driver>not cps1</Driver>
(c) <Driver>not cps1 and not cps2</Driver>
(d) <Driver>not (cps1 or cps2)</Driver>
(e) <Driver>cps1 or cps2</Driver>

(a) Include CPS1 games.
(b) Exclude CPS1 games.
(c) Exclude both CPS1 and CPS2 games.
(d) Same as (c). Just another way of writing this filter.
(e) Include CPS1 or CPS2 games.

You can list the MAME drivers with the command list-drivers.

$ ./xru-mame.py list-drivers
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Listing MAME drivers]
NOTE: clones are not included
NOTE: mechanical are not included
NOTE: devices are not included
Parsing merged MAME XML file './DAT-mame/mame-0154-merged.xml'...   done
[Final (used) drivers]
   1  calorie.c
   1  subs.c
...
 157  neogeo_noslot.c
 160  peplus.c
 199  naomi.c
 471  bfm_sc4.c

Note that you do not need to write the trailing .c when configuring driver filters.

Categories filter

Categories are controlled with the tag <Categories>. This filter and the filter language (and, or, not, (, )) works in a similar way to the <Driver> filter, but instead of drivers you specify game categories. For example:

(a) <Categories>not Casino and not PinMAME and not Fruit_Machines</Categories>
(b) <Categories>Fighter or Shooter</Categories>
(c) <Categories>Fighter and Shooter</Categories>

(a) Exclude games belonging to Casino, PinMAME or Fruit_Machines.
(b) Include only Fighter or Shooter games.
(c) Will exclude all games. Note that every game has a unique category, so
    no game can be a Fighter AND a Shooter at the same time. Be careful when 
    configuring your filters.

You can have a look at the available categories with the list-categories command.

$ ./xru-mame.py list-categories
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Listing categories from Catver.ini]
Opening ./DAT-mame/Catver.ini
[Making categories histogram]

[Final (used) categories]
    20  Climbing
...
  1342  PinMAME
  1343  Fighter
  1761  Casino
  2496  Shooter
 16779  Fruit Machines

Note that Catver.ini defines a main category and a sub-category for each game. For example:

[Category]
005=Maze / Shooter Small
10yard=Sports / Football Amer.
10yard85=Sports / Football Amer.
10yardj=Sports / Football Amer.
...     |        |
        |        +- Sub-category
        +---------- Main category

xru-mame only takes into account the main category and ignores any sub-categories. For example, the game 005 will have category Maze, 10yard Sports, etc. Also, when merging the categories into the merged XML database some categories are renamed for convenience, spaces are changed with _, and dots are removed. For example:

System / BIOS               --> BIOS
Electromechanical - PinMAME --> PinMAME
Ball & Paddle               --> Ball_and_Paddle

Any game that contains the category *Mature* either in the main category or in the sub-category will be assigned to the category Mature (no * in the final category name). In addition to the renamed, final categories you can see the original Catver.ini categories with the -v switch and even more information with -vv.

Controls filter

With the controls filter you can choose games based on the input peripherals they require. This is useful if you only have a digital joystick and will no be able to play games that require a Lightgun. Also, it allows you to remove Mahjong games and fruit machines (fruit machines typically have the controls ButtonsOnly).

The controls filter is configured with the tag <Controls>. This filter and the filter language (and, or, not, (, )) works in a similar way to the <Driver> and <Categories> filters. For example:

(a) <Controls>Joy</Controls>
(b) <Controls>Joy and not Mahjong</Controls>
(c) <Controls>not Mahjong and not Hanafuda</Controls>
(d) <Controls>not (Mahjong or Hanafuda)</Controls>
(e) <Controls>Joy and not(Mahjong or Hanafuda)</Controls>

(a) Include if Joy and maybe other controls.
(b) Include if Joy and maybe other controls, but not Mahjong.
(c) Exclude game if Mahjong or Hanafuda.
(d) Same as above
(e) Include games that use digital Joystick and exclude games which
    use Mahjong or Hanafuda controls.

You can have a look at the available controls with the list-controls command.

$ ./xru-mame.py list-controls
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Listing MAME controls]
NOTE: clones are not included
NOTE: mechanical are not included
NOTE: devices are not included
Parsing merged MAME XML file './DAT-mame/mame-0154-reduced.xml'...   done
[Input - control - type histogram (per game)]
    1  joy, mahjong, stick
    1  keypad, stick
...
  487  hanafuda, mahjong
  519  gambling
 1434  ButtonsOnly
 2874  joy
 
[Input - buttons histogram]
    9  14
   13  13
...
 1178  0
 1328  2
 
[Input - players histogram]
    1  6
    2  8
...
 2704  1
 2990  2
 
[Input - control - type histogram]
    9  Keyboard
   14  Keypad
...
  776  Mahjong
 1434  ButtonsOnly
 3134  Joy

This command additionally tells you the buttons and player statistics. Note that some games require more than one control, and this is indicated in the [Input - control - type histogram (per game)]. However, in the [Input - control - type histogram] games that use two controls are counted twice, three controls are counted three times, and so on.

Buttons filter

This filter allows you to filter games based on the number of buttons that they require. This filter works as a Python expression and all Python operators can be used with the variable buttons. Let's see this with an example. For example:

(a) <Buttons>buttons == 1 or buttons == 2</Buttons>
(b) <Buttons>buttons &lt;= 8</Buttons>

(a) Include games with 1 or 2 buttons.
(b) Include games that require 8 buttons or less. The operator <=  must be 
    written &lt;= in the configuration file because of the XML rules.

You can see the button statistics with the list-controls command. Have a look at the previous section for an example.

Players filter

This filter allows you to select games based on the number of players. Note that a game for 2 players means a game in which the two players may play simultaneously or alternated. The players filter is a Python expression over the variable players, for example:

(a) <Players>players == 1 or players == 2</Players>

(a) Include games with 1 or 2 players.

You can see the player statistics with the list-controls command. Have a look at the controls filter section for an example.

Years filter

With this filter you can choose games based on the year they were released. This is very useful to get rid of the oldies or to configure the 1980s launcher, the 1990s launcher, etc.

This filter works in a similar way to the Buttons and Players filters. Here, you can filter the variable year. For example:

(a) <Years>year == 1982</Years>
(b) <Years>year >= 1990 and year &lt; 2000</Years>

(a) Include all games released in 1982.
(b) Include games released between 1990 and 1999.

You can see the game release statistics with the list-years command.

$ ./xru-mame.py list-years
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Listing MAME controls]
NOTE: clones are not included
NOTE: mechanical are not included
NOTE: devices are not included
Parsing merged MAME XML file './DAT-mame/mame-0154-reduced.xml'...   done
[Release year histogram (raw)]
    1  2007?
    1  1973
...
    2  ????
...
    5  197?
...
  261  1996
  345  1995
  597  200?
  668  199?

[Release year histogram (trimmed)]
   22  2011
   23  2010
...
  952  1996
 1035  1995

Most games have a well determined release year, for example 1996. However, some games (most of them fruit machines) have release years of the form 199?. Also, for other games the release year is not completely determined, for example 1998?, or even not known at all (????). xru-mame will remove all the trailing question marks (for example, 1998? becomes 1998) and undetermined years will be expanded. For example, a game with a year defined as 199? will be included in the filter year == 1992. This year expansion can be controlled with the option <YearsOpts>YearExpansion</YearsOpts>. Most user will not want this option at all because games with unknown release years are usually either fruit machines or low-quality bootlegs.

Testing your filter

After you have configured your source and destination directories and your filters, you can check what games will be copied into the destination directory with the check-filter command. For the example, for the main filter configured above the results will be something like this.

$ ./xru-mame.py check-filter main
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Checking filter]
Filter name = main
[Parsing MAME merged XML]
Parsing MAME merged XML file ./DAT-mame/mame-0154-merged.xml...  done
Total number of games = 30551
Number of parents = 10038
Number of clones = 20513
[Reading ROMs in source directory]
[Applying MAME filters]
NOTE: -vv if you want to see filters in action
<Main filter>
Default filter, removing devices - Removed =   907 / Remaining = 29644
Filtering out clones             - Removed = 20513 / Remaining =  9131
User wants games with samples
Filtering out mechanical games   - Removed =  2622 / Remaining =  6509
Filtering out BIOS               - Removed =    67 / Remaining =  6442
Filtering out Non-Working games  - Removed =  2627 / Remaining =  3815
<Driver filter>
Filter = "cps1 or cps2"
Removed =  3742 / Remaining =    73
<Categories filter>
Filter = "not Mature and not Casino"
Removed =     0 / Remaining =    73
<Controls filter>
Filter = "Joy or Lightgun and not Mahjong and not Hanafuda"
Removed =     4 / Remaining =    69
<Buttons filter>
Filter = "buttons < 8"
Removed =     0 / Remaining =    69
<Players filter>
Filter = "players < 8"
Removed =     0 / Remaining =    69
<Year filter>
Filter = "year >= 1985 and year < 2005"
Year expansion deactivated
Removed =     0 / Remaining =    69
[Adding ROM dependencies]
Game mpang    depends on device qsound      - Adding  to list
[Filtered game list]
<Game> 1941     - Have ROM    - 1941: Counter Attack (World 900227) 
<Game> 1944     - Have ROM    - 1944: The Loop Master (USA 000620) 
...
<Game> xmvsf    - Have ROM    - X-Men Vs. Street Fighter (Euro 961004) 
[Report]
Number of filtered ROMs = 70
Number of have ROMs = 70
Number of missing ROMs = 0

Note than in this list only the filtered games are printed. You can see if you have that ROM or not (missing ROM) in the source directory.

Once you are happy with your filter then it's time to copy your ROMs to the destination directory.

Updating/Copying MAME ROMs

The copy <filterName> command will filter your ROMs and then copy, one by one, every filtered ROMs into the destination directory.

$ ./xru-mame.py copy main
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Copy/Update ROMs]
Filter name = main
[Parsing MAME merged XML]
Parsing MAME merged XML file ./DAT-mame/mame-0154-merged.xml...  done
Total number of games = 30551
Number of parents = 10038
Number of clones = 20513
[Reading ROMs in source directory]
[Applying MAME filters]
NOTE: -vv if you want to see filters in action
<Main filter>
Default filter, removing devices - Removed =   907 / Remaining = 29644
Filtering out clones             - Removed = 20513 / Remaining =  9131
User wants games with samples
Filtering out mechanical games   - Removed =  2622 / Remaining =  6509
Filtering out BIOS               - Removed =    67 / Remaining =  6442
Filtering out Non-Working games  - Removed =  2627 / Remaining =  3815
<Driver filter>
Filter = "cps1 or cps2"
Removed =  3742 / Remaining =    73
<Categories filter>
Filter = "not Mature and not Casino"
Removed =     0 / Remaining =    73
<Controls filter>
Filter = "Joy or Lightgun and not Mahjong and not Hanafuda"
Removed =     4 / Remaining =    69
<Buttons filter>
Filter = "buttons < 8"
Removed =     0 / Remaining =    69
<Players filter>
Filter = "players < 8"
Removed =     0 / Remaining =    69
<Year filter>
Filter = "year >= 1985 and year < 2005"
Year expansion deactivated
Removed =     0 / Remaining =    69
[Adding ROM dependencies]
Game mpang    depends on device qsound      - Adding  to list
[Creating list of ROMs to be copied/updated]
Added 70 ROMs
[Copying ROMs into destDir]
  0% <Copied> 1941.zip
  1% <Copied> 1944.zip
...
 98% <Copied> xmvsf.zip
[Report]
Copied ROMs     70

The update <filterName> command operates in a similar way to copy. However, rather than copying all the ROMs, each ROM in sourceDir is compared with the same named ROM in destDir, and the file will be copied only if the destination ROM does not exist or the file sizes are different. This is very useful when you destination directory is network-mounted. By default, only the newly copied files are displayed during the update process. If you want to check also the updated files, then use the option -v in the command line.

With both copy and update commands, you can use the flags --dryRun and --cleanROMs.

--dryRun will not perform any operation in your files. Use this flag if you want to test what will happen with your ROMs, and if you are happy with the results execute xru-mame without --dryRun option. Note that if you use --dryRun, --cleanROMs will show you inaccurate outcomes, since the destination directory is not modified.

--cleanROMs will delete any unknown ROM in the destination directory. After copying/update the filtered ROMs, the destination directory will be scanned and any ROM not found in the list of filtered ROMs will be deleted. Use this with caution! The list of deleted ROMs in destDir is always printed.

NFO files

Both Advanced Launcher and ROM Collection Browser create NFO files having the scrapped ROM information, like year of the release, genre, etc. A typical NFO file looks like this.

<?xml version="1.0" ?>
<game>
  <title>Toki (World, set 1)</title>
  <platform>MAME</platform>
  <year>1989</year>
  <publisher>TAD Corporation</publisher>
  <genre>Platform</genre>
  <plot/>
</game>

The scrapping process may be extremely slow if your ROM collection is huge. In order to speed up your MAME ROM collection scanning, you can create NFO files if you specify the --generateNFO when invoking the copy or update commands and then use the offline scrapers. Note that the generated files will have and empty plot (maybe in a future version of xru-mame the game plot will be extracted from history.dat).

Use the --cleanNFO to remove any unknown NFO files, in a similar way to the --cleanROMs flag. The list of deleted NFO files in destDir is always printed.

Artwork files

Regarding Artwork files, have a look first at the Artwork section of xru-console because in xru-mame this works in a very similar way. The main difference is that complete collections of Artwork for MAME exist, so there is no need for artwork substitution in xru-mame. Either you have the artwork or not.

Important

Artwork must have .png extension (lowercase).

If you have local Artwork available and want to copy it into destination folders (synchronised with your filtered ROM list), then configure the following into your MAME ROM filter.

<MAMEFilter>
...
  <ThumbsSource>/home/xbmc/roms/ArtWork/mame-titles/</ThumbsSource>
  <FanartSource>/home/xbmc/roms/ArtWork/mame-snaps/</FanartSource>
  <ThumbsDest  >/home/NUC-remote/ROMs/ArtWork/mame-thumbs/</ThumbsDest>
  <FanartDest  >/home/NUC-remote/ROMs/ArtWork/mame-fanart/</FanartDest>
...
</MAMEFilter>

You may check you configuration with the check-artwork <filterName> command. For example:

$ ./xru-console.py check-artwork main
XBMC ROM utilities - MAME edition version 0.1.0
[Parsing config file]
[Check-ArtWork]
Filter name = main
[Parsing MAME merged XML]
Parsing MAME merged XML file ./DAT-mame/mame-0154-merged.xml...  done
Total number of games = 30551
Number of parents = 10038
Number of clones = 20513
[Applying MAME filters]
NOTE: -vv if you want to see filters in action
<Main filter>
Default filter, removing devices - Removed =   907 / Remaining = 29644
Filtering out clones             - Removed = 20513 / Remaining =  9131
User wants games with samples
Filtering out mechanical games   - Removed =  2622 / Remaining =  6509
Filtering out BIOS               - Removed =    67 / Remaining =  6442
Filtering out Non-Working games  - Removed =  2627 / Remaining =  3815
<Driver filter>
Filter = "cps1 or cps2"
Removed =  3742 / Remaining =    73
<Categories filter>
Filter = "not Mature and not Casino"
Removed =     0 / Remaining =    73
<Controls filter>
Filter = "Joy or Lightgun and not Mahjong and not Hanafuda"
Removed =     4 / Remaining =    69
<Buttons filter>
Filter = "buttons < 8"
Removed =     0 / Remaining =    69
<Players filter>
Filter = "players < 8"
Removed =     0 / Remaining =    69
<Year filter>
Filter = "year >= 1985 and year < 2005"
Year expansion deactivated
Removed =     0 / Remaining =    69
[Adding ROM dependencies]
Game mpang    depends on device qsound      - Adding  to list
[Creating list of ROMs to be copied/updated]
Added 70 ROMs
[Artwork report]
<<  ROM  >> 1941.zip
 Original   1941
 Have T     1941.png
 Have F     1941.png
<<  ROM  >> 1944.zip
 Original   1944
 Have T     1944.png
 Have F     1944.png
...
<<  ROM  >> xmvsf.zip
 Original   xmvsf
 Have T     xmvsf.png
 Have F     xmvsf.png
Number of ROMs in destDir  = 70
Number of ArtWork found    = 70
Number of original ArtWork = 70
Number of replaced ArtWork = 0
Number of have Thumbs    = 70
Number of missing Thumbs = 0
Number of have Fanart    = 70
Number of missing Fanart = 0

Note that in xru-mame no artwork substitution is done. All artwork should be either original or missing and if not that is a bug.

When you are happy with the results, you can copy/update your artwork with the commands copy-artwork <filterName> and update-artwork <filterName>. You can use the --cleanArtWork flag if you want to delete unknown artwork into the thumbs and fanart destination directories.

xru-launcher-AL

xru-launcher-AL is a small utility that reads XBMC's Advanced Launcher configuration file and checks the launchers configured by the user. It reports if there are missing/unknown ROMs in your launcher and tells what launchers need to be updated.

$ xru-launcher-AL.py [-options] command

The following commands are supported.

  • usage

    Print usage information.

  • list

    Lists every launcher found in Advanced Launcher configuration file.

  • list-config

    Lists every launcher found in the configuration file.

  • check

    Checks the Advanced Launcher configuration file and compares against the ROM directories. It reports if Advanced Launcher should rescan the ROM launchers if outdated.

The following options are supported.

  • -h, --help

    Print short command reference.

  • -v, --verbose

    Print more information about what is going on.

  • -l, --log

    Save program output in xru-launcher-AL-log.txt

Configuration

xru-launcher-AL expects to be run on a different computer where your XBMC media centre is. The XBMC user's directory is network-mounted in the computer you run xru-launcher-AL using FUSE sshfs, nfs, samba, etc. However, it can also be run on the HTPC computer.

To start using xru-launcher-AL, create a minimalistic configuration file named xru-launcher-AL-config.xml.

<AdvancedLauncher>
<configFile>/home/NUC-remote/.xbmc/userdata/addon_data/plugin.program.advanced.launcher/launchers.xml</configFile>
</AdvancedLauncher>

The tag configFile configures the location of Advanced Launcher's configuration file. Next, use the list command to read Advanced Launcher's configuration file and list every launcher found there. You should get something like this.

$ ./xru-launcher-AL.py list       
XBMC ROM utilities - Advanced Launcher version 0.1.0
[Parsing config file]
[Listing Advanced Launcher launchers]
Parsing Advanced Launcher configuration file... done
<Launcher>
 lauch_name        = "Game Boy Advance (RetroArch VBA-Next)"
 lauch_application = /home/xbmc/bin/bin-retroarch/retroarch
 lauch_args        = -L /home/xbmc/bin/bin-libretro/vba_next_libretro.so "%rom%"
 lauch_rompath     = /home/xbmc/ROMs/roms-nintendo-gbadvance-nointro/
 lauch_thumbpath   = /home/xbmc/ROMs/artwork-AL/artwork-nintendo-gbadvance/thumbs/
 lauch_fanartpath  = /home/xbmc/ROMs/artwork-AL/artwork-nintendo-gbadvance/fanart/
 lauch_romext      = zip
...

For every launcher you want to check you need to create an entry in xru-launcher-AL configuration file. To check the launcher in the example, add this to the configuration file xru-launcher-AL-config.xml.

<AdvancedLauncher>
<configFile>/home/NUC-remote/.xbmc/userdata/addon_data/plugin.program.advanced.launcher/launchers.xml</configFile>

<launcher name="Game Boy Advance (RetroArch VBA-Next)">
<ROMsDest  >/home/NUC-remote/ROMs/roms-nintendo-gbadvance-nointro/</ROMsDest>
<FanartDest>/home/NUC-remote/ROMs/artwork-AL/artwork-nintendo-gbadvance/fanart/</FanartDest>
<ThumbsDest>/home/NUC-remote/ROMs/artwork-AL/artwork-nintendo-gbadvance/thumbs/</ThumbsDest>
</launcher>
</AdvancedLauncher>

Because xru-launcher-AL runs on a different computer that Advanced Launcher, directory names will be different, so you have to tell xru-launcher-AL the directory location on the local computer of the remote (network mounted) directories. Only the launchers listed in xru-launcher-AL's configuration file will be checked.

Once you have finished editing the configuration file, you can check the configured launchers with the list-config command. For example.

$ ./xru-launcher-AL.py list-config
XBMC ROM utilities - Advanced Launcher version 0.1.0
[Parsing config file]
[Listing configuration]
Advanced Launcher configuration file
 /home/NUC-remote/.xbmc/userdata/addon_data/plugin.program.advanced.launcher/launchers.xml
<Launcher>
 name          = Game Boy Advance (RetroArch VBA-Next)
 destDir       = /home/NUC-remote/ROMs/roms-nintendo-gbadvance-nointro/
 fanartDestDir = /home/NUC-remote/ROMs/artwork-AL/artwork-nintendo-gbadvance/fanart/
 thumbsDestDir = /home/NUC-remote/ROMs/artwork-AL/artwork-nintendo-gbadvance/thumbs/

Checking launchers

To check your launchers use the check command. You will see some information of the process and at the end a report of the launchers not checked, launchers checked and up to date, and launchers checked which need an update.

$ ./xru-launcher-AL.py check
XBMC ROM utilities - Advanced Launcher version 0.1.0
[Parsing config file]
[Checking Advanced Launcher launchers]
Parsing Advanced Launcher configuration file... done
<Launcher> 'Game Boy Advance (RetroArch VBA-Next)'
 Not found AL configuration file ROM path
 Using configured ROM local path for this launcher
 0 not found ROMs in AL configuration file
 0 missing ROMs in AL configuration file
...
<Launcher> 'SNES (Mednafen)'
 Not found AL configuration file ROM path
 Using configured ROM local path for this launcher
 0 not found ROMs in AL configuration file
 0 missing ROMs in AL configuration file
[Report]
The following launchers were not checked
 Playstation (Mednafen)
The following launchers were checked and are up to date
 MAME
 MAME CPS1
 MAME CPS2
 MAME CPS3
 MAME Neo-Geo
 SNES (Mednafen)
Advanced Launcher needs an update for the following launchers
 Game Boy Color (Mednafen)
 Game Gear (Mednafen)
 Mega Drive 32X (RetroArch Picodrive)

Examples

Simple filtering of a console ROM collection

You have a No-Intro ROM collection of the Sega Genesis and want to filter this collection to remove clone ROMs and Japanese ROMs. This is the situation.

  • The ROM collection is stored in your desktop computer in the directory /home/wintermute/No-Intro/Sega Genesis/.

  • XBMC is installed on a dedicated HTPC, which uses XBMCbuntu. XBMC runs with user xbmc.

  • The xbmc user's home directory is network-mounted in the desktop computer using FUSE's sshfs in /home/XBMC-remote/.

  • You want to store your ROM collection in the HTPC computer in the directory /home/xbmc/ROMs/roms-sega-genesis/. Note that in the desktop computer this directory is /home/XBMC-remote/ROMs/roms-sega-genesis/.

  • You prefer European/USA ROMs over Japanese ones. Also, you want to remove all Japanese games.

First, create the destination directories. Make sure your HTPC is network-mounted.

$ mkdir /home/XBMC-remote/ROMs/roms-sega-genesis

Then, create the configuration file xru-console-config.xml.

<ROMcollections>
<collection name="Sega Mega Drive" shortname="genesis">
  <ROMsSource>/home/wintermute/No-Intro/Sega Genesis/</ROMsSource>
  <ROMsDest  >/home/XBMC-remote/ROMs/roms-sega-genesis/</ROMsDest>

  <filterUpTags>Europe, World, Rev2, Rev 1, Rev A</filterUpTags>
  <filterDownTags>Japan, Beta, Proto</filterDownTags>
  <includeTags>Europe, USA</includeTags>
  <excludeTags>Japan</excludeTags>
</collection>
</ROMcollections> 

Next, check everything is all-right.

$ ./xru-console.py check-filter genesis

Finally, copy the ROMs.

$ ./xru-console.py update genesis

Splitting MAME ROMs into several launchers for each driver

You have a MAME ROM collection and want to split it into 3 folders: First one will have most games, second folder will have only CPS1 games, and third folder will have only CPS2 games. The ROM collection is stored in your desktop computer in the directory /home/wintermute/MAME/roms/. The xbmc user's home directory is network-mounted in the desktop computer using FUSE's sshfs in /home/XBMC-remote/.

First, create the destination directories. Make sure your HTPC is network-mounted.

$ pwd
/home/wintermute
$ mkdir DAT-mame
$ mkdir /home/XBMC-remote/ROMs/roms-mame-main
$ mkdir /home/XBMC-remote/ROMs/roms-mame-cps1
$ mkdir /home/XBMC-remote/ROMs/roms-mame-cps2

Create the MAME game database as described here.

Then, create the configuration file xru-mame-config.xml.

<MAMEConfig>
<General>
  <MAME_XML      >./DAT-mame/mame-0154.xml</MAME_XML>
  <MAME_XML_redux>./DAT-mame/mame-0154-reduced.xml</MAME_XML_redux>
  <Catver        >./DAT-mame/Catver.ini</Catver>
  <MergedInfo    >./DAT-mame/mame-0154-reduced.xml</MergedInfo>
</General>

<MAMEFilter name="main">
  <ROMsSource>/home/wintermute/MAME/roms-mame/</ROMsSource>
  <ROMsDest  >/home/NUC-remote/ROMs/roms-mame-main/</ROMsDest>

  <MainFilter>NoClones, NoMechanical, NoBIOS, NoNonworking</MainFilter>
  <Driver>not cps1 and not cps2 and not cps3 and not neogeo_noslot</Driver>
  <Categories>not Casino and not PinMAME and not Fruit_Machines</Categories>
  <Controls>Joy and not(Mahjong or Gambling or Hanafuda)</Controls>
  <Buttons>buttons &lt;= 8</Buttons>
  <Players></Players>
  <Years></Years>
</MAMEFilter>

<MAMEFilter name="cps1">
  <ROMsSource>/home/wintermute/MAME/roms-mame/</ROMsSource>
  <ROMsDest  >/home/NUC-remote/ROMs/roms-mame-cps1/</ROMsDest>

  <MainFilter>NoClones, NoSamples, NoMechanical</MainFilter>
  <Driver>cps1</Driver>
</MAMEFilter>

<MAMEFilter name="cps2">
  <ROMsSource>/home/wintermute/MAME/roms-mame/</ROMsSource>
  <ROMsDest  >/home/NUC-remote/ROMs/roms-mame-cps2/</ROMsDest>

  <MainFilter>NoClones, NoSamples, NoMechanical</MainFilter>
  <Driver>cps2</Driver>
</MAMEFilter>
</MAMEConfig> 

Next, check everything is all-right.

$ ./xru-mame.py check-filter main
$ ./xru-mame.py check-filter cps1
$ ./xru-mame.py check-filter cps2

Finally, copy the ROMs.

$ ./xru-mame.py update main
$ ./xru-mame.py update cps1
$ ./xru-mame.py update cps2

Changelog

Version 0.1

Released on July 2014.

  • Initial release.