HELP WANTED: MRA file converter for MiST

[jotego]

MRA file format is becoming popular on MiSTer as a way to specify ROM loading into the FPGA core. The format is increasingly supporting more file operations making it easier for the core developer.

MiST and other platforms do not have the horse power to do the conversion in real time inside their small microcontroller (some platforms don't even have one). However, we could have a small executable on a PC to run the MRA file and obtain the ROM file for the core. This would be useful in unifying the open source FPGA platforms about ROM loading, simplify life to the user and also take work load off the core developer shoulders.

Any software guru willing to take up this task?

[sebdel]

MRA file format is becoming popular on MiSTer as a way to specify ROM loading into the FPGA core. The format is increasingly supporting more file operations making it easier for the core developer.

MiST and other platforms do not have the horse power to do the conversion in real time inside their small microcontroller (some platforms don't even have one). However, we could have a small executable on a PC to run the MRA file and obtain the ROM file for the core. This would be useful in unifying the open source FPGA platforms about ROM loading, simplify life to the user and also take work load off the core developer shoulders.

Any software guru willing to take up this task?

Everytime I say I will do something, I end up dropping the ball because my life is hectic, but that's the kind of thing I can do on my work time.
So as long as you don't mind nodejs I could do something I guess.
Here's a start: a super basic tool to dump the rom part of the file to json.

Could you be more specific? What kind of output do you expect?

[slingshot]

It should take the MRA file and the .zip given in that, and should produce one or more .ROM file(s) (basically concatenated pieces from the part name attribute). Example:
https://github.com/alanswx/MraExamples/ ... de/Jin.mra
It should take jin.zip, and produce a jin.rom file which consists of jin11.6c + jin12.7c + jin13.6d + jin14.4c + jin15.3f + jin.1a + jin.1l

Btw, if it would be a really FPGA-independent format, then the "misterromdescription" should be something else, like "multiromarchivedescription", it even resembles the MRA.

[jotego]

Everytime I say I will do something, I end up dropping the ball because my life is hectic, but that's the kind of thing I can do on my work time.
So as long as you don't mind nodejs I could do something I guess.
Here's a start: a super basic tool to dump the rom part of the file to json.

Could you be more specific? What kind of output do you expect?

As Slingshot said: it has to produce a binary file out of the MAME romset (ideally zipped) by reading the MRA file and concatenating the files it points to. All those <part name="xx"> in the MRA file refer to a file inside the zip file. These parts should be dumped one after the other to form a binary file, which we can use to feed the core.

I tried running your initial commit but run into problems:

Code: Select all

jtejada@portatil:~/github/others/mra-tools$ npm install
npm WARN npm npm does not support Node.js v10.15.2
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/
npm WARN tar write after end
npm WARN tar write after end
npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jtejada/.npm/_logs/2020-01-16T21_14_09_187Z-debug.log
jtejada@portatil:~/github/others/mra-tools$ npm start test.mra
npm WARN npm npm does not support Node.js v10.15.2
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/

> mra-tools@1.0.0 start /home/jtejada/github/others/mra-tools
> babel-node src/index.js "test.mra"

sh: 1: babel-node: not found
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! mra-tools@1.0.0 start: `babel-node src/index.js "test.mra"`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the mra-tools@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jtejada/.npm/_logs/2020-01-16T21_14_33_563Z-debug.log

Thanks a lot for giving this a shot

[sebdel]

It should take the MRA file and the .zip given in that, and should produce one or more .ROM file(s) (basically concatenated pieces from the part name attribute). Example:
https://github.com/alanswx/MraExamples/ ... de/Jin.mra
It should take jin.zip, and produce a jin.rom file which consists of jin11.6c + jin12.7c + jin13.6d + jin14.4c + jin15.3f + jin.1a + jin.1l

Btw, if it would be a really FPGA-independent format, then the "misterromdescription" should be something else, like "multiromarchivedescription", it even resembles the MRA.

Ok done. Sort of. It relies on unzip and md5sum so that's linux only ATM but if it works for you and there's interest I can make it cross platform. I only tested with digdug but the md5 matches so it works in some of the cases at least. Can someone test more?
I ignored a ton of info from the MRA by the way. What do index and type do on this line: <rom index="0" zip="digdug.zip" md5="d6790fb2893b2290b4ac74544e4d78e6" type="merged|nonmerged">, do they matter?

[sebdel]

Everytime I say I will do something, I end up dropping the ball because my life is hectic, but that's the kind of thing I can do on my work time.
So as long as you don't mind nodejs I could do something I guess.
Here's a start: a super basic tool to dump the rom part of the file to json.

Could you be more specific? What kind of output do you expect?

As Slingshot said: it has to produce a binary file out of the MAME romset (ideally zipped) by reading the MRA file and concatenating the files it points to. All those <part name="xx"> in the MRA file refer to a file inside the zip file. These parts should be dumped one after the other to form a binary file, which we can use to feed the core.

I tried running your initial commit but run into problems:

Code: Select all

jtejada@portatil:~/github/others/mra-tools$ npm install
npm WARN npm npm does not support Node.js v10.15.2
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/
npm WARN tar write after end
npm WARN tar write after end
npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jtejada/.npm/_logs/2020-01-16T21_14_09_187Z-debug.log
jtejada@portatil:~/github/others/mra-tools$ npm start test.mra
npm WARN npm npm does not support Node.js v10.15.2
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/

> mra-tools@1.0.0 start /home/jtejada/github/others/mra-tools
> babel-node src/index.js "test.mra"

sh: 1: babel-node: not found
npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! mra-tools@1.0.0 start: `babel-node src/index.js "test.mra"`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the mra-tools@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jtejada/.npm/_logs/2020-01-16T21_14_33_563Z-debug.log

Thanks a lot for giving this a shot

node+npm on linux is a mess
Just do this, don't worry about the version you previously installed:
sudo apt-get install curl (if necessary)
curl -sL https://deb.nodesource.com/setup_13.x | sudo -E bash -

[slingshot]

It's good to have a helping hand in this, and thank you, but I have a concern about node.js is a really good choice as a requirement for an end-user tool. No problem with that for prototyping. (actually I wonder about what script language is OK for both in Windows and Linux).

[sebdel]

It's good to have a helping hand in this, and thank you, but I have a concern about node.js is a really good choice as a requirement for an end-user tool. No problem with that for prototyping. (actually I wonder about what script language is OK for both in Windows and Linux).

Actually, nodejs is one of the best cross platform scripting solution these days. It's up there with python.
I completely understand if you are just allergic to javascript as a language and node as a platform though. I was still in that boat like a year ago.

Regarding end users, note that the current project is intended for devs only. Once built, it will be packaged to a single file and you won't need npm anymore. It will be a single file download, but you will still need to execute it through the node runtime, just like a python app.

So short term, given that it's the dev environment on my PC right now I'll at least finish with that. It's also the language I have the most fun using ATM: With the appropriate babel settings it's functional programming all the way. You could write a MRA to ROM converter in one line if you really wanted

so, I will:
- eliminate all bash calls. That should make it cross platform by definition and also not rely on temp files. It should be straight from the unzip stream to the rom file.
- provide a proper build.
- look into solutions to compile nodejs apps to a single file executable.

Longer term, I could port it to C. My second most beloved language, but that will take way more time. I don't look forward to using libxml

[slingshot]

I have nothing against it, if it's really cross-platform, and I don't have to install half of the internet to make it work
And I agree, libs are the way to go instead of forking external apps.

[sebdel]

I have nothing against it, if it's really cross-platform, and I don't have to install half of the internet to make it work
And I agree, libs are the way to go instead of forking external apps.

I added a link to download a pre-built executable here: https://github.com/sebdel/mra-tools (>50MB ) I'm willing to bet that the C port will be <50KB.

edit: ok, added the missing dep in the zip file, so it's 2 files in the end. It's still pretty big for what it does but

[slingshot]

The binary download gives me an error. But just a thought: the parser should be already implemented in the MiSTer firmware.

[sebdel]

The binary download gives me an error. But just a thought: the parser should be already implemented in the MiSTer firmware.

How about now?
I'll look into how they do it. Who in their right mind feeds xml to a low end device though.

[sebdel]

The binary download gives me an error. But just a thought: the parser should be already implemented in the MiSTer firmware.

How about now?
I'll look into how they do it. Who in their right mind feeds xml to a low end device though.

They're using sxmlc and there's certainly a lot of code to copy from romutils. Well played, that will get me started.

edit oops, didn't mean to double post, sorry

[sebdel]

By popular demand of slingshot, here's the same sh*t in C: https://github.com/sebdel/mra-tools-c/

Only the md5 check is missing, plus some command line options, but it should work.

[slingshot]

By popular demand of slingshot, here's the same sh*t in C: https://github.com/sebdel/mra-tools-c/

Only the md5 check is missing, plus some command line options, but it should work.

This one is neat, tried it, works well. Maybe a configurable directory for MAME .zips would make it even more easy to use.

[sebdel]

By popular demand of slingshot, here's the same sh*t in C: https://github.com/sebdel/mra-tools-c/

Only the md5 check is missing, plus some command line options, but it should work.

This one is neat, tried it, works well. Maybe a configurable directory for MAME .zips would make it even more easy to use.

Sure, that was the plan all along. Check out the latest version.
Say you have mras in mra/ and roms in roms/ you can do:

Code: Select all

mra-tools mra/digdug.mra -z roms

now only the md5 check is missing. And maybe... do you know what are <rom /> "index" and "type" attributes by any chance? Is there anything to do with them?

[slingshot]

now only the md5 check is missing. And maybe... do you know what are <rom /> "index" and "type" attributes by any chance? Is there anything to do with them?

A merged ROM can contain more ROMsets for the same game. That means there can be more than one .ROM file as an output. Maybe Jose has an example MRA for this.

[hyperterminal]

Look for "MAME217RomsOnlyMerged" on archive.org for examples for merged romsets. A merged zip has the parent rom in the main folder and clones in the subfolders. Those subfolders do not contain the full romset for the clones but only the roms that differ from the rest of the archive. When clones share roms with other clones, these roms are only included in one of the clone's subfolders, hence the name merged.

[sebdel]

I just checked the Arcade cores thread on mister forum and this is way more complicated than I thought it would.
I'll implement the full thing but as of right now, we only support the "1 zip to 1 ROM" use case.

[slingshot]

One thing would be good: sometimes it's convenient to re-use the same part more than one time (e.g. address space mirroring, or just to have a placeholder). Now if one part is listed more than once, the tool complains that it cannot be found.

[sebdel]

One thing would be good: sometimes it's convenient to re-use the same part more than one time (e.g. address space mirroring, or just to have a placeholder). Now if one part is listed more than once, the tool complains that it cannot be found.

Could you tell me the name of the MRA when you test things? For this particular case you can try the branch full_mra_parser. It should behave differently in this regard and it also supports "select parts by crc" although I couldn't test that.

[slingshot]

Could you tell me the name of the MRA when you test things? For this particular case you can try the branch full_mra_parser. It should behave differently in this regard and it also supports "select parts by crc" although I couldn't test that.

I've just experimented with an MRA created by me. But easy to check it: just duplicate any part name line.

[sebdel]

Could you tell me the name of the MRA when you test things? For this particular case you can try the branch full_mra_parser. It should behave differently in this regard and it also supports "select parts by crc" although I couldn't test that.

I've just experimented with an MRA created by me. But easy to check it: just duplicate any part name line.

I merged my dev branch. The parser is mostly complete. I added support for selecting files by crc, and repeat, offset and length attributes. It will be a pretty useful tool in the end, you can use it to generate bin files from templates, stitch files together, truncate files, apply binary patches and so on.

on my todo list:
- support hex binary blobs embedded in the MRA
- support "zip" attribute per part
- implement the weird fallback logic when the file contains multiple ROM0 sections
- check md5 of the final file

[jotego]

I don't come in a couple of days and what do I find?

By popular demand of slingshot, here's the same sh*t in C: https://github.com/sebdel/mra-tools-c/

A C version. Great!

C or C++ are my preferred option for this. I think users need executables files, both in linux and windows. I tried once to distribute a Python file and people had trouble with it.

As for the tool, I do need it now as I need to prepare proper ROM files for the CPS1 core. There is one extra feature I need though: support for split data. Let me explain.

Sometimes ROM files match the system bus in odd ways. MAME has a neat way of expressing this with macros:

Code: Select all

ROM_START( sf2 )
	ROM_REGION( CODE_SIZE, "maincpu", 0 )      /* 68000 code */
	ROM_LOAD16_BYTE( "sf2e_30g.11e", 0x00000, 0x20000,...
	ROM_LOAD16_BYTE( "sf2e_37g.11f", 0x00001, 0x20000, ...

This means: make one 16-bit memory by using one byte from each file. This is the case because the board uses two 8-bit memories to form a 16-bit memory.

This can become more complicated:

Code: Select all

    ROM_REGION( 0x600000, "gfx", 0 )
    ROM_LOAD64_WORD( "sf2-5m.4a",  0x000000, 0x80000, ...
    ROM_LOAD64_WORD( "sf2-7m.6a",  0x000002, 0x80000, ...
    ROM_LOAD64_WORD( "sf2-1m.3a",  0x000004, 0x80000, ...
    ROM_LOAD64_WORD( "sf2-3m.5a",  0x000006, 0x80000, ...

This means: make a 64-bit memory by using four 16-bit memories. So take two bytes from the first file, then two from the second file, etc. Note that files are not dumped one after the other: they are interleaved.

You do get the easy case too:

Code: Select all

    ROM_REGION( 0x200000, "gfx", 0 )
    ROM_LOAD64_BYTE( "nm_09.4b",  0x000000, 0x20000,
    ROM_LOAD64_BYTE( "nm_01.4a",  0x000001, 0x20000,
    ROM_LOAD64_BYTE( "nm_13.9b",  0x000002, 0x20000,
    ROM_LOAD64_BYTE( "nm_05.9a",  0x000003, 0x20000,
    ROM_LOAD64_BYTE( "nm_24.5e",  0x000004, 0x20000,
    ROM_LOAD64_BYTE( "nm_17.5c",  0x000005, 0x20000,
    ROM_LOAD64_BYTE( "nm_38.8h",  0x000006, 0x20000,
    ROM_LOAD64_BYTE( "nm_32.8f",  0x000007, 0x20000,

This just uses one byte from each file to complete the 64 bits. Note that the final memory width is not relevant for the output file as you just output bytes. You just have to take it into account to know how many bytes to take from the files.

Now, this is a new yet needed feature for MRA's so syntax is a bit free at the moment. Something like this will do:

Code: Select all

<part>
	<!-- Merge two 16-bit based files -->
	<name "file0" width="16" endian="big"/>
	<name "file1" width="16" endian="big"/>
</part>
<part>
	<!-- Merge four 8-bit based files -->
	<name "file0"/>
	<name "file1"/>
	<name "file2"/>
	<name "file3"/>
</part>

I think you don't need to express the final memory bit width as MAME does as we just need the straight byte sequence. For 16-bit files, I think we need the option to select endianness. I don't know if there 32-bit files in more modern games. I haven't encountered them yet. It is important to preserve the file order, so if the XML parser loses the order of the statements, we may need to indicate the order as an attribute.

What do you think, sebdel?

[slingshot]

Code: Select all

ROM_START( sf2 )
	ROM_REGION( CODE_SIZE, "maincpu", 0 )      /* 68000 code */
	ROM_LOAD16_BYTE( "sf2e_30g.11e", 0x00000, 0x20000,...
	ROM_LOAD16_BYTE( "sf2e_37g.11f", 0x00001, 0x20000, ...

This means: make one 16-bit memory by using one byte from each file. This is the case because the board uses two 8-bit memories to form a 16-bit memory.

I've just solved this during ROM uploading, SpyHunter for example has 8,16 and 32 bit parts:
https://github.com/Gehstock/Mist_FPGA/b ... ST.sv#L135
https://github.com/Gehstock/Mist_FPGA/b ... ST.sv#L165