HELP WANTED: MRA file converter for MiST

[sebdel]

I have been able to render the first image out of simulation with the rom file created with your MRA tool. Thanks a lot!

Now, I'm not sure why you need the pattern attribute for first 64-bit group parts. I remember you told us it was a way to interleave files but I don't see how it is relevant there and how it works.

Code: Select all

        <group width="64">
            <!-- take two bytes from each file -->
            <part name="wlm-7.7a" pattern="01"/>
            <part name="wlm-5.9a" pattern="01"/>
            <part name="wlm-3.3a" pattern="01"/>
            <part name="wlm-1.5a" pattern="01"/>
        </group>

I started writing the MRA for Willow to test the format. However, there is a 16-bit file whose bytes need be swapped for endianness correction. I am not sure what was the final syntax you went for. What should I write?

That's great. I'm glad I can help.

So, the doc is missing, right? Here's the short(?) version:
So basically, if you say a group has a width of 64(bits), it means you will provide 64/8 = 8 bytes per value. What's inside the group describes where to read those 8 bytes from. You can also add a "repeat" attribute on groups. It will repeat that group in the output file. On the other hand, parts within groups do not support repeat, offset or length attributes. Only name and pattern.
Each character in the pattern attribute means "read 1 byte at that offset", so here, you're reading byte 0 and byte 1 (16bits) from each part, effectively interleaving them. For the next 64 bits, it will go back at the start and continue reading the files, again, using the offset in the pattern. If you wanted to change endianness you would say pattern="10".
Note1: if you don't specify a group width, it's 8 by default. If you don't specify a pattern it actually means pattern="0" (ie. read 1 byte). So, by default, you're just interleaving all the parts under the same group, byte by byte. I plan to add an attribute 'interleave="0"' for groups in order to achieve what slingshot wanted.
Note2: you can mix pattern lengths in the same group.

From all this, you get a set of rules (that are actually checked by the tool):
- (total number of bytes read from patterns in a group) * 8 must equal group width
- the offsets in a pattern must be < at the length of the pattern
- each individual part size, divided by the length of the pattern, must be equal (to avoid missing data while iterating on the group). That means a part that is read with pattern="0" (ie. byte by byte) must be twice smaller than a part that is read with pattern "01" or "10".

[jotego]

Each character in the pattern attribute means "read 1 byte at that offset", so here, you're reading byte 0 and byte 1 (16bits) from each part, effectively interleaving them. For the next 64 bits, it will go back at the start and continue reading the files, again, using the offset in the pattern. If you wanted to change endianness you would say pattern="10".

I understand it better now. Yet a couple of questions:

1. what is the default pattern when a part is say 16 bits and is embedded in a 64-bit group? Is there a default? If not, is there an error message if the pattern attribute is missing?
2. Can I use pattern for a part element that is not part of a group? Like is this a way to change the endianness of a 16-bit memory file that is a standalone file?

Code: Select all

        <part name="wlm-32.8h" pattern="10"/>

This is a case that has come up with the Willlow ROM.

3. Is it possible to have a group with only one part inside? I don't know if that makes sense, it depends on the answer to point 2, I suppose.

Finally, if the tool could output on the standard output the offsets of various parts in hexadecimal as debug information, that'd be helpful for development. Maybe with a switch argument so end users don't see it (only developers).

Thank you!

[sebdel]

1. what is the default pattern when a part is say 16 bits and is embedded in a 64-bit group? Is there a default? If not, is there an error message if the pattern attribute is missing?

a part does not have a width attribute. The pattern length *8 IS the width. So no pattern -> 8bits. The only error message is when multiple parts in a group do not respect the rule that (part length)/(pattern length) must be equal.

2. Can I use pattern for a part element that is not part of a group? Like is this a way to change the endianness of a 16-bit memory file that is a standalone file?

No. But a part can be alone in a group. I didn't test it but that should change the endianness of a part:

Code: Select all

<group width="16">
        <part name="wlm-32.8h" pattern="10"/>
</group>        

If that works that's actually pretty neat I'll add a test case for that.

regarding debug output: there's actually a trace variable at the top of main.c. If you set it to 1 or 2 you get more debug info. It doesn't have exactly what you want for now, but if you set it at 2 you get the offset used to read in a part (beware, that's quite verbose). I will add the info you want at trace level 1. Or even with the verbose argument (-v).

edit: endianess works. I added some QOL improvements today:
- log "write xx bytes @offset" in verbose mode
- -z attribute behaves more like -I from gcc: you can specify more than one and I added the mra dir and the current dir to the list of dirs where to look for the zip file. I also log this list (priority ordered) in verbose mode.

[slingshot]

I've create some MRA files for the Scramble hardware. For more widespread use, is it possible to have a Windows build? Maybe the releases section of Github repo would be a good place for that.

[sebdel]

I've create some MRA files for the Scramble hardware. For more widespread use, is it possible to have a Windows build? Maybe the releases section of Github repo would be a good place for that.

I don't have windows. I'll look into cross compiling with mingw but that could take some time. In the mean time, if someone is willing to contribute a windows port; I'll push it. I don't expect big problems but I did use the stat function so there is that.

[slingshot]

I don't have windows. I'll look into cross compiling with mingw but that could take some time. In the mean time, if someone is willing to contribute a windows port; I'll push it. I don't expect big problems but I did use the stat function so there is that.

I see. I have access to Windows 10, but I have no idea what dev environment is required. I didn't build anything on Windows for ~10 years.

[sebdel]

I don't have windows. I'll look into cross compiling with mingw but that could take some time. In the mean time, if someone is willing to contribute a windows port; I'll push it. I don't expect big problems but I did use the stat function so there is that.

I see. I have access to Windows 10, but I have no idea what dev environment is required. I didn't build anything on Windows for ~10 years.

I commited a windows binary under release/windows. It targets windows 64bits, mostly statically linked except for zlib1.dll. It's untested, I don't know what will happen with the backslash character in paths for example

edit: It's fixed now. zlib statically linked and tested with wine.

[slingshot]

Works well on Windows 10, thanks!

[slingshot]

As now there are ARC files for some features which is supported in MRAs on MiSTer, maybe it would be good if the utility could generate these files, too, using the <switches> tag or something like that. Not that I like to write XMLs, but then only one metadata file could be supplied with the cores, and the rest could be generated.
The current format described here:
https://github.com/mist-devel/mist-firm ... 0ff53c6f7e

[sebdel]

As now there are ARC files for some features which is supported in MRAs on MiSTer, maybe it would be good if the utility could generate these files, too, using the <switches> tag or something like that. Not that I like to write XMLs, but then only one metadata file could be supplied with the cores, and the rest could be generated.
The current format described here:
https://github.com/mist-devel/mist-firm ... 0ff53c6f7e

Sure, so the calipso.mra could start with:

Code: Select all

<misterromdescription>
    <name>Calipso</name>
    <rbf mod="8">Scramble</rbf>
    <switches default="3,Off">
      <dip bits="O7" name="Lives" ids="3,5"/>
      <dip bits="OA" name="Demo Sounds" ids="Off,On"/>
    </switches>
    ...    

Given a new command line attribute like -A (ARC), that would produce the calypso.rom + calipso.arc:

Code: Select all

[ARC]
RBF=SCRAMBLE
MOD=8
CONF=O7,Lives,3,5
CONF=OA,Demo Sounds,Off,On

I'm not sure about abusing the switches/dip tags. It's not really the same thing.
We could just have configuration tags instead:

Code: Select all

<misterromdescription>
    <name>Calipso</name>
    <rbf mod="8">Scramble</rbf>
    <configuration type="O7" name="Lives" values="3,5"/>
    <configuration type="OA" name="Demo Sounds" values="Off,On"/>

Which one do you prefer?

[slingshot]

As I see, <switches> is used in the MRA files, and "O" is omitted from bits. But I don't see where the mod value come from. Maybe @alanswx can give some information. I don't think the default is useful, always the first value is the default.
And I forgot to quote the CONF values, thus Demo Sounds became DemoSounds. It should be

Code: Select all

CONF="OA,Demo Sounds,Off,On"

[jotego]

I think we may have deviated from the MRA DIP syntax:

Code: Select all

	<switches default="FF,FF,C9">
		<dip bits="15"    name="Cabinet" ids="Cocktail,Upright"/>
		<dip bits="16,17" name="Coinage" ids="2c/1cr,1c/1cr,1c/2cr,Free Play" values="3,1,2,0"/>
		<dip bits="18,19" name="Lives" ids="1,2,3,5"/>
		<dip bits="20,21" name="Bonus Life After" ids="10000,15000,20000,None"/>
		<dip bits="22"    name="Difficulty" ids="Hard,Normal"/>
	</switches>

Things like OAB, should go in the XML like

Code: Select all

<dip bits="10,11"    />

There are a lot of examples here.

By the way, is there a syntax to patch a file? For example, change some bytes in a file? Just asking, not requesting.

[sebdel]

As I see, <switches> is used in the MRA files, and "O" is omitted from bits. But I don't see where the mod value come from. Maybe @alanswx can give some information. I don't think the default is useful, always the first value is the default.
And I forgot to quote the CONF values, thus Demo Sounds became DemoSounds. It should be

Code: Select all

CONF="OA,Demo Sounds,Off,On"

First, why are you adding quotes around the string here? I don't see any on your repos arc files.

Second, disregard my previous message, I misunderstood. I thought you wanted to reuse switches/dip tags to create your own mra files. But in fact you want to generate arc files using existing mra files, right?
So if I want to convert this example from the Mister Arcade core wiki:

Code: Select all

  <switches default="FF,FF,C9">
    <dip bits="15"    name="Cabinet" ids="Cocktail,Upright"/>
    <dip bits="16,17" name="Coinage" ids="2c/1cr,1c/1cr,1c/2cr,Free Play" values="3,1,2,0"/>
    <dip bits="18,19" name="Lives" ids="1,2,3,5"/>
    <dip bits="20,21" name="Bonus Life After" ids="10000,15000,20000,None"/>
    <dip bits="22"    name="Difficulty" ids="Hard,Normal"/>
  </switches>

I ignore default, read name and ids, and I guess I have to reorder ids according to values because that is not supported on Mist firmware.
So bits="15" translates to "OF". Now, what do I do with bit values > 15? Are these legit?
I see you added support for "DIP" in the firmware. I assume that is a special keyword that cannot be extracted from MRA?
I'll make sure that total CONF size does not exceed 256 bytes.
Also, yes, please someone, tell us where the MOD value is coming from.

[sebdel]

By the way, is there a syntax to patch a file? For example, change some bytes in a file? Just asking, not requesting.

You can source a file multiple times and set offset and length as you see fit. So, for example, you can patch 4 bytes at offset 1020 with:

Code: Select all

	<part name="file1.dat" length="1020" />
	<part>DE AD BE EF</part>
	<part name="file1.dat" offset="1024" />

Note: that won't work in groups.

Just asking, not requesting.

It's a bit sad that you have to say that. I know there are other communities but here you can expect respect from me at all times. That's me:

[slingshot]

As I see, <switches> is used in the MRA files, and "O" is omitted from bits. But I don't see where the mod value come from. Maybe @alanswx can give some information. I don't think the default is useful, always the first value is the default.
And I forgot to quote the CONF values, thus Demo Sounds became DemoSounds. It should be

Code: Select all

CONF="OA,Demo Sounds,Off,On"

First, why are you adding quotes around the string here? I don't see any on your repos arc files.

As I said, I forgot to add the quotes But will do it sooner or later. Or not, if the MRA can be used to generate them.

Second, disregard my previous message, I misunderstood. I thought you wanted to reuse switches/dip tags to create your own mra files. But in fact you want to generate arc files using existing mra files, right?

As I always smell some NIH syndrome in the MiSTer part, maybe we cannot re-use them fully, but yeah, that was the plan. But I'm sure we should not invent too many extensions to that file.

So if I want to convert this example from the Mister Arcade core wiki:

Code: Select all

  <switches default="FF,FF,C9">
    <dip bits="15"    name="Cabinet" ids="Cocktail,Upright"/>
    <dip bits="16,17" name="Coinage" ids="2c/1cr,1c/1cr,1c/2cr,Free Play" values="3,1,2,0"/>
    <dip bits="18,19" name="Lives" ids="1,2,3,5"/>
    <dip bits="20,21" name="Bonus Life After" ids="10000,15000,20000,None"/>
    <dip bits="22"    name="Difficulty" ids="Hard,Normal"/>
  </switches>

I ignore default, read name and ids, and I guess I have to reorder ids according to values because that is not supported on Mist firmware.

Yes, the default can be ignored, but I don't think the ids should be reordered. The order can be handled in HDL easily, I feel it's redundant.

So bits="15" translates to "OF". Now, what do I do with bit values > 15? Are these legit?
I see you added support for "DIP" in the firmware. I assume that is a special keyword that cannot be extracted from MRA?
I'll make sure that total CONF size does not exceed 256 bytes.
Also, yes, please someone, tell us where the MOD value is coming from.

It's not hex, but can continue up to 'Z'. However as the status word is 32 bit, W is the highest acceptable option char. Yes, the DIP is the placeholder for the settings from MRA. I think 256 byte is enough, but it can be increased easily if not. As the ARM CPU has limited RAM, it's better to not use all free space for the config string.

[Sorgelig]

values main task is not reordering. Some dips may occupy for example 4 bits while only like 2 or 3 values are used from all 16 possible values. So you write only 2 ids and their values.

About mod parameter:
On MiSTer MRA, this parameter is not really mod. It's just binary data sent with index 1. How it will be treated by core it's up to the core. Currently only one byte is used which basically means the mod. But imagine some core for flexibility can use 2 or 3 bytes. One byte is one feature for example audio type, another byte is video type. So in this case the binary for index 1 will have 2 bytes. So this way is more flexible than limit it to a single mod parameter with number.

MRA has patch node which looks like:
<patch offset="xxx">NN NN NN NN....</patch>
Which just replaces the bytes on given offset on earlier loaded ROM. MiSTer constructs the ROM fully in ARM memory before sending to FPGA.

There is also <interleave> node used for parts interleaving or byte shuffling. Example is in MRAs of MCR3Scroll core.

[slingshot]

I miss one thing from the DIP settings: is it a toggle switch (default, type O) or a push button (type T). It would be useful in the Defender cores for the Advance button for example. Maybe an attribute for the <dip> tag, like <dip type="T".../>.

[slingshot]

So at the end we can use the data in <rom index="1"> as the mod value if it's a single number. Feels hacky, but it's ok if it won't be changed. As we don't embed any upload data in .ARC files, the actual index and other content never will used.
But I starting to feel the best would be to forget ARC file generating.

[sebdel]

So at the end we can use the data in <rom index="1"> as the mod value if it's a single number. Feels hacky, but it's ok if it won't be changed. As we don't embed any upload data in .ARC files, the actual index and other content never will used.
But I starting to feel the best would be to forget ARC file generating.

Don't do this to me man, it's already working
Try this: https://github.com/sebdel/mra-tools-c/tree/write_arc_file/release/windows
I extracted MOD from ROM @index=1. It has to be a single byte in a single part. Let me know if this is too restrictive.
Is it important to convert RBF to upper case? I'll do it if needed.

[sebdel]

edit: deleted. sorry, misread what you were saying.

[sebdel]

edit: deleted. sorry, misread what you were saying.

[slingshot]

So at the end we can use the data in <rom index="1"> as the mod value if it's a single number. Feels hacky, but it's ok if it won't be changed. As we don't embed any upload data in .ARC files, the actual index and other content never will used.
But I starting to feel the best would be to forget ARC file generating.

Don't do this to me man, it's already working

As I checked several MRAs, the ROM format already differs. Even for Scramble, which was released 2 days ago. I have better things to do than following MiSTer changes when it's a change just because. And surely I won't change all the 19 MRAs now.
But if it's already there, you don't have to remove the code Maybe we can use it in the future sometimes.

Try this: https://github.com/sebdel/mra-tools-c/tree/write_arc_file/release/windows
I extracted MOD from ROM @index=1. It has to be a single byte in a single part. Let me know if this is too restrictive.
Is it important to convert RBF to upper case? I'll do it if needed.

I don't think it's restrictive, since we cannot use anything else
The fat driver uses strncmp for file matching, and the filesystem stores the (short) file names in uppercase, so yes, conversion is needed.

[sebdel]

values main task is not reordering. Some dips may occupy for example 4 bits while only like 2 or 3 values are used from all 16 possible values. So you write only 2 ids and their values.

About mod parameter:
On MiSTer MRA, this parameter is not really mod. It's just binary data sent with index 1. How it will be treated by core it's up to the core. Currently only one byte is used which basically means the mod. But imagine some core for flexibility can use 2 or 3 bytes. One byte is one feature for example audio type, another byte is video type. So in this case the binary for index 1 will have 2 bytes. So this way is more flexible than limit it to a single mod parameter with number.

MRA has patch node which looks like:
<patch offset="xxx">NN NN NN NN....</patch>
Which just replaces the bytes on given offset on earlier loaded ROM. MiSTer constructs the ROM fully in ARM memory before sending to FPGA.

There is also <interleave> node used for parts interleaving or byte shuffling. Example is in MRAs of MCR3Scroll core.

Thanks for the info, that helps.
I noticed you compute the MD5 before you interleave data to the destination. Is that correct? What is the ROM MD5 exactly?

[sebdel]

As I checked several MRAs, the ROM format already differs. Even for Scramble, which was released 2 days ago. I have better things to do than following MiSTer changes when it's a change just because. And surely I won't change all the 19 MRAs now.
But if it's already there, you don't have to remove the code Maybe we can use it in the future sometimes.

Understood. So no need to rush to support interleave tags ? (they're very similar to group but not quite the same)

[slingshot]

As I checked several MRAs, the ROM format already differs. Even for Scramble, which was released 2 days ago. I have better things to do than following MiSTer changes when it's a change just because. And surely I won't change all the 19 MRAs now.
But if it's already there, you don't have to remove the code Maybe we can use it in the future sometimes.

Understood. So no need to rush to support interleave tags ? (they're very similar to group but not quite the same)

I think it was mainly requested by Jotego. Also the ARC generation would be very useful for him, since he has control over both platform versions of his cores.