Page 1 of 2

STREAM file converter / ADF converter

Posted: Mon Jun 27, 2011 12:10 am
by keir
EDIT: The toolset is now hosted on GitHub:

Re: STREAM file converter / ADF converter

Posted: Mon Jun 27, 2011 2:44 pm
by keir
Hi all,

The attached archive is a just a dump of some tools I developed in the last little while to dump Amiga disk images as raw MFM, and post-convert to ADF (assuming regular AmigaDOS format) and other custom formats (work in progress!). There are also utilities to dump the file contents of an ADF into the local host filesystem, to stuff sectors of an ADF with raw data.

A quick summary:
amiga-native: Contains Amiga tool diskread for dumping a disk to a (large, 20MB!) file. Really you need an Amiga equipped with PCMCIA-CF for this to be useful.
mfmparse/kf2rawdat: Converts a Kryoflux raw STREAM file to my own fomat (as dumped by Amiga tool diskread, above). It's a super-dumb and inefficient MFM bit-cell format, but good enough for my needs...
mfmparse/mfmparse2adf: Converts my raw dump format into an ADF -- assumes the disk is not in custom format of course!
mfmparse/mfmparse: WORK IN PROGRESS to analyse my raw dumps for known custom disk formats and convert to a more useful and efficient final format. Think IPF but not as good/flexible. ;) You'll already see some nice code documenting Copylock in there...
adfread: Read file contents of an ADF and optionally dump into local host filesystem
adfwrite: Stuff data into selected sectors of an ADF image
adfbb: Bootblock utilities for an ADF. Mainly I use for stuffing bootblock sectors and recomputing the checksum.
m68k: A nice one -- MC68k disassembler/emulator library. Two example uses -- disassemble does the obvious, copylock will emulate a RNC Copylock routine, copying decrypted code to a shadow buffer for subsequent disassembly.

It's all pretty raw and unfinished. It's all public domain. I doubt it has any financial value, but if you want to use it in any way for any purpose including proprietary, feel free!

Re: STREAM file converter / ADF converter

Posted: Mon Jun 27, 2011 3:21 pm
by karadoc

It's hard to know how good the stream spec is without somebody independently implementing it, so thanks for your work, and your questions. I can now see where the spec is a little loose with definitions, so I will update it given your feedback.

The "stream position" is the byte position in the non-OOB stream data. If you strip out all the OOB data, it should (I think) then be the same as the byte position. I guess this may be where the mismatch lies - the position values in the OOB header data does not account for any OOB data (hopefully the example below will make this clearer).

I took a brief look at your source, and I see that you are keeping the stream position in the 'i' variable, and then using various offset from that. We actually use two position variables, the file (decode) position, and a different "stream position" that is only incremented for non-OOB data (and also other things related to the actual sample values of course).

So, our decoder does something like this (using terminology from the spec):

- get next byte value for decode (Encoding Marker)
- decide how many bytes are used by this next piece of data (Next Offset: the number of sample bytes, or 4 + size indicated in the header for OOB data)
- increase current cell value based on Encoding Marker if applicable
- decide if we have finished processing a cell value (New Cell)
- if we are *not* processing OOB data:
-- if we have a New Cell: add it to the cell buffer
-- otherwise: update Stream Position by adding the Next Offset
- if we *are* processing OOB data
-- decode and store any index information
-- check Stream Position against the position values contained in Stream Read or Stream End OOB data (indicate invalid stream if they do not match)
- update decoding (stream file) position using Next Offset, ready for next byte decode

Other than in checking of the decoding, we also use the stream position in some statistical numbers for output (average bytes a second), which is a bit irrelevant here.

Anyway, hopefully that helps. I'm sure IFW will correct me if I haven't got any of this quite right.

As I think we mention in the spec, we went for optimisation of processing and bandwidth over easy decoding. This is only really meant as a wire-protocol after all. The longer-term solution is a proper file format in the form of DRAFT as you may know.

Re: STREAM file converter / ADF converter

Posted: Mon Jun 27, 2011 6:17 pm
by keir
Bingo, that fixed it! Thanks karadoc!

As you suggest, I now use the OOB messages to validate the stream's integrity -- I passed all my STREAM files through it (about 100 of them) and they all passed, so I'm now matching up with stream start/end position markers perfectly.

I've again re-uploaded the archive, now including the fixed version of my conversion utility.

I will re-upload and post here whenever I get a chance to update and improve my toolset.

Re: STREAM file converter / ADF converter

Posted: Mon Jun 27, 2011 9:48 pm
by karadoc
Ah, great stuff. :)

I'll update the spec to make this clear.

Re: STREAM file converter / ADF converter

Posted: Tue Jul 05, 2011 1:28 pm
by keir
I've done a lot of work on this over the last week. The repo over at github has grown a proper disk-handling library supporting an open format for custom disk images. It handles a decent number of Amiga game disks already, although by no means all, or even, probably most. ;) I need to support a few of the trickier track formats before I can be sure that the library's approach, and container format, is good. Still, as is it supports lots of releases by Core, Gremlin, Psygnosis (but they seem to have loads of weird disk formats), Team 17, Ocean, and others.

It can convert from Kryoflux STREAM format, as well as IPFs, and it can still generate good old ADF images, from bog-standard AmigaDOS tracks.

STREAM file converter / ADF converter

Posted: Tue Jul 05, 2011 10:57 pm
by mr.vince
Impressive. Out of curiosity: Does it support generic MFM as well as verification of data stored (verify data integrity against a certain format)?

Re: STREAM file converter / ADF converter

Posted: Tue Jul 05, 2011 11:45 pm
by keir
The current architecture is that each known format has a handler which scans for validated data. The one and currently only command-line tool accepts generic MFM per track and tries the handlers in turn until at least some good data is found. Its one concession to common sense is that, for each track, it will first try the format that apparently worked for the previous track.

My fear already, and I likely don't have 10% of the Amiga disk formats covered yet, is that this is very insufficient given the tiny hamming distance between some formats and the very poor quality checksums used. One track full of zero data could look much like the next, for example.

I may have to sharpen up the acceptance policy to include most-likely voting amongst handlers, scoring handlers according to confidence in their result (e.g., how much data they validate; quality of validation they can achieve given available checksums in their respective track formats), and ultimately allowing the user to restrict format choices per-track based on a priori knowledge.

No doubt the SPS project trod this ground for the Amiga disk formats, and more generally, going on a decade ago. :) Yet still I wonder how much of an open problem auto-format detection still is. Like I say, given the hamming distances and crap redundancy information in these old track formats, manual intervention in some cases is probably inevitable.

Re: STREAM file converter / ADF converter

Posted: Tue Jul 05, 2011 11:48 pm
by IFW
Yes, this is indeed the stuff that went into the analyser about a decade ago :)

Re: STREAM file converter / ADF converter

Posted: Wed Jul 06, 2011 7:40 am
by mr.vince
This approach might also prove problematic the moment you encounter data that resolves to two or more possible format descriptors (format ambiguity). I would say there is no way around some GUI which will allow users to manipulate the automation.