Dungeon Master for Atari ST disk image produced by dtc.exe

All questions regarding the dumping of media go here.
galibert
Posts: 15
Joined: Wed Oct 12, 2011 8:17 am

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by galibert »

The stt/stx formats are kinda annoying to use for preservation/low-level fdc emulation. They're "observational", i.e. they record the results of (more or less) all the commands you can do with the wd1772 (stx compared to stt adds read times and fuzzy bits masks). So you can fake the wd answers in an emulator, but it does not tell you how to recreate the floppy itself. You can *guess*, within limits, and I'm working on an auto-guesser for mess which now uses an extremely low-level representation internally, but it's just a guess.

Annoyingly enough as far as I can see ipf can't at that point represent every st protection accurately (fuzzy bits can only be in block sizes multiple of 16 cells, you can have one cell duration for the first 32 data bytes of a sector and another for the rest...) so there isn't really a format that can handle them. Well, mess' new format can, but it's probably a little _too_ low level, and it's missing some information I should add such as media type or, more importantly, track write splice position (important to rewrite floppies afterwards).

So, well, the current situation is globally annoying :/

OG.
User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by DrCoolZic »

galibert wrote:The stt/stx formats are kinda annoying to use for preservation/low-level fdc emulation. They're "observational", i.e. they record the results of (more or less) all the commands you can do with the wd1772 (stx compared to stt adds read times and fuzzy bits masks). So you can fake the wd answers in an emulator, but it does not tell you how to recreate the floppy itself. You can *guess*, within limits, and I'm working on an auto-guesser for mess which now uses an extremely low-level representation internally, but it's just a guess.

Annoyingly enough as far as I can see ipf can't at that point represent every st protection accurately (fuzzy bits can only be in block sizes multiple of 16 cells, you can have one cell duration for the first 32 data bytes of a sector and another for the rest...) so there isn't really a format that can handle them. Well, mess' new format can, but it's probably a little _too_ low level, and it's missing some information I should add such as media type or, more importantly, track write splice position (important to rewrite floppies afterwards).

So, well, the current situation is globally annoying :/

OG.
You seems to have a deep knowledge of the ST protections ;)
Yes you are right stt and stx are good for emulator but not to reproduce an exact backup of a FD. It really depends what you want to do if your goal is to play with a game that contain "write protections" they are good enough.

I have not yet look at IPF so I cannot comment, but if what you say about cell length block limitation is true I agree this is not good :(
For example the speedlock protection from macrodos divides into four block a sector some beeing faster some beeing slower than normal (compensating blocks) and the resulting sector being of a normal length...

For format that can handle all details I can tell you for sure that the sream files contains all the needed information. I have now be playing with this format for the last few weeks (months?) and it does contain all you need. Only drawback is the size and multiple files.
Another format that can contain all the needed information is the 20 years+ old Discovery Cartridge format. It can contain all the flux transitions, if needed, as the stream format but it also has less detail sections when detail information is not required. This results in much smaller files with all needed information. I did EXACT backup and preservation of all my important Atari FD back in the 80s.

Hum this make me think that this might be an interesting format to use ??? Ijor from Atari-Forum (the creator of stx) knows well the DC format. It actually use the DC to create stx files and also have a mix pasti/dc format. We were thinking of contacting the creator of the DC to get the sources of their SW (Ijor already did a partial reverse engineering) ...
I have a bunch of programs designed around the DC format ...
galibert
Posts: 15
Joined: Wed Oct 12, 2011 8:17 am

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by galibert »

DrCoolZic wrote: I have not yet look at IPF so I cannot comment, but if what you say about cell length block limitation is true I agree this is not good :(
Good thing is IPF is designed to be cleanly extendable. So I'm sure the sps guys will add all that's needed in due time.
DrCoolZic wrote: For format that can handle all details I can tell you for sure that the sream files contains all the needed information. I have now be playing with this format for the last few weeks (months?) and it does contain all you need. Only drawback is the size and multiple files.
In addition stream files have two annoyances: they store multiple revolutions, so you need to average the flux reversal positions somehow, and they don't have a track write splice information so you're not sure from where to start writing when you're reproducing a disk.

Note that an appropriate use of compression can ensure that flux-based formats are not that much bigger than normal formats. Mess new floppy image format is flux-based, and its size is around 2-3 times the ipf equivalent only. Internally zlib is used in a per-track fashion.
DrCoolZic wrote: Another format that can contain all the needed information is the 20 years+ old Discovery Cartridge format. It can contain all the flux transitions, if needed, as the stream format but it also has less detail sections when detail information is not required. This results in much smaller files with all needed information. I did EXACT backup and preservation of all my important Atari FD back in the 80s.
Interesting. I haven't seen any DC dump "out there". Do you have dumps where more than two consecutive cells are fuzzy? I'm really curious what patterns were used.

OG.
User avatar
mr.vince
Posts: 2144
Joined: Tue Oct 05, 2010 5:48 pm

Dungeon Master for Atari ST disk image produced by dtc.exe

Post by mr.vince »

Guys, that's already in IPF. And Speedlock is of course supported as well, please check the sources... I don't see where this limitation should come from...
User avatar
mr.vince
Posts: 2144
Joined: Tue Oct 05, 2010 5:48 pm

Dungeon Master for Atari ST disk image produced by dtc.exe

Post by mr.vince »

And just for the record: STREAM was never intended to be small. It's 1:1 the communication between board and host. Our old format outlines how good raw data can be compressed, and DRAFT will even do better...

Having STREAM documented and used for productive things is just an interim solution. We had to develop KF from scratch, and if you don't compromise, things take time.

For now it's big files, but that will change.
galibert
Posts: 15
Joined: Wed Oct 12, 2011 8:17 am

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by galibert »

mr.vince wrote:Guys, that's already in IPF. And Speedlock is of course supported as well, please check the sources... I don't see where this limitation should come from...
Hmm, you're right, I don't know why but I was mentally locked on an incorrect 1 block = 1 sector concept. The size-of-weak-blocks issue is real though, check your handling of isiemWeak.

OG.
User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by DrCoolZic »

galibert wrote:Good thing is IPF is designed to be cleanly extendable. So I'm sure the sps guys will add all that's needed in due time.
Excellent
In addition stream files have two annoyances: they store multiple revolutions, so you need to average the flux reversal positions somehow, and they don't have a track write splice information so you're not sure from where to start writing when you're reproducing a disk.
I do not consider multiple revolution as an annoyance but as a mandatory feature. This is a must to find out things like fuzzy bits. If you do not want multiple revolutions to be stored just ask the DTC program to only record one.
Note that an appropriate use of compression can ensure that flux-based formats are not that much bigger than normal formats. Mess new floppy image format is flux-based, and its size is around 2-3 times the ipf equivalent only. Internally zlib is used in a per-track fashion.
Yes size is not such a big deal. The funny things is that unformated track are the one that compress the less due to random nature of the transitions :)
Interesting. I haven't seen any DC dump "out there". Do you have dumps where more than two consecutive cells are fuzzy? I'm really curious what patterns were used.
You will not find any dump of FD. The device has been designed 20 years ago for perfect duplication of FD and was used only for that purpose. This make sense as 20 years ago nobody cared about preservation, but people did care not to ruin their FD. However the capability was there/ with DC you had the capability to either copy directly or to produce a "backup file"
If you are interested you can find info about the DC http://info-coach.fr/atari/hardware/devices/dc.php and the format of the file is described here http://info-coach.fr/atari/hardware/dev ... backup.pdf. For flux transitions look at record $000A
User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by DrCoolZic »

mr.vince wrote:Guys, that's already in IPF. And Speedlock is of course supported as well, please check the sources... I don't see where this limitation should come from...
As I said I have not yet any knowledge about IPF and I am glad to learn that there is no limitation in this area.
User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by DrCoolZic »

mr.vince wrote:And just for the record: STREAM was never intended to be small. It's 1:1 the communication between board and host. Our old format outlines how good raw data can be compressed, and DRAFT will even do better...
This is perfect nothing to worry about HD are so big that size is no more a problem and there is plenty of good compression program that can help.
Having STREAM documented and used for productive things is just an interim solution. We had to develop KF from scratch, and if you don't compromise, things take time.

For now it's big files, but that will change.
Don't scare me :geek: :geek: :geek:

When you say "interim solution" and "this will change" you mean DTC will have capability to produce other files like DRAFT. I hope you do not mean that stream will disappear? The Stream file are "perfect". They are complex because they are optimized for communication but EXTREMELY well designed and contains all that is needed.

I am trying to understand the "potential" problem mentioned by Galibert about "write splice" location as I do not see what is missing from Stream file for that purpose.

Actually I want to mention another excellent capability of the KryoFlux board and Stream file which is not so easy to get from a Discovery Cartridge: the exact location of the Index in respect to flux transitions. There is a protection in Atari where the ID filed of a sector is on purpose located at the very very end of the track and the corresponding data field at the beginning of the track. In order to reproduce this pattern you need a very precise location of the index and this is feasible with the Stream format (also it is a bit complex to get it right !!! - thanks Istvan for the explanations).
User avatar
IFW
Posts: 3080
Joined: Mon Nov 08, 2010 2:42 pm

Re: Dungeon Master for Atari ST disk image produced by dtc.e

Post by IFW »

Yes, I've said it many times and I can only repeat myself here ;)
The IPF format is a moving target, and has been designed to make any kind of extension easy to do, e.g. adding different encoders/decoders.
Whenever a new functonality is needed it gets added - that simple, really :lol:
The library has been updated several times during its lifetime already, although some of the library releases were only for developers.
The usage of the library is highly recommended - ensuring that everything works as intended is just a matter of downloading the latest version.
Since the latest sources will always be posted you may choose to recompile/update your decoder etc - just make sure the functionality is identical.

Also notice that although there are many revisions, compatibility is ensured by various functions that know exactly how to deal with data which was generated by say encoder 1.0 vs encoder 2.1 and do the necessary updates, or enable functionality as required.

The IPF files have track data, which consists of logical blocks, which consists of two gap streams and one data stream. There is no requirement for all the streams to be present and the lack or presence of a specific stream can change the decoder behaviour considerably; e.g. set or remove constraints.
Notice, having constraints and variables enables the changing of the data on the fly - without modifying its meaning within reason.
Therefore, unlike any format I am aware of, decoded IPF data can and does change dynamically, according to the decoding requirements of the host software - this is ideal for catering for software emulation vs hardware emulation vs real media use.
The logical block has nothing to do with a disk sector - it's a block of data sharing the same writing properties. Obviously, you'd often find they represent real sector data, but what you'd consider as a sector may consists of as many logical blocks as is required to correctly describe the content.

Also notice, that streams can be be bitcell oriented or data oriented and the encoder - and thus the decoder - switches between whatever is needed to represent the data at will; actually according to the script describing the format and the data synthesized from the format script.

These are very similar to how FreeForm (the language used by Trace duplicators) treats data: ie bitcell level encoding and data level encoding with multiple layers of logical and/or physical encoding.

You cannot do anything else in FreeForm or in IPF at the moment, but should the need arise to add anything, the whole system was designed to be easily extensible.

Notice, Trace - and consequently FreeForm - was used to duplicate about 98% of all commercial software starting around 1982/1983, the major exception being Japan.
IPF was designed to represent the mastering data of commercial software and make it possible to rewrite such data - to memory for emulation or to disk for real - and is ideal for that.
Post Reply