All you always wanted to know about IPF

3rd Party Software, Tools & Add-ons for KryoFlux
Post Reply
User avatar
DrCoolZic
Posts: 164
Joined: Tue Jul 26, 2011 10:44 am

All you always wanted to know about IPF

Post by DrCoolZic » Mon Jan 16, 2012 5:41 pm

This thread is about “all you always wanted to know” about IPF!!!
This first post will be edited so that is always contains the latest info ... So stay tuned
Jean

edit September 2012 viewtopic.php?f=7&t=365&p=6151#p6151
New version of the IPF documentation available at my Web site

edit February 4, 2012 viewtopic.php?f=7&t=365&p=2994#p2994
- ipfdump 0.0a : read track, read address, read sectors as a WD1772 on information provided by the CAPS API
- ipfinfo v0.3b : almost same as 0.3a but only 16KB instead of 1.6MB

edit January 30, 2012 viewtopic.php?f=7&t=365&start=20#p2912
- IPFInfo v0.3a : fix bug in decoding gap + offset is now actual offset in the file so can follow with hex editor
- example dump from very interesting Theme Park Mystery

edit January 27, 2012 - see viewtopic.php?f=7&t=365&start=10#p2889
- IPF documentation v0.0a
- IPFInfo v0.2a executable.
- Example output of IPFInfo on Action Fighter

edit January 16, 2012
- IPF-42A-12016.rar : Contains the modified SPS 4.2 source tree to compile under Microsoft VC++2010 as well as the source of the IPFInfo program
- IPF Documentation V0.0 for review: IPF file format, IPFLib 4.2A Documentation, IPInfo v0.0 documentation
- IPFInfo.exe Windows IPF dump file based on the work done by Keir Fraser
Attachments
IPF Documentation v00.rar
IPF documentatation
(181.43 KiB) Downloaded 270 times
IPF-42A-120116.rar
IPFLib + IPFInfo sources
(70.31 KiB) Downloaded 260 times
Last edited by DrCoolZic on Sun Sep 08, 2013 11:46 am, edited 8 times in total.

User avatar
DrCoolZic
Posts: 164
Joined: Tue Jul 26, 2011 10:44 am

Re: All you always wanted to know about IPF

Post by DrCoolZic » Mon Jan 16, 2012 5:50 pm

In this thread I would like to discuss “all you always wanted to know” about IPF!!!

So most of the information will probably come from SPS people, but anybody having knowledge about IPF is of course welcome to answer and hopefully I should be able to also provide information

This thread is intended for programmers and is probably of no interest to ipf users.

so I expect questions like:
  • Format of IPF file
  • CAPS library compilation, usage
  • Emulator integration
But not question like:
  • How do I get submit games to get IPF
  • Where can I find IPF files
  • Etc.
To start the discussion I post :
  • a very preliminary IPF documentation
  • a CAPS library source tree that compile under Visual Studio C++ 2010
  • an IPF dump program
This documents are provided in the first post of this thread

User avatar
DrCoolZic
Posts: 164
Joined: Tue Jul 26, 2011 10:44 am

Re: All you always wanted to know about IPF

Post by DrCoolZic » Mon Jan 16, 2012 6:33 pm

You will find attached the dump of an Atari game called FoundationWaste
It is one of the game that I have used to verify my IPFInfo program based on program from Keir Fraser.
and I already have lots of questions.

So we have the IMGE records that point to DATA records to find the actual data content...

Seems like track data are divided into a number of blocks that (in most cases) correspond to the number of sectors. I said in most cases because from what I understand it might be necessary to have more blocks. For example for a sector with variable bit rates region (e.g. macrodos/speedlock) might need to be further divided into more regions ???

But for now lets assume 1 block is one sector.
If I can make sense of most of the information in the different records, I cannot make sense of the data information placed after the block descriptor???

Reminder: A track is usually composed of:
  • a post-index region (say GAP1)
  • several repeat of (for each sector):
  • -pre-address gap field (say GAP2)
  • -address field
  • -post-address + pre-data gap field (say GAP3)
  • -data field
  • -post-data gap field (say GAP4)
  • End of repeat
  • a pre-index field (say GAP5)
In IPF I find a block descriptor for each sector that contains an offset to get to the DATA and GAP information. But if I look at the content it does not make sense for me :(

The Gap offset points to strange unknown data? I am very familiar at looking at track/sector content in Hex but here I find nothing known?

The data offset points to some information placed in front of address info followed by GAP3 followed by strange info followed by data field.
Something like <unknown_info> <address_field> <partof_gap3> <data_field> ...

Code: Select all

22 08 4E 00 22 08 4E 21 38 22 08 00 00 21 06 44 
89 44 89 44 89 22 07 FE 00 00 05 02 06 AB 23 1C 
4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 
4E 4E 4E 4E 4E 4E 00 00 00 00 00 00 21 06 44 89 
44 89 44 89 42 02 03 FB 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
So it look like a mixture of gap/address/data fields with a lot of things missing like sync byte part of gaps etc.

May be I am asking stupid question but this looks strange???
First gap seems to point to info that does not look like gap info
Then data point to a mixture of everything.

I would have expected more something like what I have described in the track?
What about pre-index and post-index info?
what about sync mark?
what about gap content?
why mix gap/and data

As you can see currently I am a bit lost. But I suspect that if I look at the code I will have the answer.
Any comment to help me understand is welcome
Attachments
f1.rar
Foundation Waste dump
(15.38 KiB) Downloaded 191 times
Last edited by DrCoolZic on Fri Jan 27, 2012 9:31 am, edited 1 time in total.

keir
Posts: 35
Joined: Sun Jun 26, 2011 11:03 pm

Re: All you always wanted to know about IPF

Post by keir » Tue Jan 17, 2012 10:24 am

Hi,

The format for a data stream is (using dodgy pseudocode):

Code: Select all

 uint8_t cntlen:3; // bits 7:5 indicate #bytes for length field
 uint8_t type:5; // enum cpdatXXX (0=end,1=sync,2=data are the only field types I've used)
 uint8_t length[cntlen];
 uint8_t data[length];
 
So, for example, a sync mark 0x448944894489 might be encoded as:

Code: Select all

0x21,0x06,0x44,0x89,0x44,0x89,0x44,0x89
These chunks repeat until you see a zero type/cntlen byte.

I assume gap data is a bit similar but I haven't really investigated. I'm not sure that the division of data into blocks/sectors is important. I make a stab at it in my own library, but the only formats it definitely matters for are non-uniform density tracks where different sectors have different bitcell timings.

The key to understanding this stuff is files Capsdef.h and CapsImgS.cpp in the IPF decoder sources. Also I built my own version of the IPF library with tracing added to find out what paths get taken. :)

If you want to see my code for generating IPF data streams, it's in libdisk/container_ipf.c in my github repository. Function ipf_tbuf_bit() is the function to append a bit of data to a data-chunk stream.

User avatar
DrCoolZic
Posts: 164
Joined: Tue Jul 26, 2011 10:44 am

Re: All you always wanted to know about IPF

Post by DrCoolZic » Tue Jan 17, 2012 4:44 pm

Thanks. This is interesting information.
As I said I am just starting to look at IPF and for sure I still have a lot of things to learn ;)

keir
Posts: 35
Joined: Sun Jun 26, 2011 11:03 pm

Re: All you always wanted to know about IPF

Post by keir » Wed Jan 18, 2012 12:29 am

I have some ST IPFs which should be exercising newer IPF features like gap streams a lot more than my old Amiga IPFs. I will study them in more detail over the next few days.

I removed what I previously listed here about gap streams, it was misleadingly incomplete and wrong. ;)

I will write up a text file describing data and gap streams as comprehensively and accurately as I can. If stuff from that is useful to help with DrCoolZic's big PDF document, that's great.

keir
Posts: 35
Joined: Sun Jun 26, 2011 11:03 pm

Re: All you always wanted to know about IPF

Post by keir » Wed Jan 18, 2012 6:52 pm

I checked in some draft text to my github repository: https://github.com/keirf/Amiga-Disk-Uti ... fo/ipf.txt

It's definitely vastly more complete and correct than what I posted here yesterday. Also I have run some Atari ST IPFs through the official decoder library and their gaps are decoded as I expect (so far!).

Still, the logic around gap streams is very complicated, and I need to run the decoder with yet more 'genuine' IPFs plus some of my own devising. The rules for filling gaps with looped sample data are particularly tricky to work out.

Oh also, regarding the mix-up between gap and data in the Atari ST IPFs. Each block can only have one data area plus one gap, which isn't a great fit with the IBM format that the ST uses, which actually has a gap and write splice between each sector's header and payload. Hence the format is really data/gap/data/gap, so the first of those gaps must be encoded in the data stream. The alternative would be to encode each logical sector into two IPF blocks: one for header, and one for payload. But then the per-track block count in the IPF image would no longer correspond to the real number of logical sectors. Perhaps that is even less desirable than encoding gap info in the data stream!

User avatar
IFW
Posts: 2634
Joined: Mon Nov 08, 2010 2:42 pm

Re: All you always wanted to know about IPF

Post by IFW » Thu Jan 19, 2012 10:58 am

Few comments and hints to help seeing the bigger picture - apologies for not going into detail.
Analyser pipeline:
1, A format is described by a script, and the script may or may not contain constraints in terms of expected content, it's usually about the structure of the data rather than data itself.
2, Some formats get automatically generated based on a pattern matching and a very advanced AI able to construct new things from "design patterns", while other formats are purely hand-scripted.
3, This gets compiled into an IPF file.
4, The AI pairs up sector header and data blocks and this is enforced for several reasons - you'd see the same thing in a FreeForm script for a Trace machine.
5, Understanding and replicating the gap area in a script is done (unless manually scripted), by a very complex algorithm. Hence it is expected that the output is not simple either.
6, The IPF file contains logical blocks - they may or may not match the data sectors in terms of disk geometry
7, A logical block in an IPF file is <data><gap><gap>, but think about what this can really represent with clever manipulation: <gap><data><gap>. This matches for example generic MFM (PC, ST etc), and there is nothing stopping the encoder to supply two logical blocks for each sector, there are reasons not to though, see IPF properties below why.
8, By the same logic any data in the gap area is data, not gap, hence it must be part of logical block - whether as part of another block or a standalone one depends on lots of factors, but again see below.

IPF properties:
1, An IPF file describes how to write a disk, whether to a real media or memory is mostly irrelevant and up to the host program
2, It was designed with the mastering process in mind and simulates writing artefacts; it must be able to replicate data, but be flexible enough to adapt the stream to slightly different runtime supplied requirements - this is what happens when something is written on a real disk.
3, IPF files can be compared and for that to be reliable data and gap areas must be properly marked as data or gap. All IPF files have the types correctly set.
4, IPF files can be runtime changed by the decoder, without modifying the data content (unless that behaviour is desired). The only way to do that is via properly describing the gap areas, what is flexible and what is not.
5, I'd recommend to check the difference between the handling of inter-sector gaps and the track write-splice

keir
Posts: 35
Joined: Sun Jun 26, 2011 11:03 pm

Re: All you always wanted to know about IPF

Post by keir » Thu Jan 19, 2012 11:53 am

Thanks Istvan, that's quite useful! I guess then that you are modelling a whole-track-at-a-time mastering process, and so there is only one true write splice (and for example, only one overlap position in the track that you modify across reading multiple track revolutions in the IPF decoder). Still you do the forward and reverse gap filling even for inter-sector gaps, but then I guess I've not yet seen a case where you use loop samples except for the last block in a track, so in practice I haven't seen a case where you end up with an uneven 'splice' where forward and reverse streams meet, except for the last block in a track. Which corresponds to what you're saying. Huh, it makes sense. ;)

Okay, here's a fun question for you: I generated an AmigaDOS IPF where every block including the first starts with address mark 44894489. Only the last block per track advertises a gap (the 'track gap'). There are no gap streams, so IPF decoder fills the track gap with the gapvalue byte. Now, by my reading of the decoder library, the default value should forward *and* reverse fill the gap, so the write splice occurs exactly halfway between the end of the last block and the start of the first block. This is because InitGapStream() returns the default stream for both gap0 and gap1 when neither stream is specified in block flags. And indeed, I just confirmed this by observing the track_info.overlap value when my IPF tracks are decoded.

Worked lovely in an emulator, *BUT* when I wrote this back to real disk with DTC, the first block's address mark was not readable! My impression is that the write splice ended up at the *end* of the track gap rather than in the middle, hence bits of 44894489 are lost, or the disk controller has simply lost sync for just long enough to miss the address mark. As soon as I manually pre-pended a few nul bytes to the start of the first block's data stream, everything worked beautifully. I assumed this was a bug in my IPF generation at the time, but looking now at how the IPF decoder library generates a 'default gap' from no explicit gap streams, is this a bug/limitation of DTC? e.g., it starts writing from track_info.startbit rather than track_info.overlap?

User avatar
IFW
Posts: 2634
Joined: Mon Nov 08, 2010 2:42 pm

Re: All you always wanted to know about IPF

Post by IFW » Thu Jan 19, 2012 12:09 pm

Where DTC starts writing the data - again - depends on quite a few factors. It is possible that the data supplied would match something that it shouldn't... hard to tell just by guessing.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest