This first post will be edited so that is always contains the latest info ... So stay tuned
edit September 2012 viewtopic.php?f=7&t=365&p=6151#p6151
New version of the IPF documentation available at my Web site
edit February 4, 2012 viewtopic.php?f=7&t=365&p=2994#p2994
- ipfdump 0.0a : read track, read address, read sectors as a WD1772 on information provided by the CAPS API
- ipfinfo v0.3b : almost same as 0.3a but only 16KB instead of 1.6MB
edit January 30, 2012 viewtopic.php?f=7&t=365&start=20#p2912
- IPFInfo v0.3a : fix bug in decoding gap + offset is now actual offset in the file so can follow with hex editor
- example dump from very interesting Theme Park Mystery
edit January 27, 2012 - see viewtopic.php?f=7&t=365&start=10#p2889
- IPF documentation v0.0a
- IPFInfo v0.2a executable.
- Example output of IPFInfo on Action Fighter
edit January 16, 2012
- IPF-42A-12016.rar : Contains the modified SPS 4.2 source tree to compile under Microsoft VC++2010 as well as the source of the IPFInfo program
- IPF Documentation V0.0 for review: IPF file format, IPFLib 4.2A Documentation, IPInfo v0.0 documentation
- IPFInfo.exe Windows IPF dump file based on the work done by Keir Fraser
- IPF Documentation v00.rar
- IPF documentatation
- (181.43 KiB) Downloaded 378 times
- IPFLib + IPFInfo sources
- (70.31 KiB) Downloaded 361 times
So most of the information will probably come from SPS people, but anybody having knowledge about IPF is of course welcome to answer and hopefully I should be able to also provide information
This thread is intended for programmers and is probably of no interest to ipf users.
so I expect questions like:
- Format of IPF file
- CAPS library compilation, usage
- Emulator integration
- How do I get submit games to get IPF
- Where can I find IPF files
- a very preliminary IPF documentation
- a CAPS library source tree that compile under Visual Studio C++ 2010
- an IPF dump program
It is one of the game that I have used to verify my IPFInfo program based on program from Keir Fraser.
and I already have lots of questions.
So we have the IMGE records that point to DATA records to find the actual data content...
Seems like track data are divided into a number of blocks that (in most cases) correspond to the number of sectors. I said in most cases because from what I understand it might be necessary to have more blocks. For example for a sector with variable bit rates region (e.g. macrodos/speedlock) might need to be further divided into more regions ???
But for now lets assume 1 block is one sector.
If I can make sense of most of the information in the different records, I cannot make sense of the data information placed after the block descriptor???
Reminder: A track is usually composed of:
- a post-index region (say GAP1)
- several repeat of (for each sector):
- -pre-address gap field (say GAP2)
- -address field
- -post-address + pre-data gap field (say GAP3)
- -data field
- -post-data gap field (say GAP4)
- End of repeat
- a pre-index field (say GAP5)
The Gap offset points to strange unknown data? I am very familiar at looking at track/sector content in Hex but here I find nothing known?
The data offset points to some information placed in front of address info followed by GAP3 followed by strange info followed by data field.
Something like <unknown_info> <address_field> <partof_gap3> <data_field> ...
Code: Select all
22 08 4E 00 22 08 4E 21 38 22 08 00 00 21 06 44 89 44 89 44 89 22 07 FE 00 00 05 02 06 AB 23 1C 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 00 00 00 00 00 00 21 06 44 89 44 89 44 89 42 02 03 FB 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May be I am asking stupid question but this looks strange???
First gap seems to point to info that does not look like gap info
Then data point to a mixture of everything.
I would have expected more something like what I have described in the track?
What about pre-index and post-index info?
what about sync mark?
what about gap content?
why mix gap/and data
As you can see currently I am a bit lost. But I suspect that if I look at the code I will have the answer.
Any comment to help me understand is welcome
- Foundation Waste dump
- (15.38 KiB) Downloaded 285 times
The format for a data stream is (using dodgy pseudocode):
Code: Select all
uint8_t cntlen:3; // bits 7:5 indicate #bytes for length field uint8_t type:5; // enum cpdatXXX (0=end,1=sync,2=data are the only field types I've used) uint8_t length[cntlen]; uint8_t data[length];
Code: Select all
I assume gap data is a bit similar but I haven't really investigated. I'm not sure that the division of data into blocks/sectors is important. I make a stab at it in my own library, but the only formats it definitely matters for are non-uniform density tracks where different sectors have different bitcell timings.
The key to understanding this stuff is files Capsdef.h and CapsImgS.cpp in the IPF decoder sources. Also I built my own version of the IPF library with tracing added to find out what paths get taken.
If you want to see my code for generating IPF data streams, it's in libdisk/container_ipf.c in my github repository. Function ipf_tbuf_bit() is the function to append a bit of data to a data-chunk stream.
I removed what I previously listed here about gap streams, it was misleadingly incomplete and wrong.
I will write up a text file describing data and gap streams as comprehensively and accurately as I can. If stuff from that is useful to help with DrCoolZic's big PDF document, that's great.
It's definitely vastly more complete and correct than what I posted here yesterday. Also I have run some Atari ST IPFs through the official decoder library and their gaps are decoded as I expect (so far!).
Still, the logic around gap streams is very complicated, and I need to run the decoder with yet more 'genuine' IPFs plus some of my own devising. The rules for filling gaps with looped sample data are particularly tricky to work out.
Oh also, regarding the mix-up between gap and data in the Atari ST IPFs. Each block can only have one data area plus one gap, which isn't a great fit with the IBM format that the ST uses, which actually has a gap and write splice between each sector's header and payload. Hence the format is really data/gap/data/gap, so the first of those gaps must be encoded in the data stream. The alternative would be to encode each logical sector into two IPF blocks: one for header, and one for payload. But then the per-track block count in the IPF image would no longer correspond to the real number of logical sectors. Perhaps that is even less desirable than encoding gap info in the data stream!
1, A format is described by a script, and the script may or may not contain constraints in terms of expected content, it's usually about the structure of the data rather than data itself.
2, Some formats get automatically generated based on a pattern matching and a very advanced AI able to construct new things from "design patterns", while other formats are purely hand-scripted.
3, This gets compiled into an IPF file.
4, The AI pairs up sector header and data blocks and this is enforced for several reasons - you'd see the same thing in a FreeForm script for a Trace machine.
5, Understanding and replicating the gap area in a script is done (unless manually scripted), by a very complex algorithm. Hence it is expected that the output is not simple either.
6, The IPF file contains logical blocks - they may or may not match the data sectors in terms of disk geometry
7, A logical block in an IPF file is <data><gap><gap>, but think about what this can really represent with clever manipulation: <gap><data><gap>. This matches for example generic MFM (PC, ST etc), and there is nothing stopping the encoder to supply two logical blocks for each sector, there are reasons not to though, see IPF properties below why.
8, By the same logic any data in the gap area is data, not gap, hence it must be part of logical block - whether as part of another block or a standalone one depends on lots of factors, but again see below.
1, An IPF file describes how to write a disk, whether to a real media or memory is mostly irrelevant and up to the host program
2, It was designed with the mastering process in mind and simulates writing artefacts; it must be able to replicate data, but be flexible enough to adapt the stream to slightly different runtime supplied requirements - this is what happens when something is written on a real disk.
3, IPF files can be compared and for that to be reliable data and gap areas must be properly marked as data or gap. All IPF files have the types correctly set.
4, IPF files can be runtime changed by the decoder, without modifying the data content (unless that behaviour is desired). The only way to do that is via properly describing the gap areas, what is flexible and what is not.
5, I'd recommend to check the difference between the handling of inter-sector gaps and the track write-splice
Okay, here's a fun question for you: I generated an AmigaDOS IPF where every block including the first starts with address mark 44894489. Only the last block per track advertises a gap (the 'track gap'). There are no gap streams, so IPF decoder fills the track gap with the gapvalue byte. Now, by my reading of the decoder library, the default value should forward *and* reverse fill the gap, so the write splice occurs exactly halfway between the end of the last block and the start of the first block. This is because InitGapStream() returns the default stream for both gap0 and gap1 when neither stream is specified in block flags. And indeed, I just confirmed this by observing the track_info.overlap value when my IPF tracks are decoded.
Worked lovely in an emulator, *BUT* when I wrote this back to real disk with DTC, the first block's address mark was not readable! My impression is that the write splice ended up at the *end* of the track gap rather than in the middle, hence bits of 44894489 are lost, or the disk controller has simply lost sync for just long enough to miss the address mark. As soon as I manually pre-pended a few nul bytes to the start of the first block's data stream, everything worked beautifully. I assumed this was a bug in my IPF generation at the time, but looking now at how the IPF decoder library generates a 'default gap' from no explicit gap streams, is this a bug/limitation of DTC? e.g., it starts writing from track_info.startbit rather than track_info.overlap?