Get Information from experts about Flux transitions analysis

All questions about how to use KryoFlux go here.
User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Get Information from experts about Flux transitions analysis

Post by DrCoolZic » Mon Nov 25, 2013 5:12 pm

I am currently developing a new set of tools for analyzing flux transitions from Floppy Disk.
I have completed a prototype to validate several concepts (more on this may be in another thread as this is not the subject). Let’s call it the Flux Transition Analyzer (FTA) for now. Note that currently the FTA prototype looks and provides functionality very similar to the DTC UI program (not really a surprise?).

Currently the tool decodes the stream file into arrays of flux transitions and index information and display graphically this information. I have also added a DPLL algorithm to retrieve the clock from the transitions (useful for clock variation protections).

I have preserved about hundred Atari floppy diskettes, but up until now I did not spend time to look at the stream files.
To my surprise there is a lot to learn just by looking at the "flux graphs". :roll:

With my current knowledge I am able to interpret most data but there are still a lot of things that I have difficulties to understand.
This is the purpose of this thread: get specific information from experts of this forum.

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Mon Nov 25, 2013 5:23 pm

Now the fist subject: Unfortunate tracks / sectors ...

By definition an unformatted diskette is a diskette that has never been formatted. During formatting, the particles are aligned forming a pattern of magnetized tracks, each broken up into sectors, enabling the controller to properly read and write data.

Here is a definition from Wikipedia: “A blank "unformatted" diskette has a coating of magnetic oxide with no magnetic order to the particles”. http://en.wikipedia.org/wiki/Floppy_disk

The magnetization on the surface should be relatively uniform and there should not be any flux reversal under the head. The consequence is that we should not get any data pulse from the read circuitry. But in practice we have many pseudo random transition detected. Two things are happening:
  •  As we have no regular flux transitions, the drive's automatic gain control (ACG) is set to its maximum possible gain since it presumes a weak signal coming from the drive's read head.
     Some random flux variations exist naturally on the magnetic surface and due to the fact that the ACG is turned to its maximum they are sometimes detected as flux transitions.
The detected data seems "random" in the sense that the data is never the same twice due to drive speed (and humidity, temperature) fluctuations. Note that compared to a normal track the number of flux transition detected on an unformatted track is usually much lower.
Also the data appears to be random we can see that most of the transitions are located between 2µs and 7µs. The image below show a typical unformatted track:
atari_unf_lin.JPG
unformatted
atari_unf_lin.JPG (134.53 KiB) Viewed 3802 times
If we limit the scale of displayed transitions below 10µs we have the following image:
atari_unf_limit.JPG
atari_unf_limit.JPG (175.55 KiB) Viewed 3803 times
Here we can clearly see that there is a peak in the distribution around 2.8µs and a smaller one around 6.4µs.

Few things to note:
  •  Usually all the unformatted tracks from one diskette looks very similar (comparable histograms) but they usually differ from one diskette to another.
     If you have recorded the flux transitions of a track over several revolutions, usually you do not see much difference between a revolution and the next. But this is not true for unformatted track. Due to the random nature of unformatted track the data information recorded differ widely from one revolution to the next even though the histogram stay relatively the same.
Some protections use partially unformatted tracks:
night-shift-79.0.JPG
partially unformatted
night-shift-79.0.JPG (154.41 KiB) Viewed 3805 times
The example above shows track 79.0 of Night Shift game. We can see that we have two unformatted sections in this track. If you look at the clock as decoded by the DPLL you can easily imagine that the data read from this track contains fuzzy bits.
Last edited by DrCoolZic on Mon Nov 25, 2013 6:12 pm, edited 2 times in total.

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Mon Nov 25, 2013 5:23 pm

Duplicate post can't remove ???
Last edited by DrCoolZic on Mon Nov 25, 2013 6:21 pm, edited 2 times in total.

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Mon Nov 25, 2013 5:30 pm

Following is a zoom on an unformatted area near the end of a track taken from DrT D50 Editor floppy:
d50-zoom-unf.JPG
partially unformatted + fish bone pattern
d50-zoom-unf.JPG (93.21 KiB) Viewed 3805 times
We see that the end of the track is unformatted and have random transitions, but just before that area we also see a strange “fish bone” pattern. We have some flux transitions placed closed to 5µs and 3µs which are exactly at the border of the inspection window and this will certainly results in fuzzy bits. Note that the fish bone pattern is used so that transitions close to 5µs are “compensated” by transitions close to the 3µs and we can see that this result (thanks to the DPLL) in a relatively stable 2µs clock.

Another trick used for protection is partially formatted track: The track seems to be an unformatted but in fact it contains a sector. Often this unique hidden sector is limited to only the address section. Most copier won’t be smart enough to detect this hidden information.

Although it is easy to visually detect that a track is unformatted it is more difficult to detect this by software. The detection should be reliable and ideally detect partially unformatted track as well as partially formatted track. The best simple indicator I have found is to use a variety of Shannon Entropy measurement. This computation is done on the histogram of the flux transition repartition. A “normal track” has a coherent repartition and therefore a high entropy.

Another interesting problem is how do we reproduce unformatted areas on Floppy Disks?
You could think that we just have to use blank diskette and that we do not write anything in areas we want to keep unformatted. Unfortunately when you buy floppy diskettes they are already preformatted (usually using the IBM format and with a DOS 2.0 boot sector). I do not know why the manufacturers do that, but I guess this must be for them to run some tests? But even if you could buy unformatted diskette you would have to keep a bunch of unused diskettes which is not practical.

Therefore I think that the closest solution to simulate an unformatted area is to write pseudo-random flux transitions on the floppy. The goal here is to make sure that the Floppy Disk Controller read circuitry will not be able to decode any meaningful data. In practice you can write anything you want (for example a complete track of 0) as long as it cannot be interpreted by the FDC.
I need to experiment writing unformatted track with KryoFlux board

Note that only tracks 0 to 79 are preformatted on blank diskettes. If therefore if we sample the flux transitions of tracks beyond this limit (track 80-83) we can see that “real” unformatted track are similar to unformatted tracks but slightly different. This seems to indicate that professional duplication machines were using unformatted diskettes.
unf-80.0.JPG
unf-80.0.JPG (157.92 KiB) Viewed 3805 times
All the previous examples where taken from Double Density Floppy disks (the one used on Atari). But I have also experimented with High Density floppy disks (PC diskettes).
I have noticed that the unformatted tracks from a HD floppy look different! They seems to exhibit a much more pronounced banding effect:
dos_hd_unf.JPG
DOS High Density unformatted
dos_hd_unf.JPG (156.28 KiB) Viewed 3805 times
Is this related to the magnetic support? It should be as the sampling has been done the same floppy drive!
This needs to be further investigated.

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Mon Nov 25, 2013 5:43 pm

Summary so far
All the above is about unformatted :) Spitted in three parts because limitation on number of images :(

Questions:
  • First any feedback on the above welcome
  • Is the explanation about the reason of flux transitions on an unformatted track correct? any better idea?
  • Explanation about the distribution? Why "bands" (e.g. 2.8, 6.4, ...) and not uniform Poisson distribution
  • Why different bands for HD and DD? Magnetic support different ???
  • Why different unformat distribution for track above 79 ?
  • How does KryoFlux reproduce unformatted track?
  • Any advice on a good algorithm to detec unformatted tracks? If possible not complex (e.g. FFT, convolution) preferred simple marker like entropy
Thanks
JEan

Darkstar
Posts: 72
Joined: Thu Nov 04, 2010 7:58 pm

Re: Get Information from experts about Flux transitions anal

Post by Darkstar » Mon Nov 25, 2013 8:14 pm

First question: Why use 5 posts instead of just putting it into one? (and yes, you can delete posts...)

But I think I can answer some of your questions...
DrCoolZic wrote: Why different bands for HD and DD? Magnetic support different ???
yes, the magnetic coating (coercivity) is different[1]
DrCoolZic wrote: Why different unformat distribution for track above 79 ?
I guess because "factory new" disks are only unformatted up to track 79 and might have serial numbers or other manufacturing data left over in the higher tracks... Did you try to find out the encoding of that residual data?
DrCoolZic wrote: How does KryoFlux reproduce unformatted track?
KryoFlux (the hardware) samples each track n times and stores the data in a file. When this is converted to IPF, it is detected by their analyzer as "unformatted" and a special flag gets set in the file to convey this information to the program reading the IPF file back in, which can then re-generate a random pattern to "simulate" an unformatted track (so no actual random data needs to be stored in the IPF file at all)
DrCoolZic wrote: Any advice on a good algorithm to detec unformatted tracks? If possible not complex (e.g. FFT, convolution) preferred simple marker like entropy [/list]
I would go the other way: try to detect each and every "known" format (FM, MFM, GCR, various rotational speeds, etc.), and only output "unformatted" if none of them matches

-Darkstar

Jeff_HxC2001
Posts: 58
Joined: Thu May 12, 2011 10:56 pm

Re: Get Information from experts about Flux transitions anal

Post by Jeff_HxC2001 » Mon Nov 25, 2013 10:08 pm

Darkstar wrote: I would go the other way: try to detect each and every "known" format (FM, MFM, GCR, various rotational speeds, etc.), and only output "unformatted" if none of them matches

-Darkstar
mhhh this is not enough if you are dealing with protected disks.
The algorithm that i am using is working at the pulses level timing. This is a "simple" pulses group comparator.
By this i am able to detect unformatted track and flakey bits :

http://hxc2001.com/img/TrackViewer/HxCT ... Layout.jpg

Darkstar
Posts: 72
Joined: Thu Nov 04, 2010 7:58 pm

Re: Get Information from experts about Flux transitions anal

Post by Darkstar » Mon Nov 25, 2013 10:15 pm

Yeah, now that I'm thinking about it some more, probably the only way to reliably detect unformatted tracks is to sample multiple revolutions and see if they match up to some degree (you need a good metric for that though... or a good filter)
Still, if you get "matching" tracks you probably cannot be 100% sure that it's not unformatted (on the other hand, when you get non-matching tracks there's very little chance that it contains valid data)

Jeff_HxC2001
Posts: 58
Joined: Thu May 12, 2011 10:56 pm

Re: Get Information from experts about Flux transitions anal

Post by Jeff_HxC2001 » Mon Nov 25, 2013 10:27 pm

Darkstar wrote:Yeah, now that I'm thinking about it some more, probably the only way to reliably detect unformatted tracks is to sample multiple revolutions and see if they match up to some degree (you need a good metric for that though... or a good filter)
Still, if you get "matching" tracks you probably cannot be 100% sure that it's not unformatted (on the other hand, when you get non-matching tracks there's very little chance that it contains valid data)
Exactly ! :)

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Tue Nov 26, 2013 9:42 am

Darkstar wrote:First question: Why use 5 posts instead of just putting it into one? (and yes, you can delete posts...)
Was my intent but unfortunately post is limited to 3 pictures/attachments
I still cannot find any way to delete post. Might depends on user permission?
Darkstar wrote: yes, the magnetic coating (coercivity) is different[1]
Thanks interesting reference with good information.
Darkstar wrote:I guess because "factory new" disks are only unformatted up to track 79 and might have serial numbers or other manufacturing data left over in the higher tracks... Did you try to find out the encoding of that residual data?
Indeed on some diskette I have found what looks like data on track 80. But in most cases track beyond 79 seems to contain random data. Here is my belief on the subject. As I mentioned most if not all manufactured floppies where formatted (?for reliability test?) and therefore duplication machine like trace probably could not rely on having blank unformatted diskettes in their feeder. Therefore unformatted tracks in range 0-79 had to be written (simulated) by the duplicator where tracks in range 80-83 would be naturally unformatted tracks.
Darkstar wrote:KryoFlux (the hardware) samples each track n times and stores the data in a file. When this is converted to IPF, it is detected by their analyzer as "unformatted" and a special flag gets set in the file to convey this information to the program reading the IPF file back in, which can then re-generate a random pattern to "simulate" an unformatted track (so no actual random data needs to be stored in the IPF file at all)
Thanks but I already know about Stream and IPF format. If you look at the latest release from KryoFlux I have discovered that they now distribute my document :o (you can find latest version here) http://info-coach.fr/atari/software/pro ... otocol.pdf (new version to be published soon) and http://info-coach.fr/atari/software/pro ... tation.pdf.
Your description is what I think they do but I would be interested to get more detail information. But I guess I just need to run some tests to confirm.
Darkstar wrote: I would go the other way: try to detect each and every "known" format (FM, MFM, GCR, various rotational speeds, etc.), and only output "unformatted" if none of them matches
Interesting idea.
I did not mention but my only platform of interest is Atari ST and marginally PC. Up until last week my prototype tool was totally platform agnostic: Histogram and scattered charts with automatic axis range detection. But from now on I will concentrate only on MFM as this is the only format that the WD1772 can read. As you can see on published pictures I already have a DPLL (that implements in software the US Pattent 4,780,844 - http://info-coach.fr/atari/documents/ge ... 780844.pdf) that I use to display the locked clock. I am finishing code for inspection window data separator (more info in http://info-coach.fr/atari/hardware/FD-Hard.php#FDC_PLL) DSR and SYNC detector.
Therefore your idea is interesting because the only way for the WD1772 to read something is to find at least some SYNC mark on the track. Therefore detecting sync would also allow to find very small address segment hidden in unformatted track.
Jeff_HxC2001 wrote:The algorithm that I am using is working at the pulses level timing. This is a "simple" pulses group comparator.
By this I am able to detect unformatted track and flakey bits :
http://hxc2001.com/img/TrackViewer/HxCT ... Layout.jpg
Jeff this is quite interesting. If I understand, you check if there is some sort of correlation analysis between a group of adjacent transitions.
Mhhh fuzzy bits detection in most cases requires more than that
Darkstar wrote:Yeah, now that I'm thinking about it some more, probably the only way to reliably detect unformatted tracks is to sample multiple revolutions and see if they match up to some degree (you need a good metric for that though... or a good filter). Still, if you get "matching" tracks you probably cannot be 100% sure that it's not unformatted (on the other hand, when you get non-matching tracks there's very little chance that it contains valid data)
Sorry but this one does not seems to be a good idea. It is true that in case of unformatted tracks the content vary from revolution to revolution but this is not seems to be a good indicator.
And yes data can vary from revolution to revolution on good data sector as this is the definition of fuzzy (or whatever you call it) bits.

As I mentioned I am currently using an entropy function which is easy to compute and gives good results on all “good diskettes”. With this function I am able to detect unformatted and partially formatted track correctly if the disk is “good”. But if the disk contains garbage data (for example partially erased tracks due to time?) the algorithm fails. I will soon publish some of these “bad” diskettes so people can tell me what they think.
For example something like this ...
chicago.JPG
Bad diskette?
chicago.JPG (222.75 KiB) Viewed 3776 times
... Fools my Shannon entropy calculator :oops:

Post Reply