Get Information from experts about Flux transitions analysis

All questions about how to use KryoFlux go here.
Jeff_HxC2001
Posts: 58
Joined: Thu May 12, 2011 10:56 pm

Re: Get Information from experts about Flux transitions anal

Post by Jeff_HxC2001 » Tue Nov 26, 2013 1:45 pm

DrCoolZic wrote:
Jeff_HxC2001 wrote:The algorithm that I am using is working at the pulses level timing. This is a "simple" pulses group comparator.
By this I am able to detect unformatted track and flakey bits :
http://hxc2001.com/img/TrackViewer/HxCT ... Layout.jpg
Jeff this is quite interesting. If I understand, you check if there is some sort of correlation analysis between a group of adjacent transitions.
Mhhh fuzzy bits detection in most cases requires more than that
To simplify i take the whole track dump, cut it in small blocks (actually there is ~1000 blocks per revolutions...). For each blocks the software try to found its exact position in the next revolution. Of course this take a lot of time. The analysis can take some minutes for a whole floppy, especially if the floppy is not formatted.
But this is only a part of the whole process. the detection of timing valid/"invalid" pulses timing come later in the process.

Here is some examples of analysis :

(Blue line : Index signal
Red line : "flakey / fuzzy" bits)

Atari ST - crazy car III :
http://hxc2001.com/img/TrackViewer/Atar ... K2_afi.png
You can see the protection at the track 79 Side 0 (the sector 247).

Some others examples :
A PC88 game :
http://hxc2001.com/img/TrackViewer/881% ... aw_afi.png
unformated track : T32S0
You can see sector write splice on all the tracks (typical with PC88 games: the master generation was done with the PC88 itself... )

Another one :
http://hxc2001.com/img/TrackViewer/8SR% ... aw_afi.gif

A damaged floppy disk... :
http://hxc2001.com/img/TrackViewer/8SR% ... aw_afi.png

Atari ST - Fire and forget (One side disk) :
http://hxc2001.com/img/TrackViewer/Atar ... et_afi.png
Protection on the track 79:0
Write Splice on 78:0:5 & 6 -> High score write...

Atari ST : TheDreamTeam WWF
http://hxc2001.com/img/TrackViewer/Atar ... K3_afi.png
Protection on the track 0:0 sector 6 (copylock)

(Note : images are very big, so it is better to download them and view them with a decent software ;) )

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Tue Nov 26, 2013 3:00 pm

Jeff_HxC2001 wrote:To simplify i take the whole track dump, cut it in small blocks (actually there is ~1000 blocks per revolutions...). For each blocks the software try to found its exact position in the next revolution. Of course this take a lot of time. The analysis can take some minutes for a whole floppy, especially if the floppy is not formatted.
But this is only a part of the whole process. the detection of timing valid/"invalid" pulses timing come later in the process.
Are you working with data from Stream and/or IPF files?
When you say "cut it in small blocks" are you talking of block of flux transitions? On a typical DD one revolution is about 40000 flux transitions so if I understand correctly you group them by about 40. This gives you a pattern and you are trying to match the same pattern on the next revolution. If it matches (I guess you have to allow some margin to take in account RPM variation, etc...) then you are in "normal data block" if not, depending on the number of mismatches, this could be either fuzzy bits or unformatted area based on number of mismatched bits in the block?

I do not fully understand the images displayed :shock: but it seems that the color scheme is used to indicate things like fuzzy or above/below normal speed sections.
I guess that the black line in middle of sector is the clock. For example in copylock we see the clock goes above normal speed.
Something like this:
copylock.JPG
copylock.JPG (124 KiB) Viewed 2214 times
Your "trackviewer" is this an available tool?

Displaying information from the flux transitions of a track is challenging. You need to display a lot of information (so you cannot have too much detail) but on some places you need to see lots of details :)
On some previous tools like KFAnalyze http://info-coach.fr/atari/software/pro ... alysis.php I was using a chart package to produce long skinny output sometimes difficult to read ...
On the tool I am working on the idea is different as this is an interactive tool. You can do things like zoom on a specific areas but more interestingly it uses what is referred as semantic zoom. This means that when you reach certain level of zoom the information displayed is different and can includes more data. An easy to understand example is to display sectors of a track as block, but when zoomed details on address, data, gap, appears, and when zoomed even further bytes can be shown, and potentially data/clock bits, and even deeper flux transitions ...
Also interesting is helping user in moving around the information displayed. In most cases you do not know where too look when you are in front of 200000 points ;) .
Therefore to get to the interesting spots you need to predefined some good triggering mechanism to identify splice area, border bits, etc ...

User avatar
IFW
Posts: 3079
Joined: Mon Nov 08, 2010 2:42 pm

Re: Get Information from experts about Flux transitions anal

Post by IFW » Tue Nov 26, 2013 6:40 pm

If you sample an unformatted track with a scope you will get no (or almost none) flux reversals.
Everything from that point is up to the drive electronics.
Therefore, when you try to find distribution etc. in the sampled data from unformatted areas, basically what you do is to profile/fingerprint the drive instead of the disk - which is probably not what you want ;)
Consequently, the test will eventually fail, just by using a different drive type/model/head etc or simply switching the density mode on drives where that is possible.
You simply cannot be 100% sure whether a track is unformatted without very thorough analysis and that's the reason DTC does not omit tracks from stream dumps even if they are almost certainly unformatted.
Some protections may use as little as just a few bytes of data and leave the rest of the track unformatted... good luck with finding that properly, unless you know exactly what you are looking for. Usually duplicator info is being stored on unformatted tracks normally somewhere from track 80 and use very small formatted areas.
It is possible to detect areas that at least show consistent behaviour between the revolutions sampled, but as per above - although very unlikely - it could be down to the property of the drive, not just actually that area being written. Additionally, it is possible to detect whether such area consist of any legible encoding - the common rules for all disk encodings are identical, simply because data couldn't be written or read reliably otherwise. Due to this such a test can be done in an encoding agnostic way.
Testing for consistency in behaviour of the flux reversals and the legibility of the flux reversals in the affected area is what you want.
Now, the complexity comes from drive speed wobble, various reading artefacts and shifts, and index signal jitter - so unless you want to run such a test for a really long time, you should find ways of reducing the behaviour test and allow for some variance.

User avatar
IFW
Posts: 3079
Joined: Mon Nov 08, 2010 2:42 pm

Re: Get Information from experts about Flux transitions anal

Post by IFW » Tue Nov 26, 2013 6:49 pm

As for what KryoFlux does during write/remastering: depends on user settings, but the default is to "unformat" all tracks (no flux reversals), so no, it does not write patterns or anything like that.
The IPF library generates a fairly safe unformatted pattern that so far works with any software emulated on any platform; the reason for this is that it has to simulate the randomness observed when an unformatted track is being read on a real drive - but in an emulated environment, where randomness can only be generated.
It is possible to disable the generation of "unformatted noise" in the IPF library; DTC does that when it writes a disk as it relies on the physical properties of the real disk being written to :)

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Wed Nov 27, 2013 11:23 am

IFW wrote:If you sample an unformatted track with a scope you will get no (or almost none) flux reversals.
Everything from that point is up to the drive electronics.
Therefore, when you try to find distribution etc. in the sampled data from unformatted areas, basically what you do is to profile/fingerprint the drive instead of the disk - which is probably not what you want ;) Consequently, the test will eventually fail, just by using a different drive type/model/head etc or simply switching the density mode on drives where that is possible.
This is the consensus on the subject and therefore it should be true. However this does not explain why we get different “random profile” from disk to disk and the larger differences when switching from DD to HD. I believe that there is a combination of noise generated by the read channel pushed to maximum amplification and the head picking up small flux reversal. In fact in this case we should probably not talk about reversal but variation. Something similar to what you hear when listening to a blank tape with the volume pushed very high.
Anyway this is more a rhetorical than practical discussion about reading data on a “real unformatted track” because the end result is that we are getting more or less random noise from the drive! More interesting is what we do to “unformat a track” (discussed below).
IFW wrote:You simply cannot be 100% sure whether a track is unformatted without very thorough analysis and that's the reason DTC does not omit tracks from stream dumps even if they are almost certainly unformatted.
Some protections may use as little as just a few bytes of data and leave the rest of the track unformatted... good luck with finding that properly, unless you know exactly what you are looking for. Usually duplicator info is being stored on unformatted tracks normally somewhere from track 80 and use very small formatted areas.
This is probably true generally speaking. However the only interest of writing hidden data is to be able to read/check them. In the case of Atari the MFM FD controller provides 3 read commands (address, sector, and track). In this case your best bet to hide a small chunk of bytes is to use an address segment which result in 7 bytes preceded by three sync and these is very easy to check. An other alternative would be to hide data that you access with a read track, and in that case you need at least one sync (in order to get meaningful data) and a sequence of bytes to check. In this later case it is slightly more difficult to detect but still not that difficult because of the presence of the sync byte(s).
IFW wrote:Testing for consistency in behaviour of the flux reversals and the legibility of the flux reversals in the affected area is what you want.
Now, the complexity comes from drive speed wobble, various reading artefacts and shifts, and index signal jitter - so unless you want to run such a test for a really long time, you should find ways of reducing the behaviour test and allow for some variance.
Actually it is easy to handle the “variance” as this is exactly what the DPLL algorithm do 8-)
Based on input from this thread, I have generated a new version of my prototype yesterday that now includes a sync detector and this allow me to check the presence of sync in unformatted track.
FYI I have some “bad diskettes” with very very distorded signals and I was surprise to find that the DPLL is good enough to recover data from this messy transitions
All sector found !!!
All sector found !!!
crappy-sync.JPG (160.96 KiB) Viewed 2193 times
The sync are the red lines.
Below is the same view on 5 revolutions (separated by the black line)
crappy-5rev.JPG
crappy-5rev.JPG (173.88 KiB) Viewed 2193 times
IFW wrote:As for what KryoFlux does during write/remastering: depends on user settings, but the default is to "unformat" all tracks (no flux reversals), so no, it does not write patterns or anything like that.
Sorry I do not understand: “The default is to unformat all track”? How
“so no, it does not write patterns” so does it unformat or not???
IFW wrote:The IPF library generates a fairly safe unformatted pattern that so far works with any software emulated on any platform; the reason for this is that it has to simulate the randomness observed when an unformatted track is being read on a real drive - but in an emulated environment, where randomness can only be generated.
It is possible to disable the generation of "unformatted noise" in the IPF library; DTC does that when it writes a disk as it relies on the physical properties of the real disk being written to :)
“as it relies on the physical properties of the real disk being written to” meaning what? Unformat? Do nothing?
What user setting?
Again remember that there is no such a thing as unformatted diskettes available from manufacturer!

User avatar
IFW
Posts: 3079
Joined: Mon Nov 08, 2010 2:42 pm

Re: Get Information from experts about Flux transitions anal

Post by IFW » Wed Nov 27, 2013 2:27 pm

Yes, you are right about the effect of amplification modulated with the drive profile we get these nice drive/head/disk specific noise patterns.

"Unformat" means writing to the track without outputting any flux reversal at all. You can't do that with a normal controller (as it always outputs clock bits), but very easy with KryoFlux or a Trace machine.
The result is identical to a real unformatted disk (as long as the drive is well aligned!), except for a special case, that affects systems like the CBM drives, that use 96TPI stepper motor with 48TPI head. There is more about this topic is on the forums and the manual.
Now keep in mind that the majority of disks were professionally duplicated; so the rules of standard controllers do not apply - as long as you don't necessarily want to read everything on the target platform. Most duplicator info types are not readable by normal controllers by design in order to make sure they do not interfere with a program or controller in any way by providing meaningful data. In other words, no, you won't find DIs on a track by looking for standard controller readable marks (syncs).
The -we parameter changes the way how DTC writes a track, regarding unformatting.

User avatar
IFW
Posts: 3079
Joined: Mon Nov 08, 2010 2:42 pm

Re: Get Information from experts about Flux transitions anal

Post by IFW » Wed Nov 27, 2013 2:51 pm

Some people might find this an interesting read including how writing works:
http://bitsavers.trailing-edge.com/pdf/ ... ystems.pdf

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Wed Nov 27, 2013 5:39 pm

IFW wrote:Some people might find this an interesting read including how writing works:
http://bitsavers.trailing-edge.com/pdf/ ... ystems.pdf
Excellent document good complement of AN413 http://info-coach.fr/atari/hardware/fd-hard/AN-413.pdf and Computer Design Feb 1980 http://info-coach.fr/atari/hardware/fd- ... 8002-1.pdf

User avatar
DrCoolZic
Posts: 172
Joined: Tue Jul 26, 2011 10:44 am

Re: Get Information from experts about Flux transitions anal

Post by DrCoolZic » Wed Nov 27, 2013 6:28 pm

IFW wrote:"Unformat" means writing to the track without outputting any flux reversal at all.
You can't do that with a normal controller (as it always outputs clock bits), but very easy with KryoFlux or a Trace machine.
I must be missing something here? How is it possible to write without outputting flux reversal?
To me this sound like “you write without writing”?
Let’s forget the controller, but writing no flux transition is
- 1) not possible for the drive write circuit (if you mean writing a steady current), and
- 2) does not means anything for the write head (as the only way to write something is to have flux changes in the head).
This remind me the discussion that we had about the so called No Flux Area used in games like Turrican. Ijor was describing that these areas were generated by using “strong erasure” but at the end we concluded that it was not possible unless we use a drive with permanent magnets as a floppy drive does not have erase head (other than tunnel or straddle).
So I must be misinterpreting your explanations?
IFW wrote:Now keep in mind that the majority of disks were professionally duplicated; so the rules of standard controllers do not apply - as long as you don't necessarily want to read everything on the target platform. Most duplicator info types are not readable by normal controllers by design in order to make sure they do not interfere with a program or controller in any way by providing meaningful data. In other words, no, you won't find DIs on a track by looking for standard controller readable marks (syncs).
I understand and fully agree with this. What you are saying is that it is possible to find information not readable by the target platform but useful for other purpose. And I guess that you might be interested in reproducing this information for preservation. However as an end user on a specific platform this information is of no interest and of course as it cannot be read it can’t be used as a protection.
I have looked at:
  • -wb<bias>: write bias (default by image) - 0=neutral, 1=bias out, 2=bias in
    -we<mode>: write: erase mode (default by bias) - 0=normal, 1=used only, 2=wipe
And indeed this description is really concise!
Bias in / bias out ???
"-we normal" = Means write nothing, leaving information already on the track intact???
"-we used only" ????????? use only what?
"-we wipe" means ??? write random flux???
Last edited by DrCoolZic on Wed Nov 27, 2013 9:17 pm, edited 1 time in total.

Jeff_HxC2001
Posts: 58
Joined: Thu May 12, 2011 10:56 pm

Re: Get Information from experts about Flux transitions anal

Post by Jeff_HxC2001 » Wed Nov 27, 2013 7:04 pm

DrCoolZic wrote: Are you working with data from Stream and/or IPF files?
Yes. And in fact with all format supported by the Hxc Floppy Emulator software.
DrCoolZic wrote: When you say "cut it in small blocks" are you talking of block of flux transitions? On a typical DD one revolution is about 40000 flux transitions so if I understand correctly you group them by about 40. This gives you a pattern and you are trying to match the same pattern on the next revolution. If it matches (I guess you have to allow some margin to take in account RPM variation, etc...) then you are in "normal data block" if not, depending on the number of mismatches, this could be either fuzzy bits or unformatted area based on number of mismatched bits in the block?
Yes this is something like this.
DrCoolZic wrote: I do not fully understand the images displayed :shock: but it seems that the color scheme is used to indicate things like fuzzy or above/below normal speed sections.
I guess that the black line in middle of sector is the clock.
Not exactly : this is just the scatter plot of the pulses.
the green/red stuff are the sectors.
DrCoolZic wrote: Your "trackviewer" is this an available tool?
Yes sure, in the hxc floppy emulator software. contact me by email if you have questions.

Post Reply