Few questions about the stream protocol

[DrCoolZic]

Not sure this is the right place or should it go to support? Please feel free to move!
I have received the board and did my first image and I am now ready to start programming based on the code you have provided, but before I have some questions/remarks about the stream documentation

Before the questions I would like to make 2 remarks on the file format.
I understand that the format is optimized for communication and therefore may look odd for a file.
However I do not understand why you have defined nop2 and nop3. Nop means ignore this byte and nop2 means ignore this byte and the next one so nop2, xyz means skip nop2 and xyz. But you can achieve the exact same thing with nop, nop without definition of an extra encoding marker! Same for nop3. So that would free you two markers (0x09 and 0x0A)!
The other strange encoding is the overflow16 that increase by 0x10000 the cell value. I understand that this is probably not happening very often but as you are claiming to work with 32 bits value in order to get to the max value of 0xFFFFFFFF that’s a lot of overflow16! You might have considered (especially as this is not happening often) to use an extra byte or word that contains the upper part of the 32 bit value.

So now the questions:

• If I understand correctly the KryoFlux DTC program does not interpret the data received from the floppy drives it just write in the stream file two main items: index timing and flux transition timing?
• Looking from a high level at the information in the file (one per track) I presume we have the following sequence of blocks : StreamRead, flux transition data, index, StreamRead, flux transition data, index, …. , StreamEnd, End Is this correct?
• The number of index OOB must be one more than the number of revolution specified?
• Data before the first index are to be ignored?
• I suppose that when you talk about a “new cell” you mean a “new flux transition”?
• Mutiple StreamRead per track: does that mean tha the file contains several data block separated by index?
• One StreamEnd per track: Is it the end of all streamReads or the end of the stream file?
• Is it possible to have an index or other info after StreamEnd?
• End OOB? Why have StreamEnd and End? If StreamEnd is the end of the streamfile then why an End OOB after? Could that have been merged to an End (means end of stream file) followed by stream position and status?

Thanks, Jean

Edit: PS I have some problems with my development environment (SW and machines) will take few days before I can start coding

[IFW]

The protocol used for streaming is extremely efficient.

The format is optimized for a communication budget and a cpu budget.
The KryoFlux SoC has dedicated sampling, communication etc hardware, but each must be served via interrupts for asynchronous and efficient operation.
Therefore the firmware has various asynchronous sub-systems communicating with the hardware, with very specific requirements, constraints and cpu budget.If the firmware fails to serve these, the system will become unusable.
Think of it this way: a single track can contain about 500000 (that is five hundred thousand!) flux reversals per second for a HD disks that is being streamed to the host computer via a limited bandwidth USB link.
Every single flux reversal is served through an interrupt and has to be served within a very limited time including the encoding of the sample taken.
Every single clock cycle you can shave off the execution time of the sampler code will free 500000 clock cycles of your cpu budget.

By using a single nop code you will add at least six clock cycles for the execution budget of each sample, of which 2 will be memory accesses. In other words you suddenly use up an additional 3 million clock cycles per second. Very, very bad

Similarly, it's a very bad idea to collate overflows. It requires specific processing, and that will even take a lot more cycles.
You should also consider what an overflow really represents: about 2.7milliseconds without any flux reversals at all. One second is represented by about 370 overflows.
For comparison: the normal flow of flux reversals on a HD MFM track means about 2, 3 or 4 microseconds distance for each transition.
So we are talking about optimising 370 bytes per track to say 3 bytes... compared to say 500000 bytes needed to represent a track with proper flux reversals, ie most of your tracks on a disk.
You have 168 readable tracks on a 96 TPI disk.
Lots of clock cycles would be wasted on this "optimisation", and in return we earn exactly nothing when we are talking about 60-80mb worth of encoded flux reversals for a single HD disk dumped...

Again, how it is and why it is, all have very, very good reasons - but you always have to remember it is an embedded system with very specific requirements.

[IFW]

1, A stream file that DTC records is exactly what the KryoFlux board sends when streaming a track.
Therefore it already contains everything you will have in the file (e.g. index and flux reversal information among other things) - it's the output from the KryoFlux hardware recorded as is, without any modification.

2, No, in practice it's not always like that, especially considering the index data.
OOB data is sent whenever the communication and the cpu budget allows it, naturally the ordering of the OOB data is guaranteed, but it is very possible to have several OOB blocks at once.
That's why the decoder works the way it does.

3, The number of index OOB is very likely to be one more than the revolutions sampled, since to sample a complete revolution you need to see two index signals, for sampling 2 consecutive revolutions 3 index signals need to be passed and so on.
It does not matter for the decoder though; e.g. if there is an error during the streaming you might see less, or if for whatever reason the streaming was not stopped you might see more.

4, You can use data before the first index, but to calculate the actual rpm you need to see at least two index signals, and there is no way to guess drive speed wobble before the first signal. You can track the drive speed from the second signal and for example detect problems with the drive - DTC does that among other things.

5, Yes.

6, No.
As I said in the previous post various sub-systems work in the KryoFlux hardware and firmware on generating the stream and transmitting the stream completely asynchronously.
The stream is being generated in a ring buffer.
A Stream Read OOB is generated each time the ring buffer wraps and is being used by DTC to verify that it is in sync with the board - if there is any problem with either the data transfer or the data encoding the stream position sent will not match.
Although data integrity itself is guaranteed by USB, packets might get lost, due to OS activity etc.
It also helps calculating the exact transfer speed as seen by the board.

7, Yes, there is always one StreamEnd.
Again, it is used to verify the correctness of the transfer as well as the processing of the stream on the host side.
It also contains any error code the board wants to communicate to the host.
There is no guarantee you wouldn't see a StreamRead before the StreamEnd, it all depends on luck.

8, There shouldn't be anything after StreamEnd, but you will see a special OOB consisting of 7 OOB codes after StreamEnd. Think of that as an EOF signal; if it is not present the stream is technically incomplete.

9, The host software has to use very low resources when streaming data from the board and works on many fairly slow devices with minimum resource usage.
This would not be possible if the host was decoding the stream while it's being transmitted (very early prototype versions of DTC did that) - however it must be able to find out when the data transfer is complete, which is impossible, since we don't know the length of the transfer in advance. Using a simple timeout or signalling an USB error to signal the end of streaming would slow down the dumping process considerably, depending on the OS, and using other USB endpoints would not work very well either.
Therefore a signalling must be present in the stream itself that the host can use to detect the end of stream data or time out.
Due to the way how the USB protocol is used by the firmware, the host can always correctly detect the special EOF signal with a single aligned data comparison after each data chunk read by the host.

I highly recommend using the stream decoder source code as is, it can handle everything including all edge cases correctly.

[DrCoolZic]

Thanks,
I now better understand the reason of the encoding and the structure of the file. I hope to be able to parse a file very soon based on the provided code...

[DrCoolZic]

I started to look at sources provided (it now correctly compile under VisualStudio 2010) and I have a stupid question.
Not sure this is the right place to ask questions about the source so let me know if I should pm or ask in some other places.

When starting to decode a stream file I suppose you first need call the Init(uint8_t *strbuf, uint32_t strsize) procedure to specify the buffer pointer and size.
What I do not understand is the usage of uint8_t (unsigned char) for the buffer pointer? limited to 256 bytes, where the size is specified as a more reasonable unsigned 32 bit value?
Note that everything is consistent as *streambuf in the class CStreamDecoder is also declared as a uint8_t etc.
thanks

[IFW]

I am not sure I understand the question correctly
You indeed need to call Init, with a pointer to the buffer holding the stream and the exact size of the stream data (stream file).
uint8_t * is a pointer to a buffer containing unsigned, 8 bit values aka bytes

[IFW]

If you mean why does the stream consists of bytes; because that's an universal format, that does not depend on MSB vs LSB issues. Also most of the flux reversal periods you'd find on a disk are encoded into a single byte.

[DrCoolZic]

Ooops No excuse
Whent to bed too late yesterday

[DrCoolZic]

Question about units used in the stream file.

1) We have a master clock of 18432000 * 73 / 28 or about 48 MHZ ? that is a period of about 20.8 ns
2) flux transition sample frequency is about 24 MHz or a period of about psample = 41.6 ns
3) cell values (time between two flux transition) is given as n * psample. So for example a value of 98 is about 4.08 µs
4) TrTime value in OOB StreamRead ? Doc says elapse time since last transfer in ms. For example 169 would mean 169 ms from last ?OOB StreamRead ? What is the usage of this value ?
5) OOB Index Timer ? is this time since last data flux transition? So a value of 41 would mean 41 * psample or about 1.7 µs since last flux transition
But this has no influence on next data transition. Right?
6) OOB Index sys time : if I understand correctly this store an absolute index system clock in KF? with index clock= master clock / 16 or about 3 MHz ie. period about pindex = 333 ns
so if index_n - index_n-1 = 600237 this would mean about 199.85 ms between two indexes or a bit above 300 RPM

Still need to fully understand whats going on in DecodeIndex()

[IFW]

1-3, Yes, but keep in mind that the sampling frequency can change with future revisions of the hardware. The new beta firmware adds information about the exact clock rates being used, and DTC uses that if present in the stream otherwise uses the default values.
4, You can calculate the transfer speed between the host and the board, and also the jitter.
5, Yes.
The KryoFlux hardware and software is unique, as index signals are never bundled together with flux transitions.
There are some disks that are impossible to sample correctly without this and when we say it's very accurate we mean it
It's referred in the source as sub-cell as you need to have a smaller resolution than a flux reversal supported - the index might be at the edge, but most of the time it is somewhere between two flux reversals.
6, SCK (sample clock) and ICK (index clock) are clocked from the same source, but have different resolutions - however they are always completely in sync.
You can use default values, but if you find them defined in the stream it's advised to use the exact values there, see (1)
Index decoding is hideously complex due to the sub-cell accuracy and the various edge cases supported...