MFM Decoding Unsectored Disks

All questions about how to use KryoFlux go here.
dexterslab
Posts: 10
Joined: Tue Aug 22, 2017 8:51 am

MFM Decoding Unsectored Disks

Post by dexterslab »

I have an unusual system in my possession that has a number of 8" floppies that we're trying to archive and read with kryoflux. In addition to the archiving we want to recover/decode the data they contain so we can analyse it on the PC. The drive is all connected and working with kryoflux.

I'll go into a reasonable amount of detail here so hopefully you can get an idea of what we're trying to do.

The hardware is a drawing/graphics tool developed for the broadcast TV industry in the early 1980s, it's called a Quantel Paintbox. I have a number of newer quantel paintboxes but this particular one is the first generation called the DPB-7001 and was in use at the BBC so we're intrigued about what is on the floppies we got with it. Also if we can recover some of the data off the disks there is a small possibility we maybe able to get the system running again but this is not too much of a desire at the moment.

So the floppies are all 8" DSDD, the drive itself is an NEC FD1165, the drive is interfaced to the system through a SMD interface card (Storage Media Device) the SMD bus is also used by the enormous 14" hard disk, this in turn then connects to the paintbox through a disk sequencer card which clocks data into the memory of the actual system.

The floppy format is very non-standard, the disk operations were mostly handled in 74x hardware logic. From what we can understand the disks have a very basic structure. I am pretty sure all the disks use 77 tracks, we are sure the disks are MFM encoded but we don't believe the tracks are sectored at all, the paintbox seemed to treat the disk more like a linear device. They may be a basic lead in after the index pulse with some sync words on each track and then the rest of the track is data, once one track had been loaded it would move to the next... rinse and repeat etc.

The disks typically held either an uncompressed PAL or NTSC bitmap images or a bitmap font

So i am able to read in the disks in using the preservation format in kryoflux which is fine for archiving but but what we really need to do is decode the MFM data from the KF stream files into raw binary without getting bogged down looking for sectors, sector headers, sync words etc. When using the MFM decode options within KF it seems to always expect to see sectors and never outputs any data for us.

We just need the entire MFM decoded track so we can figure out how the lead in and sync bytes are formatted. This will allow us to continue working out the disk control logic that's within the paintbox and ultimately allow us to read the actual data from the disks.

Can KF be configured to do MFM decode without doing anything else?

SomeGuy
Posts: 252
Joined: Wed Feb 18, 2015 8:18 pm

Re: MFM Decoding Unsectored Disks

Post by SomeGuy »

This Paintbox sounds like a very interesting device. Are the 8" floppy disks hard or soft sectored? What makes you so sure the encoding is standard MFM? If the interface card is really implemented with just 74x logic rather than a typical FDC microcontroller, I would be very suspicious it could be using something else or some slightly incompatible variation.

The tools that come with the Kryoflux are not capable of dumping just a decoded MFM/FM track without normal sector organization. There are some flux image processing tools included with the PCE emulator, however, that can convert a Kryoflux stream in to hex text files of decoded FM or MFM data: http://www.hampa.ch/pub/pce/pre/pce-201 ... -win32.zip . I have used these to repair some disk images that had damaged sector headers. Keep in mind that will only work if the track really contains MFM encoded data. It is probably important to note though that the Kryoflux folks neither endorse nor provide support for third party utilities.

If you could post a sample Kroflux preservation dump, some of the folks here could try pulling it apart.

dexterslab
Posts: 10
Joined: Tue Aug 22, 2017 8:51 am

Re: MFM Decoding Unsectored Disks

Post by dexterslab »

thanks for the reply!

Yes the paintbox is a very interesting system, it went on to change the look of TV all over the world but sadly knowledge of the system has faded to almost nothing due to the small number of people who got to use it mostly due to it's enormous cost at the time.

the disks are soft sectored, we have looked in depth at the logic on the drive controller cards and it all says MFM to us, the other reason we think it would be MFM is because the same controller cards also work the hard disk which is attached to the same interface, the hard disk is also MFM. But i have learned though messing with other quantel products is to always expect the unexpected!!

I've linked three stream file images below to download, if anyone wants to take a look feel free. I should point out that the disks are very old, the scatter plots really didn't look very good at all!

https://drive.google.com/drive/folders/ ... 0ctU00zalE

ZrX
Posts: 595
Joined: Tue Dec 06, 2011 9:09 pm

Re: MFM Decoding Unsectored Disks

Post by ZrX »

Some of the disks contain seemingly unused MFM-sectors at the end of the disk. But the first tracks look pretty much just nonsense. Tho here and there I find parts that resemble GCR encoding.

Hard to tell if the first tracks are damaged over time.

You wouldn't have access to oscilloscope? That way you could hook up to the head amplifier of the floppydrive and the analog waveform would tell you if the disks are really damaged, or something else.

SomeGuy
Posts: 252
Joined: Wed Feb 18, 2015 8:18 pm

Re: MFM Decoding Unsectored Disks

Post by SomeGuy »

Ok, the dumps KF002_DPB_Brush_Master_103505 and KF016_Fount_Master_40729_MLSH_Trade_Gothic_Bold_Extended contain lots of noise on all tracks and are basically unreadable. Were these disks heavily damaged?

KF012_MLSH_Neue_Helvetica_67_Med_Co, however looks to have a good signal and seems intact. The tracks look index aligned so in theory you might be able to write a usable duplicate from this stream file with the Kryoflux.

Track 60 and up on this disk contain what looks like a remnant from a previous normal-ish format with empty sectors. I would guess a system like this would not even bother pre-formatting tracks, so these are probably just tracks that were not used yet.

I tried the PCE tools on them hoping it would give me essentially one big sector. Some of the tracks do in fact look like that, and I can even make out bits of uncompressed bit-mapped graphics characters. So, yes, this does look like MFM. However some of the tracks have sync bytes or intentional "invalid" bytes thrown around inside them. In some places it looks like they even delimit data, in other places it looks like gibberish. I can't really say if this is how it is really supposed to be or just the result of the tools getting confused and out of sync by something else. Anyway, I'm attaching the MFM output text file for this dump if you want to look at it.

Unfortunately, that is about all I can do. That is one weird disk format. :shock: I guess we'll see if anyone else has any ideas.
Attachments
KF012_MLSH_Neue_Helvetica_67_Med_Co MFM.zip
KF012_MLSH_Neue_Helvetica_67_Med_Co MFM
(143.68 KiB) Downloaded 52 times

ZrX
Posts: 595
Joined: Tue Dec 06, 2011 9:09 pm

Re: MFM Decoding Unsectored Disks

Post by ZrX »

S0_T1.png
S0_T1.png (5.98 KiB) Viewed 1163 times
S0_T0-5.png
S0_T0-5.png (9.12 KiB) Viewed 1163 times
Pro_Font.png

dexterslab
Posts: 10
Joined: Tue Aug 22, 2017 8:51 am

Re: MFM Decoding Unsectored Disks

Post by dexterslab »

SomeGuy wrote:
Thu Aug 24, 2017 1:15 am
Ok, the dumps KF002_DPB_Brush_Master_103505 and KF016_Fount_Master_40729_MLSH_Trade_Gothic_Bold_Extended contain lots of noise on all tracks and are basically unreadable. Were these disks heavily damaged?
Yes some of the disks have a significant amount of oxide on the surface, there are mostly two brands. 3M and Dysan. The Dysan disks are much worse than the 3M ones. The system was decommissioned in the mid 1990s but how it's all been stored is anyone's guess.
Track 60 and up on this disk contain what looks like a remnant from a previous normal-ish format with empty sectors. I would guess a system like this would not even bother pre-formatting tracks, so these are probably just tracks that were not used yet.
Entirely possible the disks were preformatted and quantel just wrote their own format/data over them leaving remains behind.
ZrX wrote:
Thu Aug 24, 2017 7:55 pm
S0_T1.png
S0_T0-5.png
Pro_Font.png
wow, that's awesome work! Would you be able to share the steps and settings to get that data? We would love to try this on more of the disks!

ZrX
Posts: 595
Joined: Tue Dec 06, 2011 9:09 pm

Re: MFM Decoding Unsectored Disks

Post by ZrX »

dexterslab wrote:
Fri Aug 25, 2017 8:09 am
wow, that's awesome work! Would you be able to share the steps and settings to get that data? We would love to try this on more of the disks!
That was some manual work along with a selfmade tool I've been working on for analysing media.

So far I haven't come up with how the original hardware syncs to the single sector on a track.

The format deviates from the usual MFM media uses in ways that databits also form the data as is, not interleaving the high and low nybbles like typical disk media does. Bitshifting data around the disk name suddenly popped up from that one track which helped.

The sectors also seem as if they share the same track on sides 0 and 1, consuming less space on side 1 than on side 0.

The other font disk did contain something human recognisable but is probably unreadable if it was one of those oxidized floppies. Perhaps try washing it with plain water first...

From the Brush Master image I couldn't come up with anything.

OzOnE
Posts: 4
Joined: Wed Aug 23, 2017 5:24 pm

Re: MFM Decoding Unsectored Disks

Post by OzOnE »

Finally got access to the forum. lol

I stupidly left my e-mail address in the Username box for the past two days, thinking that my account hadn't been activated yet. I was expecting an e-mail confirmation too, but that never happened. Oh well.

OK, basically, I was helping the previous owner of the Paintbox (RetroGamerVX on YouTube) to see if we could get the thing to boot.

(It's obviously been a major challenge to try to debug things without having the machine in front of me, so we never managed to get the thing to boot from it's Fujitsu SMD HDD.)

Mr Dexter then took possession of the Paintbox about a year ago, and I've been working with him to primarily try to at least make some backups of the 8" floppy disks that arrived with the unit.

(the whole story spans more than 3 or 4 years, so I won't go into it here, yet. lol)

Recently, Mr Dexter and I were very kindly loaned the use of a Kryoflux for trying to recover the contents of the 8" floppy disks.

There is also the giant Fujitsu SMD hard drive that we've yet to extract anything from, but I'm still currently working on that, and I've just bought a smaller 8" NEC SMD HDD, so I can work on a reader for that, and ultimately design an SMD drive emulator as well.


I'll try to explain some of the logic used on the Paintbox system for locking-on to the MFM tracks...

The floppy drive unit has an extra board inside that hooks up to the same SMD cable bus that the Fuji hard drive does.

The board in the floppy drive unit basically just responds to the "Seek" commands for each cylinder/ track, and then just streams the recovered MFM data to the Disc Data Buffer board in the Paintbox itself (in a very similar way to the data from the SMD hard drive, just at a much slower clock rate).

(the floppy drive itself is just a standard NEC

Here's the logic on the floppy drive board that recovers the "Clock" and "Data" from the MFM stream.
It uses an FDC9216B chip to separate the clock and data pulses, and has a fixed 8MHz clock input to that chip...

http://imgur.com/a/ftlHR

The "Read Clock" and "Read Data" outputs then go via the SMD cable, then into some voltage-level translators, and then the TTL output goes from the translators directly into the Disk Data Buffer board.

The Data Buffer board uses this bit of logic to detect the "Sync byte" in the MFM bitstream...

http://imgur.com/a/sJS5y

The various flip-flops are "armed" after each Index pulse, then the output of the multi-input NAND gate "EE" goes Low when the sync byte is detected.

Actually, the service manual describes this better...

http://imgur.com/a/BKRpF


From what I can tell, the MSB bit of the sync byte would arrive at pin 13 of that shift-register, so from that, I could see that the bit pattern would need to be "b01010011", so 0x53 in hex?

That would cause the output of NAND gate "EE" to go Low, which then allows the actual track data to be streamed into the SRAM on the same Data Buffer board, once it has locked-on to the proper byte framing.

(The MFM data is essentially pre-decoded to normal byte data by the FDC9216 in the floppy drive unit before it enters the shift reg, so the parallel output of the shift reg should reflect the true data, rather than the raw MFM?)

Presumably, you don't really NEED to know the sync byte to be able to recover the clock from the MFM stream, and thus determine which bits are the Data bits as well?

I know the FDC chip is using a PLL to do that, and I've attempted to do the same thing on an FPGA, but I couldn't get mine to output the proper data reliably. lol

@ZrX - that's awesome that you've been able to recover such clear data from the disk images. :)

We'd been messing with things like LemASM before, to try to spot any patterns in the data.

I've been using the "pfi" tool from the PCE tools that "SomeGuy" kindly posted earlier, so I can at least try to convert the KF Stream files into some form of raw data.

Having a tool (preferably with a GUI for Windows) that has options for shifting the bits left and right, so we can spot clear text in an MFM image would be fantastic, especially if it has a preview window for visualising the data as well (with variable line width).

I've been struggling for the past few days with things like Hex Workshop, and shifting the data to see if we can spot anything. I now realise that my first big mistake was trying to shift the data AFTER it had been converted from the raw MFM image using the PCE tools. lol


EDIT: tbh, I expected to be able to create fully custom format templates for the latest version of the Kryoflux software, including things like custom sync words, and address markers etc.

I've been trying "Aufit" as well this week, and it does some nice graphical previews of the disk layout (akin to what @ZrX showed above), but it still doesn't allow you to shift the MFM bits around etc.

Having a more complete analysis tool with Kryoflux that can help identify weird formats like the Paintbox disks would be ideal. ;)



I've only really messed with MFM stuff since helping with the Paintbox, so it's been a bit of a brain-twister tbh.

I can see now how a "Clock" pulse in MFM is added when the Data cells on either side both contain 0s, and even made a spreadsheet for converting between MFM "Words", and normal byte values. lol

http://imgur.com/a/gbytC

Anyway, we're VERY grateful for any help you guys are giving with this, as it's been a very long road to get to this point.

I'm sure it's also possible to make KF Stream backups of the hard-sectored disks as well, but that will require some tweaking (and / or masking tape) before the floppy drive will let us do that. lol


Regards,
OzOnE.

OzOnE
Posts: 4
Joined: Wed Aug 23, 2017 5:24 pm

Re: MFM Decoding Unsectored Disks

Post by OzOnE »

@ZrX

I didn't even realise that many common disk formats interleave the upper and lower nibbles?

A small correction to my post above...

The Paintbox schematics show that there's a second shift-register ("BD") that is daisy-chained after "CD", and that is so the board can buffer full 16-bit data from the disk, then send it directly to the Filter Board.

That 16-bit data would contain interleaved Y and U/V pixel samples, AFAIK.

So, that might give more insight into the data format for uncompressed images.


EDIT: The normal data path is via 8-bit buffer "CE", which can then be passed to either the SRAM on the Data Buffer board (same board).

The Data Buffer board contains 28KBytes of SRAM btw, so that fits with the ~15-20KB per disk track?

I believe the 68000 CPU in the Paintbox can stream the data directly via the Filter board into it's 512KB of DRAM as well?


Mr Dexter and I were at one point thinking that some (or all) of the Font disks might actually use some form of encryption, as that does seem to be the case with the newer V-Series "Harriet" style Paintbox units, which were released around 1989-1990.

We spent a long time trying to figure out how the encryption stuff works on those newer units, and they do contain a GAL chip with the machine's serial number on it. Some clever guys on the EEVblog forums figured out the encryption algo, so that's fairly well understood now.

We suspect the classic DPB Paintbox might use some level of encryption as well, but we've yet to see which of the GAL chips on the boards might contain the serial number etc.

From your post above though, it's looking like the disks do just contain the raw uncompressed bitmaps for the fonts, which is good news.


Ultimately, we're hoping for a bit of a pay-off after all these years, and that the hard-sectored disks contain some of the original BBC Logos etc.

Even though it's looking extremely unlikely that those disks were ever intended for the Paintbox, I'm sure the images will be in some sort of readable format, so that would be great to see regardless.

Regards,
OzOnE.

Post Reply