DTC on Linux/PPC

All questions about how to use KryoFlux go here.
marcus
Posts: 11
Joined: Fri Sep 13, 2013 8:16 pm

Re: DTC on Linux/PPC

Post by marcus »

Sounds like I have missed a lot of drama here. :-)

But if you want to sign an IPF file, can't you use use GPG to create a detached signature? That should work for any file.
Darkstar
Posts: 72
Joined: Thu Nov 04, 2010 7:58 pm

Re: DTC on Linux/PPC

Post by Darkstar »

No Drama, just the Devs stating their point (which is quite valid IMHO) and some people whining about it. Sure you could use detached signatures (even official lists of SHA sums would help), but that's apparently currently not much of a priority. it's most probably just lack of manpower (I hope). BUT: You can still dump to RAW files and preserve your disks that way, and at some point in the future it might be possible to have it converted to IPF (or to convert it yourself)

-Darkstar
User avatar
mr.vince
Posts: 2138
Joined: Tue Oct 05, 2010 5:48 pm

Re: DTC on Linux/PPC

Post by mr.vince »

marcus wrote:However, this is hardly a performance problem. The CPU is 90% idle while DTC is running, so there are plenty of computrons to spare, and it's only a couple of hundred K of flux data to transfer per cylinder anyway...

...
00.0 : Bad stream position
...

If you don't want to spend time fixing your bugs, then let me again suggest that you release the source code, so that other can spend that time for you. It's a win/win deal.
This is not a bug, but a performance issue. KF uses SoC with 12Mb/s USB high speed. Only recently, with new CPUs for more powerful smartphones, embedded systems have moved to really fast USB implementations. This means we only have about 1MB/s net throughput with the CPU on the board, so the host needs to be very quick. Try it with a contemporary system, Intel-based, Windows, Mac OS X... all will be fine.

We just don't have the time to fiddle and adjust for every niche/legacy system out there. And no we do not release the source at the moment. It is the only way to have to separate product lines, consumer and pro, and charge pro users for the software (which is free to private, non-commercial users). This way we can actually fund development and our own preservation efforts as much as possible.

Just for the record: I don't think there's any moral right to demand a source for something that's not been based on anyone else's work. It's all our tech, funded with our money, it's taken 12 years to develop. Others just talk, we actually made it happen.

Darkstar wrote:The KF Devs won't release the source because they're afraid of people too stupid to use KF correctly and the world being swarmed with invalid/fake/bad dumps made from non-pristine disks.
Thanks, we can take care of our own matters. :) You may be referring to the Analyser which generates IPFs, not DTC. IPF is not KryoFlux and KryoFlux is not IPF. It is correct that raw data read with KryoFlux can be fed into our Analyser, but it's a separate software, and a separate product.
Darkstar wrote:This would solve all of their problems/concerns, and it would make it possible to have other groups who want to preserve disks for different systems to use IPF for that as well (right now, you get pretty much ignored when you try submit dumps of anything not AMIGA/Atari, so you cannot get your own dumps into IPF format, only IMG/sector based)
We main issue always ways and still is: time. As long as no one of us here is doing this as the daytime job, we have to do this in our spare time. Since we all have families, we spend at our sole discretion. If you look at the free and open IPF implementation by Keir you will notice that it only supports a fraction of formats and there's not been that much progress over the last few months. I am not pointing my finger at anyone, but my estimated guess would be that Keir also has a real life and did not have the chance to poke around in disk coding techniques and protection stuff. Interesting side note: It's open, but I really don't see many contributions to the code. Why is that? MFM not sexy? ;)

There are things where you just do branches and let people hack and code. Survival of the fittest, the best code gets used and re-used and, even though the code might not be optimized or clean, something gets put up and people are happy with it. We don't see that same chance for preservation. People can't/won't redump titles for decades. People tend to remove titles from their Todo lists once titles have been imaged. You have to get it right the very first time a disk is dumped. Imagine reading an important source code, imagine you did something wrong during imaging... you blew it. We won't let that happen with KryoFlux, or let's say: We do a lot of work in engineering and lateron (regression testing) to avoid such situation.

The main issue we see, worse than undumped titles, are titles that are considered to be preserved. People won't dump them anymore and they might even reject physical donations with "thanks, we already have it". You either have a good and verified dump, or you don't.

As time allows, we can move on and make other things happen. Yes, we still have more things we'd like to do and yes, automated analysation to the extent possible, is something we'd really like to add.
Darkstar
Posts: 72
Joined: Thu Nov 04, 2010 7:58 pm

Re: DTC on Linux/PPC

Post by Darkstar »

One of the main problems I still see with KryoFlux/IPF/SPS/DTC are the non-clear distinctions between them and that there's no official "roadmap" for getting your own dumped disks into IPF format or preserved. I tried opening tickets in the past, asking for instructions how to get the raw files to the team. And every time either the tickets were ignored or my mails. True, I'm not a "big customer" who pays KF to dump hundreds of rare disks, some of the disks I have are probably already dumped, but since the only way to get an IPF file for these is for me to "prove" somehow that I'm eligible to receive the IPF file (which normally involves sending raw dumps to the devs), I wish there would be an easier, more transparent way.

But transparency is a bit lacking IMHO (the last non-press-release-style update to SoftPres' homepage was in 2011, and the last WIP entry is even older, from sometime in 2010, for example)

Sure, this again boils down to "we do this in our spare time", which I can totally understand and relate to. But to an outsider this might look pretty much like a "we don't care" attitude

-Darkstar
User avatar
mr.vince
Posts: 2138
Joined: Tue Oct 05, 2010 5:48 pm

Re: DTC on Linux/PPC

Post by mr.vince »

How can we lay out a roadmap, if we don't know about the funding to expect? KryoFlux does already generate many formats which can be used out of the box. IPF was never advertised or promised for analysation inside KryoFlux, but it does write such files perfectly. I understand people are "desperate" to get automatic IPF creation, but this isn't trivial or easy.

We've never asked private users for any money, I am sorry you did not get a reply as expected, but we have many requests and - again - limited time. Please re-send your request, I'll see if we can help.

If you want to volunteer for "public relations" that will be very much appreciated. E.g. the new versions of VICE with extended G64 support are something we made happen and we also were part of the funding of EFGAMP (www.efgamp.eu), and there's a lot more we could tell. To quote myself: There are others that just talk, we actually make things happen.
marcus
Posts: 11
Joined: Fri Sep 13, 2013 8:16 pm

Re: DTC on Linux/PPC

Post by marcus »

Hi again.

I have now created my own open source replacement for the dtc tool: https://github.com/zeldin/OpenDTC

It only supports reading in raw format, but it runs perfectly on both the X1000 and the G4. While it is true that the G4 needed a bit more buffering than the X1000 for reliable operation, it turns our that this is actually not what is wrong with dtc. At least not the only thing.

When running dtc on OSX/PPC using the streams I captured, it still displays the same errors:

Code: Select all

$ dtc -m1 -fxtrack -i0 -fdisk.img -i6
KryoFlux DiskTool Console, v2.20, uiv.1, Mar 24 2013, 19:59:05
(c) 2009-2013 KryoFlux Products & Services Ltd.
Developed by The Software Preservation Society, www.softpres.org
Licensed for private, non-commercial use only.

00.0    : Bad stream position
00.0    : Read operation failed
02.0    : Bad stream position
02.0    : Read operation failed
04.0    : Bad stream position
04.0    : Read operation failed
06.0    : Bad stream position
06.0    : Read operation failed
08.0    : Bad stream position
08.0    : Read operation failed
10.0    : Bad stream position
10.0    : Read operation failed
12.0    : Bad stream position
12.0    : Read operation failed
14.0    : Bad stream position
14.0    : Read operation failed
16.0    : Bad stream position
16.0    : Read operation failed
18.0    : Bad stream position
18.0    : Read operation failed
20.0    : Bad stream position
20.0    : Read operation failed
22.0    : Bad stream position
22.0    : Read operation failed
24.0    : Bad stream position
24.0    : Read operation failed
26.0    : Bad stream position
26.0    : Read operation failed
28.0    : Bad stream position
28.0    : Read operation failed
30.0    : Bad stream position
30.0    : Read operation failed
32.0    : Bad stream position
32.0    : Read operation failed
34.0    : Bad stream position
34.0    : Read operation failed
36.0    : Bad stream position
36.0    : Read operation failed
38.0    : Bad stream position
38.0    : Read operation failed
40.0    : Bad stream position
40.0    : Read operation failed
42.0    : Bad stream position
42.0    : Read operation failed
44.0    : Bad stream position
44.0    : Read operation failed
46.0    : Bad stream position
46.0    : Read operation failed
48.0    : Bad stream position
48.0    : Read operation failed
50.0    : Bad stream position
50.0    : Read operation failed
52.0    : Bad stream position
52.0    : Read operation failed
54.0    : Bad stream position
54.0    : Read operation failed
56.0    : Bad stream position
56.0    : Read operation failed
58.0    : Bad stream position
58.0    : Read operation failed
60.0    : Bad stream position
60.0    : Read operation failed
62.0    : Bad stream position
62.0    : Read operation failed
64.0    : Bad stream position
64.0    : Read operation failed
66.0    : Bad stream position
66.0    : Read operation failed
68.0    : Bad stream position
68.0    : Read operation failed
70.0    : Bad stream position
70.0    : Read operation failed
72.0    : Bad stream position
72.0    : Read operation failed
74.0    : Bad stream position
74.0    : Read operation failed
76.0    : Bad stream position
76.0    : Read operation failed
78.0    : Bad stream position
78.0    : Read operation failed
80.0    : Bad stream position
80.0    : Read operation failed
82.0    : Bad stream position
82.0    : Read operation failed

Enjoy your shiny new disk image!
Please consider helping us to preserve media and continue development:
www.softpres.org/donate
However, if I copy the stream files to a remote AMD64 machine where I have a shell account, and run the Linux/AMD64 version of dtc, then it works:

Code: Select all

$ dtc -m1 -fxtrack -i0 -fdisk.img -i6
KryoFlux DiskTool Console, v2.20, uiv.1, Feb 24 2013, 07:25:23
(c) 2009-2013 KryoFlux Products & Services Ltd.
Developed by The Software Preservation Society, www.softpres.org
Licensed for private, non-commercial use only.

00.0    : frev: 31932, drift: 0.067 us, tfer: 181977 B/s, rpm: 360.026
00.0    : base: 3.204 us [98.436%], band: 3.186 us, 6.409 us, 9.600 us
00.0    : CBM DOS: OK, trk: 001, sec: 21
02.0    : frev: 31967, drift: 0.333 us, tfer: 182180 B/s, rpm: 359.990
02.0    : base: 3.206 us [98.297%], band: 3.190 us, 6.412 us, 9.566 us?
02.0    : CBM DOS: OK, trk: 002, sec: 21
04.0    : frev: 31989, drift: 0.083 us, tfer: 182383 B/s, rpm: 360.003
04.0    : base: 3.194 us [98.100%], band: 3.190 us, 6.389 us, 9.551 us?
04.0    : CBM DOS: OK, trk: 003, sec: 21
06.0    : frev: 32680, drift: 0.092 us, tfer: 186113 B/s, rpm: 359.998
06.0    : base: 3.200 us [98.323%], band: 3.172 us, 6.399 us, 9.601 us?
06.0    : CBM DOS: OK, trk: 004, sec: 21
08.0    : frev: 37482, drift: 0.208 us, tfer: 213858 B/s, rpm: 359.949
08.0    : base: 3.201 us [98.117%], band: 3.136 us, 6.403 us, 9.594 us
08.0    : CBM DOS: OK, trk: 005, sec: 21
10.0    : frev: 35536, drift: 0.092 us, tfer: 203243 B/s, rpm: 359.973
10.0    : base: 3.202 us [98.355%], band: 3.145 us, 6.404 us, 9.630 us
10.0    : CBM DOS: OK, trk: 006, sec: 21
12.0    : frev: 34991, drift: 0.191 us, tfer: 200138 B/s, rpm: 359.993
12.0    : base: 3.200 us [98.246%], band: 3.150 us, 6.401 us, 9.607 us
12.0    : CBM DOS: OK, trk: 007, sec: 21
14.0    : frev: 35366, drift: 0.067 us, tfer: 201989 B/s, rpm: 359.993
14.0    : base: 3.202 us [98.218%], band: 3.147 us, 6.403 us, 9.602 us
14.0    : CBM DOS: OK, trk: 008, sec: 21
16.0    : frev: 32886, drift: 0.508 us, tfer: 187391 B/s, rpm: 360.008
16.0    : base: 3.204 us [98.306%], band: 3.148 us, 6.409 us, 9.613 us
16.0    : CBM DOS: OK, trk: 009, sec: 21
18.0    : frev: 35604, drift: 0.125 us, tfer: 203453 B/s, rpm: 360.013
18.0    : base: 3.204 us [98.205%], band: 3.139 us, 6.409 us, 9.602 us
18.0    : CBM DOS: OK, trk: 010, sec: 21
20.0    : frev: 35136, drift: 0.491 us, tfer: 200957 B/s, rpm: 360.012
20.0    : base: 3.209 us [98.159%], band: 3.139 us, 6.418 us, 9.584 us
20.0    : CBM DOS: OK, trk: 011, sec: 21
22.0    : frev: 35114, drift: 0.583 us, tfer: 200751 B/s, rpm: 360.017
22.0    : base: 3.202 us [98.291%], band: 3.143 us, 6.403 us, 9.621 us
22.0    : CBM DOS: OK, trk: 012, sec: 21
24.0    : frev: 35524, drift: 0.117 us, tfer: 203033 B/s, rpm: 360.016
24.0    : base: 3.204 us [98.156%], band: 3.140 us, 6.407 us, 9.594 us
24.0    : CBM DOS: OK, trk: 013, sec: 21
26.0    : frev: 35411, drift: 0.275 us, tfer: 202823 B/s, rpm: 360.021
26.0    : base: 3.202 us [98.477%], band: 3.146 us, 6.403 us, 9.654 us
26.0    : CBM DOS: OK, trk: 014, sec: 21
28.0    : frev: 37458, drift: 0.608 us, tfer: 213858 B/s, rpm: 360.022
28.0    : base: 3.202 us [98.114%], band: 3.130 us, 6.404 us, 9.598 us
28.0    : CBM DOS: OK, trk: 015, sec: 21
30.0    : frev: 35442, drift: 0.150 us, tfer: 202405 B/s, rpm: 360.020
30.0    : base: 3.200 us [98.180%], band: 3.141 us, 6.401 us, 9.603 us
30.0    : CBM DOS: OK, trk: 016, sec: 21
32.0    : frev: 34454, drift: 1.065 us, tfer: 195909 B/s, rpm: 360.019
32.0    : base: 3.201 us [98.263%], band: 3.143 us, 6.401 us, 9.617 us
32.0    : CBM DOS: OK, trk: 017, sec: 21
34.0    : frev: 28306, drift: 0.425 us, tfer: 161998 B/s, rpm: 360.017
34.0    : base: 3.598 us [97.306%], band: 3.579 us, 7.196 us, 10.791 us?
34.0    : CBM DOS: OK, trk: 018, sec: 19
36.0    : frev: 32210, drift: 0.474 us, tfer: 183609 B/s, rpm: 360.022
36.0    : base: 3.601 us [97.576%], band: 3.544 us, 7.201 us, 10.763 us
36.0    : CBM DOS: OK, trk: 019, sec: 19
38.0    : frev: 32407, drift: 0.583 us, tfer: 184644 B/s, rpm: 360.022
38.0    : base: 3.601 us [97.408%], band: 3.542 us, 7.201 us, 10.801 us
38.0    : CBM DOS: OK, trk: 020, sec: 19
40.0    : frev: 32478, drift: 0.067 us, tfer: 185062 B/s, rpm: 360.017
40.0    : base: 3.601 us [97.340%], band: 3.543 us, 7.201 us, 10.815 us
40.0    : CBM DOS: OK, trk: 021, sec: 19
42.0    : frev: 32556, drift: 0.075 us, tfer: 185481 B/s, rpm: 360.021
42.0    : base: 3.600 us [97.579%], band: 3.540 us, 7.199 us, 10.769 us
42.0    : CBM DOS: OK, trk: 022, sec: 19
44.0    : frev: 31648, drift: 0.441 us, tfer: 180573 B/s, rpm: 360.021
44.0    : base: 3.602 us [97.368%], band: 3.550 us, 7.203 us, 10.799 us
44.0    : CBM DOS: OK, trk: 023, sec: 19
46.0    : frev: 30806, drift: 0.108 us, tfer: 175918 B/s, rpm: 360.018
46.0    : base: 3.602 us [97.376%], band: 3.545 us, 7.205 us, 10.801 us
46.0    : CBM DOS: OK, trk: 024, sec: 19
48.0    : frev: 29414, drift: 0.916 us, tfer: 168151 B/s, rpm: 360.018
48.0    : base: 3.805 us [98.784%], band: 3.750 us, 7.609 us, 11.414 us
48.0    : CBM DOS: OK, trk: 025, sec: 18
50.0    : frev: 29250, drift: 0.724 us, tfer: 167122 B/s, rpm: 360.016
50.0    : base: 3.801 us [98.981%], band: 3.747 us, 7.601 us, 11.375 us
50.0    : CBM DOS: OK, trk: 026, sec: 18
52.0    : frev: 29478, drift: 0.499 us, tfer: 168497 B/s, rpm: 360.015
52.0    : base: 3.802 us [98.718%], band: 3.750 us, 7.603 us, 11.435 us
52.0    : CBM DOS: OK, trk: 027, sec: 18
54.0    : frev: 29484, drift: 0.166 us, tfer: 167335 B/s, rpm: 360.014
54.0    : base: 3.800 us [98.860%], band: 3.752 us, 7.600 us, 11.405 us
54.0    : CBM DOS: OK, trk: 028, sec: 18
56.0    : frev: 29867, drift: 0.100 us, tfer: 170604 B/s, rpm: 360.018
56.0    : base: 3.799 us [98.789%], band: 3.753 us, 7.597 us, 11.422 us
56.0    : CBM DOS: OK, trk: 029, sec: 18
58.0    : frev: 30768, drift: 0.516 us, tfer: 175729 B/s, rpm: 360.014
58.0    : base: 3.802 us [99.034%], band: 3.742 us, 7.605 us, 11.355 us
58.0    : CBM DOS: OK, trk: 030, sec: 18
60.0    : frev: 30080, drift: 0.067 us, tfer: 171857 B/s, rpm: 360.012
60.0    : base: 4.001 us [99.373%], band: 3.942 us, 8.001 us, 12.092 us
60.0    : CBM DOS: OK, trk: 031, sec: 17
62.0    : frev: 25852, drift: 0.125 us, tfer: 147383 B/s, rpm: 360.012
62.0    : base: 3.998 us [99.816%], band: 3.985 us, 7.995 us, 11.975 us?
62.0    : CBM DOS: OK, trk: 032, sec: 17
64.0    : frev: 25674, drift: 0.341 us, tfer: 146395 B/s, rpm: 360.009
64.0    : base: 3.999 us [99.730%], band: 3.989 us, 7.998 us, 11.948 us?
64.0    : CBM DOS: OK, trk: 033, sec: 17
66.0    : frev: 25660, drift: 0.133 us, tfer: 146232 B/s, rpm: 360.006
66.0    : base: 4.009 us [99.652%], band: 3.984 us, 8.017 us, 11.949 us?
66.0    : CBM DOS: OK, trk: 034, sec: 17
68.0    : frev: 25674, drift: 0.083 us, tfer: 146395 B/s, rpm: 360.010
68.0    : base: 4.000 us [99.821%], band: 3.990 us, 7.999 us, 11.968 us?
68.0    : CBM DOS: OK, trk: 035, sec: 17
70.0    : frev: 60146, drift: 0.924 us, tfer: 344075 B/s, rpm: 360.004
70.0    : base: 3.761 us [33.034%], band: 1.683 us, 3.761 us, 5.077 us
70.0    : CBM DOS: <unformatted>
72.0    : frev: 61945, drift: 1.490 us, tfer: 355341 B/s, rpm: 360.006
72.0    : base: 3.602 us [30.414%], band: 1.590 us, 3.602 us, 5.296 us
72.0    : CBM DOS: <unformatted>
74.0    : frev: 62040, drift: 0.674 us, tfer: 354886 B/s, rpm: 360.005
74.0    : base: 3.546 us [32.016%], band: 1.624 us, 3.546 us, 4.830 us
74.0    : CBM DOS: <unformatted>
76.0    : frev: 61895, drift: 0.782 us, tfer: 354118 B/s, rpm: 360.000
76.0    : base: 3.641 us [30.418%], band: 1.650 us, 3.641 us, 4.924 us
76.0    : CBM DOS: <unformatted>
78.0    : frev: 61897, drift: 0.974 us, tfer: 354118 B/s, rpm: 360.008
78.0    : base: 3.714 us [32.376%], band: 1.652 us, 3.714 us, 5.403 us
78.0    : CBM DOS: <unformatted>
80.0    : frev: 61780, drift: 0.075 us, tfer: 353736 B/s, rpm: 360.003
80.0    : base: 3.583 us [30.948%], band: 1.622 us, 3.583 us, 4.894 us
80.0    : CBM DOS: <unformatted>
82.0    : frev: 61637, drift: 1.956 us, tfer: 352974 B/s, rpm: 360.005
82.0    : base: 2.800 us [19.482%], band: 1.556 us, 2.800 us, 3.991 us
82.0    : CBM DOS: <unformatted>

Enjoy your shiny new disk image!
Please consider helping us to preserve media and continue development:
www.softpres.org/donate
This is using file input, with the same files, on both machines. No KryoFlux hardware is even connected. So there is no way this is a performance problem. It's a bug, plain and simple.

My personal guess would be a byte-order problem in the parsing of the stream data (the OOB packets specifically), but that would imply that you haven't actually tested the binary on that G5 at all...

Again, I'd be happy to help you with fixing your bugs. All I'd need is to take a look at the source.
User avatar
Malvineous
Posts: 156
Joined: Sun Oct 31, 2010 10:57 pm
Location: Brisbane, Australia
Contact:

Re: DTC on Linux/PPC

Post by Malvineous »

How are you running dtc on OSX/PPC? I wasn't aware there was a PPC version. Or is this the Mac version you are running? I thought that was Intel only. I've just had a quick look at the code and I can confirm that it assumes the platform will always be little-endian.

I guess performing the capture on PPC would produce big-endian stream files (which would read correctly on PPC, but give the same errors on Intel instead), so maybe the best solution is a converter utility that will convert between big- and little-endian files?
marcus
Posts: 11
Joined: Fri Sep 13, 2013 8:16 pm

Re: DTC on Linux/PPC

Post by marcus »

I'm running the official Mac OS X binary, which is a fat (dual architecture) binary as far as I can tell.

No, the capture will always produce little-endian streams, because the streams are generated by the KryoFlux itself, not by the host.
Or more specifically, the streams are mixed-endian, because the cell values are big-endian but OOB fields are little-endian...
So the solution is simply to decode the OOB headers as little endian without making assumptions about the host byteorder. I.e. instead of

Code: Select all

const uint8_t *p;
uint32_t value = *(const uint32_t*)p;
write

Code: Select all

const uint8_t *p;
uint32_t value = p[0] | (p[1]<<8) | (p[2]<<16) | (p[3]<<24);
like I do in my program. Or you can test if the host is big-endian and call __builtin_bswap32() as needed. There are plenty of well known techniques to handle it. :-)

Converting the streams to use big-endian format in the OOB blocks would just cause confusion and fragmentation I think. (Although it could be a temporary workaround until the tool is fixed, I might test it later...)
User avatar
mr.vince
Posts: 2138
Joined: Tue Oct 05, 2010 5:48 pm

Re: DTC on Linux/PPC

Post by mr.vince »

Hence I originally said: no PPC build, only causes trouble. :)

I understand you like it, but in the real world we cover > 95% of non-legacy systems out there. That's far more than most commercial software.

But I'll pass this on to the OS X maintainer.
marcus
Posts: 11
Joined: Fri Sep 13, 2013 8:16 pm

Re: DTC on Linux/PPC

Post by marcus »

Thanks. :-)
Post Reply