It is currently Fri Sep 21, 2018 3:42 am




 Page 1 of 4 [ 54 posts ]  Go to page 1, 2, 3, 4  Next
Author Message
 Post subject: Fast Reads - how?
PostPosted: Fri Sep 10, 2010 3:20 pm 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
I'm working on a driver to read 512 byte sectors at high speed. This works:

RDLP   
   LDA I_STAT
   AND #$08
   BEQ RDLP
   LDA I_DATA
   STA (BUFRLO),Y
   INY
   bne rdlp


(Needless to say the routine then bumps the MSB of the pointer and reads the next 256 bytes). However, checking the status register every iteration wastes cycles. This produces errors:

RDLP   
   LDA I_DATA
   STA (BUFRLO),Y
   INY
   bne rdlp


However, it's more or less the same as Sijmen's code in his video streaming program. Any clues?



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Mon Sep 13, 2010 3:13 am 
User avatar

Joined: Sun Nov 02, 2003 8:15 am
Posts: 1323
Location: netherlands
This is the (fast)code I use in FDISK to read/write the bootsector.
In the OS there is a verify/retry logic that slows things down.

;***********************
;* HDROUT Y=CMD X=CFG  *
;***********************
HDROUT LDA #$01
    STA PORT+3
    LDA #$00
    STA PORT+4
    STA PORT+5
    LDA #$A0    ;CHS
    STA PORT+6
HDROU3 LDA #$01
    STA PORT+2
    STX PORT+1  ;CONFIG
    STY PORT+7  ;COMMAND
HDROU1 INC BUF+126
    BNE HDROU1  ;DELAY
    LDY $14
    DEY
HDROU2 CPY $14
    BEQ HDROUE  ;TIMEOUT
    LDA PORT+7
    BMI HDROU2  ;NOT READY
HDROUE LDA PORT+1 ;ERROR
    RTS

;*****************************
;* FPARTR                     *
;*****************************
FPARTR LDY #$21
    JSR HDROUT
    BNE FPARTE
    LDX #$00
FPARTR1 LDA PORT
    STA BUF,X
    INX
    BNE FPARTR1
FPARTB RTS

;*****************************
;* FPARTW                     *
;*****************************
FPARTW LDY #$31
    JSR HDROUT
    BNE FPARTE
    LDX #$00
FPARTW1 LDA BUF,X
    STA PORT
    INX
    BNE FPARTW1
    RTS


Later,
Sijmen.


Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Mon Sep 13, 2010 6:28 am 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
Thanks for that. It's identical to the fast code I was using, but I can't get reliable reads/writes without extra code inside the loop (before reading/writing the data register) to either wait for DRQ to set or drive busy to clear. Does this suggest a problem with the hardware? It's 100% reliable if I wait for drive busy to clear before reading the next byte from the drive. Kyle was doing the same thing in his driver code so I assume there was a reason for that (perhaps he was also having problems?). It's still quick, but it would be a lot quicker without testing the status every iteration.

It should also be noted that I'm using 8-bit mode and 512 byte sectors. I didn't test it with 256 byte sectors yet. Could this be making a difference?



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Mon Sep 13, 2010 7:43 pm 

Joined: Sat Sep 13, 2003 12:21 am
Posts: 251
Could it be that the drive is actually NOT ready quick enough?

Hmmm, forget it... looking around at code others have written for embedded IDE applications, it looks like they are sending the command, waiting for DRQ and then a quick loop to copy each byte to a buffer-with no status register checks inbetween each byte. Should work... ?



_________________
MyIDE Tools
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Tue Sep 14, 2010 12:58 am 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
I thought it was a sync problem, but surely reading the data register automatically readies the next byte. It would indeed seem the mechanism isn't fast enough to keep up with the loop in this case. My fastest "safe" code takes the shortcut of waiting for the drive busy bit to clear (rather than checking DRQ), and it works 100% of the time.

Could be a card issue? But damn - I only have one CF card which works with this. Maybe I'll back track and try it with 256 byte sectors on the HDD I was using before I got the CF card working. I have another CF card in the post...



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Tue Sep 14, 2010 10:14 am 
User avatar

Joined: Sun Nov 02, 2003 8:15 am
Posts: 1323
Location: netherlands
When you get a DRQ and read the corresponding 512/256 bytes, the DRQ should be gone.
Also the DRQ should not go before read ends.
If so, there is a problem with stability (I.E. compatibility).

That's why the OS has retry logic...... ;-)

Most of the time, the setting/resetting busy signal is so fast the atari does not see it.
So when you go for speed, just check if the drive is ready after the command and go for it.
Check DRQ afterwards and the ERRor-register before and after.

success,
Sijmen.


Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Tue Sep 14, 2010 12:25 pm 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
Well, it certainly seemed unlikely that the Atari could possibly outrun the data stream from the IDE device. And yet that's what appears to happen in this case: waiting for the "Drive Busy" bit to clear before reading a byte from the register works perfectly. Since the "safe" code checks drive busy before reading every byte (or DRQ in the previous version of the driver) and never "hangs", the relevant flags are clearly changing when each data byte is read. Tests suggest that when reading 20 x 512 byte sectors, the loop has to wait for busy to clear or DRQ to become ready between 1 and 2 per cent of the time (this is an average: sometimes as few as 10 in 10,000 bytes are misread, sometimes whole sectors have errors).

If the error flag is set by the "bad" reads (which I'm not absolutely sure it will be, since the hardware probably thinks the bytes were all read successfully), it would be useful: one could simply re-read (or write) the sector. I'll program in some retry logic if the error flag works as desired.

OK: I added error checking and DRQ checks after the read, and the error flag is not set after bad reads. Nor is DRQ still set, so no way to force a retry.

A good illustration of what's happening:

Attachment:
MyIDEerrors.jpg


It appears the Atari is "missing" characters in the input stream (this is a known good FMS: no writes have been performed with test drivers on this volume). You can see that characters in the filenames have been displaced by 1.


There are images and/or files attached to this posting. You must be logged-in to view or download this content.



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Tue Sep 14, 2010 7:46 pm 

Joined: Sat Sep 13, 2003 12:21 am
Posts: 251
FJC, I'll test your driver tonight... with my hardware. Any particular test that you want me to perform?



_________________
MyIDE Tools
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Wed Sep 15, 2010 3:49 am 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
Thanks Shawn: see AA message. :)



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Wed Sep 15, 2010 10:36 am 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
I've fixed this issue. Bob Puff's stability mod was the cause of the problem, and Mathy's stability mod the solution:

http://www.mathyvannisselroy.nl/blackbox.htm

Should have a finished driver and FDISK program for SDX shortly.



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Wed Sep 15, 2010 3:59 pm 

Joined: Sun Feb 19, 2006 1:33 pm
Posts: 89
Absolutely cannot wait for this!


Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Thu Sep 16, 2010 1:24 am 
User avatar

Joined: Sun Nov 02, 2003 8:15 am
Posts: 1323
Location: netherlands
Nice, can't wait either.

The problem is incorrect D1xx or D5xx signals to the drive.
Data is send by the IDE while the Atari doesnot ask for it, or the drive thinks the atari has send etc.
That's why the OS has retry logic to intercept badly designed atari's.

To be on the safe side, your SDX-drivers should include retry logic aswel.....
After settingup the registers, check the written values.
After a reads, do a second one for verify.
After write I do a read to verify.
Check the error-register after the command and after the job.
Check DRQ is present and does not go during transfer.

Succes and later!
Sijmen.


Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Thu Sep 16, 2010 4:58 am 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
mr-atari wrote:
Nice, can't wait either.
The problem is incorrect D1xx or D5xx signals to the drive.
Data is send by the IDE while the Atari doesnot ask for it, or the drive thinks the atari has send etc.
That's why the OS has retry logic to intercept badly designed atari's.

To be on the safe side, your SDX-drivers should include retry logic aswel.....
After settingup the registers, check the written values.
After a reads, do a second one for verify.
After write I do a read to verify.
Check the error-register after the command and after the job.
Check DRQ is present and does not go during transfer.

All these checks are what cripple the I/O speed, but I fully understand the need for them: to make hardware and software which is guaranteed to work with so many different machines is difficult. And even now, I still have only one CF card out of four which works. Only when it became evident that the IDE device was experiencing a read when the Atari hadn't issued one did I suspect timing was the root cause of the problem.

The driver will have to operate in two modes, or perhaps be compiled in two different versions: one with extensive verify and retry logic, and one without. The "fast" driver works for me now because the Atari has been properly stabilized. It works the way an IDE interface is SUPPOSED to work. Specifically, there should be no need to check DRQ going low during data transfer. Doing so takes the I/O speed from c. 62KB/s in SDX to c. 48KB/s.

The extensive retry logic in the MyIDE OS explains, then, why under DOS, I was experiencing transfer speeds of c. 12KB/s. Mathy's Phi2 mod should be the recommended one for Ataris using MyIDE.

It should also be remembered that the errors I experienced couldn't be detected solely by error checking after the read operation. Hence we'll have an option in the driver to do the "slow" I/O, and super-fast I/O for those users who know or can establish that their Atari has a stable and compatible Phi2 signal.

I believe flexibility is the way to go. Some experimentation will be required from more experienced users to get the best performance out of the driver, i.e.:

Attachment:
MyIDE_fastrw.jpg


There are images and/or files attached to this posting. You must be logged-in to view or download this content.



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Thu Sep 16, 2010 8:45 am 

Joined: Wed Mar 04, 2009 7:27 am
Posts: 98
Location: United Kingdom
Sijmen and/or others: any thoughts or recommendations for an LBA partition table?

We have the GUID partition table description here http://en.wikipedia.org/wiki/GUID_Partition_Table. And we have the MBR format here: http://en.wikipedia.org/wiki/Master_Boot_Record

Both possibly contain a lot of information superfluous to our scheme. With support for 256 byte sectors, direct access (via a driver) on the PC is out of the question (although we can still have a stand-alone utility to do the job). In any case, the upper half of a 512 byte sector is hardly a useful place to store partition table information.

Here's a suggested scheme:

Suggested SDX LBA partition table (LBA block 0):

0-7: signature(?)
8-11: first addressable LBA block for partitions (1)
12-15: last addressable LBA block for partitions
16: number of partitions
17-18: drive bits (?) (support for > 8 drives under SpartaDOS X)
19: option bits:
bit: function:
0: access mode (fast/careful)
1-7: reserved

20-31: reserved (?)

starting at offset 32, we have the tables for each partition, arranged thus:

0-3: LBA start (32 bit)
4-7: # sectors (32 bit)
8-9: sector size (128 - 2048)
10: drive number (do we need this if we have drive bits, or vice versa?)
11: active flag
12: bootable flag

An immediate improvement is to do away with or shorten the "signature" bytes at the start of the sector. This would allow the header record to fit into 16 bytes. In any case, all the subsequent tables will occupy 16 bytes each, yielding a total of 15 partitions with the shorter header record. Since SDX can support many drives, this seems reasonable.

Note the redundancy of having drive bits in the header record and drive numbers in the partition entries. Consider also that the partition table will be read into RAM on boot and stored for the duration (subject to changes by FDISK, when it will be re-written), hence it may occupy 1 page of RAM. Consider also the possibility of a back-up partition table (in LBA 1?).



_________________
The Last Word, MA65 Assembler, RIF SpartaDOS Utilities, XEDIT
Offline
 Profile  
 
 Post subject: Re: Fast Reads - how?
PostPosted: Thu Sep 16, 2010 10:59 am 

Joined: Mon Jan 23, 2006 10:49 am
Posts: 187
Location: Salzburg, Austria
Hi Jon!

flashjazzcat wrote:
We have the GUID partition table description here http://en.wikipedia.org/wiki/GUID_Partition_Table. And we have the MBR format here: http://en.wikipedia.org/wiki/Master_Boot_Record

Both possibly contain a lot of information superfluous to our scheme.

There were times when I also checked these infos, but I wouldn't bother thinking about these 2 schemes anymore. GUID is nice, but far too complicated, and MBR is a real PITA. So I'd say we should just create our own, simple scheme.

I dug into my mailbox and read through the discussion I had with Sijmen in November 2008, concerning LBA support for the 5.1 MyIDE OS. Here's the (hopefully) last scheme we had and IIRC this is also implemented in my myidetool (not in the released versions, though).
Quote:
Partition table (10x8 bytes):

1x8 bytes for drive settings.
a1 a2 a3 a4 : size in LBA sectors
a5 : $51 '5.1' identification (IF>8 is new format, changed matthias)
a6 : Drive bits
a7 : Optionbyte
a8 : free
(reference to previous CHS: CL CH HD SC Dn DB IM IS)

1x8 bytes for image-setting.
b1 b2 b3 b4 : Start of Image location. LBA Sector L, M, H, U byte
=$00000000 then images are disabled/not set.
=$value, this limits the extension of logical drives.
b5 b6 : Numbers of Images L, H.
b7 b8 : free

8x8 bytes for drives.
c1 c2 c3 c4 c5 c6 c7 c8
c1 : density+drive# ($0x,SD+MD $2x,DD $8x,QD / DRIVES $x1-$x8)
c2 c3 c4 c5 : Partition location. LBA Sector L, M, H, U byte
c6 c7 c8 : Partition size L, M, H.
All sectors #0 are reserved.

The first 4 bytes (a1-a4) contain the total number of sectors of the drive (it's good to have this info because it can't be established reliably when a drive is connected to an USB adapter).

Byte 5 (offset 4) is used as a "version signature". Current MyIDE OS stores the number of partitions in it (0-8), we set it to $51 to distinguish between old and new layout (that's important for myidetool, I want to support both schemes in a single program).

The (optional) image space can hold a maximum of "number of images" (b5,b6) image-slots of 1041 sectors each.

The 8 partition blocks contain a combined density (128, 256, 512 bytes/sector) and drive-number (1-15) byte, together with 32 bit starting sector and 24 bit size info.

Both partitions and images contain a reserved sector 0 (this can be used to store image names and other stuff) which is usually not accessible through SIO. Therefore a 1040 sectors image uses 1041 sectors on disk, the same goes for partitions.

The basic layout on disk is as before (from low to high sectors):
partition table
partitions
image space
free space (for example for movies etc).

This was our idea back then, more or less just as a reference, it doesn't mean I want to keep it exactly this way :-)

There are only a few things I'd really like to have in the new layout:
- most important: a way to distinguish between old and new partition-table layout (for example through byte 5 / offset 4)
- the total number of sectors of the disk
- a way to partition the disk into partition space / optional image space /optional free space

I also thought about adding support for more than 8-15 partitions (like the Black Box), so that the "image space" could be eliminated. But this would mean that we'd also need some new software to select/configure partitions and this software would never fit into the Atari OS. So it's better to keep the "image space" for people who like to store tons of disk images on their drive.

Now some thoughts about your suggestions:
Quote:
8-11: first addressable LBA block for partitions (1)
12-15: last addressable LBA block for partitions

Good idea!

As for image space: we could either add another 8 byte block (start/end), or combine this info into 3 4-byte blocks (as image_start would usually be partition_end+1 and thus be redundant).

Quote:
16: number of partitions

I guess we can remove this info.

Quote:
17-18: drive bits (?) (support for > 8 drives under SpartaDOS X)

Good question. Basically I'd say this info is redundant as it can be created from a drivenumber/active flag in the partition entries.

Quote:
19: option bits:
bit: function:
0: access mode (fast/careful)
1-7: reserved

Good question, again. Sijmen?

Quote:
starting at offset 32, we have the tables for each partition, arranged thus:

0-3: LBA start (32 bit)
4-7: # sectors (32 bit)
8-9: sector size (128 - 2048)

32bit sector numbers is a good idea (I guess we just used 24 back then because we wanted to fit all info into an 8 byte block).
But I'd just use a single byte for the sector size (for example meaning 2^x).

Quote:
10: drive number (do we need this if we have drive bits, or vice versa?)
11: active flag
12: bootable flag

We could combine this info into 1 byte, but since each entry will be 16 bytes (I'd strongly vote for aligning these structures at 2^x) we just might keep them separately.

On question though: what should the bootable flag be used for? I'd just skip it, even in the MBR/PC world only the (Win)DOS boot code uses this flag, all other bootloaders (like grub, lilo etc) ignore it and boot whatever partition you want.

A better alternative for selecting the boot-partition would be to store the boot-partition-number (1-15 or whatever) in the "main" block.

Quote:
An immediate improvement is to do away with or shorten the "signature" bytes at the start of the sector. This would allow the header record to fit into 16 bytes. In any case, all the subsequent tables will occupy 16 bytes each, yielding a total of 15 partitions with the shorter header record. Since SDX can support many drives, this seems reasonable.

With the addition of the image space stuff the header will be 32 bytes, but still 14 partitions is better than only 8 partitions before.

Quote:
Note the redundancy of having drive bits in the header record and drive numbers in the partition entries.

Yup, we should eliminate this redundancy :-)
Quote:
Consider also that the partition table will be read into RAM on boot and stored for the duration (subject to changes by FDISK, when it will be re-written), hence it may occupy 1 page of RAM.

Not necessarily. At the cost of more program code space you can eliminate the buffer and interpret the data while you read the bytes from disk. Only when changing the partition table you'll need the full buffer, of course, but FDISK also needs some RAM, so this is no big deal IMO.

Quote:
Consider also the possibility of a back-up partition table (in LBA 1?).

This is also a good idea!

Jon, Sijmen, any ideas/suggestions/... ?

so long,

Hias


Offline
 Profile  
 
Display posts from previous:  Sort by  
 Page 1 of 4 [ 54 posts ]  Go to page 1, 2, 3, 4  Next


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  

cron