ATARI HDD low level info

GFA, ASM, STOS, ...

Moderators: simonsunnyboy, Mug UK, Zorro 2, Moderator Team

User avatar
leonard
Moderator
Moderator
Posts: 660
Joined: Thu May 23, 2002 10:48 pm
Contact:

ATARI HDD low level info

Postby leonard » Sat Dec 26, 2015 1:15 pm

Hi,

I'm thinking about an HDD version of "We Were @" demo. I want to do exactly the same trick I did for the HDD version of my Amiga Demo 2: keep the two MSA files, and put a small EXE you can run from HDD. In AMIGA Demo 2 I used upper memory to cache the complete MSA packed file, but in We were I can't do that because I already use upper memory just to cache "unpacked" floppy data. ( I keep it unpacked to keep a fast flow during the demo, avoiding slow depacking time)

I never had HDD myself so I'm looking for good docs, driver source code, about:
-HDD low level sector loading ( how to load sector #n without using driver or TOS )
-HDD FAT handling ( I have to know the exact sector list of the two MSA files on the HDD)

Anyone have info regarding that stuff?

thanks
Leonard/OXYGENE.

User avatar
Mikefulton
Captain Atari
Captain Atari
Posts: 169
Joined: Sun Nov 29, 2015 10:27 am

Re: ATARI HDD low level info

Postby Mikefulton » Sat Dec 26, 2015 1:58 pm

It's unclear why you think you need to read raw sectors rather than using file system.

And assuming there was a good reason for that, It's additionally unclear why you'd want to bypass the bios to do it.

User avatar
leonard
Moderator
Moderator
Posts: 660
Joined: Thu May 23, 2002 10:48 pm
Contact:

Re: ATARI HDD low level info

Postby leonard » Sat Dec 26, 2015 2:02 pm

oh to be precise, "We Were @" could run on 2MB machine with music, but need 4MB only to cache unpacked floppy data in order to keep the demo flow. The kernel use low memory, and it's not compatible with system.

But GGN gives me an idea I could use: backup low memory in upper memory so I can use system driver ( ULS system developped by Cyrano Jones ). I think I can use that kind of trick, avoiding writing low level stuff. I'll check that, could be fun to write!
Leonard/OXYGENE.

User avatar
exxos
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 4933
Joined: Fri Mar 28, 2003 8:36 pm
Location: England
Contact:

Re: ATARI HDD low level info

Postby exxos » Sat Dec 26, 2015 4:07 pm

I'm not sure I follow exactly, but have you tried PP's HD driver ? It uses very low memory, assuming its memory usage you are trying to conserve.
4MB STFM 1.44 FD- VELOCE+ 020 STE - Falcon 030 CT60 - Atari 2600 - Atari 7800 - Gigafile - SD Floppy Emulator - PeST - various clutter

http://www.exxoshost.co.uk/atari/ All my hardware guides - mods - games - STOS
http://www.exxoshost.co.uk/atari/last/storenew/ - All my hardware mods for sale - Please help support by making a purchase.
http://ataristeven.exxoshost.co.uk/Steem.htm Latest Steem Emulator

User avatar
Mikefulton
Captain Atari
Captain Atari
Posts: 169
Joined: Sun Nov 29, 2015 10:27 am

Re: ATARI HDD low level info

Postby Mikefulton » Sun Dec 27, 2015 12:04 am

leonard wrote:The kernel use low memory, and it's not compatible with system.


So, this demo uses some kernel that dumps the OS and then uses RAM formerly used by the OS.

What does this kernel do that couldn't have been done using TOS?

User avatar
alien
Atari maniac
Atari maniac
Posts: 97
Joined: Sat May 01, 2004 4:01 am
Location: USA
Contact:

Re: ATARI HDD low level info

Postby alien » Sun Dec 27, 2015 1:04 am

Demos often use all of RAM. TOS expects part of that RAM to contain its variables. Therefore TOS isn't any use. Also TOS runs its own interrupts that take up too much CPU or occur at inopportune times.

Instead of using TOS, demos often use custom floppy loaders. It's not an OS, just a barebones floppy disk driver that loads the demo to RAM, unpacks it, and runs it. To run this type of demo from a harddrive, a custom driver is often needed to access the harddrive.
Alien / ST-Connexion

User avatar
Mikefulton
Captain Atari
Captain Atari
Posts: 169
Joined: Sun Nov 29, 2015 10:27 am

Re: ATARI HDD low level info

Postby Mikefulton » Sun Dec 27, 2015 1:20 am

alien wrote:Demos often use all of RAM. TOS expects part of that RAM to contain its variables. Therefore TOS isn't any use. Also TOS runs its own interrupts that take up too much CPU or occur at inopportune times.

Instead of using TOS, demos often use custom floppy loaders. It's not an OS, just a barebones floppy disk driver that loads the demo to RAM, unpacks it, and runs it. To run this type of demo from a harddrive, a custom driver is often needed to access the harddrive.


None of this is news, but my experience is that there's often a big gap between what's actually necessary to produce a desired result, and what people THINK is necessary.

User avatar
alien
Atari maniac
Atari maniac
Posts: 97
Joined: Sat May 01, 2004 4:01 am
Location: USA
Contact:

Re: ATARI HDD low level info

Postby alien » Sun Dec 27, 2015 3:32 am

Mikefulton wrote:None of this is news, but my experience is that there's often a big gap between what's actually necessary to produce a desired result, and what people THINK is necessary.


I'm sorry, I thought you were genuinely interested. I'm quite confident Leonard can code.
Alien / ST-Connexion

User avatar
Mikefulton
Captain Atari
Captain Atari
Posts: 169
Joined: Sun Nov 29, 2015 10:27 am

Re: ATARI HDD low level info

Postby Mikefulton » Sun Dec 27, 2015 7:30 am

alien wrote:
Mikefulton wrote:None of this is news, but my experience is that there's often a big gap between what's actually necessary to produce a desired result, and what people THINK is necessary.


I'm sorry, I thought you were genuinely interested. I'm quite confident Leonard can code.


I am genuinely interested. When I said it wasn't news, what I meant was, I am aware of a wide variety of reasons why someone might want to dump out the OS, including those you mentioned. However, in this particular case, there's been no clear indication as to which of those reasons might be applicable.

And I have no reason to think that Leonard isn't a good programmer. But even the best programmers sometimes overlook simple solutions to a problem.

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2978
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: ATARI HDD low level info

Postby AtariZoll » Sun Dec 27, 2015 9:30 am

leonard wrote:...
-HDD low level sector loading ( how to load sector #n without using driver or TOS )
-HDD FAT handling ( I have to know the exact sector list of the two MSA files on the HDD)
Anyone have info regarding that stuff?
thanks

You need in any case some kind of hard disk driver. It may work independent from TOS. Superiour's FFLS was something like that, but it is very old, and only ACSI, max 32MB partitions. Dealing with HDD FAT is just extra complication, and I have solutions for simple RAW low level disk access for ACSI and IDE adapters where can very fast load desired disk area. User needs to defragment partition, and than access will be OK. Forcing access to fragmented files needs FAT handling too, but it means lower speed too - and in my cases speed is essential, so I never went into that way. First step in this is to find exact loc. of file on disk, then accessing that area with simple raw access driver when low RAM and TOS is not usable. Fastest way.

Other way is to use RAM swap technique - like in WHDload, ULS and mine HAGA - what has advantage that works with any hard disk driver. But needs some RAM to save low memory - TOS workspace, hard disk driver+cache + desktop workspace. It is about 150-600 KB depending from TOS version, driver, it's cache sizes ... You call function for saving system area in top RAM. then start your demo. When disk access is needed you just give parameters as file name, destination where to load, offset in file and length, and HAGA will load it. It takes some time because of necessary RAM area swaps, so not good for many short disk accessings. Therefore there is cache to make it faster. But if you want to load for instance one complete MSA image at once, you don't need cache. Some more about is here: http://atari.8bitchip.info/hagaguide.php .
Famous Schrodinger's cat hypothetical experiment says that cat is dead or alive until we open box and see condition of poor animal, which deserved better logic. Cat is always in some certain state - regardless from is observer able or not to see what the state is.

User avatar
leonard
Moderator
Moderator
Posts: 660
Joined: Thu May 23, 2002 10:48 pm
Contact:

Re: ATARI HDD low level info

Postby leonard » Sun Dec 27, 2015 11:58 am

I think I can go like that: GGN told me it's safe to keep using GEMDOS file system if I use only memory at $10000 and more. So I will review the memory layout of the demo,I have plenty of room in the 4MB version. I think I'll keep $8 to $10000 for system. My kernel will go from $1000 to top of memory, and the screens will be run at $10000 instead of $7800
Then I'm confident I can use GEMDOS fOpen, fSeek, fRead to load MSA files and keep using depacking routine while loading.

Anyone see a problem by using GEMDOS file system with all various HDD device if I keep memory from $8 to $10000? ( obviously I will restore system timer C before calling any GEMDOS func)
Leonard/OXYGENE.

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2978
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: ATARI HDD low level info

Postby AtariZoll » Sun Dec 27, 2015 1:02 pm

Some typical RAM addresses when running APP from Desktop, without any driver, ACC and resident SW loaded :

PRG:
TOS 1.00 : $1116E
TOS 1.04 : $12496
TOS 2.06 : $18772
TOS 4.02 (Falcon) : $33298 (Set to DE, ST low res.)

AUTO, same TOS order
$A204
$A956
$CDBA
$FB6C

So, even without hard disk driver you can not do it.

It is possible only if choose RAM above $20000 . And of course, I must say that it is still bad idea , because many people has much more low RAM allocated, as mentioned in my previous post here. So, you will get lot of messages that it works not. Trust me - I talk from my own experience.

Strange that ggn gives such bad advices.
Hard disk driver self usually uses not so much RAM - 10 KB is more than enough. But for today normal 512 MB partitions you need to install min. 32 KB buffers too - so min what one hard disk driver will consume is some 36 KB. And demo must be runnable from Desktop when wabt to run from hard drive.

Btw. Timer C makes problems only in TOS 2.06 . I can give you complete source file for launcher. Just give me what files you need to access during demo playback. You can keep RAM config as is, I guess. There are games setting screen to $200 - and work very well from hard disk.
RAM needed when HAGA launcher is used is: TOS, drivers backup - 150 KB with rational driver and TOS 2.06 . 32 KB HAGA library. About 2 KB for custom code accessing HAGA for file access. Then can simply exit to Desktop too. So, with 4 MB machine you can have some 3.8 MB for demo and optional disk cache. But better to calculate with bigger space for TOS+driver backup. Even if it is 800 KB, still can have 3 MB for demo ...
Famous Schrodinger's cat hypothetical experiment says that cat is dead or alive until we open box and see condition of poor animal, which deserved better logic. Cat is always in some certain state - regardless from is observer able or not to see what the state is.

User avatar
Mikefulton
Captain Atari
Captain Atari
Posts: 169
Joined: Sun Nov 29, 2015 10:27 am

Re: ATARI HDD low level info

Postby Mikefulton » Sun Dec 27, 2015 3:08 pm

You can get the start of the system TPA from the system variable membot at $432.l

Anything below the address contained there is being used by the system. Above that address is loaded programs and the Malloc pool.

Silly question, perhaps, but there's been no mention at all of what the actual memory requirements are. All the discussion has been regarding what memory you can step on, but what are the actual requirements?

User avatar
alien
Atari maniac
Atari maniac
Posts: 97
Joined: Sat May 01, 2004 4:01 am
Location: USA
Contact:

Re: ATARI HDD low level info

Postby alien » Sun Dec 27, 2015 3:38 pm

"Scheibenkleister II / Massenspeicher am ST" by Claus Brod and Anton Stepper is a book that has a lot of information about programming the harddrive. But it's in German. ISBN 3-927065-00-5

It does look like quite the rabbit-hole though. (a metaphor for an entry into the unknown, the disorientating or the mentally deranging)

A trick might be to ask GEMDOS to tell you the sequence of sectors you need to load (a small array), and then load them directly from disk with lowlevel HDD commands. I don't know whether GEMDOS provides that kind of information though...
Alien / ST-Connexion

User avatar
leonard
Moderator
Moderator
Posts: 660
Joined: Thu May 23, 2002 10:48 pm
Contact:

Re: ATARI HDD low level info

Postby leonard » Sun Dec 27, 2015 5:35 pm

@alien: this was exactly my first idea: geting the list of sectors of the two MSA files ( parsing FAT or using GEMDOS as you suggest but I don't know if GEMDOS offers such a functionality).
I'm pretty sure low level HDD cluster reading is not so complex.
All your message regarding HDD driver / cache memory footprint afraid me :)

@mikefulton: my memory need is 1MB for demo effects, 1MB for music, and the rest is used to precache large unpacked chunk of floppy data. I think I never use more than 1.5MB unpacked precaching.

So my best bet is to keep my current demo memory footprint. Then, when I want to load data from HDD, I "swap" say 256KB to top of memory ( say 4MB-256KB), then load, then swap again.
Leonard/OXYGENE.

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2978
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: ATARI HDD low level info

Postby AtariZoll » Sun Dec 27, 2015 6:17 pm

I really "enjoyed" this thread. Folks, just continue with wild guessing, and don't listen to someone who did it all already :mrgreen:
Famous Schrodinger's cat hypothetical experiment says that cat is dead or alive until we open box and see condition of poor animal, which deserved better logic. Cat is always in some certain state - regardless from is observer able or not to see what the state is.

User avatar
Cyrano Jones
Atari Super Hero
Atari Super Hero
Posts: 662
Joined: Wed May 28, 2003 8:28 pm

Re: ATARI HDD low level info

Postby Cyrano Jones » Mon Dec 28, 2015 1:58 am

Hey Leonard,

I'm around to help if you need it. E-mail me if you need anything, I don't log in here very often.

fenarinarsa
Atari freak
Atari freak
Posts: 51
Joined: Sat Mar 15, 2014 11:23 pm

Re: ATARI HDD low level info

Postby fenarinarsa » Tue Jan 05, 2016 12:05 am

Pardon my ignorance, but as far as I know by reading various documentations, if you do a low-level access to an HDD you need to support both the filesystem and the HW protocol used by the HDD.

Granted ACSI & FAT16/FAT32 is the most used on ST and I guess you can code your own driver for that, but you may hit a wall when your demo run from a non-standard mass storage device or FS. And I think now most ST users use some kind of HDD emulator based on SD card/network/emulated FS, and you also need to take into account that your demo may run from a floppy disk (high-capacity extended floppy formats are available through HW emulators like HxC).

So maybe you should consider using GEMDOS calls after all :)

User avatar
Cyrano Jones
Atari Super Hero
Atari Super Hero
Posts: 662
Joined: Wed May 28, 2003 8:28 pm

Re: ATARI HDD low level info

Postby Cyrano Jones » Tue Jan 05, 2016 4:17 am

ULS solved all that many years ago.....

Some people like reinventing the wheel.

User avatar
troed
Atari God
Atari God
Posts: 1450
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: ATARI HDD low level info

Postby troed » Tue Jan 05, 2016 4:50 am

Cyrano Jones wrote:ULS solved all that many years ago.....

Some people like reinventing the wheel.


To be fair, maybe people just haven't heard about it? Due to this thread I've now read http://dbug.kicks-ass.net/ulsdox/uls.php - but had no idea it existed until now.

/Troed

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2978
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: ATARI HDD low level info

Postby AtariZoll » Tue Jan 05, 2016 8:54 am

fenarinarsa wrote:Pardon my ignorance, but as far as I know by reading various documentations, if you do a low-level access to an HDD you need to support both the filesystem and the HW protocol used by the HDD.
Granted ACSI & FAT16/FAT32 is the most used on ST and I guess you can code your own driver for that, but you may hit a wall when your demo run from a non-standard mass storage device or FS. And I think now most ST users use some kind of HDD emulator based on SD card/network/emulated FS, and you also need to take into account that your demo may run from a floppy disk (high-capacity extended floppy formats are available through HW emulators like HxC).
So maybe you should consider using GEMDOS calls after all :)

You see very well the situation - unlike Leonard :D Now, it's easy to say that ULS solved all it years ago, but truth is that ULS solved it only partially, and there is not much help for SW using TOS calls. Or to paraphrase, what I did was not reinventing the wheel, but making complete wheel :mrgreen:
But instead another flame war let's see things in detail:
First solution for hard disk access from games when TOS calls are not possible was simple: RAMdisk. Then Superior made FFLS driver, somewhere in 1992. That was both: low level hard disk driver and FAT16 filesystem driver. Complete code was under 10 KB. But it was limited only to ACSI port and max 32 MB (non BIG GEM) partitions. So, now pretty useless.
Using GEMDOS calls when TOS workspace is destroyed by game/demo is possible in 2 ways: WHDload/ULS/HAGA system - what saves low RAM together with TOS workspace, hard disk driver to high RAM, and when SW performs disk access request it swaps low with high RAM, restores state needed for proper TOS work and does disk access with regular, installed driver + GEMDOS. After it is donem swaps again low and high RAM, restores state for game/demo and continues. Concept has advantage that works with any hard disk adapter, because uses installed driver for it. Bad side is that by disk access RAM area swaps take pretty much time, so multiple short accessings may be slow. That is overridden normally by using disk cache, RAMdisk.
What I described above is just disk access related.
But there is lot of SW using TOS calls, not only for disk access, and fails on machines with mass storage. Like fresh case R-Type Deluxe - which main executable runs at address $20000. That's too low, and usually destroys hard disk driver. But many games are set for even lower space, under $10000.
What can cause that they don't work even from floppy under TOS 2.06 for instance. Solution for such SW is TOS in high RAM. That is present in some Klapauzius patches and mine HAGA, Gamex. That is what makes SW using GEMDOS calls, but not truly relocatable to work well from hard disks.
For instance Flight Simulator 2 has regular and relocatable main executable file. But screen is tied to $78000. What results in crash when try to run it from hard drive, because code/data will go in screen space, and will be destroyed. Solution would be to mod it's code, and make screen address flexible. But it is simpler instead that to use HAGA, and can be solved in half hour.
Famous Schrodinger's cat hypothetical experiment says that cat is dead or alive until we open box and see condition of poor animal, which deserved better logic. Cat is always in some certain state - regardless from is observer able or not to see what the state is.

User avatar
Cyrano Jones
Atari Super Hero
Atari Super Hero
Posts: 662
Joined: Wed May 28, 2003 8:28 pm

Re: ATARI HDD low level info

Postby Cyrano Jones » Tue Jan 05, 2016 10:49 am

YAWN. Disassemble 'his' code and see where it came from.

I'm out of this thread, and this forum.

BYE.

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2978
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: ATARI HDD low level info

Postby AtariZoll » Tue Jan 05, 2016 10:57 am

No need to disassemble. Sources are published. Btw. proper disassembling of TOS 1.04 GEMDOS part took about 1 week.
Famous Schrodinger's cat hypothetical experiment says that cat is dead or alive until we open box and see condition of poor animal, which deserved better logic. Cat is always in some certain state - regardless from is observer able or not to see what the state is.

User avatar
Mikefulton
Captain Atari
Captain Atari
Posts: 169
Joined: Sun Nov 29, 2015 10:27 am

Re: ATARI HDD low level info

Postby Mikefulton » Tue Jan 05, 2016 12:47 pm

I can't help thinking that optimizing the demo itself to use less memory, so that you don't have to wipe out the OS to load it, is by far the more preferable solution to all the things that have been mentioned here.

Is there uncompressed data that can be compressed? You said there's 1mb of music data. What format is that? Is it compressed? Digitized audio data usually lends itself quite well to decompression on the fly.

What the heck does "precache large unpacked chunk of floppy data" really mean? I mean, clearly something's being loaded and perhaps decompressed, but what is "floppy data"? If this is really a "cache" then it should be able to use however much, or however little, memory is available. Is "cache" really the right word here?

Is there data that can be streamed in as needed instead of sitting around in memory all the time?

Clearly, you can do whatever you want, but for some reason these ideas don't seem to even be part of the discussion.

AtariZoll
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2978
Joined: Mon Feb 20, 2012 4:42 pm
Contact:

Re: ATARI HDD low level info

Postby AtariZoll » Tue Jan 05, 2016 2:48 pm

Folks, this is not simple matter. It took years to reach some level and get experience. And now some people just throws ideas without any experience.
This thread is waste of time. I explained many things, but all what see in return is insinuation about stealing code and that did not give some ideas.
Well Mikefulton: data packing is used in HAGA, in Gamex and of course even with simple Ramdisks, decades ago . Data can be "streamed' from hard disk. I gave link where can see examples. This is thread go give some help, but should respect those who worked on similar problem lot of time.
Famous Schrodinger's cat hypothetical experiment says that cat is dead or alive until we open box and see condition of poor animal, which deserved better logic. Cat is always in some certain state - regardless from is observer able or not to see what the state is.


Social Media

     

Return to “Coding”

Who is online

Users browsing this forum: No registered users and 11 guests