TidBITS#82/System_Heap ====================== Copyright 1990-1992 Adam & Tonya Engst. Non-profit, non-commercial publications may reprint articles if full credit is given. Other publications please contact us. We do not guarantee the accuracy of articles. Publication, product, and company names may be registered trademarks of their companies. Disk subscriptions and back issues are available. For more information send electronic mail to info@tidbits.uucp or Internet: ace@tidbits.uucp -- CIS: 72511,306 -- AOL: Adam Engst TidBITS -- 9301 Avondale Rd. NE Q1096 -- Redmond, WA 98052 USA ----------------------------------------------------------------- Topics: INIT Introduction Diving In Onto INITs! The Killer INIT Little Known Facts Summing It Up INIT Introduction ----------------- INITs, the System Heap, and You by Andrew Welch This document came about because of the System Heap/INIT paranoia I've seen lately. It represents the total sum of the knowledge I have accumulated from writing system level software (INITs/cdevs/System Extensions) for three years and is accurate based on my experience and the experiences of many other Macintosh programmers. It will help you make rational decisions about resolving INIT conflicts, dispel some common myths, teach you a thing or two, and explain what really happens with all this System Heap stuff. Neophytes, forgive me if I confuse you with the programmer-speak necessary to explain this topic. Programmers, forgive me if I generalize issues for the sake of clarity. INITs, cdevs, Control Panels, System Extensions - these buzz-words stand for basically the same critters: low-level programs that load automatically when the system starts up. These are the programs that you toss into your System Folder. From now on, I will refer to them all as INITs, though there are some subtle differences between them that aren't important in this discussion. INITs are notorious for causing system crashes, conflicts, and general system weirdness. When technical support people hear that an INIT is causing problems, you'll inevitably hear something to the effect of "Have you tried expanding your System Heap?" What do they mean when they say this, and is it the ultimate solution? You may learn more about the way your computer works than you want to, but if you persevere you won't regret it. Knowledge is power. Hang on tight. But first, a little background... Diving In --------- For starters, your Macintosh has a fixed amount of memory installed in it, which acts like desk space for things you are currently working on. In real life, when you want to work on something you might take it out of your filing cabinet and put it on your desk where you can work on it effectively. This is similar to what a computer does: all of your programs and documents are stored on your hard disk, and when you want to work on them they are read in from disk and placed in memory. Life being what it is, you can't have everything. You only have a limited amount of memory that must be rationed out to the programs that you run. In fact, each program you run grabs a fixed chunk of memory when it loads, and it doesn't let go of the memory until you quit the program. This is the application's world to do with as it pleases. This is also known in technoweenie terms as a heap. Each application is said to have its own memory heap. You can set the size of this heap (how much memory the application will use, or how much of your desk space you're giving up to use the program). You can do this for any program by choosing "Get Info" for it in the Finder and entering how much memory you want it to use. That assumes you are running under System 7 or System 6's MultiFinder. Under older systems (sometimes called UniFinder) the programs that you run use all of your Macintosh's memory, so you can only run one program at a time. Well actually I just lied. Sorry about that, I realize I hardly know you and I've lied to you already. Your Macintosh has another heap that is always around (even if you aren't running any programs) called the System Heap. This is important. The System Heap is a chunk of memory that the system software (the nitty gritty, low level stuff) uses for its own purposes. Things with nasty names like device drivers, ROM patch code, etc. dwell there. It is a murky, convoluted place of which only Apple knows every nook and cranny, but a very important one nevertheless. So, a portion of your computer's memory (or RAM) is always occupied by the System Heap, an area of memory designated for the low-level system software from Apple and other nasties. The rest of your computer's memory is rationed off in chunks to the applications that you run on a first-come, first-serve basis. Normally, we developers stay in our own little application heaps, insulated from the complexities of system-level software. There are times however that the only possible way to write the programs we dream of is to delve into the world of system software and the System Heap. Yes friends, we are talking about INITs (you knew I'd get around to it some day, right?). Onto INITs! ----------- If a program wants to achieve some kind of a global effects (like QuicKeys allowing you to define macros that work in any program), it has to find out how to graft itself into your system and keep a portion of memory for itself that will stay around even when a program quits (remember folks, the memory a program allocates for itself is freed up again when it quits). We are talking sophisticated stuff here. Writing system software is extremely tricky stuff, and even for Apple who knows every little subtle nuance in the Macintosh, it is a tough thing to do right. That is why System 7 took so long and why Apple should be praised for System 7 - it works well, which is nothing short of a miracle. But I digress. You get the idea, writing system software is not for the faint of heart. In any case, INITs take a portion of System Heap memory for themselves when the computer starts up. It is also then that they "hook" themselves into the system software. When your computer first starts up, the System Heap is a certain fixed size. However, during the start up process, the System Heap grows automagically to accommodate an INIT that needs more memory for itself. For technoweenies, the mechanism that loads INITs at start up time and expands the System Heap as needed is called INIT31. There is a well known mechanism that any INIT writer worth his salt takes advantage of: 'sysz', or System Zone Expansion. This is a little parcel that all INIT writers should include in their products (it takes all of two minutes to put it in), because it tells the INIT31 mechanism how much memory the INIT needs to load. INIT31 grows the System Heap so that there is at least this much memory free, and then loads and runs the INIT. In this way, any well- written INIT will tell INIT31 how much memory it needs, get it, and be happy. All is well in INIT-land. In a perfect world, our story would end here. But as I'm sure you suspect, there are a few twists. INITs have the power to affect the operation of your entire computer, and unfortunately they can also crash the whole thing too. INITs can run into several different snags regarding the System Heap, and I will go into them one by one so that you can understand what goes wrong and why. One thing we should get straight now - if your machine crashes because of an INIT, a programmer out there somewhere is at fault. Count on it. There are well established rules for writing INITs, so the real problem is that not all programmers understand the rules. Information on how to write INITs properly is also hard to come by, and programmers are only human - we're doing our best but we make mistakes. The Killer INIT --------------- When an INIT causes a problem, you'll hear ten people shout in unison, "Have you increased the size of your System Heap?" There are utilities out there that let you manually make the System Heap bigger in an effort to fix crashes due to INITs. However this isn't much of a solution. In some cases, increasing the size of your System Heap by 20K or so can be beneficial, because it gives the System Heap a little breathing room. If you still experience problems, increasing the System Heap more probably won't help you. It is kind of like this: You are in a room with a sadistic murder wielding a nasty looking knife, who definitely doesn't mean well for you. The bigger the room, the better chance you have to escape from him. Heck, if he is out of shape and the room is big enough, he may never catch you. But one day when you least expect it... The same goes for the System Heap size method of "curing" crashes. Here are the common causes of problems INITs can run into, and what you can do about them: PROBLEM #1 Many INITs are simply badly written. No amount of fudging will help you avoid crashes from a badly written INIT, or two INITs that don't get along together. True, increasing the size of your System Heap may delay the inevitable. For instance, some INITs don't even bother to check if they actually have enough memory to do what they want, and increasing your System Heap a little (20K or so) should help this. But if an INIT is badly written, there may be nothing you can do about it. Increasing your System Heap memory to fix a blatant programming error is just a waste of memory. PROBLEM #2 Many INITs don't bother to have 'sysz's, thus the INIT31 mechanism has no way of knowing how much memory to give to the INIT. In this case, increasing the size of your System Heap DOES INDEED cure! However the 'sysz' method has been documented by Apple for years, and I'd be suspicious of an INIT that doesn't take advantage of 'sysz' anyway. PROBLEM #3 To understand this problem, I'll need to tell you a bit more about heaps. Heaps are called heaps for good reason. Imagine a big laundry basket into which you throw all your clothes. Heaps work in a similar manner. Fortunately for programmers, a maid called Memory Manager takes care of this mess. I call her Martha (just one of those things that keeps you sane when dealing with something that is NEVER wrong). When you want to put something in a heap, you ask Martha for the space for it, and she gives it to you if she can. Like all good maids, Martha knows that there is less space in a messy heap. So if she can't give you what you ask for immediately, she takes everything in the heap and cleans it up a bit to free up more room. This is a nice thing, but we all know the problem of coming into your room after someone else cleaned it - you can never find anything! Many INITs have this problem as well, and it is absolutely their fault, not Martha's. She can tell you where anything is, but some INITs don't bother to ask, they simply assume they know the location of their little chunk of memory and **crash**. There is one other twist to this situation as well. If Martha cleans everything up, and there still isn't enough room to satisfy a request for memory, she starts throwing things out. But it again isn't her fault, these things are specifically marked "Throw me out if you need the space." If an INIT doesn't take this into account, **crash**. Since the System Heap is shared by system software and all of your INITs, it is particularly active. However the whole scheme can and should work. Quite well in fact. In this situation, expanding your System Heap can delay crashes, because if Martha never has to move things or throw things out, badly written software will work most of the time. Realize though that you are sacrificing your precious memory to correct a defect in the software for which you paid good money. PROBLEM #4 This is a tricky one that many programmers don't understand fully. Earlier I described the 'sysz' method for telling INIT31 how much memory an INIT needs to loads. It works as advertised, but there is a rub. The System Heap is shared memory, it is one big basket into which many hens put their eggs. Lets say we have an INIT called "Longcut" that has a 'sysz' specifying that it needs 50K of memory. INIT31 will free up 50K of memory, load the hypothetical "Longcut" INIT, and all should be well. However, let's say "Longcut" doesn't use all 50K of the memory it asks for at start up time. Let's say it uses 10K of memory at start up time to install itself, reserving the rest for later on when it actually works its magic. This is actually a very common situation. Logic would tell you that there should be 40K of memory left over that "Longcut" can count on being there, right? WRONG! As I said before, the System Heap is shared memory. If an INIT asks for more memory than it actually uses at start up time, whatever it doesn't use isn't reserved for it automatically; instead it is thrown "back in the pot," and can be used up by anyone. So later on when "Longcut" thinks it has room to spare, it actually might not and then, **crash**. Of course, some programmers know ways to properly program around this situation, but this particular situation is not well known. In this case, increasing System Heap memory can be beneficial as well, but only in a limited sense. Only the programmers know how much memory their INITs really need, and they should be the ones to fix the problem. Little Known Facts ------------------ Many people do not realize this, but with all versions of MultiFinder, the System Heap can actually grow even _after_ start up time, easing INIT memory conflicts. If an INIT asks for more memory than is available, the System Heap will automagically get bigger. However there are two bad things about this: 1) the System Heap will grow, but never shrink again and 2) Several well respected Macintosh programmers have stated that they've seen cases where MultiFinder CANNOT expand the System Heap automatically, and die in the process. This is apparently extremely rare however. This makes claims of "you've got to increase your System Heap" even less powerful, because the system will increase it for you if need be. System 7 changes the game entirely, the System Heap can grow and shrink at will to accommodate requests for memory, eliminating many INIT memory problems entirely. However System 7 threw INIT writers several other curves, so you'll most likely need to upgrade your favorite INITs for use under System 7.0. You shouldn't have to increase your System Heap manually for System 7.0, but you never know... an extra 20K or so never hurt anyone. Summing It Up ------------- Properly written INITs shouldn't exhibit any of the above stated problems. But if your Macintosh needs a little tweaking to get it working smoothly again, there is nothing wrong to giving a little more memory to the System Heap. In no case should you ever have to allocate more than 50K of extra memory to your System Heap. After a point, no amount of fudging will save you. Try to identify the culprit INIT that causes problems, and trash it. Or better yet, make sure you have the latest version of every INIT you use. Realize that INIT crashes/conflicts _do_not_ stem from lack of System Heap memory, but from improperly written software. Giving the System Heap extra memory may circumvent some problems, but it is not the source of them. When you choose "About the Finder" from the Apple Menu, there should be a little bit of white space (free memory) in the "System" memory bar. 5-10% or so should be fine. With that in mind, here are a few recommendations. INITs are particularly sensitive pieces of code, so expect them to be upgraded more frequently then other software products, and keep up with the current versions. Your Macintosh will be more stable if you use the current versions of the INITs you use, and if you use INITs only from well known and established vendors who support their products. Taking these precautions will result in a far more stable Macintosh than any amount of System Heap tweaking ever could. 20K - 50K of extra System Heap space (free System Heap memory), OK. That is a nice margin of error. More than that? Forget it, tell them to fix it! If you are interested in increasing the size of your System Heap, here are three free programs that manually adjust System Heap size: Heap Fixer BootMan Heap Tool Contact the author at: Andrew Welch Mark 3 Software 29 Grey Rocks Road Wilton, CT 06897 ...and if you include a disk and a self-addressed, stamped mailer, I will send you the latest versions of my shareware programs! .. This text is encoded in the setext format. Please send email to or contact us at one of the above addresses to learn how to get more information on the setext format.