Obsolete the Dilemma!

It's all very well to enunciate lovely-sounding values like Joy, Growth and Choice ... but in real life we're all faced with difficult decisions. We're faced with choosing one being's joy over another's, or choosing joy versus growth in a given situation, and so forth.

There's no perfect, one-size-fits-all solution to such dilemmas.

But Cosmism does provide one valuable principle, that is very frequently appropriate for beings in the phase of evolution that humans currently occupy.

This is the principle of obsoleting the dilemma.

Rather than trying to resolve the dilemma, use a change in technology or perspective to redefine the reality within which the dilemma exists.

This may of course lead to new and different dilemmas -- which is a natural aspect of the universe's growth process.

This approach has tremendous power and we'll revisit it frequently in the following pages.

To make the idea clear, first of all I'll explore it in the context of a couple simple, everyday issues that -- in the human world right now -- seem to have a tremendous power to divide thoughtful, compassionate people.

Cosmism doesn't solve these issues -- but it does advocate a systematic route to resolving them ... not by solving them but rather by obsoleting them.

The Dilemma of Abortion

Abortion is one of the most obvious cases of a divisive ethical dilemma, in modern society.

Even among individuals who reject traditional religious notions of the human soul and the special sacredness of human life, it poses a huge ethical challenge.

On the one hand, compassion dictates that killing babies is wrong ... and the fact is that we don't really know when a fetus develops enough "reflective awareness" that killing it becomes more like killing a person than like killing a sheep, or more like killing a sheep than like killing a fish, etc.

On the other hand, forcing an adult human woman to create an infant when they don't wish to, is a clearly uncompassionate violation of that woman's personal choice and happiness, of her ability to grow in the directions she chooses.

So, there is a balance to be struck, and different caring, thoughtful people want to strike it in different ways.

The Dilemma of Vegetarianism

Vegetarianism presents a similar dilemma to abortion, though one that the mainstream of modern society seems less concerned about (due to our habitual species-centrism).

Clearly, killing animals to eat them is uncompassionate and, in itself, "wrong" according to principles of joy, growth and choice. Cows, pigs and chickens may not be as smart as we humans are -- but they have their own experiences, emotions and wills, and we're pretty damn nasty to abort these so we can have a tastier dinner.

Yes, nature is bloody and violent ... animals kill each other ... but we have the capability to much more fully understand what we're doing and to make a more considered choice....

Yet there are plenty of borderline cases. It's not clear, for instance, whether fish experience pain in the same way that birds and mammals do. It's not clear in what sense a fish has a theater of reflective consciousness. Personally, I don't feel confident that killing a fish is cutting off a stream of tremendous joy and experience, any more so than cutting down a tree or picking a carrot out of the ground.

One argument against vegetarianism is that we're evolved to be omnivores and some level of meat consumption is necessary for us to feel "natural." I personally do feel that way: in the past when I've eaten a vegetarian diet for a while, I have felt a certain disturbing lack of energy and aggressive initiative. Eating fish cures that for me just as well as eating other meat ... but without some fish or other meat intake, I really don't feel like "myself." So the question becomes: how much cruelty to fish would I incur to gain a certain amount of personal energy?

After all, we're also evolved to kill each other when we get mad -- but we make a point of suppressing this evolutionary urge in the interest of our mutual growth, joy and choice, etc.

Obsoleting the Dilemmas

What does Cosmism have to contribute to these familiar dilemmas?

It doesn't provide any trick for drawing the line between right and wrong in these tricky situations. Joy, growth and choice and other Cosmist principles are all very well; but these dilemmas are cases where two or more different options exist, each bringing joy/growth/choice and other goodies to some minds at the cost of others ... and so there's a difficult judgment to be made.

What Cosmism suggests is an alternate path: obsolete the dilemma.

This is already happening, to some extent. We should try to make it happen far more as the future unfolds.

Birth control largely obsoletes the dilemma of abortion, though it doesn't quite work well enough yet. The ability to remove an embryo from the mother without pain or danger, and incubate it in a lab from a very early stage, would obsolete the abortion dilemma in a different way.

The capability to grow cloned steaks, fish cutlets and chicken breasts and such in the lab would obsolete the dilemma of vegetarianism ... as would advances in synthetic food technology; or pharmacology that conferred the recreational, physiological and neurochemical benefits of various forms of food without requiring actual food ingestion.

There is a powerful general principle here. Ethical dilemmas are never going to be completely avoidable, but the advance of technology can blunt them pretty thoroughly, if it's done with a specific eye toward obsoleting the dilemmas.

The Dilemma of Poverty and Charity

The issue of poverty and charity can be perceived in much the same way as abortion and vegetarianism.. The ethical dilemma of whether to send 80% of my income to help starving children in Africa (I never do so, but feel some guilt over this), will be neatly obsoleted by advanced technology that eliminates material scarcity.

Why don't I send 80% of my income to help starving children in Africa (nor even 10%, for that matter)? Due to the usual mixed motivations. Part of it is surely plain old selfishness; I don't claim to be a wholly altruistic individual. And part of it is a sense that the world as a whole would not be better off if those of us fortunate enough to live in wealthy nations (or the upper classes of poorer nations) were to revert to the economic mean.

If I sent most of my income to help starving children I would be less happy on a day by day basis -- but also, I would also be in much less of a mental and practical position to create new technologies, Cosmist Manifestos, and so forth ... things that I value a great deal. My own children would be in a much worse position to create such things as well, if I were to deprive them of books, computers and education so as to feed the starving kids in Africa. And yet, I'm never quite sure it's right to value these creations over peoples' lives.

What I'm quite sure of, is that it's right to obsolete the dilemma.

Dialectics Redux?

If you've had some contact with Marxist or Hegelian philosophy you may find something familiar in the "obsolete the dilemma" notion.

Hegel, as a key point of his philosophy, described how thesis and antithesis lead to synthesis via the "dialectical" process. The synthesis obsoletes the dilemma between the thesis and antithesis. The dilemma between Being and Nothingness is resolved to yield Becoming. The dilemma between lords and vassals is resolved to yield new social classes emerging due to the advance of industrial technology. Marx saw the advance of society as a result of a series of dialectical dilemma-resolutions.

In Hegelian dialectics, dilemmas are obsoleted by redefining realities so that previously oppositional realities become unified in a new set of structures and dynamics. This is indeed closely related to the Cosmist notion of obsoleting the dilemma (and arguing the precise relationship would be an onerous task of technical philosophy that I won't undertake here!).

However, one big difference between the Hegelian/Marxist perspective and the Cosmist perspective is the amount of determinism that the former perceived to exist in the world. The notion of a precise, orderly series of dilemmas, getting obsoleted and then leading to new dilemmas in a predictable fashion -- this is anathema to the Cosmist perspective, which is all about embracing the unknown and growing oneself so as to understand and become new things that would have been wholly incomprehensible to one's prior self. Often, once a previous dilemma has become obsoleted, the world looks like a totally new place ... and the path forward is one that you never could have imagined to exist before.

Society has not evolved according to anything like the particular path that Marx and Hegel foresaw. It has evolved according to a "quasi-dialectical" process of iterative dilemma-obsoletion, though ... and will continue to do so ... so those guys did get some things fundamentally right.

Figuring out how to obsolete the dilemmas facing us is an ongoing intellectual, emotional, social and spiritual challenge -- not at all a matter of following some predetermined and inevitable path.

Wisdom

If intelligence is about achieving complex goals -- what is wisdom?

Wisdom has many aspects....

One of them is effectively leveraging experience (episodic knowledge) toward intelligence.

Another is breadth -- generality of intelligence, rather than narrow-focus on particular sorts of complex goals.

And another has to do with what sorts of goals to focus on. Wisdom has to do with focusing on greater, broader goals -- broader in extent (beyond the individual self, encompassing other individual minds and even the Cosmos), and broader in temporal extent (long-term goals rather than short-term goals).

So one view of wisdom is:

Being able to use experience, along with other mental aspects, to understand and achieve a broad variety of things, with a focus on broader and longer-term things rather than self-focused or immediate things...


One interesting consequence of this view of wisdom is: it makes clear that seeking to go beyond the human condition as presently conceived, may be the best way to achieve greater wisdom....

So if one says that a strategy like "obsoleting the dilemma" is wise, this means that it is useful for pursuing wisdom -- for understanding and achieving a broad variety of things, with a focus on broader and longer term things rather than self-focused or immediate things....

And more broadly: The basic goals of Cosmism can be seen as a natural route to the achievement of increased wisdom....

Respecting Ecosystems

One of the major dilemmas that needs to be obsoleted as we move forward is that between technology and environmental preservation.

Ecosystems are beautiful, complex things -- in fact, they are highly intelligent minds, though apparently lacking theaters of reflective, deliberative consciousness.

Further, ecosystem-minds blend richly into human minds ... we have evolved to include animals, plants and systems thereof in our extended minds. Many of us find we think and feel much more clearly when in the forest or other natural surroundings -- and there's no strange mystery here; it's because our extended minds encompass aspects of nature.

The advent of industrial and advanced technology is causing great damage to the ecosystem of Earth, leading to conflicts between some of those who deeply love the advance of technology and some of those who deeply love the Earth.

However, I think this is a temporary, albeit tragic, phenomenon. As technology advances further it will incur less and less environmental destruction, not more and more.

In fact, we can already see this happening. Technologically advanced nations are not the worst environmental offenders -- those are developing nations, who use cruder technologies more egregiously inflicting of environmental damage. Wealthier, more advanced nations value their clean air and water and parks to a greater extent.

The dilemma of "technology and environment" will be obsoleted as technology develops further, and becomes more flexible and efficient in its use of resources.

The Sociocultural Mind

The Sociocultural Mind

One of the charming peculiarities of modern Western culture -- and especially American culture, which I've lived in most of my life, and which has played a pivotal role in the development of humanity's advanced technologies -- is its emphasis on the individual rather than the social group.

And yet, if you took a human infant and raised them isolated from culture, what would they be? A primitive being, not so different from an ape. As has been repeatedly demonstrated, without appropriate social interaction in early childhood, humans never develop full linguistic and cognitive abilitites. And in most cases, people cut off from social interaction for more than a few years wind up going mad.

The truth is that human society and culture are a vast meta-mind with greater computational power, insight and complexity than any individual human mind.

So far as we can tell, the emergent mind of human society currently lacks the level of global coherence that an individual human mind has. There's no theater of reflective social consciousness. But even if so, human society contains a huge number of complex high-level self-organizing patterns, that course through all us individuals, and guide us in multiple ways, many of which our selves and deliberative consciousnesses are barely or un aware.

The Inescapability of Culture

Technological and cultural progress have largely been driven by individuals who deviate from social norms -- who push beyond their society and culture, often suffering greatly for this.

But yet, the directions that they're pushing in, the ideas and movements they're pushing forward, are invariably defined in terms of human culture and society. We, as humans, simply don't know anything else.

Chinese, aboriginal and American culture (to pluck three random examples) may seem wildly different to us -- but of course, in the bigger space of all possible modes of sociocultural interaction, they're really mightily mutually similar.

It's possible that some radically different culture and set of mind-states could be implemented in a set of interacting human brains and bodies ... but we have no way of discovering that, because a culture is not something one person can build on their own, nor something that a group of people can simply get together and spontaneously create. It's far bigger and more complex and subtle than that.

Rebellion as Conformity

In the end the existence of rebellious individuals pushing beyond the norms is "just" a mechanism that modern society and culture uses to grow, extend and expand itself. Those of us who act as rebels, are following the deep self-organizing patterns of the sociocultural order just as thoroughly as those who act as conformists. This wouldn't have been so true for some wild-eyed rebel emerging in a steady-state culture like aboriginal culture ... but it's very true now, given that modern culture relies on ongoing rebellion to generate the constant stream of scientific, cultural and aesthetic advance on which it's predicated.

Rational, orderly-looking growth curves like the ones Ray Kurzweil draws, projecting the advances of various technologies, already factor in the likelihood of a large number of individuals risking (and in many cases suffering) ostracism to spend their lives promoting ideas that their peers consider moronic, maniacal and/or misguided.

Sociocultural Self-transcendence

The modern sociocultural mind has a pattern of violating and transcending itself.

That is: it creates dilemmas, then advances in such a way as to obsolete them ... and in the process it creates new dilemmas, etc.

While Hegel and Marx's sociology made a lot of errors, it did get this general pattern of "dialectical advance" about right. They saw how society advances to obsolete dilemmas -- e.g. the dilemma of feudal lords versus vassals was obsoleted via the industrial revolution. Which in turn led to new dilemmas to be obsoleted.

Cosmism itself exemplifies this self-violating/self-transcending aspect of the modern sociocultural mind: it both violates and emerges from the dominating patterns in modern society and culture.

And it seems likely that this overall pattern is going to lead to the ultimate and thorough transcendence of the modern sociocultural mind -- as it pushes us humans (who compose its "cells") to fundamentally modify or replace our physical substrates ... a move that will dramatically transform the nature of the sociocultural mind we compose.

The individual mind and the socio-mind will transcend together.


The dilemmas of "individual versus society" and "mind versus body", which lie at the root of many of our contemporary social and psychological problems, are likely to soon get obsoleted -- and replaced with other, more advanced dilemmas! As a part of the natural growth of the thought process of the social mind with which we all, as individuals, interpenetrate.

How Should Society Be Structured?

Democracy ... capitalism ... communism ... socialism ... anarchism ... the list goes on and on ...

Is there any really good way to structure a human society?

If not, then what's the best of the bad lot?

Or, to put it another way: Is there some way that a sociocultural mind can be fashioned out of human "cells", that leads to rampant joy, growth and choice ... and with less of the opposites of these values than we now see in the human world around us?

Certainly, Cosmist values seem to argue in favor of societies allowing their members a fair bit of personal freedom, and encouraging progress rather than a steady state ... and treating each other with compassion rather than cruelty or repression.

But there are difficult issues when one digs into the details, and broad Cosmist philosophy doesn't solve them.

Balancing compassion versus choice in government is difficult -- taxing people to pay to feed and educate poor children is an imposition on freedom; yet leaving the poor children starving and ignorant is uncompassionate. One would like society to self-organize in such a way that such dichotomies don't exist -- and perhaps this is happening, coupled with the advance of technology; but it's happening frustratingly slowly for those now in disadvantaged positions.

It seems clear that, even without further advanced technologies, we could do considerably better than current social orders as solving these difficult problems.

But ultimately, it seems there are limits to how well a society of humans can be structured , given our intrinsic cognitive limitations. And there are even stricter limits to how well a society of humans can be structured under conditions of scarce resources.

I.e.: any sociocultural mind, composed at base of "legacy humans" like we are today, is bound to be at least a bit psychologically screwed-up ... and even more so if we're involved in struggling over scraps of matter shaped into particular configurations.

The best course, it seems, is to obsolete the dilemmas of society by redefining the human mind and abolishing material scarcity.

Improving Democracy?

Churchill said "Democracy is the worst form of government, except for all the others."

Clearly there's something to this, which is why the democracy meme is spreading around the world, and is correlated with other indices of human well-being.

But even if this is accepted: what kind of democracy, then?

Should the Net be used to allow direct democracy? -- where people vote on a variety of specific issues rather than delegating so much to their representatives (who so often seem to confuse and corrupt matters, sometimes doing so even in spite of good intentions)? But who has time to study the details of complex legislative issues so as to vote on them intelligently?

Presumably new mixtures of direct and representative democracy will emerge as advanced technology more thoroughly permeates our culture. To an extent -- though an incomplete extent, giving the limitations of human mind -- Internet technology can likely obsolete some of the dilemmas of democracy.

Human Life After Scarcity?

One thing that could improve the human political situation a lot more than tweaking the details of democracy, would be the drastic reduction of material scarcity.

The reason human life is less brutal now, by and large, than in the past, is because advances in technology have reduced gradually scarcity. We have developed various useful social institutions that embody a great deal of calculation and wisdom ... but arguably, the reason these institutions are proving workable and stable, is the concurrent technological advance.

It seems quite possible that democracy is only stably achievable under conditions of relative material abundance. The ancient Greeks achieved democracy's practical preconditions via slavery. In the modern world we've achieved them via technology.

Most likely, resources will never be infinite -- there will always be some contention for resources of some sort. Most probably some form of scarcity will always exist. But relative to our innate human desires, scarcity could dwindle close to zero. Given nanotech molecular assemblers plus ultra-realistic virtual worlds, provided free of charge to all, there would be a lot less motivation for anyone to risk their comfort by fighting over resources.

In this sort of scenario, the task of managing a society, using ever more refined and liberating mechanisms, will become far easier.

All sorts of possibilities that would seem "utopic" from the present perspective may become possible. Many of the social dilemmas that now strike us as inevitable, may be drastically obsoleted.

A Post-Scarcity Network of Enclaves?

For instance: Perhaps in a post-scarcity scenario, it would work for small enclaves of humans to form -- communities centered around different belief systems and life-patterns.

Each enclave could basically have its own little world, and there wouldn't be practical competition between enclaves due to the minimization of scarcity.

In the modern situation, this sort of "enclave" based social arrangement would run into problems due to issues like pollution that span enclaves. But it seems feasible that advanced technology could resolve this ... "all" one needs are extremely inexpensive nonpolluting molecular assemblers, for example.

You might argue that this is a fanciful notion, because scarcity will never truly be eliminated. But, it may be possible to eliminate scarcity from the perspective of everyday human life. There's a limit to how much an individual human can consume, and still remain human.

What if scarcity were reduced sufficiently that no one wants for practical physical comforts ... and there is more than enough advanced media and entertainment and intellectual and artistic technology to go around: more than anyone could possibly use in thousands of years? In this sort of regime, it seems quite possible that the urge to invade other enclaves and take their resources would be extremely rare.

Better Societies through Better Brains

More pessimistically, there is the possibility that we humans are collectively so perverse that we will still battle each other viciously even once scarcity is virtually abolished.

If so, then the only solution to making a happy society -- to really obsoleting the social dilemmas -- is to modify the human mind/brain ... solving the problems of society at the source.

There is no doubt that, if the human mind/brain were dramatically improved, avenues for deeper social interaction and cultural invention would open up, making modern societies seem more obsolete than primitive tribal life seems today.

I.e., in this scenario: The emergent sociocultural mind would become a far more growing, joyful, purposeful "individual".

Practical Politics Today

The above thoughts on society may seem utopian, unrealistic, futuristic ... what about the fray of real-world politics, right now?

Cosmism dictates only a few broad principles in this regard ... and in practice, reconciling them may be difficult! For instance,

  • Allow development of advanced technologies except in cases of extreme danger
  • Extend compassionate help to all, except where the imposition on individuals is tremendous
  • Don't build structures or dynamics in stone -- leave each aspect of society free for adaptation and growth
  • Make joy, growth and choice explicit goals of the social order

Broad principles like these manifest themselves in many ways in the particular situations we now confront -- but that would lead us beyond the present manifesto, to a different sort of manifesto!

Sexuality & Beyond

A futurist friend of mine likes to tell people of his aspiration to somehow remove all sexuality from his brain.

Most people think he's nuts in this regard (including his wife, with whom he has an active and healthy sex life), but he's convinced that sexuality is one of the main factors slowing down our progress toward the Singularity, immortality, superhuman benevolent AIs and all the other good stuff.

One point he always makes in these conversations it that it's amazing, when you really think about it, how much of modern human society is structured around sexuality.

Marriage, kids, dating ... buying nice clothes and making oneself up to impress the opposite sex ... buying cars or houses or the latest cellphone to impress the opposite sex with one's success ... etc.

Sexuality pervades nearly everything in our lives, implicitly even when not explicitly.

As is now common knowledge, the power sex has over us is rooted in the power our DNA has over us. We are evolved to obsess over reproducing, over extending our DNA to future generations. Even though most humans in First World countries now use birth control for nearly all their sexual encounters, and many humans choose not to reproduce at all, we are still strikingly controlled by the mind-patterns ensuing from our DNA's urge to persist itself.

But evolution has tangled sex up with all manner of other aspects of our psyches. As Freud, Reich and others pointed out so thoroughly, human motivation is deeply tied with our inner sexual energy. Eunuchs seem to generally lack aggressive, enthusiastic motivation even for things outside the realm of sex. But when my anti-sex futurist friend speaks of blotting out sexuality from his mind, he doesn't want to blot out his passion and energy generically -- he wants to focus it on things other than simulations or enactions of the reproductive act.

And yet -- for all the distraction that it provides -- sex, at its best, is a profound "altered state of consciousness" experience, just as self-melting and reality-changing as meditation or psychedelics or any other extreme of human experience. It brings you outside of your ordinary self and state-of-consciousness, putting you in touch with wholly different ways of being, interacting, experiencing.

Beyond Sex

A couple decades ago there was a color glossy periodical called "Future Sex Magazine." It didn't survive too long, partly because there wasn't that much to say on the topic. Teledildonics is a fascinating idea but it's easy enough to get the idea across ... and apparently there was a limited audience for monthly photos of hot models making it with robots and electronic AI-powered uber-vibrators.

What's more interesting is the prospect of new forms of experience, providing the same things sex provides -- but also going beyond sex in significant, surprising ways.

As satisfying as sex and sexuality are for human bodies and minds, there seems little doubt that, eventually, transhuman minds will discover new forms of pleasure and fulfillment going far beyond what we now get from sexuality. Sexuality is amazingly wonderful but it's also a mess ... it wraps up confusion and pain with pleasure in complex ways, surely more than is necessary based on the innate interconnection of pain and pleasure that exists in any finite mind.

It may seem cold or eccentric to say so, but the fact is that orgasms and genitals and romantic relationships -- as glorious as they are -- are hideously badly designed. It's obvious that other entities better serving the same purposes (and more) could be developed...!

But what do we do now, while we're stuck in legacy human form? Should we embrace asceticism, so as to more effectively work toward transhumanity without distraction? Or should we make the most of the pleasures afforded by the human form, while they're the only ones we have ... so long as we can do so without killing our chances to find something even better later on?

To what extent is a healthy sex life needed to give the human mind/body an effective grounding for making the difficult judgments that will come in the next decades as technology develops?

There are no easy and immediate answers to these questions ... each of us has to make our own judgment.

All Cosmism does is urge us to make such judgments based on a rich understanding of the issues from multiple perspectives. And urge us to obsolete the dilemma, by fixing the underlying problem, which in this case is the rootedness of our experience in a body and brain that are very difficult for us to reflectively modify and manipulate according to our deliberative desires and conclusions.

What Would Asexual Aliens Think of Human Sexuality?

One interesting exercise is to view sexuality, or other practical issues, as if one were an asexual alien viewing humans objectively and for the first time. What would such an alien think about the role of sexuality in human society and psychology?

Assuming the alien perceived the human race as being on the cusp of a technological Singularity or something similar, what would the alien think about the optimal human relationship with sexuality and other in-built obsessions of the human brain/mind?

The Essence of Sex

Considering the potentially broad scope of alien sex brings us to a more fundamental question: What is the "abstract essence" of sexuality, apart from its particular manifestation in our human embodiments?

Is it just, say, "the mutual exchange between two intelligences of positive emotion and pleasure-center stimulation, coupled with emotional open-ness?"

That sort of abstract sex seems like an unabashedly wonderful thing ... but of course, there is so much (good and bad) about human sexuality that isn't implied by such an abstract essence.... For one thing such a notion of sex is uncoupled from reproduction. It is also not intrinsically tied to any notion of long-term commitment -- although there may be an implicit relationship there, since emotion-sharing may work best between two minds that know each other well, in the way that only long-term togetherness can bring.

Probably there are forms of experience that embody the abstract essence of sexuality -- and intensify it more than is possible within the constraints of humanity -- and without the downsides of human sexuality (though perhaps with new downsides that we can't now imagine).

What can I say? Bring it on!

Goals and Meta-Goals

Ever have the experience that you seriously think you're trying to achieve one thing, but then in hindsight, years later, you look back and feel like your past self was actually trying to achieve something else entirely?

I.e., you weren't really after what you thought you were after?

I've had that experience often -- for example, during part of the time I was involved with my first startup company, Webmind Inc. I thought I was chasing two goals: 1) making a lot of money, and 2) getting advanced AI built. In hindsight though, often the actions I was taking weren't balancing these two goals very well. Much of the time I was straightforwardly pursuing AI research -- and fooling myself that the things I was doing were really serving the money goal as well, to a greater extent than was really the case. Of course this was easy to do at that time, because it was the late-1990s dot-com boom, and no one really knew the secret to making money in that period anyway.

My explicit goals with that company were making money and getting advanced AI built, with roughly equal weighting. That's what I thought I was doing, at the time.

My implicit goals, the goals it looks in hindsight like my actions were pursuing, were weighted a bit differently: maybe 70% AI and 30% money. That's what I now think I was doing.

In regard to that company, at that point in time, I had a poorly-aligned goal system.

These concepts have a fundamental importance.

As an external observer, one can look at a system and identify the goals it seems to be pursuing: i.e., in mathematical terms, the functions it seems to be trying to maximize. These are the system's implicit goals.

A system can pursue implicit goals, in this sense, even if it lacks any concept of what a "goal" is.

Some systems, on the other hand, also have explicit goals, meaning that they model themselves as pursuing certain goals.

A well-aligned goal system is a set of explicit goals that fairly accurately reflect the implicit goals of the intelligent system containing the goal system.

Achieving a well-aligned goal system generally requires long practice and deep self-understanding.

Naturally, this is much easier if one's (explicit and implicit) goals are simple ones!

Meta-Goals

Well-alignedness is an example of a meta-goal: a property of a goal system that is not tied to the specific content of the goals, but rather to the general nature of the goal system.

Another meta-goal is consistency.

Consider a goal system as a set of top-level goals, together with other subgoals that are derived from these. Then, a goal system is fundamentally inconsistent to the extent that achieving any one of the top-goals, decreases the level of achievement of any of the other top-level goals. Fundamental consistency is the opposite of this.

There is also a notion of subgoal inconsistency: the extent to which, on average, achieving one the subgoals derived from a particular top-level goal, decreases the level of achievement of the subgoals derived from other top-level goals.

Even if a system is fundamentally consistent, if it's not subgoal consistent, it may have a very hard time achieving its goals.

A related problem is subgoal alienation -- sometimes a subgoal is derived as a way of achieving some other goal or subgoal, but then the former one persists even after the latter one is abandoned. Subgoal alienation leads to the accidental creation of new top-level goals, which leads to inconsistency and poor goal system alignment.

I've known a number of people who originally took high-paid, unrewarding jobs for the reason of saving money up to pursue some other goal. But after a while the other goal became less and less important to them, and the high-paying job became an end in itself. Sometimes this is subgoal alienation; sometimes it's a matter of a poorly-aligned goal system (maybe the high pay was really their main goal all along, and they were just fooling themselves about the other goal).

Sexuality is a massive case of subgoal alienation, on the cultural and biological level. It emerged as a subgoal of reproduction, but -- especially since the advent of birth control -- it has liberated itself from these roots and now serves as a top-level goal itself for most adult humans.

Are Humans Goal-Oriented?

Humans, by nature, are not that thoroughly goal-oriented. We have certain in-built biological goals but we've done well at subverting these. Some of us adopt our own invented or culturally-acquired goals and put a lot of effort into pursuing them. But a lot of our behavior is just plain spontaneous and not terribly goal-oriented.

Of course pretty much any behavior could be modeled as goal-oriented; i.e. every series of actions can be viewed as an attempt to maximize some quantity. But the question is whether this is a simple and natural way to model the behavior in question -- does it pass the Occam's Razor test?

Will Advanced AIs Be Goal-Oriented?

One can imagine artificial minds with a vastly more thorough goal-orientation than humans, and a much greater attention to meta-goals. Most likely such minds will exist one day -- alongside, potentially, artificial minds that are much less goal oriented, much more loosely organized, than humans.

It also seems likely that, even in a mind more thoroughly devoted to goal-achievement than the human mind, a certain percentage of resources will wind up being allocated to spontaneous activity that is not explicitly goal-oriented in any of its details ... because this spontaneous activity may generate creative ideas that will be helpful for achievement of various goals.

Evolution of Top-Level Goals

I've spoken about about goal systems as possessing "top level goals," from which other goals are derived as subgoals.

Are these top-level goals then invariant as a mind evolves and learns?

This is not how humans work, and needs not be how AIs work.

Indeed, if one has a mind that gradually increments its processing power, one might expect that as it grows more intelligent it will discover new potential top-level goals that its earlier versions would not have been able to understand.

How can top-level goals get revised?

Not in a goal-oriented way, because if goal G is put in charge of revising goal G1, this really just means that goal G1 is not top-level -- G is top-level.

Top-level goals can get revised via spontaneous, non-goal-oriented activity -- and the occurrence of this phenomenon in intelligent systems seems to be a fundamental aspect of the growth of the cosmos.

Goals grow, goals drift -- and thus the universe evolves, via both pursuit and spontaneous development of goals.